VMs with TrueNAS iSCSI disks are not compatible with Proxmox HA

[antoninchadima]

Everything is fine without HA.

With HA the VM doesn't start.

task started by HA resource agent
TASK ERROR: freenas: unknown iscsi provider. Available [comstar, istgt, iet, LIO] at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 35.

[TheGrandWazoo]

Did you install the plugin on each node in the cluster?

[antoninchadima]

yes it is installed on all 12 nodes
i can even live migrate a vm from one node to another

but a VM with TrueNAS iSCSI disks is not compatible with being added as a resource in Proxmox HA

[TheGrandWazoo]

I have HA running on my cluster without issues. Are you sure the target host has the Plugin installed? Due to the fact that the error is:
TASK ERROR: freenas: unknown iscsi provider. Available [comstar, istgt, iet, LIO] at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 35.
And the [comstar, istgt, iet, LIO] does NOT have the FreeNAS leads me to believe that node did not get the Plugin installed...

Snippet from the patch for ZFSPlugin.pm showing the original and the patch.

-    die "$provider: unknown iscsi provider. Available [comstar, istgt, iet, LIO]";
+    die "$provider: unknown iscsi provider. Available [comstar, freenas, istgt, iet, LIO]";

[antoninchadima]

thank you again for your time

the plugin is surely installed on every node
the storage is visible on every node in the server view on the right side
and in the Datacenter (cluster1) Search overview:

The HA is defined as follows (there are no other nodes):

the storage is defined for all nodes no exception:

Maybe i can try to reinstall the plugin on each node again (if it is possible)
and tell you the results

[antoninchadima]

did apt --reinstall install freenas-proxmox on every node
and got the same message with no errors:

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 28 not upgraded.
Need to get 2,924 B of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://dl.cloudsmith.io/public/ksatechnologies/truenas-proxmox/deb/debian any-version/main amd64 freenas-proxmox all 2.2.0-1 [2,924 B]
Fetched 2,924 B in 0s (32.2 kB/s)           
(Reading database ... 66952 files and directories currently installed.)
Preparing to unpack .../freenas-proxmox_2.2.0-1_all.deb ...
Unpacking freenas-proxmox (2.2.0-1) over (2.2.0-1) ...
Reinstalling original Proxmox VE files so upgrade can be performed
Setting up freenas-proxmox (2.2.0-1) ...
Already up to date.
Configuring freenas-proxmox 2.2.0-1
Restarting pvedaemon...
Restarting pveproxy...
Restarting pvestatd...
Restarting pvescheduler...

no signing key and repository issues, on all nodes fine

Maybe a nasty proxmox issue?
The HA existed before adding the iscsi storage...
so i can try to delete the HA group disable HA
and enable it again (after having this plugin installed)
and tell you the results

[antoninchadima]

this doesn't help
i can even reproduce it in a clean fresh installed proxmox 3 node cluster

i can create a VM with this iscsi storage on every node
i can even manually migrate it to every node

The HA Settings in Options is set to Shutdown Policy=migrate
Cluster Resource Scheduling: HA Scheduling= Static Load, Rebalance on Start=true

i create a HA group with all the nodes included (restricted=true, nofailback=true)
i add the VM to this HA group (max-started=1, max-relocate=1, requested state=started)
everything goes fine up to this point

aftewards i stop the VM

HA 122 - Stop:

Requesting HA stop for VM 122
TASK OK

VM 122 - Shutdown:

task started by HA resource agent
VM still running - terminating now with SIGTERM
TASK OK

everything is still OK

and finally i try to start the VM

HA 122 - Start:

Requesting HA start for VM 122
TASK OK

VM 122 - Migrate:

task started by HA resource agent
2024-03-25 09:35:27 use dedicated network address for sending migration traffic (172.16.5.40)
2024-03-25 09:35:28 starting migration of VM 122 to node 'pve10' (172.16.5.40)
2024-03-25 09:35:28 migration finished successfully (duration 00:00:01)
TASK OK

VM 122 - Start:

task started by HA resource agent
TASK ERROR: freenas: unknown iscsi provider. Available [comstar, istgt, iet, LIO] at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 35

and the VM with iscsi storage plugin doesn't start and is in HA error state
the same for a clean new Proxmox install, 3 nodes, no other VM
and i don't have any problem with nfs and ceph with HA on the same cluster

[antoninchadima]

I have checked every single /usr/share/perl5/PVE/Storage/ZFSPlugin.pm
if patched and contains die "$provider: unknown iscsi provider. Available [comstar, freenas, istgt, iet, LIO]";
and i didnt find a single one with only: die "$provider: unknown iscsi provider. Available [comstar, istgt, iet, LIO]";

mystery
maybe some cache? will restart the nodes

[antoninchadima]

Ok a complete restart of all proxmox nodes helped, even on a completely new installation.

thank you very much and you can close the ticket.
maybe this should be mentioned in the readme.

[antoninchadima]

Ok there is no more the HA issue, but:

task started by HA resource agent
iscsiadm: No session found.
iscsiadm: No session found.
iscsiadm: No session found.
iscsiadm: No session found.
iscsiadm: No session found.
iscsiadm: No session found.
/dev/rbd4
TASK OK

[mir07]

I think I have a solution for this issue. The problem is when the hooks for the freenas plugin kicks in an patches the upstream files it runs these systemd commands:

• systemctl restart pvedaemon.service
• systemctl restart pvestatd.service
• systemctl restart pveproxy.service

But the problem is that ha-manager runs with the original ZFSPlugin.pm so to update ha-manager the following systemd sequence should have been run:

• systemctl restart pvedaemon.service
• systemctl restart pve-ha-crm.service
• systemctl restart pve-ha-lrm.service
• systemctl restart pvestatd.service
• systemctl restart pveproxy.service

[antoninchadima]

yes, that's it!

Thank you very much.

[TheGrandWazoo]

Reopen. Need to provide a fix in the installer.