pve-doc 5.2-9 has been release. Need to create a new patch for the version.

Update patch for libpve-storage-perl from 5.0-30 to 5.0-32
Update patch for pve-docs from 5.2-9 to 5.2-10
Update patch for pve-manager from 5.2-10 to 5.2-12

I was having a issue with the multipath patch when not using multipathing. It seems to be failing on the run_command() and not executing the rest of the to give a path or return the values needed.

When I create VirtIO SCSI hard drive, it is not recognized as bootable device in VM.
A can boot VM from CD, install system on hdd, but I can't boot system from the hdd.

Booting from VirtIO Block device works OK.

When you go into an 'Edit' mode on the iSCSI panel an error is displayed when you goto save..

It looks like the 'target' property and seems to be only after a upgrade not while I was in edit. This could be a regression issue in the pve-docs.

I've noticed new file in repo: perl5/PVE/Storage/ZFSPoolPlugin.pm. Is it necessary?

Hi

Could you update for 6.0-5 if possible. Would greatly appreciate.

Look like they changed ZFSPlugin.pm

Getting this error on VM Start
task started by HA resource agent TASK ERROR: freenas: unknown iscsi provider. Available [comstar, istgt, iet, LIO] at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 35.

This is the output of the ZFSPlugin.pm patch

Hunk #1 succeeded at 10 with fuzz 1.
Hunk #2 FAILED at 27.
Hunk #3 succeeded at 40 (offset -1 lines).
Hunk #4 succeeded at 62 (offset -3 lines).
Hunk #5 succeeded at 160 (offset -5 lines).
Hunk #6 succeeded at 187 (offset -5 lines).
Hunk #7 FAILED at 237.
Hunk #8 succeeded at 258 (offset -11 lines).
2 out of 8 hunks FAILED -- saving rejects to file /usr/share/perl5/PVE/Storage/ZFSPlugin.pm.rej

Regards
UdK

Using patch that is attached to pull request #11, provide multipathing function to ZFSPlugin.pm.

Update patch to be compatible to pve-docs: 5.2-8

Environment:
FreeNAS build FreeNAS-11.1-U5
Proxmox Virtual Environment 5.2-1

ZFS over iSCSI definition in Proxmox:
ID: zfs-over-scsi0
Portal: <>
Pool: pool0
Target: iqn.2005-10.org.freenas.ctl:target2
FreeNAS user: root
FreeNAS password: <>

FreeNAS config:
Portal Group ID: 1
Target Name: target2
Extent Name: extent0
Extent Type: file
Extent Path: /mnt/fast0/extent0
Target/Extent: links target2 and extent0

Issue:
When creating a VM, the disk image appears in the Proxmox backend. However, the create task creates with:

TASK ERROR: create failed - error with cfs lock 'storage-zfs-over-scsi0': Unable to find the target id for iqn.2005-10.org.freenas.ctl:target2 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 124.

Relevant entry on the Proxmox side in /var/log/syslog/:

Jul 9 10:04:31 prox01test pvedaemon[94112]: root@pam starting task UPID:prox01test:00016FCB:016C7286:5B42A68F:qmcreate:112:root@pam:
Jul 9 10:04:31 prox01test pvedaemon[94155]: FreeNAS::lun_command : create_lu(/dev/zvol/fast0/vm-112-disk-1)
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_globalconfig : Response code: 401
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_globalconfig : Response content:
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target : Response code: 401
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target : Response content:
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target_to_extent : Response code: 401
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target_to_extent : Response content:
Jul 9 10:04:32 prox01test pvedaemon[94155]: FreeNAS::API::freenas_get_first_available_lunid : return 0
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target : Response code: 401
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target : Response content:
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_globalconfig : Response code: 401
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_globalconfig : Response content:
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target : Response code: 401
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target : Response content:
Jul 9 10:04:32 prox01test pvedaemon[94155]: FreeNAS::API::freenas_list_lu : sucessfull
Jul 9 10:04:32 prox01test pvedaemon[94155]: FreeNAS::list_lu(/dev/zvol/fast0/vm-112-disk-1):name : lun not found
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_globalconfig : Response code: 401
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_globalconfig : Response content:
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target : Response code: 401
Jul 9 10:04:32 prox01test pvedaemon[94155]: [ERROR]FreeNAS::API::get_target : Response content:
Jul 9 10:04:32 prox01test pvedaemon[94155]: VM 112 creating disks failed
Jul 9 10:04:32 prox01test pvedaemon[94155]: create failed - error with cfs lock 'storage-zfs-over-scsi0': Unable to find the target id for iqn.2005-10.org.freenas.ctl:target2 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 124.
Jul 9 10:04:32 prox01test pvedaemon[94112]: root@pam end task UPID:prox01test:00016FCB:016C7286:5B42A68F:qmcreate:112:root@pam: create failed - error with cfs lock 'storage-zfs-over-scsi0': Unable to find the target id for iqn.2005-10.org.freenas.ctl:target2 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 124.

Entries on the FreeNAS side in /var/log/debug.log:

Jul 9 10:04:32 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 247, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 477, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] self.is_authenticated(request)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 563, in is_authenticated
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] raise ImmediateHttpResponse(response=auth_result)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 247, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 477, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] self.is_authenticated(request)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 563, in is_authenticated
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] raise ImmediateHttpResponse(response=auth_result)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 247, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 477, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] self.is_authenticated(request)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 563, in is_authenticated
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] raise ImmediateHttpResponse(response=auth_result)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 247, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 477, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] self.is_authenticated(request)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 563, in is_authenticated
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] raise ImmediateHttpResponse(response=auth_result)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 247, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 477, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] self.is_authenticated(request)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 563, in is_authenticated
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] raise ImmediateHttpResponse(response=auth_result)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 247, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 477, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] self.is_authenticated(request)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 563, in is_authenticated
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] raise ImmediateHttpResponse(response=auth_result)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 247, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 477, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] self.is_authenticated(request)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 563, in is_authenticated
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] raise ImmediateHttpResponse(response=auth_result)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "./freenasUI/api/utils.py", line 247, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] request_type, request, *args, **kwargs
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 477, in dispatch
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] self.is_authenticated(request)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 563, in is_authenticated
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] raise ImmediateHttpResponse(response=auth_result)
Jul 9 10:04:32 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse

Provide some enhancements make the FreeNAS fields visible in the GUI modal when FreeNAS-API is selected.

Possible to package the files and work into a .deb package.
Few givens:

Storage.pm is in the libpve-storage-perl package
ZFSPlugin.pm is in the libpve-storage-perl package
apidoc.js is in the pve-docs package
pvemanagerlib.js is in the pve-manager package

We can retrieve the version of these packages so the FreeNAS package can properly update the files.
Also looking to mesh/cherry pick the apidoc.js, pvemanagerlib.js and ZFSPlugin.pm to keep full copy of the files instead of diffs.

I've got a serious issue while cloning VM. Looks like I am able to clone only running VM. When VM is offline, I've got error:

# qm clone 9000 200
create full clone of drive scsi0 (freenas-zfs:vm-9000-disk-1)
transferred: 0 bytes remaining: 8589934592 bytes total: 8589934592 bytes progression: 0.00 %
qemu-img: iSCSI misaligned request: iscsilun->block_size 4096, offset 0, count 2147483136
qemu-img: error getting block status at offset 0: Invalid argument
qemu-img: iSCSI misaligned request: iscsilun->block_size 4096, offset 0, count 2147483136
clone failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f raw -O raw iscsi://10.0.254.101/iqn.2005-10.org.freenas.ctl:pve/0 iscsi://10.0.254.101/iqn.2005-10.org.freenas.ctl:pve/2' failed: exit code 1

But when I start the VM up, cloning works OK:

create full clone of drive scsi0 (freenas-zfs:vm-9000-disk-1)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred: 5046272 bytes remaining: 8584888320 bytes total: 8589934592 bytes progression: 0.06 % busy: 1 ready: 0 
drive-scsi0: transferred: 839909376 bytes remaining: 7750025216 bytes total: 8589934592 bytes progression: 9.78 % busy: 1 ready: 0 
drive-scsi0: transferred: 1384251392 bytes remaining: 7205683200 bytes total: 8589934592 bytes progression: 16.11 % busy: 1 ready: 0 
drive-scsi0: transferred: 4717543424 bytes remaining: 3872391168 bytes total: 8589934592 bytes progression: 54.92 % busy: 1 ready: 0 
drive-scsi0: transferred: 8129609728 bytes remaining: 460324864 bytes total: 8589934592 bytes progression: 94.64 % busy: 1 ready: 0 
drive-scsi0: transferred: 8589934592 bytes remaining: 0 bytes total: 8589934592 bytes progression: 100.00 % busy: 1 ready: 0 
drive-scsi0: transferred: 8589934592 bytes remaining: 0 bytes total: 8589934592 bytes progression: 100.00 % busy: 0 ready: 1 
all mirroring jobs are ready 
Qemu Guest Agent is not running - VM 9000 qmp command 'guest-ping' failed - got timeout
suspend vm
trying to acquire lock...
drive-scsi0: Cancelling block job
drive-scsi0: Done.
resume vm
trying to acquire lock...
found pending changes, discarding for clone

That issue is breaking usage of templates, as template cannot be power on.

Noticed an erroneous character after the FreeNAS Username and password entered in from the UI.

FreeNAS (v. 11.2-U3) is reporting that the zvol Proxmox is making are double the real size of the actual size. IIRC, this happened previously, so I think it's a regression with the current code.

Mike

Environment:
FreeNAS build FreeNAS-11.1-U5
Proxmox Virtual Environment 5.2-1

Extent and Target/Extent are created successfully in FreeNAS. However, I am unable to start the VM in the Proxmox web UI.

It appears that the LUN number is missing from the iSCSI URL. It is trying to use:

iscsi://10.0.70.88/iqn.2005-10.org.freenas.ctl:target3

When I add the LUN ID at the end, so that it becomes:

iscsi://10.0.70.88/iqn.2005-10.org.freenas.ctl:target3/9

And start the command manually, using:

/usr/bin/kvm -id 102 -name test102 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=5e6d2afd-29bc-4ae3-a12c-8584ba9c905d' -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/102.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 512 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:ec896ecda4a1' -drive 'file=/mnt/pve/iso/template/iso/ubuntu-18.04-live-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=iscsi://10.0.70.88/iqn.2005-10.org.freenas.ctl:target3/9,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-generic,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=96:E7:59:A7:F0:98,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'

It works.

I don't know where the LUN is meant to be stored/injected in the URL.

Output of /var/log/syslog on the Proxmox server:

Jul 9 14:10:55 prox01test pvedaemon[6041]: start VM 102: UPID:prox01test:00001799:000288FC:5B42E04F:qmstart:102:root@pam:
Jul 9 14:10:55 prox01test pvedaemon[1906]: root@pam starting task UPID:prox01test:00001799:000288FC:5B42E04F:qmstart:102:root@pam:
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::lun_command : list_lu(/dev/zvol/fast0/vm-102-disk-1)
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::get_target() : sucessfull
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::get_globalconfig : target_basename=iqn.2005-10.org.freenas.ctl
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::get_target() : sucessfull
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_get_targetid target3 : 1
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu for target 1
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::get_target_to_extent() : sucessfull
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::get_extent : sucessfull
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 1 : 1
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 2 : 2
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 3 : 3
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 4 : 4
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 5 : 5
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 6 : 6
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 7 : 7
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 8 : 8
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 9 : 9
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 10 : 10
Jul 9 14:10:55 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu : sucessfull
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::list_lu(/dev/zvol/fast0/vm-102-disk-1):name : lun found /dev/zvol/fast0/vm-102-disk-1
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::lun_command : list_view(/dev/zvol/fast0/vm-102-disk-1)
Jul 9 14:10:55 prox01test pvedaemon[6041]: FreeNAS::API::get_target() : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_globalconfig : target_basename=iqn.2005-10.org.freenas.ctl
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_target() : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_get_targetid target3 : 1
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu for target 1
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_target_to_extent() : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_extent : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 1 : 1
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 2 : 2
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 3 : 3
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 4 : 4
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 5 : 5
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 6 : 6
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 7 : 7
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 8 : 8
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 9 : 9
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 10 : 10
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::list_lu(/dev/zvol/fast0/vm-102-disk-1):lun-id : lun found
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::lun_command : list_lu(/dev/zvol/fast0/vm-102-disk-1)
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_target() : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_globalconfig : target_basename=iqn.2005-10.org.freenas.ctl
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_target() : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_get_targetid target3 : 1
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu for target 1
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_target_to_extent() : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_extent : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 1 : 1
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 2 : 2
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 3 : 3
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 4 : 4
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 5 : 5
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 6 : 6
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 7 : 7
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 8 : 8
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 9 : 9
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 10 : 10
Jul 9 14:10:56 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::list_lu(/dev/zvol/fast0/vm-102-disk-1):name : lun found /dev/zvol/fast0/vm-102-disk-1
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::lun_command : list_view(/dev/zvol/fast0/vm-102-disk-1)
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_target() : sucessfull
Jul 9 14:10:56 prox01test pvedaemon[6041]: FreeNAS::API::get_globalconfig : target_basename=iqn.2005-10.org.freenas.ctl
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::get_target() : sucessfull
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_get_targetid target3 : 1
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu for target 1
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::get_target_to_extent() : sucessfull
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::get_extent : sucessfull
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 1 : 1
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 2 : 2
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 3 : 3
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 4 : 4
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 5 : 5
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 6 : 6
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 7 : 7
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 8 : 8
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 9 : 9
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu 1 : 10 : 10
Jul 9 14:10:57 prox01test pvedaemon[6041]: Use of uninitialized value $iscsi_lunid in concatenation (.) or string at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 459.
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::API::freenas_list_lu : sucessfull
Jul 9 14:10:57 prox01test pvedaemon[6041]: FreeNAS::list_lu(/dev/zvol/fast0/vm-102-disk-1):lun-id : lun found
Jul 9 14:10:57 prox01test systemd[1]: Started 102.scope.
Jul 9 14:10:57 prox01test systemd-udevd[6059]: Could not generate persistent MAC address for tap102i0: No such file or directory
Jul 9 14:10:57 prox01test kernel: [ 1664.193266] device tap102i0 entered promiscuous mode
Jul 9 14:10:58 prox01test kernel: [ 1664.222538] vmbr0: port 3(tap102i0) entered blocking state
Jul 9 14:10:58 prox01test kernel: [ 1664.222543] vmbr0: port 3(tap102i0) entered disabled state
Jul 9 14:10:58 prox01test kernel: [ 1664.223041] vmbr0: port 3(tap102i0) entered blocking state
Jul 9 14:10:58 prox01test kernel: [ 1664.223045] vmbr0: port 3(tap102i0) entered forwarding state
Jul 9 14:10:58 prox01test kernel: [ 1664.518521] vmbr0: port 3(tap102i0) entered disabled state
Jul 9 14:10:58 prox01test pvedaemon[6041]: start failed: command '/usr/bin/kvm -id 102 -name test102 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=5e6d2afd-29bc-4ae3-a12c-8584ba9c905d' -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/102.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 512 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:ec896ecda4a1' -drive 'file=/mnt/pve/iso/template/iso/ubuntu-18.04-live-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=iscsi://10.0.70.88/iqn.2005-10.org.freenas.ctl:target3/,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-generic,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=96:E7:59:A7:F0:98,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1
Jul 9 14:10:58 prox01test pvedaemon[1906]: root@pam end task UPID:prox01test:00001799:000288FC:5B42E04F:qmstart:102:root@pam: start failed: command '/usr/bin/kvm -id 102 -name test102 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=5e6d2afd-29bc-4ae3-a12c-8584ba9c905d' -smp '1,sockets=1,cores=1,maxcpus=1' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/102.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 512 -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:ec896ecda4a1' -drive 'file=/mnt/pve/iso/template/iso/ubuntu-18.04-live-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=iscsi://10.0.70.88/iqn.2005-10.org.freenas.ctl:target3/,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-generic,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=96:E7:59:A7:F0:98,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1
Jul 9 14:11:00 prox01test systemd[1]: Starting Proxmox VE replication runner...

Revise patch for FreeNAS patches to pvemanagerlib.js to patch into version 5.2-8

https://perldoc.perl.org/perlrun.html

-T
turns on "taint" so you can test them. Ordinarily these checks are done only when running setuid or setgid. It's a good idea to turn them on explicitly for programs that run on behalf of someone else whom you might not necessarily trust, such as CGI programs or any internet servers you might write in Perl. See perlsec for details. For security reasons, this option must be seen by Perl quite early; usually this means it must appear early on the command line or in the #! line for systems which support that construct.

Here is what I got when running the second patch command:

root@proxmox1:~/freenas-proxmox# patch -b /usr/share/perl5/PVE/Storage/ZFSPlugin.pm < perl5/PVE/Storage/ZFSPlugin.pm.patch
patching file /usr/share/perl5/PVE/Storage/ZFSPlugin.pm
Hunk #1 succeeded at 10 with fuzz 1.
Hunk #2 FAILED at 26.
Hunk #3 succeeded at 41 (offset 1 line).
Hunk #4 succeeded at 65 (offset 3 lines).
Hunk #5 succeeded at 165 (offset 5 lines).
Hunk #6 succeeded at 191 (offset 5 lines).
Hunk #7 FAILED at 233.
Hunk #8 succeeded at 264 (offset 10 lines).
2 out of 8 hunks FAILED -- saving rejects to file /usr/share/perl5/PVE/Storage/Z

After applying the patches manually, the UI got messed up when trying to look at VMs or adding VMs so I had to back them out.

In my case I was testing the SSL portion of the code but did not have HTTPS selected on the FreeNAS server. This was providing an error that "No Lun's were found" on the server.

Hi, is there any documentation about using/configuring multipath with this ZFS over ISCSI plugin?

Thanks.

Receiving the following messages in /var/log/syslog

Oct 16 06:25:14 pve pveproxy[7260]: file /etc/pve/storage.cfg line 12 (section 'iSCSIPool') - unable to parse value of 'freenas_password': unexpected property 'freenas_password'
Oct 16 06:25:14 pve pveproxy[7260]: file /etc/pve/storage.cfg line 13 (section 'iSCSIPool') - unable to parse value of 'freenas_use_ssl': unexpected property 'freenas_use_ssl'
Oct 16 06:25:14 pve pveproxy[7260]: file /etc/pve/storage.cfg line 14 (section 'iSCSIPool') - unable to parse value of 'freenas_user': unexpected property 'freenas_user'
Oct 16 06:25:23 pve pveproxy[7261]: file /etc/pve/storage.cfg line 12 (section 'iSCSIPool') - unable to parse value of 'freenas_password': unexpected property 'freenas_password'
Oct 16 06:25:23 pve pveproxy[7261]: file /etc/pve/storage.cfg line 13 (section 'iSCSIPool') - unable to parse value of 'freenas_use_ssl': unexpected property 'freenas_use_ssl'
Oct 16 06:25:23 pve pveproxy[7261]: file /etc/pve/storage.cfg line 14 (section 'iSCSIPool') - unable to parse value of 'freenas_user': unexpected property 'freenas_user'
Oct 16 06:25:38 pve pveproxy[7259]: file /etc/pve/storage.cfg line 12 (section 'iSCSIPool') - unable to parse value of 'freenas_password': unexpected property 'freenas_password'
Oct 16 06:25:38 pve pveproxy[7259]: file /etc/pve/storage.cfg line 13 (section 'iSCSIPool') - unable to parse value of 'freenas_use_ssl': unexpected property 'freenas_use_ssl'
Oct 16 06:25:38 pve pveproxy[7259]: file /etc/pve/storage.cfg line 14 (section 'iSCSIPool') - unable to parse value of 'freenas_user': unexpected property 'freenas_user'

Everything seems to be working but just need to check if a schema or library changed that the patches and plugin needs to reflect.

I followed the instructions on a freshly installed and updated (with enterprise updates) Proxmox VE 5.2-9 and there is no change in the ZFS over iSCSI interface.

I also tried a different browser after a server restart to see if that would change anything.

Thanks.

I was able to setup ZFS over iSCSI: the 3 nodes of my cluster have access to the storage.
I tried to move a harddrive of one of the VM to the new zfs over iscsi storage, and got the following:

create full clone of drive ide0 (san-x3650m5-hdd:vm-101-disk-0) Use of uninitialized value $target_id in numeric eq (==) at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 508. TASK ERROR: storage migration failed: error with cfs lock 'storage-san-x3650m5-ssd': Unable to find the target id for 3650.target.all at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 149.

multipath -ll output
root@dc-pmontt-flex-nodo02:/# multipath -ll mpath0 (36589cfc000000d68e17c6b320cb68bde) dm-5 FreeNAS,iSCSI Disk size=4.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw -+- policy='round-robin 0' prio=50 status=active
|- 4:0:0:30 sdc 8:32 active ready running
- 5:0:0:30 sdd 8:48 active ready running

followed the indications: setup ssh keys, validated ssh login working, applied the patches and so on, the cluster aparently is OK with the newly added storage.

Any ideas?
Thanks

edit: using:
Kernel Version Linux 5.0.18-1-pve #1 SMP PVE 5.0.18-2 (Fri, 2 Aug 2019 14:51:00 +0200) PVE Manager Version pve-manager/6.0-5/f8a710d7

Originally posted by @goberreuter in #39 (comment)

Should we also taint check other variables from the API calls?

I just tried to install the patch to run FreeNAS as the backend for Proxmox... But the FreeNAS option does not show in the options... what have I missed?

See the output from the patches below... I'm running Proxmox 5.3-9

Thanks for any feedback!

Ron

root@proxmox2:~/freenas-proxmox-master# patch -b /usr/share/pve-manager/js/pvemanagerlib.js < pve-manager/js/pvemanager
lib.js.patch
ch -b /usr/shapatching file /usr/share/pve-manager/js/pvemanagerlib.js
Hunk #1 succeeded at 5916 (offset 22 lines).
Hunk #2 succeeded at 30154 (offset 68 lines).
Hunk #3 succeeded at 30167 (offset 68 lines).
Hunk #4 succeeded at 30204 (offset 68 lines).
Hunk #5 succeeded at 30221 (offset 68 lines).
Hunk #6 succeeded at 30231 (offset 68 lines).
Hunk #7 succeeded at 30246 (offset 68 lines).
Hunk #8 succeeded at 30303 (offset 68 lines).
Hunk #9 succeeded at 30313 (offset 68 lines).
root@proxmox2:~/freenas-proxmox-master# patch -b /usr/share/perl5/PVE/Storage/ZFSPlugin.pm < perl5/PVE/Storage/ZFSPlugi
n.pm.patch
patching file /usr/share/perl5/PVE/Storage/ZFSPlugin.pm
root@proxmox2:~/freenas-proxmox-master# patch -b /usr/share/pve-docs/api-viewer/apidoc.js < pve-docs/api-viewer/apidoc.
js.patch
patching file /usr/share/pve-docs/api-viewer/apidoc.js
Hunk #1 succeeded at 33722 (offset 245 lines).
Hunk #2 succeeded at 33903 (offset 245 lines).
Hunk #3 succeeded at 34116 (offset 245 lines).
root@proxmox2:~/freenas-proxmox-master# apt-get install librest-client-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  librest-client-perl
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.3 kB of archives.
After this operation, 32.8 kB of additional disk space will be used.
Get:1 http://ftp.debian.org/debian stretch/main amd64 librest-client-perl all 273-1 [10.3 kB]
Fetched 10.3 kB in 0s (21.7 kB/s)
Selecting previously unselected package librest-client-perl.
(Reading database ... 46931 files and directories currently installed.)
Preparing to unpack .../librest-client-perl_273-1_all.deb ...
Unpacking librest-client-perl (273-1) ...
Setting up librest-client-perl (273-1) ...
Processing triggers for man-db (2.7.6.1-2) ...
root@proxmox2:~/freenas-proxmox-master# cp perl5/PVE/Storage/LunCmd/FreeNAS.pm /usr/share/perl5/PVE/Storage/LunCmd/Free
NAS.pm
root@proxmox2:~/freenas-proxmox-master# systemctl restart pvedaemon
root@proxmox2:~/freenas-proxmox-master# systemctl restart pveproxy
root@proxmox2:~/freenas-proxmox-master# systemctl restart pvestatd
root@proxmox2:~/freenas-proxmox-master#

This may be something I did or havnt done properly on FreeNAS.
I just configured this into PVE 6 (thanks for your time and good work!)

I created a VM with a disk attached to FreeNAS. However, the machine/conf never showed up on that node. I see a disk was made and is sitting as a zvol on the main pool.

Error during creation:
TASK ERROR: unable to create VM 101 - error with cfs lock 'storage-freenas': Unable to find the target id for iqn.2005-10.org.freenas.ctl at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 149.

So I went to remove the disk through proxmox, and got this error which prevented its deletion:
Could not find lu_name for zvol vm-101-disk-0 at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 116. (500)

Update to be compatible to version 5.2-7

Update patch for libpve-storage-perl to 5.0-33
Update patch for pve-docs to 5.3-1
Update patch for pve-manager to 5.3-5

Great job :) but

after patch proxmox 5.3-5 to 5.3-8 validation in ZFS over iSCSI not working properly.
There is validation problem on all targets accept FreeNAS-API. "Add" button is inactive (LIO, Comstar etc)

Revise patch for /usr/share/perl5/PVE/Storage/ZFSPlugin.pm to version patch into version 5.0-25

Hi

I found your script and it's exactly what i was looking for !

Unfortunatly i fail to run it.

I follow your instruction :

https://github.com/TheGrandWazoo/freenas-proxmox/blob/master/README.md
then i try setup the zfs over iscsi with freenas provider.

iqn.2005-10.org.freenas.ctl:scsi

It's just an example, i try many parameter but i never found somethings working .
strangely when i try get information on content in the storage badly mounted i get this :

command '/usr/bin/ssh -o 'BatchMode=yes' -i /etc/pve/priv/zfs/10.0.0.7_id_rsa [email protected] zfs list -o name,volsize,origin,type,refquota -t volume,filesystem -Hr' failed: exit code 255 (500)

i was thinking ssh was no more used . so what's happens ?

when i try the proxmox iscsi alone it found correctly my target ... by setting 10.0.0.7 as portal.
any idee

best regards and happy new years !

your email answer :
Romain,

One enhancement that I working on is to remove the dependency from SSH and use the API’s. Unfortunately it is in a backend Proxmox lib so it is taking a bit of time to create the enhancement so I don’t break other plugins. It just has to do with the listing of the contents of the ZFS pools and zols.

Please follow the steps at https://pve.proxmox.com/wiki/Storage:_ZFS_over_iSCSI that have to do with SSH between Proxmox VE and FreeNAS. The code segment should start out ‘mkdir /etc/pve/priv/zfs’

next question 👍

thanks you very much.
It's working now !

just have some questions. Actually i have done it by using root access for create keys as writed on the proxmox wiki url but is that possible to use lower security users as you looks like suggest by yourself "It just has to do with the listing of the contents of the ZFS pools and zols" ?
Really better that proxmox doesn't get root access to freenas.
Same way how use other api user account on freenas for does't set in form api username,password option with root ?
And did the option api username is used to for the ssh user login ?

best regards,

When trying to create a VM disk the following error occurs:

TASK ERROR: unable to create VM 100 - error with cfs lock 'storage-vmstorage_iscsi': Unable to connect to the FreeNAS API service at '192.168.77.2' using the 'http' protocol at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 241.

The disk name does appear in the FreeNAS tree though:

Setup is as follows:

• DAC 10GbE (Intel X520) connection running between the two servers, with a static IP set on each end
• FreeNAS 11.2-U5
• ProxMox 5.4-3 (config for ZFS connection below)
  

Hello all,

As I could read in the description, the plugin should work with PVE 5.3.

My Setup is following:
FreeNAS-11.2-U3 server with multipath iSCSI over Dual Infiniband ConnectX-2 HCA (as it is a PCIe 2.0 card, it gives me a workload of round about 25 to 28 Gbit/s over 8 iperf windows. I think this is a good base for iSCSI instead of NFS. The ZFS pool has 8 mirrored VDEVs (8x2 SAS HDDs) and a mirrored ZLOG (PCIe SSDs), the server has 256GB RAM. iSCSI is new to me and also your plugin patch, so maybe I made a mistake somewhere. PVE is version 5.4-5 and has three cluster nodes.

Here is the iSCSI testing setup:
In FreeNAS => Sharing => Block (iSCSI):
Target Global Configuration:
Base Name = iqn.2019-05.[TLD.DOMAIN.HOST]
ISNS Server = [NONE]
Pool Available Space Threshold (%) = 75
Portals:
Discovery Auth Method = NONE
Discovery Auth Group = NONE
IP Address = [SUBNET1 IP]
Port = 3260
IP Address = [SUBNET2 IP]
Port = 3260
Initiators:
Group ID = 1
Initiators = ALL
Authorized Networks = ALL
Authorized Access:
Nothing changed or added.
Target:
Target Name = test-target1
Portal Group ID = 1
Initiators Group ID = 1
Auth Method = NONE
Extents:
Extent Name = test-target1
Extent Type = Device
Device = [POOLNAME]/[ZVOLNAME] (20T)
Available Space Threshold (%) = 75
LUN RPM = SSD

The test-target1 is recognized by the PVE cluster nodes.
If I want to create a new VM, I have to create a ZFS over iSCSI device first in the storage section of PVE. This is no problem because it only creates the meta data entries.
If I create a new VM and select that previously created device I will get following error message:

cannot create '[POOLNAME]/[ZVOLNAME]/vm-100-disk-0': parent is not a filesystem
TASK ERROR: unable to create VM 100 - error with cfs lock 'storage-vm100-disk0': command '/usr/bin/ssh -o 'BatchMode=yes' -i /etc/pve/priv/zfs/XXX.XXX.XXX.XXX_id_rsa [email protected] zfs create -s -b 4k -V 52428800k [POOLNAME]/[ZVOLNAME]/vm-100-disk-0' failed: exit code 1

The parent is the zvol device which cannot be a dataset and which cannot be mounted by default.
The command which failed was executed over a SSH connection and is a shell command. Is that correct? As I read this plugin patch should use the API and not SSH? This is confusing to me.
The patch should contain a documentation with a little bit more information to get a clearer picture how it should work. At the moment it is not working as expected, I assume that this is related to the newer PVE version, but because of the basic documentation I could be mistaken.
Thank you for your work. It is exactly what I`m missing in Proxmox.

Best regards, Ralf

When resizing a disk of an online guest, the zvol itself is correctly resized, but the guest isn't notified about the new size.

I'm testing with VirtIO SCSI disk on CentOS 7 guest. When using an LVM thin storage, just after a resize, I see in logs :

août 20 12:39:37 localhost.localdomain kernel: sd 2:0:0:1: Capacity data has changed
août 20 12:39:37 localhost.localdomain kernel: sd 2:0:0:1: [sdb] 6365184 512-byte logical blocks: (3.25 GB/3.03 GiB)
août 20 12:39:37 localhost.localdomain kernel: sdb: detected capacity change from 3246391296 to 3258974208

When doing exactly the same with ZFS over iSCSI with FreeNAS API, I get

août 20 12:43:30 localhost.localdomain kernel: sd 2:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical

And the disk still has the previous size. I either need to reboot, or rescan with :

echo 1 > /sys/class/block/sdb/device/rescan

It seems strange that both storages doesn't behave the same from the guest POV.

Hi,
I am getting this error message when I try to create a new disk (clone, new vm, new disk in existing VM):

update VM 103: -scsi1 midgard-iscsi:1 TASK ERROR: error with cfs lock 'storage-midgard-iscsi': Unable to create lun /dev/zvol/vmtank/proxmox/disks/vm-103-disk-2 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 160.

error with cfs lock 'storage-midgard-iscsi': Unable to create lun /dev/zvol/vmtank/proxmox/disks/vm-126-disk-1 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 160. at /usr/share/perl5/PVE/API2/Qemu.pm line 1253. (500)

ssh'ing in with ssh -i /etc/pve/priv/zfs/10.0.5.1_id_rsa -p [email protected] works fine.

Tested on three different Proxmox hosts, all at:

Kernel Version Linux 4.15.18-9-pve #1 SMP PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100)
PVE Manager Version pve-manager/5.3-6/37b3c8df

Patched files two days ago with the latest available from your github.

FreeNAS 11.2-FINAL.

Had it running earlier (prior to 11.2-FINAL update) so I currently have 6 disks running over iSCSI that was created with your patches. Not sure what broke it as I didn't notice it until now (existing disk are working fine).

Thanks for any help!

The password field displays the password in clear text and after saving the password field still shows the password in clear text)

Why have the REST/Client.pm be part of this patch when you can just install it using 'apt-get install librest-client-perl'.

Hello.

I am running FreeNAS 11.2U6 and ProxMox 5.4-13 (I've attached 'pveversion -v' to the end of this).

Sometime around August my system developed the problem where iSCSI extents and zvols were not being deleted from FreeNAS (by using the ProxMox WebGUI) upon the 4 conditions:

1. deleting a VM
   Error Message from WebGUI:

Could not remove disk 'VMs:vm-150-disk-0', check manually: error with cfs lock 'storage-VMs': Unable to delete lun /dev/zvol/Base_Vol_1/ProxMox-VMs/vm-150-disk-0 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 212.
error with cfs lock 'storage-VMs': Unable to delete lun /dev/zvol/Base_Vol_1/ProxMox-VMs/vm-150-disk-0 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 212.

2. Converting a VM to a template
   Error Message from WebGUI:

TASK ERROR: error with cfs lock 'storage-VMs': Unable to delete lun /dev/zvol/Base_Vol_1/ProxMox-VMs/vm-139-disk-1 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 212.

3. Rolling back to a previous snapshot
   Error Message from WebGUI:

TASK ERROR: Unable to delete lun /dev/zvol/Base_Vol_1/ProxMox-VMs/vm-140-disk-1 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 212.

4. Deleting a snapshot
   Error Message from WebGUI:

/ProxMox-VMs/vm-140-state-snap1 at /usr/share/perl5/PVE/Storage/LunCmd/FreeNAS.pm line 212.

When I have to do any of the above things, I have to perform the action in the ProxMox WebGUI and then complete the action by manually deleting the extents and zvols in the FreeNAS WebGUI.

This has caused a side issue of ProxMox being unaware of the deleted disks. IE If I delete vm 140 from ProxMox, manually delete the extents and zvols for the vm 140 disks (eg disk 0 and disk 1), when I create a new vm 140, the first disk is vm-140-disk-2.

To try and fix this, I have made sure to update ProxMox, FreeNAS, and done a 'git pull' for the freenas-proxmox git repo. I then followed the instruction in the README. I got error messages that the files had already been patched, but attempted to patch anyway. I followed the remaining instructions in the README.

That did not work. I then re-installed the pve-manager, libpve-storage-perl, and pve-docs packages using 'apt-get --reinstall install'. I then patched the files again without any errors. I followed the remaining instructions in the README.

Still, this problem persists.

This is perplexing. All other functions where ProxMox integrates with FreeNAS work correctly - I can create snapshots, clone VMs, make new VMs, make new disks in VMs, etc.

Please help :)

Results of 'pveversion -v':

proxmox-ve: 5.4-2 (running kernel: 4.15.18-21-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-9
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-55
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

Have an optional field to use a separate IP address to call the FreeNAS APIs instead of the portal IP. That way the FreeNAS Web GUI can be attached to a separate NIC and not the iSCSI NIC.
The portal IP will still be used for iSCSI and SSH cmd because of the backend Proxmox Storage plugins and libs.

An error is generated when trying to rollback a linked clone from a template. I get the following error:

Could not find lu_name for zvol base-9000-disk-0/vm-119-disk-0 at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 116.

119 is the clone, 9000 is the template.

I tested it with a full clone and it works fine.

I'll probably poke around the code for this but I'm going to report it in case someone else gets it/has gotten it.

Do you have to perform the instruction steps for each node in a cluster? Is it good enough to perform the steps on the master node?

Most of the ZFS and iSCSI commands are done via the API, but there is still a piece that uses the SSH command. I believe it is for the "Content" portion of storage. Not sure if it can be removed/SUPERed but will look into it in my free time.

If I move a working UEFI VM from local-zfs stoarge to FreeNAS (11.1-U6) it will not boot into the OS. It will only boot to the UEFI shell.
Here is the syslog of the move of the efidisk and the scsi0 disk (there are no other disks associated with this VM):

Dec 17 09:15:22 pve pvedaemon[13866]: <root@pam> move disk VM 108: move --disk efidisk0 --storage freenas-vm-storage
Dec 17 09:15:22 pve pvedaemon[13866]: <root@pam> starting task UPID:pve:0000455B:00FAEE12:5C17D9AA:qmmove:108:root@pam:
Dec 17 09:15:22 pve pvedaemon[17755]: FreeNAS::lun_command : create_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:15:22 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:22 pve pvedaemon[17755]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:22 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:22 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:22 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::freenas_get_first_available_lunid : return 4
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0):name : lun not found
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::create_lu(lun_path=/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0, lun_id=4) : blocksize convert 4k = 4096
Dec 17 09:15:23 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:25 pve pvedaemon[17755]: FreeNAS::API::create_extent(lun_path=/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0, lun_bs=4096) : sucessfull
Dec 17 09:15:25 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::API::create_target_to_extent(target_id=5, extent_id=9, lun_id=4) : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::create_lu(lun_path=/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0, lun_id=4) : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::lun_command : add_view()
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::lun_command : list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:26 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0):name : lun found /dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::lun_command : list_view(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0):lun-id : lun found 4
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::lun_command : list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:27 pve pvedaemon[17755]: FreeNAS::list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0): naa found 0x6589cfc00000011e2d78563962a47980
Dec 17 09:15:30 pve pvedaemon[13866]: <root@pam> end task UPID:pve:0000455B:00FAEE12:5C17D9AA:qmmove:108:root@pam: OK
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::lun_command : list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0):name : lun found /dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::lun_command : list_view(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0):lun-id : lun found 4
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::lun_command : list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:41 pve pvedaemon[13866]: FreeNAS::list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0): naa found 0x6589cfc00000011e2d78563962a47980
Dec 17 09:15:44 pve pvedaemon[13866]: <root@pam> move disk VM 108: move --disk scsi0 --storage freenas-vm-storage
Dec 17 09:15:44 pve pvedaemon[13866]: <root@pam> starting task UPID:pve:00004742:00FAF700:5C17D9C0:qmmove:108:root@pam:
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::lun_command : create_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::freenas_get_first_available_lunid : return 5
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:45 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1):name : lun not found
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::create_lu(lun_path=/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1, lun_id=5) : blocksize convert 4k = 4096
Dec 17 09:15:46 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:47 pve pvedaemon[18242]: FreeNAS::API::create_extent(lun_path=/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1, lun_bs=4096) : sucessfull
Dec 17 09:15:47 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:48 pve pvedaemon[18242]: FreeNAS::API::create_target_to_extent(target_id=5, extent_id=10, lun_id=5) : sucessfull
Dec 17 09:15:48 pve pvedaemon[18242]: FreeNAS::create_lu(lun_path=/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1, lun_id=5) : sucessfull
Dec 17 09:15:48 pve pvedaemon[18242]: FreeNAS::lun_command : add_view()
Dec 17 09:15:48 pve pvedaemon[18242]: FreeNAS::lun_command : list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:15:48 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:48 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:48 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:48 pve pvedaemon[18242]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1):name : lun found /dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::lun_command : list_view(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1):lun-id : lun found 5
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::lun_command : list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:15:49 pve pvedaemon[18242]: FreeNAS::list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1): naa found 0x6589cfc000000af7bf4b0d8d32fb8cff
Dec 17 09:16:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 17 09:16:03 pve systemd[1]: Started Proxmox VE replication runner.
Dec 17 09:17:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 17 09:17:01 pve CRON[10388]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec 17 09:17:04 pve systemd[1]: Started Proxmox VE replication runner.
Dec 17 09:17:48 pve pvedaemon[2186]: worker exit
Dec 17 09:17:48 pve pvedaemon[2183]: worker 2186 finished
Dec 17 09:17:48 pve pvedaemon[2183]: starting 1 worker(s)
Dec 17 09:17:48 pve pvedaemon[2183]: worker 13799 started
Dec 17 09:17:58 pve pvedaemon[13866]: <root@pam> end task UPID:pve:00004742:00FAF700:5C17D9C0:qmmove:108:root@pam: OK

Here is syslog when I start the moved VM now on FreeNAS storage

Dec 17 09:21:37 pve pvedaemon[18680]: start VM 108: UPID:pve:000048F8:00FB80C3:5C17DB21:qmstart:108:root@pam:
Dec 17 09:21:37 pve pvedaemon[16207]: <root@pam> starting task UPID:pve:000048F8:00FB80C3:5C17DB21:qmstart:108:root@pam:
Dec 17 09:21:37 pve pvedaemon[18680]: FreeNAS::lun_command : list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:21:37 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:37 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:37 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:37 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:37 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:37 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0):name : lun found /dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::lun_command : list_view(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0):lun-id : lun found 4
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::lun_command : list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0)
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:38 pve pvedaemon[18680]: FreeNAS::list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-0): naa found 0x6589cfc00000011e2d78563962a47980
Dec 17 09:21:41 pve pvedaemon[18680]: Use of uninitialized value $type in string eq at /usr/share/perl5/PVE/QemuServer.pm line 2026.
Dec 17 09:21:41 pve pvedaemon[18680]: FreeNAS::lun_command : list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:21:41 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:41 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:41 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:41 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:41 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:41 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:41 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1):name : lun found /dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::lun_command : list_view(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1):lun-id : lun found 5
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::lun_command : list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:42 pve pvedaemon[18680]: FreeNAS::list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1): naa found 0x6589cfc000000af7bf4b0d8d32fb8cff
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::lun_command : list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:45 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1):name : lun found /dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::lun_command : list_view(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::list_lu(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1):lun-id : lun found 5
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::lun_command : list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1)
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_globalconfig : target_basename=iqn.2017-12.com.lahansons
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_target() : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_target_to_extent() : change undef iscsi_lunid to 0
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::api_call : setup : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::get_extent : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::API::freenas_list_lu : sucessfull
Dec 17 09:21:46 pve pvedaemon[18680]: FreeNAS::list_extent(/dev/zvol/DataDump/data/proxmox-vm/vm-108-disk-1): naa found 0x6589cfc000000af7bf4b0d8d32fb8cff
Dec 17 09:21:50 pve systemd[1]: Started 108.scope.
Dec 17 09:21:50 pve systemd-udevd[19097]: Could not generate persistent MAC address for tap108i0: No such file or directory
Dec 17 09:21:52 pve pvedaemon[16207]: <root@pam> end task UPID:pve:000048F8:00FB80C3:5C17DB21:qmstart:108:root@pam: OK

This is the configuration of the VM after the move.

root@pve:~# cat /etc/pve/qemu-server/108.conf 
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 4
efidisk0: freenas-vm-storage:vm-108-disk-0,size=128K
ide2: none,media=cdrom
memory: 8192
name: daily-driver
net0: virtio=52:83:30:59:AF:08,bridge=vmbr10
numa: 0
ostype: l26
scsi0: freenas-vm-storage:vm-108-disk-1,size=60G
scsihw: virtio-scsi-pci
smbios1: uuid=5866a3b0-9af5-4cd7-9d6f-55a99551cea9
sockets: 1
unused0: local-zfs:vm-108-disk-0
unused1: local-zfs:vm-108-disk-1
vga: qxl
vmgenid: 2651efe3-d1a1-46a3-a08e-000a7b322431

I also tried removing and recreating the EFI disk on FreeNAS, but there was no change in behavior. I also tried using cache=writeback on scsi0 without success. See attached screenshots. The last screen shot is the VM booting after moving it back to local-zfs.

I honestly don't know if this is an issue with the FreeNAS patches (using commit 82fe272) or a problem with Proxmox, but I'm seeing the errors I outlined here when utilizing FreeNAS as my VM storage. When performing any operation from the command line, i.e. qm start XXX or qm clone XXX YYY... I see unrecognized command 'list_extent' It does not seem to bother my Linux VM's that I create on the FreeNAS storage, but the VM's that I move from internal local-zfs Proxmox storage will "start," but only boot to the UEFI shell or iPXE. Moving or cloning VM's to/from the local-zfs storage does not exhibit this error, hence I think it may be relevant to these patches. Let me know if you need more information.

Just tried on a test VM, and everything seems to be working as expected. Thanks for your work on this !
Now, I'm a bit reluctant to start relying on this as long as it's not upstream, have you tried to push this upstream already ? Any blocker for it to be merged ?

Update pve-manager.js.patch to reflect pve-manager version 5.2-6
Update pve-docs.js.patch to reflect pve-docs.js version 5.2-5

Wondering if there is an interest to have periodic snapshots available from the Proxmox VE GUI when FreeNAS is selected. After some testing I noticed that the periodic snapshots on the FreeNAS can NOT be rolled back seemingly due to the Proxmox VE holding on to the iSCSI target. But you can rollback a Proxmox VE Snapshot so I am assuming Proxmox VE backend does some magic to make that happen.

Community's thoughts?

Well in talking to the FreeNAS developers on what is happening with the issue of a zvol being deleted where it be in a move, delete or other means from the Proxmox side.
I am wondering if I should had a feature to "Create" a target for each VM. It would be a check box that would either parse/select/create a Single Target per VM or use the target under the global Target field.

FreeNAS is also adding a feature to "force" the deletion of the zvol if the target is "In Use" but that is not due to 11.3_1/2.

What do you y'all think?

Hello,

I am currently experiencing what seems to be a simple permissions issue when trying to access my shares. I have followed this guide here: https://forum.proxmox.com/threads/guide-setup-zfs-over-iscsi-with-pve-5x-and-freenas-11.54611/ to the tee but it still seems that I cannot connect.

Got through the patches and SSH key creation just fine, but when trying to add the share in Proxmox there is a permissions error:

I copy-pasted the error message shown in Proxmox into the console and get this:

I have tried or verified:

• Manually logging into FreeNAS through the Proxmox console using ssh@[name of freenas server], works just fine using password authentication and I can successfully run the zfs list command,
• That I enabled password authentication and allowed login as root on the FreeNAS end.
• The public key has been uploaded to the FreeNAS server and that it matches the one in Proxmox.

My setup is as follows:

• DAC 10GbE (Intel X520) connection running between the two servers, with a static IP set on each end
• FreeNAS 11.2-U5
• ProxMox 5.4-3 (config for ZFS connection below)
  

I would imagine I'm just missing something simple here, as I am still new to Linux and making mistakes here and there. Any help would be appreciated. Thanks!