Truncated disk ID about csi-driver HOT 9 CLOSED

I think we should use the best-match approach. First match the full id, then try to match the trimmed id.

from csi-driver.

udev is what creates the names under /dev/disk/by-id , specifically '/lib/udev/scsi_id` binary [1]. Since this is a known naming scheme I just always stripped and assembled the path by taking the first 21 chars of a uuid of the disk id, which is its serial number. See here [2]
In your case it seems that that udev didn't create a world-wide-name(wwn) symlink using the short version.
Please follow those steps to gather more info:

• uname -a
• cat /etc/os-release
• ls -la /dev/disk/by-id/
• find the device that the created alias is pointing at, for example /dev/sda
• do what udev does /lib/udev/scsi_id --export --whitelisted -d /dev/sda (or whatever device name found earlier)

This udev rule combine all of this together /lib/udev/rules.d/60-persistent-storage.rules . On every RHCOS this uses the ID_SERIAL_SHORT to create the symlink. We need to compare this to what you have to understand where this is coming from.

[1] https://github.com/systemd/systemd/tree/a859abf062cef1511e4879c4ee39c6036ebeaec8/src/udev/scsi_id
[2]

from csi-driver.

[root@worker3 ~]# uname -a
Linux worker3.k8.local 5.4.8-200.fc31.x86_64 #1 SMP Mon Jan 6 16:44:18 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[root@worker3 ~]# cat /etc/os-release
NAME=Fedora
VERSION="31.20200113.3.1 (CoreOS)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora CoreOS 31.20200113.3.1"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='31.20200113.3.1'

[root@worker3 ~]# ls -la /dev/disk/by-id/
total 0
drwxr-xr-x. 2 root root 200 May 8 05:30 .
drwxr-xr-x. 8 root root 160 May 8 05:18 ..
lrwxrwxrwx. 1 root root 9 May 8 05:23 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx. 1 root root 9 May 8 05:30 scsi-0QEMU_QEMU_HARDDISK_aefb7ca6-ee05-485e-b186-9d93dd22f33a -> ../../sda
lrwxrwxrwx. 1 root root 9 May 8 05:23 virtio-e38c0155-e805-401f-8 -> ../../vda
lrwxrwxrwx. 1 root root 10 May 8 05:23 virtio-e38c0155-e805-401f-8-part1 -> ../../vda1
lrwxrwxrwx. 1 root root 10 May 8 05:23 virtio-e38c0155-e805-401f-8-part2 -> ../../vda2
lrwxrwxrwx. 1 root root 10 May 8 05:23 virtio-e38c0155-e805-401f-8-part3 -> ../../vda3
lrwxrwxrwx. 1 root root 10 May 8 05:23 virtio-e38c0155-e805-401f-8-part4 -> ../../vda4

[root@worker3 ~]# /lib/udev/scsi_id --export --whitelisted -d /dev/sda
ID_SCSI=1
ID_VENDOR=QEMU
ID_VENDOR_ENC=QEMU\x20\x20\x20\x20
ID_MODEL=QEMU_HARDDISK
ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
ID_REVISION=2.5+
ID_TYPE=disk
ID_SERIAL=0QEMU_QEMU_HARDDISK_aefb7ca6-ee05-485e-b186-9d93dd22f33a
ID_SERIAL_SHORT=aefb7ca6-ee05-485e-b186-9d93dd22f33a
ID_SCSI_SERIAL=aefb7ca6-ee05-485e-b186-9d93dd22f33a

[root@worker3 ~]# /lib/udev/scsi_id --version
v243.5-1.fc31

Its looking like in this version of fedora coreos ID_SERIAL_SHORT is a bit longer than expected
Also the udev rules look like they are using ID_SERIAL in FCOS

# SCSI devices
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?", IMPORT{program}="scsi_id --export --whitelisted -d $devnode", ENV{ID_BUS}="scsi"
KERNEL=="cciss", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}!="?", IMPORT{program}="scsi_id --export --whitelisted -d $devnode", ENV{ID_BUS}="cciss"
KERNEL=="sd|sr*|cciss*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="?", SYMLINK+="disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}"
KERNEL=="sd|cciss*", ENV{DEVTYPE}=="partition", ENV{ID_SERIAL}=="?*", SYMLINK+="disk/by-id/$env{ID_BUS}-$env{ID_SERIAL}-part%n"

from csi-driver.

What is the output of sudo udevadm info -q all /dev/sda ?

from csi-driver.

Followup from out-of-band conversation: neither Fedora nor upstream seem to be using ID_SERIAL_SHORT in their udev rules.

from csi-driver.

could not reporduce this on FC31 from April

[root@test-fchos31-short-serial-id ~]# ll /dev/disk/by-id/
total 0
lrwxrwxrwx. 1 root root  9 May 25 10:44 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx. 1 root root  9 May 25 10:44 ata-QEMU_DVD-ROM_QM00004 -> ../../sr1
lrwxrwxrwx. 1 root root  9 May 25 10:44 scsi-0QEMU_QEMU_HARDDISK_177a53cb-8350-47d0-a -> ../../sda

[root@test-fchos31-short-serial-id ~]# cat /etc/os
os-release  ostree/     
[root@test-fchos31-short-serial-id ~]# cat /etc/os-release 
NAME=Fedora
VERSION="31.20200505.3.0 (CoreOS)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora CoreOS 31.20200505.3.0"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='31.20200505.3.0'

[root@test-fchos31-short-serial-id ~]# udevadm info -q all /dev/sda
P: /devices/pci0000:00/0000:00:06.0/virtio2/host2/target2:0:0/2:0:0:1/block/sda
N: sda
L: 0
S: disk/by-id/scsi-0QEMU_QEMU_HARDDISK_177a53cb-8350-47d0-a
S: disk/by-path/pci-0000:00:06.0-scsi-0:0:0:1
E: DEVPATH=/devices/pci0000:00/0000:00:06.0/virtio2/host2/target2:0:0/2:0:0:1/block/sda
E: DEVNAME=/dev/sda
E: DEVTYPE=disk
E: MAJOR=8
E: MINOR=0
E: SUBSYSTEM=block
E: USEC_INITIALIZED=29143536
E: ID_SCSI=1
E: ID_VENDOR=QEMU
E: ID_VENDOR_ENC=QEMU\x20\x20\x20\x20
E: ID_MODEL=QEMU_HARDDISK
E: ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
E: ID_REVISION=2.5+
E: ID_TYPE=disk
E: ID_SERIAL=0QEMU_QEMU_HARDDISK_177a53cb-8350-47d0-a
E: ID_SERIAL_SHORT=177a53cb-8350-47d0-a
E: ID_SCSI_SERIAL=177a53cb-8350-47d0-ab5d-3a6009bc5d91
E: ID_BUS=scsi
E: ID_PATH=pci-0000:00:06.0-scsi-0:0:0:1
E: ID_PATH_TAG=pci-0000_00_06_0-scsi-0_0_0_1
E: DEVLINKS=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_177a53cb-8350-47d0-a /dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:1
E: TAGS=:systemd:

from csi-driver.

@bingman-ux can you upgrade your FCOS and see if that still happens?

from csi-driver.

To add a bit of flavor to this issue, I upgraded ovirt from 4.3.10 (which was working beautifully) to 4.4.0 yesterday and now am running into this issue. It appears that 'something' has changed that means that the paths are now full length.

The VM is using a 4.4 compatibility version, with the Q35 with Legacy bios chipset, and the q35-rhel8.1.0 emulated machine.

I can see the mechanism working in the background to attach the volume to the machine, but from OKD's perspective the volume attaches, but then fails to mount:
MountVolume.MountDevice failed for volume "pvc-3a65ddfb-8268-43f3-a748-750b591eaeca" : rpc error: code = Unknown desc = lstat /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_91e39dce-970a-41ba-8: no such file or directory

Following through some of the troubleshooting above:

Linux bee-2lfzn-worker-0-ctvn9 5.6.13-200.fc31.x86_64 #1 SMP Thu May 14 23:26:14 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[root@bee-2lfzn-worker-0-ctvn9 ~]# cat /etc/os-release
NAME=Fedora
VERSION="31.20200521.20.0 (CoreOS)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora CoreOS 31.20200521.20.0"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='31.20200521.20.0'

[root@bee-2lfzn-worker-0-ctvn9 ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  100G  0 disk
├─sda1   8:1    0  384M  0 part /boot
├─sda2   8:2    0  127M  0 part /boot/efi
├─sda3   8:3    0    1M  0 part
└─sda4   8:4    0 99.5G  0 part /sysroot
sdb      8:16   0    1G  0 disk
sdc      8:32   0    5G  0 disk
sdd      8:48   0    5G  0 disk
sr0     11:0    1 1024M  0 rom

[root@bee-2lfzn-worker-0-ctvn9 ~]# ls /dev/disk/by-id
ata-QEMU_DVD-ROM_QM00005                                             scsi-0QEMU_QEMU_HARDDISK_0d9a7600-370e-45fa-942c-7f7adbbab308-part2  scsi-0QEMU_QEMU_HARDDISK_91e39dce-970a-41ba-88c9-484e8a2e47a1
scsi-0QEMU_QEMU_HARDDISK_0d9a7600-370e-45fa-942c-7f7adbbab308        scsi-0QEMU_QEMU_HARDDISK_0d9a7600-370e-45fa-942c-7f7adbbab308-part3  scsi-0QEMU_QEMU_HARDDISK_a7ba2ac9-25be-4d60-9500-313d8251ca02
scsi-0QEMU_QEMU_HARDDISK_0d9a7600-370e-45fa-942c-7f7adbbab308-part1  scsi-0QEMU_QEMU_HARDDISK_0d9a7600-370e-45fa-942c-7f7adbbab308-part4  scsi-0QEMU_QEMU_HARDDISK_aab8c5a3-caa2-4249-bdcb-744d5f22e8c3

[root@bee-2lfzn-worker-0-ctvn9 ~]# /lib/udev/scsi_id --export --whitelisted -d /dev/sdc
ID_SCSI=1
ID_VENDOR=QEMU
ID_VENDOR_ENC=QEMU\x20\x20\x20\x20
ID_MODEL=QEMU_HARDDISK
ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
ID_REVISION=2.5+
ID_TYPE=disk
ID_SERIAL=0QEMU_QEMU_HARDDISK_91e39dce-970a-41ba-88c9-484e8a2e47a1
ID_SERIAL_SHORT=91e39dce-970a-41ba-88c9-484e8a2e47a1
ID_SCSI_SERIAL=91e39dce-970a-41ba-88c9-484e8a2e47a1

I think this is the pertinent part of the persistent storage rules file

# SCSI devices
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?*", IMPORT{program}="scsi_id --export --whitelisted -d $devnode", ENV{ID_BUS}="scsi"

Either way, I did a search through the file and there are no mentions of ID_SHORT_SERIAL, confirming what @lucab found.

Anything else that I can do to help troubleshoot this one?

from csi-driver.

I've managed to work around this issue by removing the volume id string trim from pkg/service/node.js:166 (container at quay.io/crreeves/ovirt-csi-driver and quay.io/crreeves/ovirt-csi-operator.

I'm not really sure of the best way to productionize it, maybe grepping the storage rules file for ID_SHORT_SERIAL, if it doesn't exist then don't trim the ID, otherwise apply the 20 char limit?

from csi-driver.