Pete's Log: Multipath woes

Entry #2005, (Coding, Hacking, & CS stuff)
(posted when I was 43 years old.)

The pi cluster (currently thinking of naming it LRPC - Laundry Room Pi Cluster) gave me some more woes this evening. I made some progress with MetalLB that I am happy with, but Longhorn suddenly started giving me volumes that wouldn't mount in my pods. kubectl describe would give me output of this nature:

Normal SuccessfulAttachVolume 92s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-ffd85c77-e041-4889-afeb-145ed6cf3714" Warning FailedMount 22s (x8 over 86s) kubelet MountVolume.MountDevice failed for volume "pvc-ffd85c77-e041-4889-afeb-145ed6cf3714" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-ffd85c77-e041-4889-afeb-145ed6cf3714" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ffd85c77-e041-4889-afeb-145ed6cf3714/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.5 (07-Jan-2020) /dev/longhorn/pvc-ffd85c77-e041-4889-afeb-145ed6cf3714 is apparently in use by the system; will not make a filesystem here!

The internet could not seem to find any instances of that error message pertaining to Kubernetes. The error on the last line comes from mkfs.ext4 and usually indicates you've asked it to format a device that has a partition table. But it seems it can happen if the device is otherwise busy.

Luckily Longhorn has this nice feature that lets you attach a volume directly to one of your cluster nodes. So I did that, which allowed me to try to troubleshoot why the volume was misbehaving. So I attached the volume to one of my worker nodes, where it became /dev/sdb.

Attempting to mount /dev/sdb gave me

mount: /dev/sdb already mounted or /mnt/tmp/ busy

fdisk told me there was no partition or anything on the device.

mkfs.ext4 /dev/sdb gave me the same "is apparently in use by the system; will not make a filesystem here!" error.

Finally, lsof gave me some insight: multipath had /dev/sdb open, which I'm guessing is what was blocking things. That discovery led me to searching for longhorn multipath, which led me to this Longhorn KB article on troubleshooting volume with multipath. I followed their guidance and added

blacklist { devnode "^sd[a-z0-9]+" }

to /etc/multipath.conf on each of my nodes. Then a quick systemctl restart multipathd.service and suddenly everything is working as it should.