Since 1.5.3 I always have "XXX is apparently in use by the system; will not make a filesystem here!"

[o10s]

Hello,
I am tired of the new longhorn honestly it has been working perfectly fine until 1.5.3 since then I ALWAYS have the following kind of errors:

MountVolume.MountDevice failed for volume "pvc-a21154e9-21e0-4ae7-b56b-459b211f84e8" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-a21154e9-21e0-4ae7-b56b-459b211f84e8" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/c8ddd8ae2636ae5da1d521fd4366178afe59764308a2d6a77f214e1294c258f7/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.46.4 (18-Aug-2021) /dev/longhorn/pvc-a21154e9-21e0-4ae7-b56b-459b211f84e8 is apparently in use by the system; will not make a filesystem here! )

And I have absolutely no idea why...

I always have to do the following on my nodes: systemctl restart multipathd.service for it to work again

Anyone has any idea how to really solve it once and for all ?

Thank you

[roger-ryao]

Hi @o10s :
Have you followed the steps outlined in https://longhorn.io/kb/troubleshooting-volume-with-multipath/?

[o10s]

Hi @roger-ryao

Yes I did follow that which looked like it solved the issue back when I first encountered the problem (the restart multipath command comes from there)

However whenever a pod restarts or a new pvc is created I have the same kind of error message...

Which never happened before 1.5.3

[mantissahz]

@o10s Could you show the info by multipathd -t multipath -t?

[o10s]

@roger-ryao so the zip file is too big so I had to put it on weTransfer here: https://we.tl/t-gFSJLNvMkm
@mantissahz it says -t is an invalid argument...

[mantissahz]

@o10s sorry about the typo, it should multipath -t (not multipathd)

[o10s]

OH GOD WAIT
I think I am dumb as hell, in the multipath.conf the devnode is NOT on the blacklist but on a "default" section I will change it and check it out again

[o10s]

Yeah it solved it... sorry for the trouble !