Globbing in fstab together with mountall (running mounts in parallel) causing directories not found

[lsylsy2]

Describe the bug

I'm using fstab configuration like this in a snapraid+mergerfs build.
/media/disk*/parity2 /media/parity2 fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs,func.getattr=newest 0 0
there are directories under each disk called parity0, parity1, etc. which indicates different levels of parity and used to store data with different importance.
However, when the system is booting, mount.fuse is ran together with the disks, so when mergerfs tries to find "/media/disk*/parity2", it will see nothing (because /media/disk0 is empty waiting to be mount, and so on), so fails.
after the system is booted up, umount /media/parity2 then mount /media/parity2 fixes the problem.
not using globbing fixes the problem, but it's ugly.
/media/disk1/parity0:/media/disk2/parity0:/media/disk3/parity0:/media/disk4/parity0:/media/disk5/parity0:/media/disk6/parity0:/media/disk7/parity0:/media/disk8/parity0

To Reproduce

# ls /media/disk*
/media/disk1:
lost+found  parity0  parity1  parity1.content  parity2  parity2.content
/media/disk2:
lost+found  parity0  parity1  parity1.content  parity2  parity2.content
/media/disk3:
lost+found  parity0  parity1.content  parity1.parity  parity2.content  parity2.parity
/media/disk4:
lost+found  parity0  parity1  parity1.content  parity2  parity2.content
/media/disk5:
lost+found  parity0  parity1  parity1.content  parity2.content  parity2.parity
/media/disk6:
lost+found  parity0  parity1  parity1.content  parity2  parity2.content
/media/disk7:
lost+found  parity0  parity1  parity1.content  parity2  parity2.content
/media/disk8:
lost+found  parity0  parity1  parity1.content  parity2  parity2.content

# cat /etc/fstab
# some entries
/dev/disk/by-uuid/1 /media/disk1/ ext4 noatime,defaults 0 0
/dev/disk/by-uuid/2 /media/disk2/ ext4 noatime,defaults 0 0
/dev/disk/by-uuid/3 /media/disk3/ ext4 noatime,defaults 0 0
/dev/disk/by-uuid/4 /media/disk4/ ext4 noatime,defaults 0 0
/dev/disk/by-uuid/5 /media/disk5/ ext4 noatime,defaults 0 0
/dev/disk/by-uuid/6 /media/disk6/ ext4 noatime,defaults 0 0
/dev/disk/by-uuid/7 /media/disk7/ ext4 noatime,defaults 0 0
/dev/disk/by-uuid/8 /media/disk8/ ext4 noatime,defaults 0 0

# this will succeed
/media/disk1/parity0:/media/disk2/parity0:/media/disk3/parity0:/media/disk4/parity0:/media/disk5/parity0:/media/disk6/parity0:/media/disk7/parity0:/media/disk8/parity0 /media/parity0 fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs,func.getattr=newest 0 0
# these will fail
/media/disk*/parity1 /media/parity1 fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs,func.getattr=newest 0 0
/media/disk*/parity2 /media/parity2 fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs,func.getattr=newest 0 0

Expected behavior

use some options to wait for dependencies to be mounted, while still keep the beauty of globbing.

System information:

• OS, kernel version: uname -a
Linux * 5.13.0-40-generic #45~20.04.1-Ubuntu SMP Mon Apr 4 09:38:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• mergerfs version: mergerfs -V

mergerfs version: 2.28.1
FUSE library version: 2.9.7-mergerfs_2.28.0
fusermount version: 2.9.9
using FUSE kernel interface version 7.29

• mergerfs settings in fstab
• List of drives, filesystems, & sizes:
  • df -h

/dev/sdd                                                                         3.7T  1.7T  2.0T  48% /media/disk1
/dev/sdh                                                                         3.7T  1.7T  2.0T  48% /media/disk5
/dev/sdf                                                                         3.7T  1.8T  1.9T  48% /media/disk7
/dev/sde                                                                         3.7T  1.7T  2.0T  48% /media/disk2
/dev/sdi                                                                         3.7T  1.7T  2.0T  48% /media/disk6
/dev/sdg                                                                         3.7T  1.7T  2.0T  48% /media/disk8
/dev/sdb                                                                         3.7T  2.0T  1.7T  55% /media/disk3
/dev/sdc                                                                         3.7T  1.8T  1.9T  48% /media/disk4
1/parity0:2/parity0:3/parity0:4/parity0:5/parity0:6/parity0:7/parity0:8/parity0   30T   14T   15T  49% /media/parity0
1/parity1:2/parity1:4/parity1:5/parity1:6/parity1:7/parity1:8/parity1             26T   12T   14T  48% /media/parity1
1/parity2:2/parity2:4/parity2:6/parity2:7/parity2:8/parity2                       22T   11T   12T  48% /media/parity2

• lsblk -f

sdb    ext4           * 1.6T    54% /media/disk3
sdc    ext4           * 1.9T    47% /media/disk4
sdd    ext4           * 1.9T    47% /media/disk1
sde    ext4           * 1.9T    47% /media/disk2
sdf    ext4           * 1.9T    47% /media/disk7
sdg    ext4           * 1.9T    47% /media/disk8
sdh    ext4           * 1.9T    47% /media/disk5
sdi    ext4           * 1.9T    47% /media/disk6

• A strace of the application having a problem:
  • strace -fvTtt -s 256 -o /tmp/app.strace.txt <cmd>
  • strace -fvTtt -s 256 -o /tmp/app.strace.txt -p <appPID>
• strace of mergerfs while app tried to do it's thing:
  • strace -fvTtt -s 256 -p <mergerfsPID> -o /tmp/mergerfs.strace.txt

Additional context

Add any other context about the problem here.

[trapexit]

What do you suggest mergerfs do? This is hardly unique to it. The same would happen with any dependency not yet mounted. systemd has the ability to manage dependencies and that is the standard way to handle such things.

[lsylsy2]

I've searched some issues and sof sites, it looks like systemd doesn't support globbing, so no meaning to solve at that stage (If I write configuration with no globbing, I can just live with the ugly mergerfs configuration.)
How will mergerfs do in this scenario (parallel mounting in fstab) if I'm using /mnt/disk*, just like in README.md ? That should be a common usage of globbing.
From my understanding, systemd handle things before these parameters, and globbing happens in mergerfs code.
Add a parameter like "waitmount=/media/disk*"?
Or mergerfs is just writing paths into its memory, and don't care if that directory is a null mounting point at one time, but mounted with a disk next time?

[trapexit]

Add a parameter like "waitmount=/media/disk*"?

So to have it wait for all found paths to be mounted to before moving forward? Is it only checking against root to compare the device/filesystem or just the parent path? Cause mounts can be at any level and mounts in mounts. There would have to be a timeout otherwise it could block forever on mount failure. And then does it mount anyway or fail?

I've already started to accommodate for some of this in mergerfs v3 where there is a much more expressive config system but doing all that in the current release isn't so easy. It'd be much simpler for you just to code up what the conditions are for mounting and add it as a systemd service like many people do with mergerfs already... just doing it after your conditions are met.

[trapexit]

You can pretty easily create this dependency in systemd. I had been meaning to put this up. Perhaps you can use this? You can do whatever you want in the prep script.

https://github.com/trapexit/mergerfs/wiki/systemd

[davidkna]

@trapexit Wouldn't a mount unit work better than an exec unit here? You can also forward some parameters to systemd via fstab (x-systemd.requires-mounts-for=, x-systemd.requires=, x-systemd.after=).

[trapexit]

I don't see how. The "problem" is that the op doesn't want to have to enumerate all dependencies. So you need to come up with a way to discover those dependencies which a mount unit would not offer. You can't articulate "for each directory (that I know to be a target for a filesystem mount that I've defined elsewhere) wait some amount of time till all the mounts shows up (because I have it in fstab and have the drive connected and don't expect it to fail much) and then start mergerfs."

Alternatively you could write a script to generate those dependencies and update fstab or create a mount unit with them but in either case you've got to script something. This isn't something that can be managed otherwise with traditional means.

[trapexit]

The point of the glob feature in mergerfs is to make mounting large numbers of filesystems easier. You have to have a directory present for a mount to happen so this works without concern with a race condition because by the time userland really starts the mounts are up and mergerfs simply doesn't care whether or not a branch is different from the root filesystem or not. It just reacts when requested.

If you want that same convenience for branch locations inside a filesystem... meaning it literally doesn't exist at the time mergerfs starts... then that's a whole other ball of wax. mergerfs could try to be smart but it still will have to depend on what is known. Like... /mnt/disk*/foo could strip /foo and then run the glob, find the /mnt/disk* that exist, and then append /foo back on. But that means something different from what /mnt/disk*/foo means as a glob. That might be sufficient for many usecases but such behavior would need to be created. It's not an existing thing.