New files added to mergerfs exported over NFS not showing up on ubuntu/debian NFS client with kernel >=5.6, but accessible directly.

[spirkaa]

New files added to mergerfs exported over NFS not showing up on ubuntu/debian NFS client with kernel >=5.6, but accessible directly.

Observed in real world after upgrade Proxmox from 6.4 to 7.2.

Tested nfs v3 and v4 mounted locally and over network with default mount opts and config of nfs client on Ubuntu 22.04 (5.15), Debian 11 (5.10), Ubuntu 20.04 (mainline 5.6). Kernels <=5.5 working as intended.

ubuntu@nfs-srv:~$ ls -la /mnt/nfs/mergerfs; find /mnt/nfs/mergerfs
total 8
drwxrwxrwx 2 root root 4096 May 22 06:16 .
drwxr-xr-x 3 root root 4096 May 22 06:16 ..
/mnt/nfs/mergerfs
ubuntu@nfs-srv:~$ touch /mnt/nfs/mergerfs/test-on-nfs
ubuntu@nfs-srv:~$ ls -la /mnt/nfs/mergerfs; find /mnt/nfs/mergerfs
total 8
drwxrwxrwx 2 root root 4096 May 22 06:16 .
drwxr-xr-x 3 root root 4096 May 22 06:16 ..
/mnt/nfs/mergerfs
ubuntu@nfs-srv:~$ touch /export/mergerfs/test-local-on-mergerfs
ubuntu@nfs-srv:~$ ls -la /mnt/nfs/mergerfs; find /mnt/nfs/mergerfs
total 8
drwxrwxrwx 2 root root 4096 May 22 06:16 .
drwxr-xr-x 3 root root 4096 May 22 06:16 ..
/mnt/nfs/mergerfs
ubuntu@nfs-srv:~$ ls -la /mnt/nfs/mergerfs/test-on-nfs /mnt/nfs/mergerfs/test-local-on-mergerfs
-rw-rw-r-- 1 ubuntu ubuntu 0 May 22 06:18 /mnt/nfs/mergerfs/test-local-on-mergerfs
-rw-rw-r-- 1 ubuntu ubuntu 0 May 22 06:18 /mnt/nfs/mergerfs/test-on-nfs
ubuntu@nfs-srv:~$ touch /mnt/disk1/test-local-on-disk1
ubuntu@nfs-srv:~$ ls -la /mnt/nfs/mergerfs; find /mnt/nfs/mergerfs
total 8
drwxrwxrwx 2 root   root   4096 May 22 06:19 .
drwxr-xr-x 3 root   root   4096 May 22 06:16 ..
-rw-rw-r-- 1 ubuntu ubuntu    0 May 22 06:19 test-local-on-disk1
-rw-rw-r-- 1 ubuntu ubuntu    0 May 22 06:18 test-local-on-mergerfs
-rw-rw-r-- 1 ubuntu ubuntu    0 May 22 06:18 test-on-nfs
/mnt/nfs/mergerfs
/mnt/nfs/mergerfs/test-local-on-disk1
/mnt/nfs/mergerfs/test-local-on-mergerfs
/mnt/nfs/mergerfs/test-on-nfs

To Reproduce

On fresh Ubuntu 22.04 VM

sudo mkdir -p /export/mergerfs /mnt/{disk1,disk2,nfs}
sudo chmod 777 /export/mergerfs /mnt/{disk1,disk2,nfs}
curl -LO https://github.com/trapexit/mergerfs/releases/download/2.33.5/mergerfs_2.33.5.ubuntu-jammy_amd64.deb
sudo dpkg -i mergerfs_2.33.5.ubuntu-jammy_amd64.deb
sudo mergerfs -o nonempty,allow_other,use_ino,noforget,inodecalc=path-hash,cache.files=off,moveonenospc=true,category.create=mfs,dropcacheonclose=true,minfreespace=1G,fsname=mergerfs /mnt/disk\* /export/mergerfs
sudo apt -y install nfs-common nfs-kernel-server
echo -e "/export localhost(ro,fsid=0,no_subtree_check)\n/export/mergerfs localhost(rw,fsid=1,root_squash,all_squash,no_subtree_check,anonuid=1000,anongid=1000)" | sudo tee /etc/exports
sudo exportfs -ra
sudo mount localhost:/ /mnt/nfs
nfsstat -m

echo -e "\nTEST"
echo -e "\nTEST -  initial state (empty dir) of nfs ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Create file on nfs"
touch /mnt/nfs/mergerfs/test-on-nfs
sleep 2
echo -e "TEST - File (1) missing in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Create local file on mergerfs"
touch /export/mergerfs/test-local-on-mergerfs
sleep 2
echo -e "TEST - Files (2) still missing in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - But this files (2) are actually accessible"
ls -la /mnt/nfs/mergerfs/test-on-nfs /mnt/nfs/mergerfs/test-local-on-mergerfs
echo -e "\nTEST - Create local file on branch"
touch /mnt/disk1/test-local-on-disk1
sleep 2
echo -e "TEST - Now all files (3) show up in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Local ls (just in case)"
ls -la /export/mergerfs
find /export/mergerfs

Expected behavior

New files added to mergerfs exported over NFS must showing up on NFS client with kernel >=5.6.

System information:

• OS, kernel version: Linux nfs-srv 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• mergerfs version: mergerfs version: 2.33.5
• mergerfs settings: nonempty,allow_other,use_ino,noforget,inodecalc=path-hash,cache.files=off,moveonenospc=true,category.create=mfs,dropcacheonclose=true,minfreespace=1G,fsname=mergerfs /mnt/disk\* /export/mergerfs
• List of drives, filesystems, & sizes:

Filesystem           Size  Used Avail Use% Mounted on
tmpfs                198M 1012K  197M   1% /run
/dev/sda1            9.8G  2.1G  7.7G  22% /
tmpfs                989M     0  989M   0% /dev/shm
tmpfs                5.0M     0  5.0M   0% /run/lock
/dev/sda15           105M  5.3M  100M   5% /boot/efi
tmpfs                198M  4.0K  198M   1% /run/user/1000
mergerfs             9.8G  2.1G  7.7G  22% /export/mergerfs
localhost:/          9.8G  2.1G  7.7G  22% /mnt/nfs
localhost:/mergerfs  9.8G  2.1G  7.7G  22% /mnt/nfs/mergerfs

[trapexit]

I'm dealing with some issues with my main server so I'm a bit busy with that so I likely won't get a chance to look at this for a week or so but... can you trace mergerfs? See if it is getting requests from NFS when you say it doesn't return anything?

Can you do other requests? find and ls only stress opendir/readdir. What about stat, open, chmod, etc.?

[spirkaa]

Stat and chmod calls - no difference.

mergerfs.strace.txt

Updated script:

sudo mkdir -p /export/mergerfs /mnt/{disk1,disk2,nfs}
sudo chmod 777 /export/mergerfs /mnt/{disk1,disk2,nfs}
curl -LO https://github.com/trapexit/mergerfs/releases/download/2.33.5/mergerfs_2.33.5.ubuntu-jammy_amd64.deb
sudo dpkg -i mergerfs_2.33.5.ubuntu-jammy_amd64.deb
sudo mergerfs -o nonempty,allow_other,use_ino,noforget,inodecalc=path-hash,cache.files=off,moveonenospc=true,category.create=mfs,dropcacheonclose=true,minfreespace=1G,fsname=mergerfs /mnt/disk\* /export/mergerfs
sudo strace -fvTtt -s 256 -p $(pgrep mergerfs) -o ~/mergerfs.strace.txt &
sudo apt -y install nfs-common nfs-kernel-server
echo -e "/export localhost(ro,fsid=0,no_subtree_check)\n/export/mergerfs localhost(rw,fsid=1,root_squash,all_squash,no_subtree_check,anonuid=1000,anongid=1000)" | sudo tee /etc/exports
sudo exportfs -ra
sudo mount localhost:/ /mnt/nfs
nfsstat -m

echo -e "\nTEST"
echo -e "\nTEST -  initial state (empty dir) of nfs ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Create file on nfs"
touch /mnt/nfs/mergerfs/test-on-nfs
sleep 2
echo -e "TEST - File (1) missing in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Create local file on mergerfs"
touch /export/mergerfs/test-local-on-mergerfs
sleep 2
echo -e "TEST - Files (2) still missing in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - But this files (2) are actually accessible"
ls -la /mnt/nfs/mergerfs/test-on-nfs /mnt/nfs/mergerfs/test-local-on-mergerfs
chmod 666 /mnt/nfs/mergerfs/test-on-nfs /mnt/nfs/mergerfs/test-local-on-mergerfs
stat /mnt/nfs/mergerfs/test-on-nfs /mnt/nfs/mergerfs/test-local-on-mergerfs
echo -e "TEST - Files (2) not showing up in listing even after direct requests"
ls -la /mnt/nfs/mergerfs
echo -e "\nTEST - Create local file on branch"
touch /mnt/disk1/test-local-on-disk1
sleep 2
echo -e "TEST - Now all files (3) show up in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Local ls (just in case)"
ls -la /export/mergerfs
find /export/mergerfs

sudo pkill -9 strace
cat ~/mergerfs.strace.txt

[trapexit]

Preferably the trace would be only of the issue itself. I can't tell from the trace what's what otherwise. There are no timestamps on the client side to correlate with the trace.

trace mergerfs, issue a touch, stat, and maybe ls to the nfs share.

[hydra3333]

Hello. Is this issue still alive ?

I'm considering using mergerfs on a Raspberry Pi 4 with 2 USB3 disks (10Tb and 12Tb both formatted NTFS) having filesystems merged with mergerfs and presented to Chromecast-for-google-tv devices (using VLC) via NFS. i.e. the Pi4 will be a home media server.

This issue could make that scenario not very handy ;)

[trapexit]

It's open, yes, because there were no followups and no verification of claimed issue.

[hydra3333]

OK I'll give it a whirl over the next week or two providing we survive just-acquired covid :)

[seancrites]

Having a relatively same setup, multiple drives with common directories, using mergerfs to pool them into one directory and then share the pooled directory via NFS. If I created the directory or file on the NFS client, it showed up instantly on both the client & server. When I created it on the server, it took a while to show up on the client side. I learned to work around this by creating everything on the client side. I had a chance to rebuild a NAS, using the same setup (MergerFS + NFS) and the trouble followed.

Turned out the issue was with the NFS client mount options, I needed to include 'lookupcache=none':

192.168.1.2:/music /mnt/music nfs4 auto,lookupcache=none 0 0

This made changes on the NFS server appear instantly on the client side. Files & directories have their own respective min & max cache timeout values. ref: https://www.man7.org/linux/man-pages/man5/nfs.5.html#DATA_AND_METADATA_COHERENCE. Hope that helps.

[hydra3333]

Thanks !
Now referred to in https://forums.raspberrypi.com/viewtopic.php?p=2020848#p2020848

[FlexibleToast]

I'm having the same or similar issue. It is exactly as described here: https://serverfault.com/questions/680969/nfs-client-does-not-see-new-files-created-on-server-caching. Which seems to suggest it may be related to ctimes? It's kind of strange because if I mount with the previously suggested option mount -o lookupcache=none 192.168.50.9:/Media /var/kerbol/media it works for the first file, but not anything after that. It's bizarre, it happens with touch, dd, curl, but if I use vi to create a file or touch the containing directory all the files are visible.

microos-pi4-1:/var/kerbol> sudo mount -o lookupcache=none 192.168.50.9:/Media /var/kerbol/media
microos-pi4-1:/var/kerbol> touch media/test
microos-pi4-1:/var/kerbol> ls media/
Anime  Anime_Movies  Audio_Books  Books  Movies  Music  Pictures  test  TV  Videos
microos-pi4-1:/var/kerbol> touch media/test2
microos-pi4-1:/var/kerbol> ls media/
Anime  Anime_Movies  Audio_Books  Books  Movies  Music  Pictures  test  TV  Videos
microos-pi4-1:/var/kerbol> dd if=/dev/random of=media/test3 bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.397838 s, 26.4 MB/s
microos-pi4-1:/var/kerbol> ls media/
Anime  Anime_Movies  Audio_Books  Books  Movies  Music  Pictures  test  TV  Videos
microos-pi4-1:/var/kerbol> curl gooogle.com -o media/test_file
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   220  100   220    0     0   2616      0 --:--:-- --:--:-- --:--:--  2650
microos-pi4-1:/var/kerbol> ls media/
Anime  Anime_Movies  Audio_Books  Books  Movies  Music  Pictures  test  TV  Videos
microos-pi4-1:/var/kerbol> touch media/
microos-pi4-1:/var/kerbol> ls media/
Anime  Anime_Movies  Audio_Books  Books  Movies  Music  Pictures  test  test2  test3  test_file  TV  Videos
microos-pi4-1:/var/kerbol> touch media/test4
microos-pi4-1:/var/kerbol> ls media/
Anime  Anime_Movies  Audio_Books  Books  Movies  Music  Pictures  test  test2  test3  test_file  TV  Videos
microos-pi4-1:/var/kerbol> vim media/test5
microos-pi4-1:/var/kerbol> ls media/
Anime  Anime_Movies  Audio_Books  Books  Movies  Music  Pictures  test  test2  test3  test4  test5  test_file  TV  Videos
microos-pi4-1:/var/kerbol> rm media/test*
microos-pi4-1:/var/kerbol> ls media/
Anime  Anime_Movies  Audio_Books  Books  Movies  Music  Pictures  TV  Videos

Some, but not all file operations seem to force NFS to re-read the contents of the directory, while others it just ignores.

Here is my mergerfs mount options if you're interested: defaults,allow_other,cache.files=partial,noforget,inodecalc=path-hash,use_ino,dropcacheonclose=true. I tried reading through the docs and changing settings that were suggested for NFS.