Help with performance / much faster using samba

[vikingloudmouth]

I have an SSD, and two hard drives mounted as /var/media/SSD, /var/media/HDD1, and /var/media/HDD2 and created a pool from the two HDDs
mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs,fsname=media_pool_fs /var/media/HDD1/media:/var/media/HDD2/media/ /storage/media_pool

Without using the mergerfs mount, I can copy (using cp) large files (mkv's) from the SSD to the HDD's at ~90MB/s. My problem is when copying from the SDD to /storage/media_pool I only get ~15MB/s. I have tried playing with the options listed here, and the only one I have found to help is enabling cache.writeback which got me to ~30MB/s. This increased to ~40MB/s with nullrw=true.

I also have all of these folders accessible via a samba share. Using this samba share, I can copy from the SSD to /storage/media_pool and keep the original speed of ~90MB/s.

What settings are suggested for me to try and get these speeds locally using cp ? I haven't had luck looking up other suggestions, everyone else seems to have to opposite problem of the samba share being slower

[trapexit]

I really have nothing to add to what I've already described in the docs.

nullrw=true means it is literally doing nothing when doing reads and writes as I describe in the docs. It is purely for testing theoretical rw perf. If you only get 40MB/s then your system sounds pretty slow to begin with. It can easily be hundreds of MB/s on random lowend systems.

The fastest it can possibly be is cache.files=off, nullrw=true, a 1MB or so readahead, and ensuring the software uses large buffer sizes. If that is slow... there is nothing I can do.

[trapexit]

The value of interest when stracing is the read and write buffer sizes. The 3rd argument of the call. "write(X,Y,Z) = W" or similar with read. Z and W are the important parts. How much is it trying to read or write and how much actually is read or written.

Regarding the HDD perf. Sounds to me like the latency on the drive / drive interconnect is likely really high. USB often is. Hubs probably don't help. If you provide a trace like I show in the docs for mergerfs you can compare the time it takes for individual commands to run. Could be other things but without the log I can't say too much. For instance... as I mention in the docs if xattr calls are being made for security it could seriously impact write perf.

[vikingloudmouth]

When using the mergerfs mount on the HDD and stracing the dd test with bs=1M that third argument to write is 1048576. Looking at the runtimes in the strace file though, the problem of speed isn't the read or write commands, it is the fsync that is being called after every write - it takes 10 times as long as the write. If I change the command to dd if=/dev/zero of=/storage/media_pool/test.file bs=1M count=500 (dropping the conv=fsync), I get ~110MB/s, admittedly I don't know if this is still a valid test.

Performing the strace on cp wasn't as helpful to me, all it shows is sendfile64(4, 3, NULL, 16777216) = 16777216 <1.092659> repeating until the copy is done. I also ran the strace on mergerfs while cp was running by attaching to the pid via strace -fvTtt -s 256 -p <mergerfsPID> -o /storage/copy_mergerfs_log.txt, all it contains is the following pattern. Is this expected? Or am I doing something wrong here?

14607 09:40:50.323287 restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...>
14605 09:40:50.323406 restart_syscall(<... resuming interrupted restart_syscall ...>) = 4096 <0.000126>
14605 09:40:50.323752 restart_syscall(<... resuming interrupted restart_syscall ...>) = 24 <0.000182>
14608 09:40:50.324130 <... restart_syscall resumed>) = 4176 <0.002767>
14605 09:40:50.324251 restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...>
14608 09:40:50.324329 restart_syscall(<... resuming interrupted restart_syscall ...>) = 4096 <0.000130>
14608 09:40:50.324697 restart_syscall(<... resuming interrupted restart_syscall ...>) = 24 <0.000133>
14606 09:40:50.325003 <... restart_syscall resumed>) = 4176 <0.002531>
14608 09:40:50.325121 restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...>
14606 09:40:50.325230 restart_syscall(<... resuming interrupted restart_syscall ...>) = 4096 <0.000125>
14606 09:40:50.325528 restart_syscall(<... resuming interrupted restart_syscall ...>) = 24 <0.000073>
14606 09:40:50.325766 restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...>
14607 09:40:50.326128 <... restart_syscall resumed>) = 4176 <0.002756>
14607 09:40:50.326277 restart_syscall(<... resuming interrupted restart_syscall ...>) = 4096 <0.000163>
14607 09:40:50.326657 restart_syscall(<... resuming interrupted restart_syscall ...>) = 24 <0.000071>
14607 09:40:50.326894 restart_syscall(<... resuming interrupted restart_syscall ...> <unfinished ...>
14605 09:40:50.326968 <... restart_syscall resumed>) = 4176 <0.002655>
14605 09:40:50.327068 restart_syscall(<... resuming interrupted restart_syscall ...>) = 4096 <0.000282>
14605 09:40:50.327502 restart_syscall(<... resuming interrupted restart_syscall ...>) = 24 <0.000064>
14608 09:40:50.327692 <... restart_syscall resumed>) = 4176 <0.002508>

I originally did play around with xattr, security_capability , and the options listed under performance in the docs. The only one I found made any difference was enabling cache.writeback as I originally said. I couldn't find any mention to it in the docs, but is the size of this cache configurable?

[trapexit]

Right, fsync forces the data to the disk. So it tests the true speed. Vs not using it and allow the kernel to cache writes and eventually flush which doesn't give a true representation of the performance. The point of the test is to test mergerfs. Not to test the kernel's caching and your system's memory size and cache pressure.

The logs aren't supposed to be useful to you. They are useful to me. Which is why I request them as part of offering support. They show exactly what the app is requesting from the kernel and how long.

Are you sure the pid was correct? pgrep -alf mergerfs will give you the process id to attach to.

writeback cache is not configurable. it's just a feature of the kernel and likely bounded by the size of fuse messages. if page caching is enabled the kernel is already caching in general. writeback really just aggregates small writes.

Can you please just pick a set of options and give me the strace logs?

[vikingloudmouth]

Ok, from the start.

Runtime environment:

• copied the static release to /storage/bin/mergerfs.
• No containers involved (yet).
• /storage/bin/mergerfs/mergerfs --version , output mergerfs v2.38.0
• uname -a, output Linux CoreELEC 4.9.269 #1 SMP PREEMPT Sat Mar 5 16:44:59 CET 2022 aarch64 GNU/Linux
• lsb_release -a, output CoreELEC (official): 19.4-Matrix
• df -h, output

Filesystem                Size      Used Available Use% Mounted on
devtmpfs                  1.9G      4.0K      1.9G   0% /dev
tmpfs                   760.6M     16.6M    744.0M   2% /run
/dev/CE_FLASH           511.0M    221.0M    289.9M  43% /flash
/dev/loop0              199.0M    199.0M         0 100% /
/dev/CE_STORAGE          24.0G     13.1G     10.9G  55% /storage
tmpfs                     1.9G         0      1.9G   0% /dev/shm
tmpfs                     4.0M         0      4.0M   0% /sys/fs/cgroup
tmpfs                     1.9G      2.3M      1.9G   0% /var
tmpfs                     1.9G         0      1.9G   0% /tmp
/dev/sdc1                 4.5T      3.8T    734.7G  84% /var/media/Expansion2
/dev/sda1               234.6G     13.6G    221.0G   6% /var/media/SandiskSSD
/dev/sdb1                 4.5T      4.3T    269.4G  94% /var/media/Expansion

Mergerfs mount:
/storage/bin/mergerfs/mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs,fsname=media_pool_fs,minfreespace=100G /var/media/Expansion2/media /storage/media_pool

Check pid: pgrep -alf mergerfs, output 6824 /storage/bin/mergerfs/mergerfs -o cache.files=partial,dropcacheonclose=true,category.create=mfs,fsname=media_pool_fs,minfreespace=100G /var/media/Expansion2/media /storage/media_pool

strace of mergerfs: strace -fvTtt -s 256 -p 6824 -o /storage/bin/mergerfs/mergerfs_cp_log.txt

In another terminal: strace -fvTtt -s 256 -o /storage/bin/mergerfs/cp_log.txt cp /var/media/SandiskSSD/test.mkv /storage/media_pool

When this finished (200seconds later, on the 510MB test.mkv), hit ctrl + c, on the terminal with the mergerfs strace running.

The strace logs are available here

[trapexit]

There is clearly something wrong about that strace of mergerfs. It's not usable.

I would suggest playing with strace options to see if you get proper logs from it.