Efficiency of FUSE (high-level) API / bottlenecks in WinFsp

[mhx]

I did a couple of performance measurements with the DwarFS FUSE driver that I'm just porting to Windows with the help of WinFsp.

On Linux, I've so far been using the low-level FUSE API exclusively. It seems there has been work on making the low-level API available on Windows as well as part of the winfuse project, but it doesn't look like there's much happening these days. Do you reckon there will be progress on supporting the FUSE low-level API in the future?

As an example, I've got a DwarFS file system with ~80,000 files and overall 800 MB of data (the full Boost 1.82.0 sources). On a Linux machine, doing

tar cf - mnt/ | wc -c

I can scan and read the whole file system in just about 3 seconds (and that includes the time it takes to decompress the data; the compressed file system is only 100 MiB in size).

Doing the exact same thing on Windows on the same machine (albeit in a VirtualBox instance), using WinFsp with the high-level FUSE API, creating the tar archive takes around 30 seconds, so it's an order of magnitude slower.

So I wondered where the bottleneck is and did some measurements. On Linux, I see:

$ ./dwarfs-win-alpha -f boost.dwarfs mnt -ocachesize=1g -operfmon=fuse
I 10:57:06.841465 dwarfs (v0.7.0-RC4-144-g1d9228e-dirty on branch mhx/windows-build, fuse version 35)
I 10:57:06.844778 file system initialized [3.292ms]
[fuse.op_read] 
      samples: 85029
      overall: 914.1ms
  avg latency: 10.75us
  p50 latency: 4.096us
  p90 latency: 8.192us
  p99 latency: 32.77us

[fuse.op_lookup]
      samples: 82605
      overall: 173.7ms
  avg latency: 2.103us
  p50 latency: 2.048us
  p90 latency: 4.096us
  p99 latency: 8.192us

[fuse.op_getattr]
      samples: 82582
      overall: 92.73ms
  avg latency: 1.123us
  p50 latency: 1.024us
  p90 latency: 2.048us
  p99 latency: 2.048us

[fuse.op_open]
      samples: 75435
      overall: 70.48ms
  avg latency: 934.3ns
  p50 latency: 1.024us
  p90 latency: 1.024us
  p99 latency: 2.048us

[fuse.op_readdir]
      samples: 14538
      overall: 65.49ms
  avg latency: 4.505us
  p50 latency: 2.048us
  p90 latency: 16.38us
  p99 latency: 65.54us

[fuse.op_init]
      samples: 1
      overall: 72.59us
  avg latency: 72.59us
  p50 latency: 131.1us
  p90 latency: 131.1us
  p99 latency: 131.1us

These measurements look reasonable. Reading/decompressing all regular file data takes about one second. lookup/getattr are called approximately once for each inode. open is called once per regular file.

Let's look at the Windows version. I've made sure that real-time protection in Windows Defender was off during the test:

C:\Users\mhx\git\dwarfs\build-vcpkg>.\dwarfs.exe boost.dwarfs tmp -o cachesize=1g -o perfmon=fuse
I 10:55:24.594684 dwarfs (v0.7.0-RC4-144-g5ae78b9 on branch mhx/windows-build, fuse version 32)
I 10:55:24.602112 file system initialized [4.454ms]
The service dwarfs has been started.
The service dwarfs has been stopped.
[fuse.op_getattr]
      samples: 2288176
      overall: 4.402s
  avg latency: 1.924us
  p50 latency: 800ns
  p90 latency: 6.4us
  p99 latency: 25.6us

[fuse.op_read]
      samples: 158804
      overall: 898.4ms
  avg latency: 5.657us
  p50 latency: 3.2us
  p90 latency: 6.4us
  p99 latency: 25.6us

[fuse.op_open]
      samples: 301790
      overall: 782.6ms
  avg latency: 2.593us
  p50 latency: 1.6us
  p90 latency: 6.4us
  p99 latency: 25.6us

[fuse.op_readdir]
      samples: 14292
      overall: 154ms
  avg latency: 10.77us
  p50 latency: 6.4us
  p90 latency: 25.6us
  p99 latency: 102.4us

[fuse.op_init]
      samples: 2
      overall: 49.6us
  avg latency: 24.8us
  p50 latency: 400ns
  p90 latency: 51.2us
  p99 latency: 51.2us

[fuse.op_statfs]
      samples: 60
      overall: 28.1us
  avg latency: 468.3ns
  p50 latency: 400ns
  p90 latency: 1.6us
  p99 latency: 1.6us

[fuse.op_readlink]
      samples: 2
      overall: 2.1us
  avg latency: 1.05us
  p50 latency: 200ns
  p90 latency: 3.2us
  p99 latency: 3.2us

First of all, despite the fact that it's doing slightly more read calls, the overall read latency is exactly the same, just under one second.

However, it's spending 4.4 seconds just doing getattr calls. It's doing almost 30x as many getattr calls as the Linux version. That's a surprise.

It's also doing a lot more open calls. Again, a bit surprising.

Still, the overall time spent inside the FUSE driver is just over 6 seconds, so more than 23 seconds are spent elsewhere. I presume that a lot of time is spent mangling paths from wstring to UTF-8, but I can't believe that's the only culprit.

I'm wondering what the best way would be to get the performance of the Windows driver closer to the Linux performance.

• What's the source of all the getattr calls? Is there something I could do to reduce their number? DwarFS file systems are read-only, so the attributes for an inode never actually change.

• Would a low-level FUSE API, if it was available, improve the situation dramatically? At least, there would have to be much less path mangling, but I wonder if this would help e.g. reduce the number of getattr calls.

• Would switching to the "native" WinFsp API improve things? DwarFS has been designed to be usable without FUSE, so the FUSE driver itself is just a thin layer making calls into the DwarFS API.

As another example that the bottleneck is not DwarFS itself, this is dwarfsextract using libarchive to build a pax archive from the file system, completely bypassing the FUSE driver:

C:\Users\mhx\git\dwarfs\build-vcpkg>.\dwarfsextract.exe --perfmon=filesystem_v2 -i boost.dwarfs -f pax > NUL
[filesystem_v2.readv_future]
      samples: 150870
      overall: 142.4ms
  avg latency: 944.1ns
  p50 latency: 800ns
  p90 latency: 800ns
  p99 latency: 6.4us

[filesystem_v2.getattr]
      samples: 165210
      overall: 45.53ms
  avg latency: 275.6ns
  p50 latency: 400ns
  p90 latency: 800ns
  p99 latency: 1.6us

[filesystem_v2.open]
      samples: 150870
      overall: 9.228ms
  avg latency: 61.17ns
  p50 latency: 200ns
  p90 latency: 200ns
  p99 latency: 400ns

[filesystem_v2.statvfs]
      samples: 2
      overall: 600ns
  avg latency: 300ns
  p50 latency: 400ns
  p90 latency: 800ns
  p99 latency: 800ns

The whole operation takes about one second.

[billziss-gh]

Apologies for the late response; I have been inactive for a couple months due to a family matter.

On Linux, I've so far been using the low-level FUSE API exclusively. It seems there has been work on making the low-level API available on Windows as well as part of the winfuse project, but it doesn't look like there's much happening these days. Do you reckon there will be progress on supporting the FUSE low-level API in the future?

WinFuse is an effort to bring the low-level FUSE API to Windows. It works on top of WinFsp and requires a modified version of libfuse.

I originally created WinFuse primarily as a method to add FUSE capability to WSL1. WinFuse exposes the FUSE protocol and makes it accessible from the native Windows API (DeviceIoControl) and the WSL1 Linux API (/dev/fuse). Then libfuse can be used on both Windows and Linux (with modifications to support a different mounting protocol and to use DeviceIoControl on Windows instead of read/write).

Unfortunately when Microsoft moved from WSL1 (which is based on Windows kernel emulation of the Linux API) to WSL2 (which is based on a virtualized Linux kernel) it removed my major motivation to continue with the WinFuse project.

Doing the exact same thing on Windows on the same machine (albeit in a VirtualBox instance), using WinFsp with the high-level FUSE API, creating the tar archive takes around 30 seconds, so it's an order of magnitude slower.

Ideally such tests should be performed in an identical environment. But I understand that this may not always be possible if your primary environment is Linux and you are just trying to get a feel for Windows performance via virtualization.

---

Let us discuss the broad differences between file system implementation on Windows and Linux, which will help us better understand what you are seeing and suggest potential improvements.

File system operation classes and their performance

First let us consider file system operations in two broad classes: operations that act on the names of files (e.g. open, unlink, rename, etc.) and operations that act on the content of files (e.g. read, write, memory mapped I/O, etc.). Let's call the first class of operations "namespace" operations and the second class of operations "I/O" operations.

In general "I/O" operations are quite similar between Linux and Windows. For this reason you will often see very similar performance when using Windows vs Linux. This is also true with WinFsp file systems, if you enable the kernel cache manager (on FUSE -o FileInfoTimeout=-1).

OTOH "namespace" operations follow a different model between Linux and Windows. The first important difference is that on Linux paths are broken down into path components represented by (inode-number, name) tuples (hence the need for the lookup operation). In contrast the Windows kernel treats a file path as an identifier and does not attempt to break it down into path components before passing it into the file system driver (FSD); for example, the Win32 path C:\Path\To\File might become the internal kernel path \Device\HarddiskVolumeX\Path\To\File and the Windows kernel will parse out the \Device\HarddiskVolumeX part (and use it to locate the file system device), but it will pass the \Path\To\File part intact to the FSD. This way Windows can avoid some unnecessary parsing of the "file path" part and this can be a benefit in some circumstances (both performance-wise, but also functionally: for example, named pipes on Windows are implemented as a sort of an FSD).

A second important difference is that on Linux many namespace operations are treated as primitive operations, whereas on Windows they are not. As an example consider unlink. On Linux this is a primitive operation, but the Windows equivalent DeleteFile consists of the operations Open, SetInformation / Disposition, Close; these operations open the file, set the "disposition" flag on it (which instructs the file system to mark the file as pending deletion) and then close the file (at which point the file system actually deletes the file). Obviously even if all other file system code was the same, crossing the user/kernel boundary 3 times instead of 1 time is bound to be slower.

As a rule of thumb you should expect "namespace" operations on Linux to be faster than on Windows. OTOH, you should expect "I/O" operations on Linux to be on par with those on Windows.

Listing directories

An important "namespace" operation is that of listing files in a directory. This operation is very different between Linux and Windows: On Linux a readdir fetches the names of files, but does not provide full stat information. OTOH, on Windows a FindFirstFile/FindNextFile fetches the names of files together with stat information. This can be an advantage or disadvantage depending on your scenario.

WinFsp and FUSE

WinFsp provides a native API that matches the operations that a Windows kernel mode FSD would have to implement. There are two FUSE layers on top of this native API, the high-level FUSE API provided with the core WinFsp project and the low-level FUSE API provided by the WinFuse project.

Both FUSE wrappers have to do considerable work to match the Windows file system model to the Linux one, that goes far beyond translating strings to wstrings. For some insight into this see the SSHFS Port Case Study. The low-level FUSE wrapper has to go into even greater lengths as it has to translate every Windows path it sees into (inode-number, name) tuples via lookup.

How to improve performance

• The low-level FUSE API is very far from the Windows file system model, so I would not recommend it if you care about performance. There is also the issue that I no longer work on WinFuse, because the primary reason for its existence (WSL1) is effectively dead (or at least feels so to me).

• The high-level FUSE API is a good alternative and one that I use myself for new cross-platform file systems (usually via Go and cgofuse). It does not have the best possible performance on Windows, but it can be pretty fast if one takes care with some operations. Notably the high-level FUSE API includes a special "readdir-plus" operation that allows a user mode file system to provide both name and stat information; this avoid a torrent of getattr's during directory listing.

• The native API provides the best performance on Windows. See performance testing here.

Finally if you have a read-only file system you do not have to worry about cache coherency and you can always use -o FileInfoTimeout=-1 for best performance of both "namespace" and "I/O" operations.

[mhx]

Hi again and thank you so much for the detailed reply!

So it looks like there isn't actually much left for me to do to improve performance, at least for as long as I want to stick to the FUSE API.

I've already been using FUSE_FILL_DIR_PLUS in the high-level API, however, I'm seeing almost two orders of magnitude more getattr calls than with the Linux FUSE implementation. Setting -o FileInfoTimeout=-1 isn't different from not setting it. Setting -o FileInfoTimeout=0, just to try and see any effect, results in 5.4 million getattr calls vs. 4.3 million calls when not setting it (or setting it to -1).

As another quick test, I made a simple file system containing only a single small file in a subdirectory. I'm running this with -o FileInfoTimeout=-1 and detailed logging so I can see every single call in the FUSE API. I'll then do a single cat mountpoint\dwarfs\version.h, which results in 9 getattr calls and 3 open calls before finally doing a read:

I 20:31:06.894114 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:1144] file system initialized [64.45ms]
D 20:31:06.915196 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:203] op_init_common
D 20:31:06.923852 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:793] op_statfs(/)
D 20:31:06.947028 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/)
D 20:31:06.958028 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:443] op_readlink
D 20:31:06.968974 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/.)
The service dwarfs has been started.
D 20:31:09.928656 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 20:31:09.939114 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs)
D 20:31:09.950301 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 20:31:09.960029 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:520] op_open
D 20:31:09.971444 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:793] op_statfs(/)
D 20:31:09.971440 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:793] op_statfs(/)
D 20:31:09.991691 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 20:31:10.009112 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs)
D 20:31:10.022120 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 20:31:10.032996 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:520] op_open
D 20:31:10.055761 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 20:31:10.072823 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs)
D 20:31:10.086524 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 20:31:10.109517 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:520] op_open
D 20:31:10.129980 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:580] op_read
D 20:31:10.172079 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:587] read(/dwarfs/version.h [2], 4,096, 0) -> 327

[billziss-gh]

Does your file system support readlink and symlinks? If yes, then the FUSE layer has to send considerably more getattr’s to determine if a particular path component is a symlink.

[mhx]

Does your file system support readlink and symlinks? If yes, then the FUSE layer has to send considerably more getattr’s to determine if a particular path component is a symlink.

Ohhh, it does indeed!

And I presume from what I'm seeing that those calls are not cached?

I'll check later if the behaviour changes when I disable symlinks. As I know beforehand if a file system contains symlinks or not, I can choose to disable readlink completely if it doesn't.

[billziss-gh]

According to Windows file system rules when a file is opened via a path that contains symlinks, the symlinks must first be resolved and the real path must be returned to Windows (via the special STATUS_REPARSE status code). The Windows kernel will then attempt to open the file via the real path.

This means that the user mode DLL contains a version of realpath (only more complicated, because it has to support the more general concept of reparse points). This is where all extra getattr calls originate from. (It is possible to optimize this behavior if you are using the native API and you have a fast version of realpath in your file system.)

And I presume from what I'm seeing that those calls are not cached?

These extra getattr calls come from the user mode DLL. Most caching in WinFsp is done in kernel mode. The reasoning was that caching should be added when it was necessary to minimize user/kernel mode interaction, but user mode code should be kept simple: because if user mode caching was needed the user mode file system could have its own caching layer without performance implications.

Over the lifetime of the WinFsp project the complexity of the user mode DLL has increased and even more so in the FUSE layer. It may be worthwhile to add some caching and especially a getattr cache. It has been asked before: #326

[mhx]

So with the readlink call disabled, I'm indeed seeing a reduction in getattr calls, down from 9 to 6. There's still 3 open calls, which is a bit surprising. Could multiple open calls on the FUSE layer be caused by a single Windows system call to open/read a file?

The service dwarfs has been started.
D 12:39:51.915462 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 12:39:51.920177 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 12:39:51.924208 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:520] op_open(/dwarfs/version.h)
D 12:39:51.928570 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:793] op_statfs(/)
D 12:39:51.928660 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:793] op_statfs(/)
D 12:39:51.933137 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 12:39:51.941124 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 12:39:51.945370 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:520] op_open(/dwarfs/version.h)
D 12:39:51.949603 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 12:39:51.953067 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:334] op_getattr(/dwarfs/version.h)
D 12:39:51.956222 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:520] op_open(/dwarfs/version.h)
D 12:39:51.959467 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:580] op_read
D 12:39:51.962727 [C:\Users\mhx\git\dwarfs\src\dwarfs_main.cpp:587] read(/dwarfs/version.h [2], 4,096, 0) -> 327

[billziss-gh]

Recall that on Windows every namespace operation requires opening the file, performing the operation and then closing the file. For example, consider an imaginary Windows equivalent of stat: it would be more like a series of calls open, fstat, close rather than a single stat call.

A lot of applications are coded against the Win32 API, which follows the above pattern and does not reuse the same HANDLE across operations; so you get a lot of extra open and close calls. Furthermore Windows has a lot of components such as filter drivers, etc. that see file system traffic and like to open files for their own purposes (e.g. AntiVirus).

Because in the WinFsp design access control happens in user mode (to allow flexibility and custom access control policies) all open requests have to be forwarded to user mode. So you get to see most opens in your user mode file system (some may still be optimized away by the FSD in some circumstances).

If you want to see where your file system traffic originates from, I recommend the FileSpy tool (original website seems to be gone, but the tool here seems to be the right one: https://community.chocolatey.org/packages/filespy).

[billziss-gh]

Correct link for FileSpy (don’t know why my search engine could not find it at first): http://www.zezula.net/en/tools/filespy.html

[mhx]

I've added some code that checks if there are any symlinks present in a file system image, and if not, it will simply not register the readlink operation with the FUSE layer. This should the case for most images built on Windows. In my test case, this reduced the number of getattr calls from 4.3 million to about 850,000. This is a significant improvement, so thanks for pointing that out!