Caching performance should be better

[raykrueger]

Mountpoint for Amazon S3 version

mount-s3 1.3.2

AWS Region

us-west-2

Describe the running environment

Ubuntu 22.04 and Amazon Linux 2023

Mountpoint options

sudo mount-s3 --cache /mnt/raid0 raykrueger20240124--usw2-az1--x-s3 /mnt/s3

What happened?

Using the --cache arg provides no performance benefit.

I have my cache dir set to /mnt/raid0, which is a 2 nvme raid0 mount. When I read a 10G file from my S3 mount it takes 9-10 seconds. When I read the file every time after the first, I expect that time to be the same speed as reading from that /mnt/raid0 location. Every subsequent read takes the same amount of time as the first. Also, it takes the same amount of time if I have no --cache at all.

I see the same behavior mounting directory (express) or regular S3 buckets.

Relevant log output

# mount-s3 --cache /mnt/raid0 raykrueger20240124--usw2-az1--x-s3 /mnt/s3
bucket raykrueger20240124--usw2-az1--x-s3 is mounted at /mnt/s3

# time cat /mnt/s3/data/file_1 > /dev/null

real	0m9.108s
user	0m0.007s
sys	0m4.287s
# time cat /mnt/s3/data/file_1 > /dev/null

real	0m7.780s
user	0m0.012s
sys	0m4.683s
# time cat /mnt/s3/data/file_1 > /dev/null

real	0m7.410s
user	0m0.014s
sys	0m4.509s

The 2s drop is margin for error variation.

The cache is populated.

# du -sh /mnt/raid0/mountpoint-cache/
11G	/mnt/raid0/mountpoint-cache/

This is the cache performance I expect.

# time cat /mnt/raid0/mountpoint-cache/V1/9c/eaf5ecb13463aa2b4834c069b935949355d31e8b9c201fe4f3b461ad5d0132/* > /dev/null

real	0m1.587s
user	0m0.010s
sys	0m1.576s

[raykrueger]

Based on looking at metrics in the debug logs it looks like the cache is "working", blocks are pulled and read from cache. The performance is not close to what I would expect for reading from disk. The same 10G file copied to my /mnt/raid0 disk comes off as such...

# time cat /mnt/raid0/file_1 > /dev/null

real	0m1.150s
user	0m0.000s
sys	0m1.150s

Next steps I'm going to try adding some flamegraphs to the tracing and see where this peaking.

[raykrueger]

After sprinkling in a bunch of instrument attributes, spans, and a flamegraph layer; I can confirm that all of our time is spent in deserializing at mountpoint_s3::data_cache::disk_data_cache.

[arsh]

Thanks for reporting this. In [1], we are introducing cache benchmarks. With this in place, we want to explore improvements by experimenting with changing the serialization library, for example.

[1] #783

[raykrueger]

I should mention that the tracing changes I made are in my fork here https://github.com/raykrueger/mountpoint-s3/tree/flamegraphs

I had no intention of sending a PR for the changes as they were pretty heavy handed.