Metadata Caching in case of not existing file

[nguyenminhdungpg]

If I use --cache option, will mountpoint-s3 not cache metadata of a requested file that is not existing?
In s3fs, there is option disable_noobj_cache that help me to achieve this.

Thank you very much

[dannycjones]

Hey @nguyenminhdungpg. Mountpoint will indeed cache the non-existence of a file. We've sometimes described this as 'negative caching'. This behavior was introduced in Mountpoint v1.5.0.

It sounds like you would prefer that Mountpoint would not cache these entries, and instead go to S3 each time if it has not seen that the file exists. Can you tell us a bit more about your use case? What problems does caching the negative entries introduce for you?

[nguyenminhdungpg]

Hi @dannycjones thank you for your information.

My usecase for using disable_noobj_cache option of s3fs at this time is very weird and it is not good.

There are 2 services run in parallel and there is a chance that one requests file in s3fs mounted folder before the other finished adding the file to S3 using AWS SDK for S3.
If I don't use disable_noobj_cache, then the non-existences of the file and maybe its parent folders are cached, requests to the file with high frequency can make it (and its parent folders) seem to be not-existed forever. This also breaks a logic in my service that copies files from or create new files in the seem-to-be-not-existed-forever folders in s3fs mounted folder.

On another hand, I am wondering without 'negative caching', can an attacker brute-force requests to non-existing files and increase my S3 cost?

[nguyenminhdungpg]

Hi @dannycjones
Is there any news about how to flag off negative cache?

Currently I am setting TTL to 6 months but with negative cache is always enabled, some logics in my service are broken.

Thank you.

[monthonk]

Hey, we still don't have any updates to share yet. For your use case, I wonder why you set the TTL to be really long? We usually recommend using long TTL for static bucket that its content doesn't change that often. Is it possible to reduce TTL and then you can change logic in your service to do retry?

Another option you have is calling readdir to discover the new files because we don't serve readdir results from the cache, but it's more expensive operation.

[nguyenminhdungpg]

Hi @monthonk , thank you for your information.

I have a service to upload files to S3 using AWS SDK. File will be stored in folder tree with multi level style. File will not be modified or deleted. mount-s3 is used to serve files for my website and my website is very high traffics. My CDN cache is working not well, it still miss many requests and load on mount-s3 is very high at peak time. That's why I set mount-s3 TTL that long, I need files loaded as fast as possible.

mount-s3 negative cache does not only cache metadata of files, also folder. So when my service check if a file or folder is existing, folders' non-existence are cached by negative cache and break the logic of next requests (a lot of requests).
I can't use readdir because we store a very large number of files and it is very high traffics system, readdir can make performance bad.

I just need a way to flag off negative cache like disable_noobj_cache in s3fs.

As I see in many system, 404 is normally not cached or cached but with very small TTL. It is reasonable if we can flag off negative cache or negative cache has its own TTL.

[monthonk]

Thanks for clarifying. I assume that your two services are running on different systems, so that writer couldn't access mountpoint directory, and that's why you use AWS SDK to upload the files. Is that correct?

[nguyenminhdungpg]

@monthonk yes, that's correct. I use AWS SDK for upload service and I use s3fs/mount-s3 for file serving service (readonly).

[monthonk]

Thanks for your information @nguyenminhdungpg, I think an option to turn off negative caching is a valid feature request. We will come back to you once we have any news on this.

[nguyenminhdungpg]

@monthonk Thank you. I look forward to hearing from you.

[nguyenminhdungpg]

Hi @monthonk , I'd like to ask a question.
If O_DIRECT flag is used in opening file, Mountpoint will check S3 to ensure the object exists and return the latest object content. Does it update the file's latest metadata into cache after this?
Thank you.

[monthonk]

Hey @nguyenminhdungpg, yes, if you use O_DIRECT to open a file it will bypass the metadata cache and also update the cache after that.

[nguyenminhdungpg]

Hi @monthonk , thank you. I've just done an workaround to bypass the negative caching using O_DIRECT and it fetches the file from S3, update the cache. The workaround help resolving my issue but it is great if I can flag of negative caching or set negative caching TTL using mount-s3 command options. (e.g set negative cache TTL to be 30s or 1m). Wish next version of mount-s3 come with this feature.