Too many del stat cache entry and head requests for just 1 file requested

[nguyenminhdungpg]

Additional Information

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.93 (commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

2.9.9

Kernel information (uname -r)

6.2.16-19-pve

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"

How to run s3fs, if applicable

[] command line
s3fs $aws_bucket /home/mounted -o endpoint=ap-southeast-1 -o allow_other -o umask=000 -o dbglevel=info -o use_cache=/home/mounted-cache -o disable_noobj_cache -o max_background=1000 -o parallel_count=30 -o multireq_max=50 -o logfile=/var/log/mounted.log
[] /etc/fstab

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

2024-04-19T08:37:36.180Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5]
2024-04-19T08:37:36.179Z [INF] s3fs.cpp:s3fs_release(3062): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5]
2024-04-19T08:37:36.179Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.179Z [INF]       fdcache_entity.cpp:RowFlush(1413): [tpath=][path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5][physical_fd=26]
2024-04-19T08:37:36.179Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.108Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] []
2024-04-19T08:37:36.108Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:36.108Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:36.108Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084]
2024-04-19T08:37:36.108Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.107Z [INF] s3fs.cpp:s3fs_flush(2984): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5]
2024-04-19T08:37:36.060Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4]
2024-04-19T08:37:36.060Z [INF] s3fs.cpp:s3fs_release(3062): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4]
2024-04-19T08:37:36.057Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.057Z [INF]       fdcache_entity.cpp:RowFlush(1413): [tpath=][path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4][physical_fd=26]
2024-04-19T08:37:36.057Z [INF] s3fs.cpp:s3fs_flush(2984): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4]
2024-04-19T08:37:36.019Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.014Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.014Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.010Z [INF]       fdcache_entity.cpp:SetMCtime(835): [path=/Upload/img_2024_04_11_10_39_47.jpg][physical_fd=26][mtime=1712807063][ctime=1712807063]
2024-04-19T08:37:36.008Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.936Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] []
2024-04-19T08:37:35.936Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.936Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.936Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084]
2024-04-19T08:37:35.936Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.935Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.935Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=-1]
2024-04-19T08:37:35.935Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] []
2024-04-19T08:37:35.935Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.935Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.935Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084]
2024-04-19T08:37:35.935Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.934Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.934Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=-1]
2024-04-19T08:37:35.931Z [INF] s3fs.cpp:s3fs_open(2805): [path=/Upload/img_2024_04_11_10_39_47.jpg][flags=0x8800]
2024-04-19T08:37:35.931Z [INF] s3fs.cpp:s3fs_open(2805): [path=/Upload/img_2024_04_11_10_39_47.jpg][flags=0x8800]
2024-04-19T08:37:35.930Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.854Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] []
2024-04-19T08:37:35.854Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.853Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.852Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084]
2024-04-19T08:37:35.852Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.852Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/Upload/img_2024_04_11_10_39_47.jpg]

Details about issue

In haproxy log, there is only one request to img_2024_04_11_10_39_47.jpg but in logs of s3fs, there are 4 times of Head Request sent to S3 and 4 times executed delete stat cache entry of the file.
My website has very large number of uploaded images and when it takes high traffics, s3fs is very slow and there so so much of Head requests like above.
Could you please tell me why there are so much Head requests and how to optimize this.
Thank you very much.