s3fs freezes when running a working command with the -f foreground option

[BrianHVB]

Details about issue

While troubleshooting an issue with mounting a specific bucket, I attempted to use the -f (foreground) option. This resulted in the terminal running the foreground s3fs process completely freezing. It was not responsive to any ctrl+c or similar commands. Additionally, attempting to perform any disk operations from another terminal (such as ls /tmp or df -h) would hang indefinitely. The only way to recover would be to ps -aux | grep s3fs and kill -9 the stuck s3fs process.

This happens with any s3fs command run in the foreground, even commands that otherwise successfully mount the S3 bucket. Taking a working command (tested several) and just adding the -f option causes the program to freeze

Additional Information

Version of s3fs being used (s3fs --version)

s3fs --version

Amazon Simple Storage Service File System V1.90 (commit:unknown) with GnuTLS(gcrypt)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

dpkg -s fuse

Package: fuse
Status: deinstall ok config-files
Priority: optional
Section: utils
Installed-Size: 130
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Architecture: amd64
Source: fuse (2.9.9-6)
Version: 2.9.9-6+b1
Config-Version: 2.9.9-6+b1

dpkg -s fuse3

Package: fuse3
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 100
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Architecture: amd64
Version: 3.14.0-4
Replaces: fuse
Provides: fuse (= 3.14.0-4)
Depends: libc6 (>= 2.34), libfuse3-3 (= 3.14.0-4), adduser, mount (>= 2.19.1), sed (>= 4)
Breaks: fuse

Kernel information (uname -r)

uname -r

6.1.0-17-cloud-amd64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"

Executed command

# works fine
s3fs -o dbglevel=debug -o curldbg -o iam_role=auto <our_bucket>:/<our_prefix>/ /tmp/s3
# no output

ls /tmp/s3
# list of files are displayed
umount /tmp/s3

# freezes -- cannot ctl+c;
# same command, but with -f
s3fs -f -o dbglevel=debug -o curldbg -o iam_role=auto <our_bucket>:/<our_prefix>/ /tmp/s3

2024-01-10T20:51:11.017Z [INF] s3fs version 1.90(unknown) : s3fs -f -o dbglevel=debug -o curldbg -o iam_role=auto <redacted> /tmp/s3
2024-01-10T20:51:11.018Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [DBG] 
2024-01-10T20:51:11.018Z [INF]     s3fs.cpp:set_mountpoint_attribute(4093): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40755)
2024-01-10T20:51:11.018Z [DBG] curl.cpp:InitMimeType(410): Try to load mime types from /etc/mime.types file.
2024-01-10T20:51:11.018Z [DBG] curl.cpp:InitMimeType(415): The old mime types are cleared to load new mime types.
2024-01-10T20:51:11.020Z [INF] curl.cpp:InitMimeType(434): Loaded mime information from /etc/mime.types
2024-01-10T20:51:11.020Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission.
2024-01-10T20:51:11.023Z [INF] s3fs.cpp:s3fs_init(3382): init v1.90(commit:unknown) with GnuTLS(gcrypt)
2024-01-10T20:51:11.023Z [INF]       curl.cpp:LoadIAMRoleFromMetaData(2919): Get IAM Role name
2024-01-10T20:51:11.023Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2024-01-10T20:51:11.023Z [DBG] curl.cpp:RequestPerform(2283): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/
2024-01-10T20:51:11.023Z [CURL DBG] *   Trying 169.254.169.254:80...
2024-01-10T20:51:11.023Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
2024-01-10T20:51:11.023Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
2024-01-10T20:51:11.023Z [CURL DBG] > Host: 169.254.169.254
2024-01-10T20:51:11.023Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-01-10T20:51:11.023Z [CURL DBG] > Accept: */*
2024-01-10T20:51:11.023Z [CURL DBG] > 
2024-01-10T20:51:11.024Z [CURL DBG] < HTTP/1.1 200 OK
2024-01-10T20:51:11.024Z [CURL DBG] < Content-Type: text/plain
2024-01-10T20:51:11.024Z [CURL DBG] < Accept-Ranges: none
2024-01-10T20:51:11.024Z [CURL DBG] < Last-Modified: Wed, 10 Jan 2024 20:32:46 GMT
2024-01-10T20:51:11.024Z [CURL DBG] < Content-Length: 19
2024-01-10T20:51:11.024Z [CURL DBG] < Date: Wed, 10 Jan 2024 20:51:11 GMT
2024-01-10T20:51:11.024Z [CURL DBG] < Server: EC2ws
2024-01-10T20:51:11.024Z [CURL DBG] < Connection: close
2024-01-10T20:51:11.024Z [CURL DBG] < 
2024-01-10T20:51:11.024Z [CURL DBG] * Closing connection 0
2024-01-10T20:51:11.024Z [INF]       curl.cpp:RequestPerform(2316): HTTP response code 200
2024-01-10T20:51:11.024Z [INF]       curl.cpp:SetIAMRoleFromMetaData(1802): IAM role name response = "<redacted_valid_role_name>"
2024-01-10T20:51:11.024Z [INF] s3fs.cpp:s3fs_init(3399): loaded IAM role name = <redacted_valid_role_name>
2024-01-10T20:51:11.024Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
2024-01-10T20:51:11.024Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
2024-01-10T20:51:11.024Z [INF] s3fs.cpp:s3fs_check_service(3516): check services.
2024-01-10T20:51:11.024Z [INF] curl.cpp:CheckIAMCredentialUpdate(1770): IAM Access Token refreshing...
2024-01-10T20:51:11.024Z [INF]       curl.cpp:GetIAMCredentials(2822): [IAM role=<redacted_valid_role_name>]
2024-01-10T20:51:11.024Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 30
2024-01-10T20:51:11.024Z [DBG] curl.cpp:RequestPerform(2283): connecting to URL http://169.254.169.254/latest/api/token
2024-01-10T20:51:11.024Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache
2024-01-10T20:51:11.024Z [CURL DBG] *   Trying 169.254.169.254:80...
2024-01-10T20:51:11.024Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1)
2024-01-10T20:51:11.024Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2024-01-10T20:51:11.024Z [CURL DBG] > Host: 169.254.169.254
2024-01-10T20:51:11.024Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-01-10T20:51:11.025Z [CURL DBG] > Accept: */*
2024-01-10T20:51:11.025Z [CURL DBG] > Transfer-Encoding: chunked
2024-01-10T20:51:11.025Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2024-01-10T20:51:11.025Z [CURL DBG] > Expect: 100-continue
2024-01-10T20:51:11.025Z [CURL DBG] > 
2024-01-10T20:51:11.025Z [CURL DBG] < HTTP/1.1 100 Continue
Killed

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

journalctl | grep s3fs

# last few lines after the successful ls command
Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: fdcache.cpp:OpenExistFdEntity(628): [path=/<redacted_file_name>][flags=0x0]
Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: fdcache.cpp:Open(526): [path=/<redacted_file_name>[size=-1][time=-1][flags=0x0]
Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: s3fs.cpp:s3fs_getattr(801): [path=/<redacted_file_name>] uid=1000, gid=1000, mode=100640 

# next line is a single time jump to the failed/frozen s3fs
Jan 10 20:51:05 ip-172-31-70-135 s3fs[17565]: destroy

[ggtakec]

@BrianHVB Are you trying to use fuse3?
s3fs-fuse does not yet support fuse3.(Code modifications will be required to support it.)
Although it may seem to be working normally, problems will probably occur in various places when using fuse3.