rm: can't remove 'test': I/O error

[FlowerBirds]

Additional Information

Version of s3fs being used (s3fs --version)

s3fs 1.19

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

LAPTOP-TC4A0SCV:/data/test4/s3fs-fuse-1.93# apk list | grep fuse-
fuse-exfat-1.3.0-r2 x86_64 {fuse-exfat} (GPL-2.0-or-later)
fuse-openrc-3.11.0-r0 x86_64 {fuse3} (GPL-2.0-only LGPL-2.1-only) [installed]
fuse-2.9.9-r1 x86_64 {fuse} (GPL-2.0-only LGPL-2.1-only) [installed]
ceph-fuse-16.2.10-r1 x86_64 {ceph} (LGPL-2.1-only AND LGPL-2.0-or-later AND GPL-2.0-only AND GPL-3.0-only AND CC-BY-SA-1.0 AND BSL-1.0 AND GPL-2.0-or-later WITH Autoconf-exception-2.0 AND BSD-3-Clause AND MIT AND custom)
fuse-overlayfs-1.8.2-r0 x86_64 {fuse-overlayfs} (GPL-3.0-or-later)
fuse-static-2.9.9-r1 x86_64 {fuse} (GPL-2.0-only LGPL-2.1-only)
fuse-overlayfs-doc-1.8.2-r0 x86_64 {fuse-overlayfs} (GPL-3.0-or-later)
kio-fuse-5.0.1-r2 x86_64 {kio-fuse} (GPL-3.0-only)
unionfs-fuse-doc-2.1-r0 x86_64 {unionfs-fuse} (BSD-3-Clause)
fuse-doc-2.9.9-r1 x86_64 {fuse} (GPL-2.0-only LGPL-2.1-only)
fuse-common-3.11.0-r0 x86_64 {fuse3} (GPL-2.0-only LGPL-2.1-only) [installed]
fuse-exfat-doc-1.3.0-r2 x86_64 {fuse-exfat} (GPL-2.0-or-later)
fuse-dev-2.9.9-r1 x86_64 {fuse} (GPL-2.0-only LGPL-2.1-only) [installed]
rbd-fuse-16.2.10-r1 x86_64 {ceph} (LGPL-2.1-only AND LGPL-2.0-or-later AND GPL-2.0-only AND GPL-3.0-only AND CC-BY-SA-1.0 AND BSL-1.0 AND GPL-2.0-or-later WITH Autoconf-exception-2.0 AND BSD-3-Clause AND MIT AND custom)
fuse-exfat-utils-1.3.0-r2 x86_64 {fuse-exfat} (GPL-2.0-or-later)
py3-confuse-1.7.0-r0 x86_64 {py3-confuse} (MIT)
unionfs-fuse-2.1-r0 x86_64 {unionfs-fuse} (BSD-3-Clause)
confuse-dev-3.3-r0 x86_64 {confuse} (ISC)
gvfs-fuse-1.50.1-r0 x86_64 {gvfs} (LGPL-2.0-or-later)
confuse-doc-3.3-r0 x86_64 {confuse} (ISC)
confuse-3.3-r0 x86_64 {confuse} (ISC)

Kernel information (uname -r)

5.15.133.1-microsoft-standard-WSL2

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.16.0
PRETTY_NAME="Alpine Linux v3.16"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

How to run s3fs, if applicable

[x] command line
[] /etc/fstab

/usr/bin/s3fs -d data /data -f -o url=http://191.168.3.132:9100,passwd_file=/etc/passwd-s3fs,endpoint=us-east-1,allow_other,use_cache=/tmp,max_stat_cache_size=1000,stat_cache_expire=900,retries=5,connect_timeout=10,use_path_request_style

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

2024-01-05T03:15:36.545Z [INF]       curl.cpp:RemakeHandle(2165): Retry request. [type=0][url=http://191.168.3.132:9100/data/test4/s3fs-fuse-1.93/test/][path=/test4/s3fs-fuse-1.93/test/]
2024-01-05T03:15:36.545Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [DELETE] [/test4/s3fs-fuse-1.93/test/] [] []
2024-01-05T03:15:36.545Z [INF]       curl_util.cpp:url_to_host(334): url is http://191.168.3.132:9100
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(file is corrupted)</Message><Key>test4/s3fs-fuse-1.93/test/</Key><BucketName>data</BucketName><Resource>/data/test4/s3fs-fuse-1.93/test/</Resource><Region>us-east-1</Region><RequestId>17A75515C1848546</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>2024-01-05T03:15:36.549Z [INF]       curl.cpp:RequestPerform(2590): HTTP response code 500 was returned, slowing down
2024-01-05T03:16:24.549Z [INF] curl.cpp:RequestPerform(2719): ### retrying...

Details about issue

Step1 unzip file

tar xzf s3fs-fuse-1.93.tar.gz

Step2 remove s3fs-fuse-1.93 folder

LAPTOP-TC4A0SCV:/data/test4# rm -rf s3fs-fuse-1.93/test/
rm: can't remove 's3fs-fuse-1.93/test': I/O error

By observing the Wireshark packet capture, it was found that during the decompression process, the test folder was first created by calling PUT /data/test4/s3fs-fuse-1.93/test/, and then the files under the /data/test4/s3fs-fuse-1.93/test/ folder were created by calling PUT. After the files under the test folder were uploaded, PUT /data/test4/s3fs-fuse-1.93/test/ and PUT /data/test4/s3fs-fuse-1.93/ were called once again and successfully created. When manually creating a folder using mkdir test, only one PUT method was called, so the manually created folder can be deleted. However, when deleting the folder after decompression, it cannot be deleted, resulting in an I/O error.

Question：Why is it called twice to create a folder during decompression?

[FlowerBirds]

LAPTOP-TC4A0SCV:/data/test4# ll
total 278
-rw-r-----    1 root     root        284353 Jan  5 09:39 s3fs-fuse-1.93.tar.gz
LAPTOP-TC4A0SCV:/data/test4# mkdir pp
LAPTOP-TC4A0SCV:/data/test4# ll
total 278
drwxr-xr-x    1 root     root             0 Jan  5 14:20 pp
-rw-r-----    1 root     root        284353 Jan  5 09:39 s3fs-fuse-1.93.tar.gz
LAPTOP-TC4A0SCV:/data/test4# chmod 777 pp
LAPTOP-TC4A0SCV:/data/test4# ll
total 278
drwxrwxrwx    1 root     root             0 Jan  5 14:20 pp
-rw-r-----    1 root     root        284353 Jan  5 09:39 s3fs-fuse-1.93.tar.gz
LAPTOP-TC4A0SCV:/data/test4# ll pp
total 0
LAPTOP-TC4A0SCV:/data/test4# rm -rf pp
rm: can't remove 'pp': I/O error
LAPTOP-TC4A0SCV:/data/test4#

I find chmod will send PUT method to create folder again, then delete failed.

[FlowerBirds]

minio/minio#18739

[ggtakec]

@FlowerBirds
Both creating and deleting directories are working normally.
We need a little more information to solve your problem.

If you are able, try the following:

• Try using the code from the master branch of s3fs (If building is difficult, please try the following while using v1.91 you are currently using)
• Try dbglevel and curldbg options and collect logs.
  This log can help detect problems.

Thanks in advance for your assistance.