Input/output error with Scaleway S3 bucket when trying to write

[Doctor-Who]

Additional Information

Version of s3fs being used (s3fs --version)

V1.93 (commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

2.9.9

Kernel information (uname -r)

6.6.3-1-default

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="openSUSE Tumbleweed"
# VERSION="20231207"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20231207"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20231207"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"

How to run s3fs, if applicable

Used in fstab as below :

s3fs#reolink /srv/backup/reolink fuse _netdev,allow_other,use_path_request_style,passwd_file=/root/.passwd-s3fs,url=https://s3.fr-par.scw.cloud/ 0 0
s3fs#gislab-homeassistant /srv/backup/homeassistant fuse _netdev,allow_other,use_path_request_style,passwd_file=/root/.passwd-s3fs,url=https://s3.fr-par.scw.cloud/ 0 0

cp files or even a simple touch text file :
touch: failed to close 'test.txt': Input/output error
cp: cannot create directory '/srv/backup/reolink/2023/12': Input/output error

information : rm command well works

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Cannot find any log, I'm running on a Pi3B+ :/

Details about issue

Since few weeks, upload to my s3 bucket is not working anymore. Was working during 2 years without any problem.
It is suppose to copy from FTP (camera video) to S3 mount point to send them into the cloud.
I've try to umount and mount without any success.
Even rebooting the Raspberry not working.

I've see other issue but without any solution (or any solution I cannot understand)

Thanks for this amazing opensource project.

[klausbadelt]

similar issue on AWS S3, simple cp fails after ~50 secs.

[ggtakec]

@Doctor-Who
The version of s3fs described in this issue is 1.93, but does the fact that it has been working for several years mean that s3fs was upgraded when it stopped working?

I need to know the reason why it doesn't work in detail, so if you can, please specify options such as dbglevel or curldbg and provide the log of the problem area.

Thanks in advance for your assistance.

[Doctor-Who]

@ggtakec Thanks for your message.
It happens when my openSUSE Tumbleweed updated. It do upgrade every month by cron event. And it stops working since snapshot in december.

Now my version is :

NAME="openSUSE Tumbleweed"
# VERSION="20240131"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20240131"
PRETTY_NAME="openSUSE Tumbleweed"

Perhaps kernel incompatibility ?

[juliogonzalez]

Perhaps kernel incompatibility ?

Please provide the info requested by @ggtakec :-)

It would probably also help if you can list the differences in the installed packages packages, if you still have the right btrtfs snapshots available:

• One where everything was working
• One right after it stopped working

One of the runtime requirements for s3fs is fuse. But fuse didn't get a bump in December according to https://build.opensuse.org/package/show/openSUSE%3AFactory/fuse, so could be the kernel or something else.

And s3fs 1.93 was submitted 7 months ago, so the build dependencies should not be a problem either.

[andibraeu]

Hi,

I have the same problem, but it's a server running on debian (dev-instance at scaleway):

Version of s3fs being used (s3fs --version)

V1.89 (commit:unknown) with GnuTLS(gcrypt) (delivered with debian)
V1.94 (commit:70a30d6) with OpenSSL

GNU/Linux Distribution, if applicable (cat /etc/os-release)

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Kernel information (uname -r)

4.19.0-26-amd64

What I know: Writing worked until Feb 16, 8:32 UTC, as I have a cronjob running there updating a timestamp every minute. I also don't think there was an update of s3fs. But maybe there were changes at scaleway?

[andibraeu]

I found the root case in the logs:

curl.cpp:RequestPerform(2536): HTTP response code 400, returning EIO. Body Text: <?xml version='1.0' encoding='UTF-8'?>#012<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'</Message><Region>fr-par</Region><RequestId>...</RequestId></Error>

Maybe Scaleway accepted other regions, earlier. And now they updated or corrected their security config. I simply appended endpoint=fr-par to my config and now it works.

[blacklight]

I can confirm that @andibraeu's workaround (adding endpoint=<cc>-<city> to the mount configuration) works.

Same issue here as the others - I can't recall exactly when s3fs stopped working when writing to my Scaleway buckets, but it was probably around November/December.

Maybe worth adding the endpoint= configuration for Scaleway to the README/FAQ?

[andibraeu]

I think it's not a workaround, it's the solution ;)

It's also documented by scaleway: https://www.scaleway.com/en/docs/tutorials/object-storage-s3fs/#s3fs-fuse