Input/output error-S3FS

[kulanjith404]

Hi,

I just installed s3fs on Debian GNU/Linux 12 (bookworm) successfully. But when I'm going to create a directory it's getting an error like this. Please help!

mkdir: cannot create directory ‘qw’: Input/output error

This is the method that I mount it to this:

[sdlAnti]

Hello, i have same problem, if i build s3fs from src (i need version vith key bucket_size) mounted FS have IO error, if i use version from repo(Version: 1.90-1) everythin is ok, but 1.90 doe's not have key bucket_size

cat /etc/os-release && s3fs --version && ls -l / | grep backup
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Amazon Simple Storage Service File System V1.94 (commit:9ab5a2e) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
ls: cannot access '/backup': Input/output error
d?????????   ? ?    ?        ?            ? backup

[ggtakec]

@kulanjith404
When starting s3fs, try adding the dbglevel=info (or =dbg) or curldbg option to output the log.
I think those logs will help resolve this issue.

@sdlAnti
The content of your issue seems to be a different issue than this one.

ls: cannot access '/backup': Input/output error

This error may be caused by the fact that you created the /backup folder, possibly from a client other than s3fs.
Try compat_dir, complement_stat, umask options etc.

[kulanjith404]

@ggtakec
Thank you so much for responding to this.

As you mentioned I tried with the curldbg as an option with the s3fs mounting command. But how can I get the logs?

[ggtakec]

@kulanjith404
Since s3fs is running in the background, could you find the log messages in the system log file(under /var directory)?
Alternatively, you can start s3fs in the foreground(with the -f option) and output logs to stdout.
(At this time, you can also output to a file using the logfile option.)

[kulanjith404]

@ggtakec

I captured the logs at the moment that I'm going to create the folder. please find the below log detail. The folder name that I'm going to create Is 'testfolder'.

2024-04-15T17:16:49.459Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.459Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.459Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.459Z [CURL DBG] > HEAD /sense-test/testfolder HTTP/1.1
2024-04-15T17:16:49.459Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.459Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.459Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.459Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7bcdb57ae85502f13737babf8cc3b248623b7143e87b92e31d19e267d70fcc4c
2024-04-15T17:16:49.459Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.459Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.459Z [CURL DBG] >
2024-04-15T17:16:49.498Z [CURL DBG] < HTTP/1.1 404 Not Found
2024-04-15T17:16:49.498Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.498Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.498Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F1DC8104C158AA774D0D
2024-04-15T17:16:49.498Z [CURL DBG] < x-reserved-indicator: 10202
2024-04-15T17:16:49.498Z [CURL DBG] < x-amz-error-code: NoSuchKey
2024-04-15T17:16:49.498Z [CURL DBG] < x-amz-error-message: The specified key does not exist.
2024-04-15T17:16:49.498Z [CURL DBG] < x-amz-error-detail-Key: 1024/testfolder
2024-04-15T17:16:49.498Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSdBpyAvhdRPqBSWqeilT3zLaJ+uvkM1
2024-04-15T17:16:49.498Z [CURL DBG] < Date: Mon, 15 Apr 2024 17:16:50 GMT
2024-04-15T17:16:49.498Z [CURL DBG] < Content-Length: 0
2024-04-15T17:16:49.498Z [CURL DBG] <
2024-04-15T17:16:49.498Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact
2024-04-15T17:16:49.498Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.498Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.498Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.498Z [CURL DBG] > HEAD /sense-test/testfolder/ HTTP/1.1
2024-04-15T17:16:49.498Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.498Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.498Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.498Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=b652135220dc533ba25961a0ba0073912c273c191759ec8d4674c6342ae7af60
2024-04-15T17:16:49.498Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.499Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.499Z [CURL DBG] >
2024-04-15T17:16:49.546Z [CURL DBG] < HTTP/1.1 404 Not Found
2024-04-15T17:16:49.546Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.546Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.546Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F2038104C15A68BD3ED8
2024-04-15T17:16:49.546Z [CURL DBG] < x-reserved-indicator: 10202
2024-04-15T17:16:49.546Z [CURL DBG] < x-amz-error-code: NoSuchKey
2024-04-15T17:16:49.546Z [CURL DBG] < x-amz-error-message: The specified key does not exist.
2024-04-15T17:16:49.546Z [CURL DBG] < x-amz-error-detail-Key: 1024/testfolder
2024-04-15T17:16:49.546Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSy8q2NtH2+wyqzNM7D2BhPBzQMIcFIN
2024-04-15T17:16:49.546Z [CURL DBG] < Date: Mon, 15 Apr 2024 17:16:50 GMT
2024-04-15T17:16:49.546Z [CURL DBG] < Content-Length: 0
2024-04-15T17:16:49.546Z [CURL DBG] <
2024-04-15T17:16:49.546Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact
2024-04-15T17:16:49.546Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.546Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.546Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.546Z [CURL DBG] > HEAD /sense-test/testfolder_%24folder%24 HTTP/1.1
2024-04-15T17:16:49.546Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.546Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.546Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.546Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=abf6d246e9e2f26ebc0adb48d19789dc9c5938ead28adf583f50ec42ea7b7317
2024-04-15T17:16:49.546Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.546Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.546Z [CURL DBG] >
2024-04-15T17:16:49.551Z [CURL DBG] < HTTP/1.1 404 Not Found
2024-04-15T17:16:49.551Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.551Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.551Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F2338104C15DBAD55D79
2024-04-15T17:16:49.551Z [CURL DBG] < x-reserved-indicator: 10202
2024-04-15T17:16:49.551Z [CURL DBG] < x-amz-error-code: NoSuchKey
2024-04-15T17:16:49.551Z [CURL DBG] < x-amz-error-message: The specified key does not exist.
2024-04-15T17:16:49.551Z [CURL DBG] < x-amz-error-detail-Key: 1024/testfolder_$folder$
2024-04-15T17:16:49.551Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSkl0TULkkEkZLa2gETnnDZtUMWrOpzU
2024-04-15T17:16:49.551Z [CURL DBG] < Date: Mon, 15 Apr 2024 17:16:50 GMT
2024-04-15T17:16:49.551Z [CURL DBG] < Content-Length: 0
2024-04-15T17:16:49.551Z [CURL DBG] <
2024-04-15T17:16:49.551Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact
2024-04-15T17:16:49.551Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.551Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.551Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.551Z [CURL DBG] > GET /sense-test/?delimiter=/&max-keys=2&prefix=testfolder/ HTTP/1.1
2024-04-15T17:16:49.551Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.551Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.551Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.551Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a9f6b0ae75f2250bbd9e8ad32cae4bcf06c1eda2f034a176d1f2b27be989cf21
2024-04-15T17:16:49.551Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.551Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.551Z [CURL DBG] >
2024-04-15T17:16:49.555Z [CURL DBG] < HTTP/1.1 200 OK
2024-04-15T17:16:49.555Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.555Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.555Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F2378104C15FEBA0D1FC
2024-04-15T17:16:49.555Z [CURL DBG] < x-amz-bucket-region: as-south-210
2024-04-15T17:16:49.555Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSq2ESeUANYZaKGwWVD9YX+/3g1OrRhb
2024-04-15T17:16:49.555Z [CURL DBG] < Content-Type: application/xml
2024-04-15T17:16:49.555Z [CURL DBG] < Date: Mon, 15 Apr 2024 17:16:50 GMT
2024-04-15T17:16:49.555Z [CURL DBG] < Content-Length: 286
2024-04-15T17:16:49.555Z [CURL DBG] <
2024-04-15T17:16:49.555Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact
2024-04-15T17:16:49.555Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.555Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.555Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.555Z [CURL DBG] > HEAD /sense-test/testfolder HTTP/1.1
2024-04-15T17:16:49.555Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.555Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.555Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.555Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7bcdb57ae85502f13737babf8cc3b248623b7143e87b92e31d19e267d70fcc4c
2024-04-15T17:16:49.555Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.555Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.555Z [CURL DBG] >
2024-04-15T17:16:49.573Z [CURL DBG] < HTTP/1.1 404 Not Found
2024-04-15T17:16:49.573Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.573Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.573Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F23B8104C161E9F5C53E
2024-04-15T17:16:49.573Z [CURL DBG] < x-reserved-indicator: 10202
2024-04-15T17:16:49.573Z [CURL DBG] < x-amz-error-code: NoSuchKey
2024-04-15T17:16:49.573Z [CURL DBG] < x-amz-error-message: The specified key does not exist.
2024-04-15T17:16:49.573Z [CURL DBG] < x-amz-error-detail-Key: 1024/testfolder
2024-04-15T17:16:49.573Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSQB5D4d2BV63OWlPdjUvrL8yPoUPrKd
2024-04-15T17:16:49.573Z [CURL DBG] < Date: Mon, 15 Apr 2024 17:16:50 GMT
2024-04-15T17:16:49.573Z [CURL DBG] < Content-Length: 0
2024-04-15T17:16:49.573Z [CURL DBG] <
2024-04-15T17:16:49.573Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact
2024-04-15T17:16:49.573Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.573Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.573Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.573Z [CURL DBG] > HEAD /sense-test/testfolder/ HTTP/1.1
2024-04-15T17:16:49.573Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.573Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.573Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.573Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=b652135220dc533ba25961a0ba0073912c273c191759ec8d4674c6342ae7af60
2024-04-15T17:16:49.573Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.573Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.573Z [CURL DBG] >
2024-04-15T17:16:49.595Z [CURL DBG] < HTTP/1.1 404 Not Found
2024-04-15T17:16:49.595Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.595Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.595Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F24E8104C163607844E4
2024-04-15T17:16:49.595Z [CURL DBG] < x-reserved-indicator: 10202
2024-04-15T17:16:49.595Z [CURL DBG] < x-amz-error-code: NoSuchKey
2024-04-15T17:16:49.595Z [CURL DBG] < x-amz-error-message: The specified key does not exist.
2024-04-15T17:16:49.595Z [CURL DBG] < x-amz-error-detail-Key: 1024/testfolder
2024-04-15T17:16:49.595Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSn1rP2kIbbUWt772AQouHpF0h+RhuUh
2024-04-15T17:16:49.595Z [CURL DBG] < Date: Mon, 15 Apr 2024 17:16:50 GMT
2024-04-15T17:16:49.595Z [CURL DBG] < Content-Length: 0
2024-04-15T17:16:49.595Z [CURL DBG] <
2024-04-15T17:16:49.595Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact
2024-04-15T17:16:49.595Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.595Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.595Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.595Z [CURL DBG] > HEAD /sense-test/testfolder_%24folder%24 HTTP/1.1
2024-04-15T17:16:49.595Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.595Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.595Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.595Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=abf6d246e9e2f26ebc0adb48d19789dc9c5938ead28adf583f50ec42ea7b7317
2024-04-15T17:16:49.595Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.595Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.595Z [CURL DBG] >
2024-04-15T17:16:49.600Z [CURL DBG] < HTTP/1.1 404 Not Found
2024-04-15T17:16:49.600Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.600Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.600Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F2648104C1651C51179D
2024-04-15T17:16:49.600Z [CURL DBG] < x-reserved-indicator: 10202
2024-04-15T17:16:49.600Z [CURL DBG] < x-amz-error-code: NoSuchKey
2024-04-15T17:16:49.600Z [CURL DBG] < x-amz-error-message: The specified key does not exist.
2024-04-15T17:16:49.600Z [CURL DBG] < x-amz-error-detail-Key: 1024/testfolder_$folder$
2024-04-15T17:16:49.600Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSgZlS7oNiW94ckjQqNdjk1iQK6flMa0
2024-04-15T17:16:49.600Z [CURL DBG] < Date: Mon, 15 Apr 2024 17:16:50 GMT
2024-04-15T17:16:49.600Z [CURL DBG] < Content-Length: 0
2024-04-15T17:16:49.600Z [CURL DBG] <
2024-04-15T17:16:49.600Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact
2024-04-15T17:16:49.600Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.600Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.600Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.600Z [CURL DBG] > GET /sense-test/?delimiter=/&max-keys=2&prefix=testfolder/ HTTP/1.1
2024-04-15T17:16:49.600Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.600Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.600Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.600Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a9f6b0ae75f2250bbd9e8ad32cae4bcf06c1eda2f034a176d1f2b27be989cf21
2024-04-15T17:16:49.600Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.600Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.600Z [CURL DBG] >
2024-04-15T17:16:49.605Z [CURL DBG] < HTTP/1.1 200 OK
2024-04-15T17:16:49.605Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.605Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.605Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F2698104C16774EE41C2
2024-04-15T17:16:49.605Z [CURL DBG] < x-amz-bucket-region: as-south-210
2024-04-15T17:16:49.605Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCS+MwAjxKmfIarLLKkx0cEN81AJ7lO5X
2024-04-15T17:16:49.605Z [CURL DBG] < Content-Type: application/xml
2024-04-15T17:16:49.605Z [CURL DBG] < Date: Mon, 15 Apr 2024 17:16:50 GMT
2024-04-15T17:16:49.605Z [CURL DBG] < Content-Length: 286
2024-04-15T17:16:49.605Z [CURL DBG] <
2024-04-15T17:16:49.605Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact
2024-04-15T17:16:49.606Z [CURL DBG] * Found bundle for host: 0x7f19381f4500 [serially]
2024-04-15T17:16:49.606Z [CURL DBG] * Can not multiplex, even if we wanted to
2024-04-15T17:16:49.606Z [CURL DBG] * Re-using existing connection #1 with host obs.as-south-210.orel.cloud
2024-04-15T17:16:49.606Z [CURL DBG] > PUT /sense-test/testfolder/ HTTP/1.1
2024-04-15T17:16:49.606Z [CURL DBG] > Host: obs.as-south-210.orel.cloud
2024-04-15T17:16:49.606Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-04-15T17:16:49.606Z [CURL DBG] > Accept: /
2024-04-15T17:16:49.606Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=Y7437CA6LTJJGKHAQHBC/20240415/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-atime;x-amz-meta-ctime;x-amz-meta-gid;x-amz-meta-mode;x-amz-meta-mtime;x-amz-meta-uid, Signature=a999e6aef0f220c9d96af9f8a97d85f96bafa131309540ab69b838bde9c88d47
2024-04-15T17:16:49.606Z [CURL DBG] > Content-Type: application/x-directory
2024-04-15T17:16:49.606Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-04-15T17:16:49.606Z [CURL DBG] > x-amz-date: 20240415T171649Z
2024-04-15T17:16:49.606Z [CURL DBG] > x-amz-meta-atime: 1713201409
2024-04-15T17:16:49.606Z [CURL DBG] > x-amz-meta-ctime: 1713201409
2024-04-15T17:16:49.606Z [CURL DBG] > x-amz-meta-gid: 0
2024-04-15T17:16:49.606Z [CURL DBG] > x-amz-meta-mode: 493
2024-04-15T17:16:49.606Z [CURL DBG] > x-amz-meta-mtime: 1713201409
2024-04-15T17:16:49.606Z [CURL DBG] > x-amz-meta-uid: 0
2024-04-15T17:16:49.606Z [CURL DBG] > Content-Length: 0
2024-04-15T17:16:49.606Z [CURL DBG] >
2024-04-15T17:16:49.608Z [CURL DBG] < HTTP/1.1 400 Bad Request
2024-04-15T17:16:49.608Z [CURL DBG] < Server: OBS
2024-04-15T17:16:49.608Z [CURL DBG] < x-reserved: amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc
2024-04-15T17:16:49.608Z [CURL DBG] < x-amz-id-2: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCRg1jeQ7UeFvCtg+wSX/N8hrT8UtZgrV
2024-04-15T17:16:49.608Z [CURL DBG] < x-amz-request-id: 0000018EE2C2F26E8104C169D4CBE84E
2024-04-15T17:16:49.608Z [CURL DBG] < Content-Type: application/xml
2024-04-15T17:16:49.608Z [CURL DBG] < Content-Length: 303
2024-04-15T17:16:49.608Z [CURL DBG] <
2024-04-15T17:16:49.608Z [CURL DBG] * Connection #1 to host obs.as-south-210.orel.cloud left intact

[bdwheele]

I'm also experiencing this bug when I try to connect to versitygw. It started happening when they added this code:

	utils.SetResponseHeaders(ctx, []utils.CustomHeader{		
		{
			Key:   "X-Amz-Access-Point-Alias",
			Value: "false",
		},
		{
			Key:   "X-Amz-Bucket-Region",
			Value: region,
		},
	})

which returns a few extra X-Amz headers. When I comment out src/curl.cpp:3425

            //meta[key] = value;        // key is lower case for "x-amz"

it works as expected.