s3fs-fuse to HCP (Hitachi Content Platform) issue #2423
Comments
|
Adding this extra debug logging from this morning: [root@centos8s hcpdata]# s3fs testns -o use_cache=/tmp -o use_path_request_style -o allow_other -o no_check_certificate -o multireq_max=5 -o passwd_file=/etc/s3fs-passwd -o dbglevel=info -f -o curldbg /hcpdata -o url=https://testtenant.hcp1.demo.com I see these successful connection attempts in the HCP logs like it mounted the bucket fine: 192.168.1.101 - security [27/Feb/2024:10:26:06 -0500] "GET /testns/ HTTP/1.1" 200 248 testtenant@hs3 10 101 0 |
|
I added my SSL certificate to the bundle in /etc/pki/tls/certs/ca-bundle.crt and tried again with these results: [root@centos8s hcpdata]# s3fs testns -o use_cache=/tmp -o use_path_request_style -o allow_other -o multireq_max=5 -o passwd_file=/etc/s3fs-passwd -o dbglevel=info -f -o curldbg /hcpdata -o url=https://testtenant.hcp1.demo.com |
|
FYI, I just tested and versions 1.85, 1.87, and 1.89 work FINE with NO issue. |
|
I tested the latest 1.93 against our HCP Cloudscale Product and it works with no problem, but not our HCP product. Both are SIGV4 compliant. |
|
@HDS-nlucas Below are the three s3fs execution commands you reported: The only difference between (1) and (2) is the presence or absence of the debug output option, but the log for (2) appears to be the result of a successful startup. I don't know why it failed in (1) (I can't determine it from the logs in (2) and (3)), but what is the problem? |
|
@ggtakec Correct, (2) also gave the IO error. (3) also gave the IO error. So, with that said, version 1.93 and 1.91 works fine with our newer HCP Cloudscale product. |
|
@HDS-nlucas |
|
@ggtakec Thanks for the followup. I just tested 1.94 against our HCP platform and it still gives the error: -bash: cd: /hcpdata: Input/output error This error is AFTER mounting the bucket successfully to the /hcpdata folder. On the HCP, I see it mounts successfully: |
Could you get the s3fs log(started with |
|
S3FS Version: 1.94-1.el9 Log from S3FS: [root@localhost ~]# s3fs testns -o use_cache=/tmp -o use_path_request_style -o allow_other -o multireq_max=5 -o no_check_certificate -o passwd_file=/etc/passwd-s3fs -o dbglevel=info -f -o curldbg /hcpdata -o url=https://testtenant.hcp1.demo.com/ Log from HCP S3 storage: 192.168.1.102 - security [15/Apr/2024:14:32:14 -0400] "GET /testns/ HTTP/1.1" 200 248 testtenant@hs3 52 101 0 Log from trying to access mounted bucket: |
Additional Information
Version of s3fs being used (
s3fs --version)[root@centos8s ~]# s3fs --version
Amazon Simple Storage Service File System V1.93 (commit:unknown) with OpenSSL
I installed via:
yum install epel-release
yum install s3fs-fuse
Version of fuse being used (
pkg-config --modversion fuse,rpm -qi fuseordpkg -s fuse)[root@centos8s ~]# rpm -qi fuse
Name : fuse
Version : 2.9.7
Release : 17.el8
Architecture: x86_64
Install Date: Thu 22 Feb 2024 05:20:13 PM EST
Group : Unspecified
Size : 208380
License : GPL+
Signature : RSA/SHA256, Fri 28 Apr 2023 07:35:03 AM EDT, Key ID 05b555b38483c65d
Source RPM : fuse-2.9.7-17.el8.src.rpm
Build Date : Tue 18 Apr 2023 02:26:34 PM EDT
Build Host : x86-04.stream.rdu2.redhat.com
Relocations : (not relocatable)
Packager : builder@centos.org
Vendor : CentOS
URL : http://fuse.sf.net
Summary : File System in Userspace (FUSE) v2 utilities
Kernel information (
uname -r)[root@centos8s ~]# uname -r
4.18.0-540.el8.x86_64
GNU/Linux Distribution, if applicable (
cat /etc/os-release)[root@centos8s ~]# cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
How to run s3fs, if applicable
s3fs testns -o use_cache=/tmp -o use_path_request_style -o allow_other -o no_check_certificate -o multireq_max=5 -o passwd_file=/etc/s3fs-passwd /hcpdata -o url=https://testtenant.hcp1.demo.com
s3fs syslog messages (
grep s3fs /var/log/syslog,journalctl | grep s3fs, ors3fs outputs)Feb 26 11:47:02 centos8s s3fs[10154]: s3fs version 1.93(unknown) : s3fs -o use_cache=/tmp -o use_path_request_style -o allow_other -o no_check_certificate -o multireq_max=5 -o passwd_file= /etc/s3fs-passwd -o url=https://testtenant.hcp1.demo.com testns /hcpdata
Feb 26 11:47:02 centos8s s3fs[10154]: Loaded mime information from /etc/mime.types
Feb 26 11:47:02 centos8s s3fs[10155]: init v1.93(commit:unknown) with OpenSSL, credential-library(built-in)
Details about issue
All seems to work ok, except after mounting the bucket to /hcpdata, when I do a df -kh it shows up correctly:
[root@centos8s ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 11M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/mapper/cs-root 70G 6.0G 64G 9% /
/dev/mapper/cs-home 421G 3.1G 418G 1% /home
/dev/nvme0n1p1 1014M 274M 741M 27% /boot
tmpfs 364M 32K 364M 1% /run/user/1000
s3fs 4.0G 0 4.0G 0% /hcpdata
Then I do a cd /hcpdata, I get the error:
[root@centos8s ~]# cd /hcpdata
-bash: cd: /hcpdata: Input/output error
This happens on 1.93 and 1.91 code levels. I am an expert on Hitachi Content Platform (HCP) which is S3 SIGv4 compliant storage. I can do a Teams meeting to share and display the issue if necessary.
When I mount the bucket, this is the log information from HCP indicating it mounted the namespace./bucket correctly:
192.168.1.101 - security [26/Feb/2024:11:02:26 -0500] "GET /testns/ HTTP/1.1" 200 248 testtenant@hs3 51 101 0
192.168.1.101 - security [26/Feb/2024:11:02:34 -0500] "HEAD /testns// HTTP/1.1" 200 0 testtenant@hs3 3 101 0
The text was updated successfully, but these errors were encountered: