Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fails to mount volume after reboot of IMDSv2 instance #2391

Open
mosaic-nick opened this issue Dec 15, 2023 · 1 comment
Open

Fails to mount volume after reboot of IMDSv2 instance #2391

mosaic-nick opened this issue Dec 15, 2023 · 1 comment

Comments

@mosaic-nick
Copy link

Additional Information

Version of s3fs being used (s3fs --version)

V1.93 (commit:unknown)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

2.9.2

Kernel information (uname -r)

5.10.201-191.748.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"

How to run s3fs, if applicable

s3fs#:/srcdata /var/opt/mosaic/srcdata fuse uid=,gid=**********,allow_other,mp_umask=277,umask=277,use_cache=/tmp,iam_role=auto,url=https://s3.eu-west-1.amazonaws.com,endpoint=eu-west-1,dbglevel=debug 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: s3fs version 1.93(unknown) : s3fs -o rw,uid=995,gid=1002,allow_other,mp_umask=277,umask=277,use_cache=/tmp,iam_role=auto,url=https://s3.eu-west-1.amazonaws.com,endpoint=eu-west-1,dbglevel=debug,dev,suid ************:/srcdata /var/opt/mosaic/srcdata
Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [DBG]
Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]:    PROC(uid=0, gid=0) - MountPoint(uid=995, gid=1002, mode=40705)Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: Loaded mime information from /etc/mime.types
Dec 15 10:37:02 ip-10-7-4-122 s3fs[2577]: init v1.93(commit:unknown) with OpenSSL, credential-library(built-in)
Dec 15 10:37:02 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2620): ### CURLE_COULDNT_CONNECT
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: ### retrying...
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]:      Retry request. [type=-1][url=http://169.254.169.254/latest/api/token][path=]
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: Failed to reset handle and internal data for retrying.
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: curl.cpp:GetIAMv2ApiToken(3055): Error(-5) occurred, could not get IAMv2 api token.
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: s3fs_cred.cpp:GetIAMCredentialsURL(409): AWS IMDSv2 token retrieval failed: -5
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]:      Get IAM Role name
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2620): ### CURLE_COULDNT_CONNECT
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: ### retrying...
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]:      Retry request. [type=14][url=http://169.254.169.254/latest/meta-data/iam/security-credentials/][path=]
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2598): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>#012<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"#012#011"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">#012<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">#012 <head>#012  <title>401 - Unauthorized</title>#012 </head>#012 <body>#012  <h1>401 - Unauthorized</h1>#012 </body>#012</html>
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: curl.cpp:GetIAMRoleFromMetaData(3199): Error(-5) occurred, could not get IAM role name from meta data.
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: s3fs.cpp:s3fs_init(4296): could not load IAM role name from meta data.
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: s3fs.cpp:s3fs_exit_fuseloop(4277): Exiting FUSE event loop due to errors
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: destroy

Details about issue

Upon rebooting an instance using IMDSv2, the s3fs mount is not mounted successfully. From the logs, it looks like it tries to get an IMDSv2 token, but curl fails to connect, suggesting that the IMDS endpoint is not available at that point. It then seems to retry, but still can't connect, and then errors, but tries to continue to retrieve the IAM role anyway, which then fails due to a lack of the IMDSv2 token.

Logging on to the instance and running "mount -a" then successfully mounts the volume.
@mosaic-nick
Copy link
Author

It seems that the issue is that it is trying to mount before the network service is running. I have been able to work around this by setting the following options on the mount in fstab:

_netdev,x-systemd.after=network-online.target

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mosaic-nick