Assuming role configured in AWS profile does not work outside of aws partition (China, US Gov Cloud, etc.)

[csy97]

Mountpoint for Amazon S3 version

mount-s3 1.6.0

AWS Region

cn-northwest-1

Describe the running environment

When I use the command AWS_DEFAULT_REGION=cn-northwest-1 AWS_PROFILE=bwmtest2 mount-s3 --region cn-northwest-1 --debug --log-directory . / testtmppp /mnt/ mount-s3 will fail when mounted

[root@VM-0-5-centos .aws]# AWS_DEFAULT_REGION=cn-northwest-1 AWS_PROFILE=bwmtest2 mount-s3 --region cn-northwest-1   --debug --log-directory ./   testtmppp /mnt/
Error: Failed to create S3 client

Caused by:
    0: initial ListObjectsV2 failed for bucket testtmppp in region cn-northwest-1
    1: Client error
    2: No signing credentials found
Error: Failed to create mount process

But the fact that I was able to get the IAM role correctly

AWS_DEFAULT_REGION=cn-northwest-1 AWS_PROFILE=bwmtest2 aws sts get-caller-identity
{
    "Account": "714736990101",
    "UserId": "AROA2M2NE66K5E6KBFALA:botocore-session-1713966126",
    "Arn": "arn:aws-cn:sts::714736990101:assumed-role/s3fullrole/botocore-session-1713966126"
}

Mountpoint options

AWS_DEFAULT_REGION=cn-northwest-1 AWS_PROFILE=bwmtest2 mount-s3 --region cn-northwest-1   --debug --log-directory ./   testtmppp /mnt/

What happened?

Note that this problem only occurs when I need to ASSUME, if I use ak/sk directly I don't have this problem!
This is my credentials file.

[default]
aws_access_key_id = ak
aws_secret_access_key = sk
[bwmtest2]
output=json
region=cn-northwest-1
role_arn=arn:aws-cn:iam::714736990101:role/s3fullrole
source_profile=default

I think this is a bug in China, the resources in China are segregated from global, when I refer to a source_profile mount-s3 in credentials it sends the request to global's sts endpoint instead of China's endpoint, thus causing the authentication to fail.

The steps to reproduce this are very simple in the China region, through the source_profile assume role can appear this

I saw through a packet grab that the request from sts was sent to sts.amazonaws.com instead of the China endpoint I uploaded the attachment.
sts.zip

Relevant log output

2024-04-24T13:40:16.462962Z  INFO mountpoint_s3::cli: mount-s3 1.6.0
2024-04-24T13:40:16.463094Z DEBUG mountpoint_s3::cli: CliArgs { bucket_name: "testtmppp", mount_point: "/mnt/", prefix: None, region: Some("cn-northwest-1"), endpoint_url: None, force_path_style: false, transfer_acceleration: false, dual_stack: false, requester_pays: false, bucket_type: None, no_sign_request: false, profile: None, read_only: false, storage_class: None, allow_delete: false, allow_overwrite: false, auto_unmount: false, allow_root: false, allow_other: false, maximum_throughput_gbps: None, max_threads: 16, part_size: 8388608, uid: None, gid: None, dir_mode: None, file_mode: None, foreground: false, expected_bucket_owner: None, log_directory: Some("./"), log_metrics: false, debug: true, debug_crt: false, no_log: false, cache: None, metadata_ttl: None, max_cache_size: None, user_agent_prefix: None, sse: None, sse_kms_key_id: None }
2024-04-24T13:40:18.465168Z ERROR awscrt::socket: id=0x7f04040023e0 fd=11: timed out, shutting down.
2024-04-24T13:40:18.465287Z ERROR awscrt::channel-bootstrap: id=0x55daf004b580: Connection failed with error_code 1048.
2024-04-24T13:40:18.465300Z ERROR awscrt::http-connection: static: Client connection failed with error 1048 (AWS_IO_SOCKET_TIMEOUT).
2024-04-24T13:40:18.465315Z  WARN awscrt::connection-manager: id=0x55daeffdf900: Failed to obtain new connection from http layer, error 1048(socket operation timed out.)
2024-04-24T13:40:18.465334Z  WARN awscrt::connection-manager: id=0x55daeffdf900: Failed to complete connection acquisition with error_code 1048(socket operation timed out.)
2024-04-24T13:40:18.465355Z  WARN awscrt::Unknown: id=0x55daeffd4650: IMDS Client failed to acquire a connection, error code 1048(socket operation timed out.)
2024-04-24T13:40:18.465378Z ERROR awscrt::Unknown: (id=0x55daeffd4650) IMDS client failed to update the token from IMDS.
2024-04-24T13:40:20.565746Z ERROR awscrt::socket: id=0x7f0404002a30 fd=11: timed out, shutting down.
2024-04-24T13:40:20.565827Z ERROR awscrt::channel-bootstrap: id=0x55daf004b580: Connection failed with error_code 1048.
2024-04-24T13:40:20.565836Z ERROR awscrt::http-connection: static: Client connection failed with error 1048 (AWS_IO_SOCKET_TIMEOUT).
2024-04-24T13:40:20.565842Z  WARN awscrt::connection-manager: id=0x55daeffdf900: Failed to obtain new connection from http layer, error 1048(socket operation timed out.)
2024-04-24T13:40:20.565852Z  WARN awscrt::connection-manager: id=0x55daeffdf900: Failed to complete connection acquisition with error_code 1048(socket operation timed out.)
2024-04-24T13:40:20.565857Z  WARN awscrt::Unknown: id=0x55daeffd4650: IMDS Client failed to acquire a connection, error code 1048(socket operation timed out.)
2024-04-24T13:40:20.565940Z DEBUG mountpoint_s3_client::instance_info: EC2 instance info not retrieved: ImdsQueryFailed(CrtError(Error(1048, "aws-c-io: AWS_IO_SOCKET_TIMEOUT, socket operation timed out.")))
2024-04-24T13:40:20.565989Z  WARN mountpoint_s3::cli: failed to detect network throughput. Using 10 gbps as throughput. Use --maximum-throughput-gbps CLI flag to configure a target throughput appropriate for the instance. Detection failed due to: failed to get instance type: IMDS query failed: Unknown CRT error
2024-04-24T13:40:20.566013Z  INFO mountpoint_s3::cli: target network throughput 10 Gbps
2024-04-24T13:40:20.566164Z TRACE mountpoint_s3_client::s3_crt_client: constructing client with auth config Default
2024-04-24T13:40:20.574065Z ERROR awscrt::AuthCredentialsProvider: Failed to resolve either region, role arn or token file path during sts web identity provider initialization.
2024-04-24T13:40:20.574112Z ERROR awscrt::AuthCredentialsProvider: Failed to resolve config profile during process credentials provider initialization.
2024-04-24T13:40:20.584323Z TRACE mountpoint_s3_client::s3_crt_client: resolved endpoint uri=Uri("https://testtmppp.s3.cn-northwest-1.amazonaws.com.cn")
2024-04-24T13:40:20.584392Z TRACE mountpoint_s3_client::s3_crt_client: resolved auth scheme auth_scheme=AuthScheme { disable_double_encoding: true, scheme_name: SigV4, signing_name: "s3", signing_region: "cn-northwest-1" }
2024-04-24T13:40:20.584645Z DEBUG list_objects{id=0 bucket="testtmppp" continued=false delimiter="" max_keys="0" prefix=""}: mountpoint_s3_client::s3_crt_client::list_objects: new request
2024-04-24T13:40:21.463395Z  INFO mountpoint_s3::metrics: process.memory_usage: 10850304
2024-04-24T13:40:21.463421Z  INFO mountpoint_s3::metrics: s3.client.buffer_pool.mem_limit: 2013265920
2024-04-24T13:40:21.463425Z  INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_allocated: 0
2024-04-24T13:40:21.463428Z  INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_cutoff: 33554432
2024-04-24T13:40:21.463431Z  INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_num_blocks: 0
2024-04-24T13:40:21.463434Z  INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_reserved: 0
2024-04-24T13:40:21.463436Z  INFO mountpoint_s3::metrics: s3.client.buffer_pool.primary_used: 0
2024-04-24T13:40:21.463439Z  INFO mountpoint_s3::metrics: s3.client.buffer_pool.secondary_reserved: 0
2024-04-24T13:40:21.463442Z  INFO mountpoint_s3::metrics: s3.client.buffer_pool.secondary_used: 0
2024-04-24T13:40:21.463445Z  INFO mountpoint_s3::metrics: s3.client.num_auto_default_network_io: 0
2024-04-24T13:40:21.463447Z  INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_copy_network_io: 0
2024-04-24T13:40:21.463450Z  INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_get_network_io: 0
2024-04-24T13:40:21.463453Z  INFO mountpoint_s3::metrics: s3.client.num_auto_ranged_put_network_io: 0
2024-04-24T13:40:21.463456Z  INFO mountpoint_s3::metrics: s3.client.num_requests_being_prepared: 0
2024-04-24T13:40:21.463458Z  INFO mountpoint_s3::metrics: s3.client.num_requests_being_processed: 0
2024-04-24T13:40:21.463461Z  INFO mountpoint_s3::metrics: s3.client.num_requests_stream_queued_waiting: 0
2024-04-24T13:40:21.463464Z  INFO mountpoint_s3::metrics: s3.client.num_requests_streaming_response: 0
2024-04-24T13:40:21.463466Z  INFO mountpoint_s3::metrics: s3.client.num_total_network_io: 0
2024-04-24T13:40:21.463469Z  INFO mountpoint_s3::metrics: s3.client.request_queue_size: 0
2024-04-24T13:40:21.463472Z  INFO mountpoint_s3::metrics: system.available_memory: 1708064768
2024-04-24T13:40:23.358447Z ERROR awscrt::socket: id=0x7f03fc02aad0 fd=13: timed out, shutting down.
2024-04-24T13:40:23.358524Z ERROR awscrt::channel-bootstrap: id=0x55daf004c340: Connection failed with error_code 1048.
2024-04-24T13:40:23.358533Z ERROR awscrt::http-connection: static: Client connection failed with error 1048 (AWS_IO_SOCKET_TIMEOUT).
2024-04-24T13:40:23.358539Z  WARN awscrt::connection-manager: id=0x55daf011bfd0: Failed to obtain new connection from http layer, error 1048(socket operation timed out.)
2024-04-24T13:40:23.358564Z  WARN awscrt::connection-manager: id=0x55daf011bfd0: Failed to complete connection acquisition with error_code 1048(socket operation timed out.)
2024-04-24T13:40:23.358575Z  WARN awscrt::Unknown: id=0x55daf012a5a0: IMDS Client failed to acquire a connection, error code 1048(socket operation timed out.)
2024-04-24T13:40:23.358598Z ERROR awscrt::Unknown: (id=0x55daf012a5a0) IMDS client failed to update the token from IMDS.
2024-04-24T13:40:25.458932Z ERROR awscrt::socket: id=0x7f03fc02aad0 fd=13: timed out, shutting down.
2024-04-24T13:40:25.459014Z ERROR awscrt::channel-bootstrap: id=0x55daf004c340: Connection failed with error_code 1048.
2024-04-24T13:40:25.460350Z ERROR awscrt::http-connection: static: Client connection failed with error 1048 (AWS_IO_SOCKET_TIMEOUT).
2024-04-24T13:40:25.460361Z  WARN awscrt::connection-manager: id=0x55daf011bfd0: Failed to obtain new connection from http layer, error 1048(socket operation timed out.)
2024-04-24T13:40:25.460373Z  WARN awscrt::connection-manager: id=0x55daf011bfd0: Failed to complete connection acquisition with error_code 1048(socket operation timed out.)
2024-04-24T13:40:25.460377Z  WARN awscrt::Unknown: id=0x55daf012a5a0: IMDS Client failed to acquire a connection, error code 1048(socket operation timed out.)
2024-04-24T13:40:25.460402Z ERROR awscrt::AuthCredentialsProvider: (id=0x55daf004c190) Default chain credentials provider failed to source credentials with error 6153(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_IMDS_SOURCE_FAILURE, Valid credentials could not be sourced by the IMDS provider)
2024-04-24T13:40:25.460411Z ERROR awscrt::AuthSigning: (id=0x7f0408000d80) Credentials Provider failed to source credentials with error 6153(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_IMDS_SOURCE_FAILURE, Valid credentials could not be sourced by the IMDS provider)
2024-04-24T13:40:25.460417Z ERROR awscrt::S3MetaRequest: id=0x55daf02e30f0 Meta request could not sign HTTP request due to error code 6146 (Attempt to sign an http request without credentials)
2024-04-24T13:40:25.460422Z ERROR awscrt::S3MetaRequest: id=0x55daf02e30f0 Could not prepare request 0x7f03fc000a80 due to error 6146 (Attempt to sign an http request without credentials).
2024-04-24T13:40:25.460517Z DEBUG list_objects{id=0 bucket="testtmppp" continued=false delimiter="" max_keys="0" prefix=""}: mountpoint_s3_client::s3_crt_client: S3 request failed request_type=Default crt_error=None range=None duration=4.875575188s ttfb=None request_id=<unknown>
@

[csy97]

To add to this, when I enable IMDS authentication I seem to be able to find the correct endpoint , but when I disable IMDS versions v1 and v2 the failure occurs!

[dannycjones]

Thanks for the bug report, @csy97. Yes, it looks like a bug. Specifically, it looks like assuming a role is not working outside of the global regions - i.e. China, US Gov Cloud.

We'll continue looking into this - I'll provide an update soon.

[csy97]

hello ,I noticed that this aws-c-auth seems to have fixed the problem and the project has updated the branch information for crt dependency, I would like to enquire when the new binary/rpm version of mount-s3 will be released!

[passaro]

Hi @csy97, we have integrated the fix and we are adding integration tests for it. We plan to include it in the next release, but we cannot share a date for it, yet.