ERROR awscrt::AuthCredentialsProvider: static: invalid credential_source property: EcsContainer

[at16cib]

Mountpoint for Amazon S3 version

mount-s3 1.6.0

AWS Region

eu-central-1

Describe the running environment

Batch job, ECS task using a Docker image Ubuntu 22.04.
The credentials to access the s3 bucket:

• ECS task has a job/task role that is allowed to assume a given role that has all relevant access to S3
• The assumed role is passed as an argument to the batch job, and during runtime an /root/.aws/config file is created with the following content:

[profile mountS3Profile]
role_arn=arn:aws:iam::XXXXXXXX:role/test/dev-prc-774-ppaas-s3-test-data-access
credential_source=EcsContainer

• The profile mountS3Profile is used to setup mount-s3

Mountpoint options

mount-s3 --debug-crt --debug XXXXXXXX-storage --prefix i-06d2778a53d93ee09/ --read-only --profile mountS3Profile --region eu-central-1 /data/input --log-directory /tmp/logs

What happened?

Using the profile with credential_source=EcsContainer is resulting in the following error:

Error: Failed to create S3 client
	
	Caused by:
	0: invalid AWS credentials
	1: CRT error 34: aws-c-common: AWS_ERROR_INVALID_ARGUMENT, An invalid argument was passed to a function.
	Error: Failed to create mount process

The logs show

ERROR awscrt::AuthCredentialsProvider: static: invalid credential_source property: EcsContainer

• I can confirm that the role setup in general works. And it is possible to list files with the profile mountS3Profile. By running the following command in the container:

aws s3api list-objects-v2 --bucket XXXXXXXX-storage --prefix i-06d2778a53d93ee09/ --profile mountS3Profile

• For testing, I replaced credential_source=EcsContainer with credential_source=Ec2InstanceMetadata. With this the error was only that is not able to fetch credentials. Which is of course correct, but at least it accepted the credential_source type.

2024-04-22T18:10:46.038+02:00	2024-04-22T16:10:46.034996Z ERROR awscrt::AuthSigning: (id=0x7fea20000bf0) Credentials Provider failed to source credentials with error 6155(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_HTTP_STATUS_FAILURE, Unsuccessful status code returned from credentials-fetching http request)
2024-04-22T18:10:46.038+02:00	2024-04-22T16:10:46.035006Z ERROR awscrt::S3MetaRequest: id=0x55a47220d7d0 Meta request could not sign HTTP request due to error code 6146 (Attempt to sign an http request without credentials)
2024-04-22T18:10:46.038+02:00	2024-04-22T16:10:46.035012Z ERROR awscrt::S3MetaRequest: id=0x55a47220d7d0 Could not prepare request 0x7fea34002110 due to error 6146 (Attempt to sign an http request without credentials).

Relevant log output

2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.437353Z DEBUG awscrt::AWSProfile: Creating profile collection from file at "/root/.aws/config"
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.437372Z DEBUG awscrt::task-scheduler: id=0x7f0fc0000d20: Running epoll_event_loop_unsubscribe_cleanup task with <Canceled> status
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.437375Z DEBUG awscrt::AWSProfile: Creating profile collection from file at "/root/.aws/credentials"
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.437396Z DEBUG awscrt::AWSProfile: property "role_arn" has value "" replaced during merge
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.437399Z DEBUG awscrt::AWSProfile: property "credential_source" has value "" replaced during merge
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.437402Z INFO awscrt::AuthCredentialsProvider: static: profile mountS3Profile has role_arn property is set to arn:aws:iam::XXXXXXXX:role/test/dev-prc-774-ppaas-s3-test-data-access, attempting to create an STS credentials provider.
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.437405Z DEBUG awscrt::AuthCredentialsProvider: static: computed session_name as aws-common-runtime-profile-config-19
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.437407Z INFO awscrt::AuthCredentialsProvider: TLS context not provided, initializing a new one for querying STS
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.441308Z INFO awscrt::AuthCredentialsProvider: static: credential_source property set to EcsContainer
2024-04-23T13:46:39.444+02:00	2024-04-23T11:46:39.441317Z ERROR awscrt::AuthCredentialsProvider: static: invalid credential_source property: EcsContainer

[at16cib]

It seems that the repository that is used for authentication is not supporting the credential_source=EcsContainer and therefore running into this error here
From the docs it is very misleading, and it sounded like ECS is supported
Should I open an feature request in https://github.com/awslabs/aws-c-auth to support ECS?

[ahmarsuhail]

Thanks for opening the issue @at16cib and for the additional info, we're looking into it and will check with the team that works on aws-c-auth.

[dannycjones]

I just wanted to share an update here. It is a bug specifically when using the credential_source field with EcsContainer - this should be supported. We're working with the owners of aws-c-auth on a fix. We'll let you know when we have more information.