storage/s3: endpoint_url from AWS config file

[jsproede]

Hi 👋

We've used this tool a lot since we've discovered s5cmd for our S3 buckets. As we're not using AWS, we need to set the S3_ENDPOINT_URL each time we open a new Terminal.

As the AWS-CLI officially supports endpoint_url in the AWS config file for a profile, I thought it would be great if s5cmd would support endpoint_url in AWS config profiles as well. Unfortunately the AWS-SDK does not support endpoint_url, therefore we're using the go-ini module to read the AWS config file.

Therefore we extended the newSession function to use either the configured AWS config file (via AWS_CONFIG_FILE env-variable) or the default AWS config file ($HOME/.aws/config) and lookup/use the endpoint_url for the specified profile (only if configured and not specified by $S3_ENDPOINT_URL or the --endpoint-url-flag).

Additionally we created a test which covers the config file including a endpoint_url and a profile which has not set the endpoint_url.

What do you think about this?

If there is anything missing or we need to change something in this pull request, feel free to let us know 🙂

[jsproede]

I forgot, that AWS config profiles must be prefixed with profile. After pushing my changes the "ci / test (windows/go-1.17.x) (pull_request)" is now failing 🤔 I'll mark my PR as ready again. Can someone look into it (maybe restart the failing job)?

[tilimil]

What about the "region" and "ca_bundle" in the config file as well? It appears these aren't being consumed by s5cmd either. Is there a more complete way to bring in all the setting in the config file and pass those to the SDK?

[yakimant]

Thanks for the PR, looking forward for it being merged.
Thanks to the reviewers in advance!

[jsproede]

What about the "region" and "ca_bundle" in the config file as well? It appears these aren't being consumed by s5cmd either. Is there a more complete way to bring in all the setting in the config file and pass those to the SDK?

@tilimil Great suggestion. We've mainly focused on the endpoint-URL. I'm not sure, if the SDK could properly handle region and ca_bundle itself and therefore go-ini would not be necessary for these two configuration properties. I need to take a deeper look into the code of s5cmd to properly answer that. Maybe one of the maintainers can help us with that? :)

[MrWaip]

Hi! It's nice feature to have

[k0ste]

Needs rebase

[jsproede]

@k0ste Since the PR has now been open since July 2023, I wanted to save myself the time of working on a regular rebase as long as there is no feedback, because I don't even know yet whether the PR would be accepted at all.