Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s5cmd ls --region argument, and implied lists respect the cp argument #702

Open
jordan-jack-schneider opened this issue Feb 14, 2024 · 0 comments

Comments

@jordan-jack-schneider
Copy link

jordan-jack-schneider commented Feb 14, 2024

If you're doing a copy from one region to another with --no-clobber there are a lot of failure modes with region resolution. The resolution order of ls is

Region detection
While executing the commands, s5cmd detects the region according to the following order of priority:
--source-region or --destination-region flags of cp command.
AWS_REGION environment variable.
Region section of AWS profile.
Auto detection from bucket region (via HeadBucket API call).
us-east-1 as default region.

So this is actually impossible if the region section of your aws profile is set (which you might want as a default). Without the config, it relied on HeadBucket calls, which can sometimes cause performance problems when making a lot of copies with s5cmd run.

Ideally, s5cmd would take in a --region argument, and s5cmd cp $SRC $DST --destination-region $REGION --no-clobber would pass the --destination-region flag to the internal list needed by --no-clobber.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant