awswrangler copy_objects

[nmakarun]

I am copying objects from one prefix to another using awswrangler and it works perfectly fine, but I need to get rid of the source prefix when I copy to target and keep just the Key name from source prefix.

wr.s3.copy_objects(full_paths_list, source_path, target_path)

Any idea on how can I achieve this, I looked at the documentation (https://aws-sdk-pandas.readthedocs.io/en/stable/stubs/awswrangler.s3.copy_objects.html)

Couldnt find anything tangible.

[malachi-constant]

Is this what you are trying to do?

>>> wr.s3.copy_objects(['s3://hansonlu-test-data-bucket/source/extra-prefix/file1.json'], 's3://hansonlu-test-data-bucket/source/extra-prefix/', 's3://hansonlu-test-data-bucket/target/')
['s3://hansonlu-test-data-bucket/target/file1.json']

[nmakarun]

Thanks for this, I am using 2.15.1 version in glue version 3.0, Can I update this by using --update parameter or do I need to use a .whl file by downloading in S3.

[malachi-constant]

You should be able to update a Glue Job's version of the library by adding the version number in --additional-python-modules.

[nmakarun]

Thanks, That worked but interestingly. It doesnt work, when I pass the source and target s3 paths as variables, but works when I pass the hard coded values
E.g.
source_path = f"{bucket_uri}/{source_prefix}/"
target_path = f"{bucket_uri}/{new_prefix}/"
wr.s3.copy_objects(full_paths_list, source_path, target_path)
Above scenario doesnt work.

E.g.
wr.s3.copy_objects(full_paths_list, 's3://bucket-name/source/', 's3://bucket-name/target/')

Above case works, not sure if this is the expected behaviour.

[nmakarun]

Found the issue, it works when I use single quotes with format method but not with f string formatting method

source = 's3://{}/source'.format('bucketname')
target = 's3://{}/target'.format('bucketname')
wr.s3.copy_objects(full_paths_list, source , target)

this works.

[malachi-constant]

Hmm.. f-strings should work the same way. Here's a working example.

>>> import awswrangler as wr
>>> bucket = "hansonlu-test-data-bucket"
>>> source_path = f"s3://{bucket}/source/extra-prefix/"
>>> target_path = f"s3://{bucket}/target/"
>>> paths_list = [f"{source_path}file1.json"]
>>> wr.s3.copy_objects(paths_list, source_path, target_path)
['s3://hansonlu-test-data-bucket/target/file1.json']