example of uploading uncompressed local file to s3 as gzip

[tooptoop4]

I have local uncompressed file, I would like to save its gzip version on s3. but don't want to store the gzip locally

[mpenkov]

Basic example:

with open('local_file', 'rb') as fin:
   with open('s3://bucket/key.gz', 'wb') as fout:
       fout.write(fin.read())

[piskvorky]

…and to make use of streaming, for large inputs:

with open('local_file', 'rb') as fin:
   with open('s3://bucket/key.gz', 'wb') as fout:
       for line in fin:
           fout.write(line)

smart_open was made for easier streaming of primarily text files, with text transformations and other actions in between. So just-copy-the-file-verbatim isn't its primary use case. I expect the native aws cli + gzip piping will do a better job there.

[tooptoop4]

do u have example with sed/bash transform on each row?