Write to ZipFile backed by S3 stream

[hiselitelordship]

I have tried to merge two examples from the documentation together:

• Open a S3 object for writing as a file stream.
• Open a ZipFile and pass it the file stream (which is streaming data to S3).

My use case is essentially to take hundreds of files from a S3 bucket and zip them all up. I am doing this in Lambda (not in VPC with EFS) and therefore I don't have unbounded space to just download all the files, zip them locally and upload the resulting ZIP file as there is every chance I will run out of space. Therefore I want to stream hundreds of files down and stream it back out to a single ZIP.

The examples get tantalisingly close to my requirement. Any help appreciated.

from smart_open import open
import zipfile

with open(my_s3_file, 'w') as fout:
    with zipfile.ZipFile(fout, 'w') as zip:
        zip.writestr('hello/en.txt', 'hello world!\n')
        zip.writestr('hello/ru.txt', 'здравствуй, мир!\n')

But I get this:

Traceback (most recent call last):
File "/home/auser/environments/temp/streamzips3.py", line 14, in
zip.writestr('hello/en.txt', 'hello world!\n')
File "/usr/lib/python3.10/zipfile.py", line 1812, in writestr
with self.open(zinfo, mode='w') as dest:
File "/usr/lib/python3.10/zipfile.py", line 1515, in open
return self._open_to_write(zinfo, force_zip64=force_zip64)
File "/usr/lib/python3.10/zipfile.py", line 1610, in _open_to_write
self.fp.write(zinfo.FileHeader(zip64))
TypeError: write() argument must be str, not bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/auser/environments/temp/streamzips3.py", line 13, in
with zipfile.ZipFile(fout, 'w') as zip:
File "/usr/lib/python3.10/zipfile.py", line 1310, in exit
self.close()
File "/usr/lib/python3.10/zipfile.py", line 1835, in close
self._write_end_record()
File "/usr/lib/python3.10/zipfile.py", line 1929, in _write_end_record
self.fp.write(endrec)
TypeError: write() argument must be str, not bytes

[piskvorky]

ZIP files are binary files, so don't you have to pass it a binary fout handle? Not w = text.

[hiselitelordship]

Perfect! Thank you.

This does exactly what I need. Adjusted the "open" call to write binary with "wb" . Testing with large files seems to show a constant memory usage (doesn't increase because it is truly streaming and therefore won't hit a Lambda limit unless I hit a duration limit of course).

from smart_open import open
import zipfile

basepath="/home/auser/some/files/"
files=["a.txt","b.txt"]

tp = {'min_part_size': 5 * 1024**2}
with open(S3URL, 'wb', transport_params=tp) as fout:
	with zipfile.ZipFile(fout, 'w') as zip:
		for afile in files:
			afile = basepath+afile
			zip.write(afile)