Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Case: fetching many files from s3 and generating zip #291

Open
gabriels1234 opened this issue Sep 19, 2022 · 1 comment
Open

Use Case: fetching many files from s3 and generating zip #291

gabriels1234 opened this issue Sep 19, 2022 · 1 comment

Comments

@gabriels1234
Copy link

I'm building a solution that requires broad compatibility.
Got a few questions.
@jimmywarting
I was able to test downloading many files and getting them as a zip. This saves a lot of processing power/server costs, since there's nothing to do with the files, just give to the user as a zip.
I was also able to plug this into a react app.

I have these questions, base on comments seen from @jimmywarting :

  1. what improvements are there in native-file-system-adapter that pertain to fetching remote files.
  2. Is it possible to use code from that repo to make StreamSaver not have the need for the mitm html without losing in features/compatibility? (namely, a cleaner solution)
  3. In what cases the files fetched will be sitting on RAM? how to detect/know?

I know that at some point this will not the way of getting to this solution, but I must aim for maximum compatibility

Thanks!

@gabriels1234
Copy link
Author

UPDATE: I was able to make the download and zip of many files (say: 3000 100kb files) in parallel not using await for each fetch. initially, it caused the ERR::INSUFFICIENT..., and then I made it so each promise returned by the fetch was put in an array of promises. So, for every 1 MB of data downloaded, I awaited (await Promise.all(allPromises)). It made the trick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant