-
-
Notifications
You must be signed in to change notification settings - Fork 407
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RAM ballooning on downloading large zip files [OOM] #293
Comments
If you do have control over the server then i do recommend that you should try to save the file the way server usually tells browser to save files by responding with a all the things StreamSaver actually dose is mimic how server saves files (the good old fashion way) for ref. a download can not be initiated with a http call to the server with either fetch or XMLHttpRequest, it has to be a navigation. I notice that you are sending a payload and some authentication headers. so a normal GET request would not work. but you are also sending a authentication request header that makes it a bit more trickier cuz you can't send a request header with forms. so my recommendation is that you create a other solution for submitting a form (with the payload that you need to send to the server) with out necessary request headers, either by using cookies or creating some one time / expiring url or that you include some kind of token/api key inside of the form or in the url so it dose not have to be sent inside of a request header. then browser will be able to download files without the need of using streamsaver. |
Could the following Chromium issue be a possible root cause? It appears that Chromium first writes files to a sandboxed filesystem in order to first perform a security scan on it, before fully flushing the file into the filesystem. Would be great to brainstorm a possible work around for this as I'm transforming/decrypting a file as it's downloaded, and then piping it to the filesystem with Stream Saver. Using I'm experiencing a similar issue to OP where the RAM usage is ballooning under a process called https://bugs.chromium.org/p/chromium/issues/detail?id=1168715 edit: I see you're already involved in that thread 🙏 |
I wanted to come back and leave an update on my progress. I've solved my high RAM usage. Deep in my code there was a closure that was causing memory to balloon. Something similar to this: function getChunk () {
function fetchChunk() {
}
} I also changed my pump() logic from const pump = async (): Promise<void> => {
let res = await reader.read();
if(res.done) {
return await writer.close();
}
else {
await writer.write(res.value);
return pump();
}
} instead of: const pump = (): Promise<void> =>
reader.read().then((res) => (res.done ? writer.close() : writer.write(res.value).then(pump)));
pump(); |
beacuse http cannot paused by code. so my solved is using http range download and using streamSave.js to save.
by the way, use is best pratice |
Heyo,
Been tinkering with different configs/imports/ways to call/etc and going through older issues related to my problem for over a day now but can't seem to find how my implementation is messed up. On a 10GB zip download, the script will error out after about 2GB. I'm seeing two peculiarities with how I implemented this library
Uncaught TypeError: [StreamSaver] You didn't send a messageChannel at onMessage (mitm.html?version=2.0.0:66:11
For the post message error, I get about 7 of those when the download first starts and then another at a regular interval after that which makes sense with what you're doing under the hood. Just not sure why it's erroring, and wondering if it's how I am calling streamsaver. Here's the code that I've got so far:
Environment: NPM 8.17; streamsaver - 2.0,6; react- 17.0.2; working with http not https
Edit 1: Backend info -
The backend is Django hosted on a separate computer that I do have control over, but can't open directly up for the end user to download from. The response from Django is a StreamingHttpResponse.
Definitely a me problem and not a you problem. Just seeing if what I coded is wrong in some way?
The text was updated successfully, but these errors were encountered: