Download of large files slow in Blink engine browsers (Chrome, Edge)

[stefanfis]

In my app I'm dealing with large data sets. I learned that with StreamSaver.js downloading is rather slow on Chrome compared to Safari. I've done some tests on macOS and Windows. The downloaded payload consists of 1 million lines, each 100 bytes long, resulting in a 100MB file.

macOS Big Sur (MacBook Air M1)
Chrome: 76s
Safari: 17s

Windows 10 (Ryzen 3600)
Chrome: 90s
Edge: 131s

On macOS I did also tests with 10MB files, where Chrome took about 8s and Safari downloaded instantly. All those download times were taken manually.

Trying to find out what takes that long, I measured the time between createWriteStream and stream closing as can be seen in this code:

    let startTime = new Date().getTime()
    let fileStream = streamSaver.createWriteStream('test.txt')
    let writer = fileStream.getWriter()
    for( let i = 0; i < payload.length; ++i ) {
        let output_row = payload[i]+"\n"
        writer.write(new TextEncoder().encode(output_row))
    };
    writer.close()
    let endTime = new Date().getTime()

The differences were modest, Safari needed about 1 second, Chrome about 2.

I even wrote my own simple implementation of a service worker downloader where the download isn't streamed. Chrome now downloads 100MB within less than a second.

Do you have any ideas why downloading takes so long, especially on Chrome?

[jimmywarting]

Comparing safari to chrome is like comparing canary against IE9

Safari don't support downloading files generated by service worker so what it dose instead is to accumulate all chunks into one Blob and download everything at once. (without service worker)

So yea, safari is a bit faster but you are not able to download small bits of pieces periodically. Safari needs to buffer up everything into a Blob and download it afterwards

[jimmywarting]

I think it would be great if you could mention that somewhere in the documentation

Yea it isn't mention anywhere, Would be a good idea.

What causes the slow download times at all

One reason why it could be slower is that it copy everything over postMessage instead of using transferable

---

it's also unlikely that you have all the data at once

    for( let i = 0; i < payload.length; ++i ) {
        let output_row = payload[i]+"\n"
        writer.write(new TextEncoder().encode(output_row))
    }

if you do then you could basically do: new Blob(payload) + <a href="objecturl" download="name"> instead

[stefanfis]

All data is stored locally in an array. It's basically some kind of browser-based file editor, where the use case includes some really huge files, even into the low single-digit GB range. I'm doing a lot of experiments how far we can go nowadays.

Here's how I created the data for the test case I mentioned in my initial post. The data is already created when I started measuring download times. This resembles my real world use case very closely.

let payload = create_payload(num,99)
// Write to file stream as shown in my initial post
function create_payload(num, len) {
    let payload = []
    for( let i=0; i<num; ++i ) {
        let str = Array(len).fill().map(() => String.fromCharCode(parseInt(Math.random()*26) + 65)).join("")
        payload.push( str )
    }
    return payload
}

if you do then you could basically do: new Blob(payload) + <a href="objecturl" download="name"> instead

OMG, I never thought that I could do it that way. I'm running some experiments about memory consumption for both solutions and reporting them back here.

[jimmywarting]

Depending on where the data comes from then it could be tuff to create large blobs, FileSaver.js had this limitation where it could only create 500 MB blobs

but if the blob is constructed with other blobs backed up by other places then you can create much larger files.

If you create smaller blob chunks then browser have a chance to offload some memory to the disk by transfering the blob to a temporary location on the HDD (since the data can then be read lazy)

• https://docs.google.com/presentation/d/1MOm-8kacXAon1L2tF6VthesNjXgx0fp5AP17L7XDPSM/
• use a byte sequence instead of a buffer node-fetch/fetch-blob#40

[jimmywarting]

btw, a better more realistic way would be to do stream the data and not have it ready all at once

    for( let i = 0; i < 99; ++i ) {
        let output_row = i+"\n"
        const data = crypto.getRandomValues(new Uint8Array(99))
        await writer.write(data)
    }

like u are doing in create_payload

[stefanfis]

Thanks for those links. According to https://source.chromium.org/chromium/chromium/src/+/master:storage/browser/blob/README.md, on modern desktop OS the maximum Blob size is 2 GB nowadays.

[jimmywarting]

where dose your data come from? the server?

[stefanfis]

I promised to do some performance tests. To make things more interesting, I used a payload of 500MB, constructed with the function shown above. I logged window.performance.memory.usedJSHeapSize during multiple "breakpoints".

Here's the download time and JS memory consumption for the StreamSaver.js based solution:

            Download time: 370 seconds (~12MBits/s)    
            Start: usedJSHeapSize: 4,488,836
            Payload created: usedJSHeapSize: 983,401,545
            Data written: usedJSHeapSize: 1,811,490,676
            After download: usedJSHeapSize: 590,707,244

This looks rather good from the JS heap perspective. Browser memory consumption (according to the Chrome Task Manager) rose up to 6GB, though, but everything worked flawlessly. Only if it wouldn't take so extremely long.

            Download time: 8 seconds
            Start: usedJSHeapSize: 4,482,826
            Payload created: usedJSHeapSize: 710,483,665
            Blob created: usedJSHeapSize: 1,282,549,874
            After download: usedJSHeapSize: 1,282,793,912
            Some 30s after download: 690,431,570

That's what you expect regarding memory consumption, I guess. Creating a Blob from the Array doubles your heap memory usage, but everything worked and it was fast, really, really fast.

(I also tested your await writer ... idea. It seems it was about 10% faster, but that's still very slow.)

So, I guess I'll end up with a dual approach in my application: For payloads up to several hundred MB, I'll go the very fast Blob download route, while using StreamSaver.js for really huge datasets.