Memory overload when downloading large files.

[aplekhanov]

Hello,

I keep challenging your great product and found recently that when I try to download a large file (like ~150 Mb for instance on screenshot below) from the server, the debug navigator shows that it makes a huge impact to device memory. Moreover, it multiplies on number of simultaneous downloads.

Well, is it a bug or architecture pitfall? I mean is it possible to fix it?

Thanks and happy coding!

[aplekhanov]

Just start unsecured server with .serveDirectory on iPad and get file using URLSession from 'http://localhost:9000/big.pdf'. The same picture on macOS. Seems it load the whole file into memory for every downloading session.

[yvbeek]

Hi @aplekhanov, Yes big files are fully loaded in memory. The implementation of the file handler is pretty basic.

Unfortunately to properly send big files to your client you need chunked transfer encoding, I haven't implemented that in the HTTP classes. I'm not really sure how much work it would be to add it to the framework, I haven't explored the topic in too much detail yet.

Perhaps I can have a look at it in the weekend.
Or if you want to show off your coding skills, pull requests are most welcome 😉 😄

[aplekhanov]

Hi @Zyphrax,
Yeah, I found already that part of code in the HTTPFileHandler response.body = try Data(contentsOf: fileURL) . I guess there must be NSFileHandle to get chunked data from file... and some special http header for this case... right? Very tempting 😈

But right now I need to finish A LOT OF other interesting work. Let's say it is not high priority and most important thing is that it could be realized by you, by me or by someone else in future 🍻

[yvbeek]

I've just pushed a commit that adds support for range requests on files. That might help in some cases and for example allows for streaming video on iOS. It contains FileHandle code to read parts of the files.

However this does not resolve the memory issue yet. To fix that we need to build in proper streaming, for example by adding something like a bodyStream variable to HTTPResponse.

I will probably add that later on. The problem is that the socket layer (CocoaAsyncSocket) makes it difficult to implement this. I'll probably have to replace that first.

[aplekhanov]

I will probably add that later on. The problem is that the socket layer (CocoaAsyncSocket) makes it difficult to implement this. I'll probably have to replace that first.

Great to hear it! Have you dig into a new Network framework?

[yvbeek]

Great to hear it! Have you dig into a new Network framework?

Yeah, I'm working on a wrapper for the Apple Network framework and a DispatchIO implementation for iOS 11 and lower versions. The SSL implementation is going to be a nightmare, but I'll try 😅

[JUSTINMKAUFMAN]

@Zyphrax I am interested to hear whether or not you decided to pursue this. Not an urgent feature for me, but would be nice to have 😄.

[yvbeek]

Hi @JUSTINMKAUFMAN. Darwin sockets are quite complex and especially the SecureTransport SSL layer is going to be a challenge. I'm still working on it, but haven't had much time to spend on it.

As soon as I have a unified network layer, I'll dive into proper buffering for large files.

[shalom-aviv]

Hi @JUSTINMKAUFMAN. Darwin sockets are quite complex and especially the SecureTransport SSL layer is going to be a challenge. I'm still working on it, but haven't had much time to spend on it.

As soon as I have a unified network layer, I'll dive into proper buffering for large files.

Hey

Do you have a progress with this issue?

PS
Telegraph is a Great work!
Thank you

[shalom-aviv]

I'm checked Kitura and Vapor for big files transfering
Kitura seems has the same issue

Vapor support streams for files
I'm not sure, but maybe next link help you
https://github.com/vapor/vapor/tree/main/Sources/Vapor

ChannelResponseBodyStream in
https://github.com/vapor/vapor/blob/73d57959a96ceca719558e70c320b2f729d2b25b/Sources/Vapor/HTTP/Server/HTTPServerResponseEncoder.swift

BodyStreamWriter in
https://github.com/vapor/vapor/blob/73d57959a96ceca719558e70c320b2f729d2b25b/Sources/Vapor/Response/Response%2BBody.swift

https://github.com/vapor/vapor/blob/73d57959a96ceca719558e70c320b2f729d2b25b/Sources/Vapor/HTTP/BodyStream.swift

https://github.com/vapor/vapor/blob/73d57959a96ceca719558e70c320b2f729d2b25b/Sources/Vapor/Request/Request%2BBodyStream.swift

[blaineam]

Welp ran into a road block for one of my apps thinking I could do something like this feature request to stream a on device decrypted mp4 file. I can decrypt the file but because it has to wait for the whole file to be processed before responding which makes the app feels super slow. Having the ability to write to the output as it decrypts chunks of a mp4 file would be incredible because the video could start playing while decryption is still occurring. Not sure how I could write to the response piece by piece but that functionality would be phenomenal.