FileStreamResult resulting in significantly more I/O than client is requesting

[coonmoo]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I have troubles understanding why returning a FileStreamResult from my API is causing a large amount of bytes being read from the underlying FileStream without the client actually starting to read from the HttpResponseMessagestream.

I also observed different results running on Kestrel vs IIS Express.

Running on Kestrel the output is:
Total bytes read: 524288

Running on IIS Express the whole file is read:
Total bytes read: 2498125

Based on the client code below which does not actually read from the stream I would not expect that the server reads the whole file and writes it to the response body.

Is it possible to prevent this scenario and actually only write bytes to the response body which the client requested to read?
(calling stream.ReadAsync(...))

Response buffering is not enabled and the default response buffer settings for Kestrel are lower than what I am seeing here.

The problem is that this is causing a lot of unnecessary disk I/O if clients are not going to read the stream or are aborting the request.

Client

HttpClient httpClient = new();
HttpResponseMessage response = await httpClient.GetAsync("https://localhost:44367/files/test", HttpCompletionOption.ResponseHeadersRead);
await using Stream stream = await response.Content.ReadAsStreamAsync();
await Task.Delay(1000);

Server

[ApiController]
[Route("[controller]")]
public class FilesController : ControllerBase
{
    [HttpGet("test")]
    public async Task<IActionResult> Get()
    {
        FileStream fileStream = new LoggingFileStream("ForBiggerBlazes.mp4", FileMode.Open);
        FileStreamResult fileStreamResult = new(fileStream, "video/mp4");
        return fileStreamResult;
    }
}

public class LoggingFileStream : FileStream
{
    private int totalReadCount;
    public override async Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
    {
        int read = await base.ReadAsync(buffer, offset, count, cancellationToken);
        totalReadCount += read;
        Console.WriteLine("Total bytes read: " + totalReadCount);
        return read;
    }
}

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();

WebApplication app = builder.Build();
app.UseHttpsRedirection();

app.UseRouting();
app.UseAuthentication();
app.UseAuthorization();

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
});

app.Run();

Expected Behavior

No response

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

8.0.204

Anything else?

No response

[Tratcher]

This has to do with built in write buffers in the stack. We don't know when the client is going to call Read, we only know they want us to send the file, so as much is sent as possible until all the buffers between the client and the server fill up.

• HttpClient buffers
• Client TCP buffers
• Proxy buffers
• Server TCP buffers
• Server response buffers
• etc.

[coonmoo]

Thanks for the quick reply, that makes sense.

I see no effect changing the Kestrel MaxResponseBufferSize to a smaller value.

Are you aware of any other response buffer settings which can be configured on the web server level (IIS or Kestrel)?

We stream a lot of video files and see our disk capacity easily being saturared when users skip around in videos.

Causing for example 10MB being read from the FileStream but the request being immediately canceled.
The user does not need the data but we read it anyway from the disk due to response buffering.

[Tratcher]

How much control do you have over the client? Can you have them send range requests?

[coonmoo]

Most clients are browser based where we do not have much control.

Embedding the video in the browser like that:
<video src="https://localhost:44367/files/test" controls></video>

results in the browser issuing an initial request on page load.

In the above mentioned request the browser only reads about 130KB of data (for rendering the initial video thumbnail).
But on the server side I still see the whole 2.4MB video file being read from disk and being entirely written to the response body.

Often browser clients leave the page after a short time, having the effect about using 20x more disk I/O than necessary.

Note that this is getting worse with larger video files, I sometimes see 10MB written to the response body where only about 150KB of data was actually read by the client.

[Tratcher]

Try this:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.fileresult.enablerangeprocessing?view=aspnetcore-8.0#microsoft-aspnetcore-mvc-fileresult-enablerangeprocessing

[coonmoo]

Thanks, I tried it and verified that this is giving me the same results.

The browser client initially sends this range header request:

Range: bytes=0-

Getting the following response from the API:

Content-Length: 2498125
Content-Range: bytes 0-2498124/2498125

The backend still reads 2.4MB from the disk and writes the file completely to the response body while only 130KB are actually arriving/being read at the browser client.

[Tratcher]

Here's an interesting example where you can limit the range:
https://stackoverflow.com/questions/48156306/html-5-video-tag-range-header

This would require some manual limiting on the filestream, or you could modify the request range header before generating the response.