dotnet-gcdump seem not reflect the memory usage of my dump

[julienGrd]

hello guys, i try to understand whats seem to be a memory leak on my .net 8.0 blazor server app.

The context : at least on one client, the memory usage increase during the days until he reached the maximum
the app is deployed on windows server 2022 with IIS

To analyse that I made a dump with gc-dump when the memory of the IIS process was at 5.9 GigaBytes. Only one website is deployed so there was only one IIS process
at this time 58 client was connected

When i analyse this dump with visual studio, i dont find my 5.9 GigaBytes. actually im not even sure how to interpret the results.

Can we assume the number circle in blue above should be the total memory used ?

When I ask to show the "dead objects", It say me there is around 700 MegaBytes used, which is a lot but still not explain the 5.9 GigaBytes.

You can find the complete dump here https://drive.google.com/file/d/1Joa2EEwAKqkdC4eShFVz0hmOahOG0ho9/view?usp=sharing , do you find the 5.9 GB memory used in this dump ?

Thanks for your help !

[noahfalk]

When you observed the 5.9 GB, where did you read that number from? From Task Manager perhaps?

Although I won't know for sure until we clarify where the 5.9GB came from, I suspect 5.9GB was measuring the total committed virtual memory used by your process whereas the 14MB is refering to memory used by live managed objects. Memory used by live managed objects, or even the GC heap in its entirety is still only one part of a process' total memory usage. It probably breaks down something like this:

                                All Commited Virtual Memory (5.9GB)
                               /                                \
               Managed heap committed VM                         All other commited VM (aka 'native memory')
              /                 \
      Memory for objects        Caches, fragmentation and bookkeeping
      /               \
 Live objects (14MB)  Dead objects (700MB)

Likely a large portion of your 5.9GB is contained in the "All other committed VM" category. This includes things like allocations from native allocators (calls to malloc() in C or new in C++), calls to VirtualAlloc() that weren't made by the .NET GC, and memory allocated by the OS for loading modules or storing thread stacks.

If you run dotnet-counters one of the default counters you will see is gc-committed which corresponds to that "Managed heap committed VM" node of the tree. Assuming that number is only a small fraction of the 5.9GB that means most of your memory usage is not coming from the GC heap and is instead coming from the "All other committed VM" portion of memory usage.

Although not the only possibility, a common way this happens is when a modestly sized managed object is keeping a large block of native memory alive. For example imagine we have:

class Bitmap : IDisposable
{
    IntPtr _imageDataPointer;
    ...
}

This managed bitmap type might only be ~50 bytes, but it is holding a pointer to some memory that was allocated with a native heap allocator like malloc() that is 1MB in size. If your GC heap had 1000 of these objects on it then it still might be very small because it only takes 50KB of managed heap memory to store these objects. But the objects themselves are holding native memory pointers that total 1GB in size. Because IntPtr is an opaque pointer to native memory .NET/GC has no visibility into what is there. You could try to spot this type of leak by looking for .NET types that implement IDisposable which are growing in number over time. You could also try working with memory analysis tools that directly understand native heap allocations such as:
https://bugslasher.net/2011/08/09/how-to-troubleshot-native-memory-leaks-on-windows-gflags-and-umdh/
https://learn.microsoft.com/en-us/sysinternals/downloads/vmmap

Tools like that might give you more direct visibility into what is using the bulk of the 5.9GB in your process. Hope that helps!

[julienGrd]

thanks for this very detailed response !
i confirm you the 5.9 Gb come from the task manager, i didnt run dotnet-counters because its a production server and i didnt want install the sdk for now, but maybe when i will continue my tests i will do it.

I dont have so much time for now to investigate more deeply, please dont close this issue for now, i will come back with more information when i will have the time to investigate all the solutions you present in your response.

other question : how i can interpret these "dead objects" : is it objects not properly dispose for example ?

thanks again !

[noahfalk]

i didnt run dotnet-counters because its a production server and i didnt want install the sdk for now

You don't need the sdk in order to get these tools btw. You can download the executables directly from http links. For example https://aka.ms/dotnet-counters/win-x64. Links for other architectures and OSes are located here: https://learn.microsoft.com/dotnet/core/diagnostics/dotnet-counters#install

how i can interpret these "dead objects" : is it objects not properly dispose for example ?

"Dead objects" refers to objects that are no longer reachable via references from managed code. For example:

void Main(string[] args)
{
    object o = new object(); 
    // the object is live right now because it can be accessed through the o reference
    o = null;
    // the object is now dead because it is still on the GC heap but you have no way to reference it
}

When the GC runs it reclaims memory used by dead objects. Having objects in this state is normal as the GC only runs periodically. For lots more info about how the GC works https://github.com/Maoni0/mem-doc/blob/master/doc/.NETMemoryPerformanceAnalysis.md

[dotnet-policy-service]

Hi @julienGrd. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[noahfalk]

I think the original question has been answered and there isn't anything else actionable on this issue so I'm closing it. If something related comes up in the future you can always open a new issue and refer back to this one. Thanks!