SignalR Memory keeps increasing

[cceraja]

Hi,

I notice that Signalr memory in IIS keeps increasing and never goes down even when the users are disconnected. Almost all the users (approx. 200 users) are connected using websockets.

---

Asp.net.SignalR.Client.dll - 2.2.3
Asp.net.SignalR.Core.dll - 2.2.3
Asp.net.SignalR.SystemWeb.dll - 2.2.3

Setup: Single server, dedicated app pool for Signalr WCF
O/S: Windows 2012 R2
Server : Ec2 T2XLarge ( 4 vCPU, 16 GB Mem)
Available Free memory : > 50%

---

Please refer to the IIS memory consumption taken for the application.

Regards,
Raja

20180730-1.xlsx

[cceraja]

Any update on why this is happening?

[penguinawesome]

same issue here

[analogrelay]

Can you use a memory profiler to determine which objects are being left alive? SignalR does intentionally hold messages in memory after a client disconnects in order to be able to re-broadcast those messages in the event of a reconnecting client. These messages are stored in a ring buffer so that it only buffers a fixed number of messages, but it is entirely expected that memory would not be released as soon as a client is disconnected.

Are you hitting specific performance issues directly attributed to this memory usage? SignalR should also be using weak references to ensure that this memory is freed when the Garbage Collector needs additional space. Can you provide information on the failures you are seeing?

[moozzyk]

Likely #3850.

[analogrelay]

As described above, this looks to be just the standard SignalR message buffer. SignalR uses a fixed-size ring buffer to hold messages (so old messages are only removed when space is needed) in order to allow messages to be replayed to clients that reconnect. You can modify this buffer size using the IConfigurationManager.DefaultMessageBufferSize option.

Closing this as we haven't heard any updates on the objects that are staying alive. Feel free to comment further if you have question. We can always re-open this if there's an issue to resolve!

[rafael-biz]

I still think this is a critical bug and that this issue should not be closed. It is a big memory leak that can easily consume a lot of memory and crash an application due to an OutOfMemoryException. Especially because the default value of DefaultMessageBufferSize is 1000 messages per Hub. Right now, I'm struggling with that.

[AgSync-Aaron]

In my situation our web app (hosting SignalR only) never releases strings referenced by HostContext inside of a DefaultSubscription. These are 35K of duplicate strings. We don't have 35K connections or users, why on earth would 35K context objects be held onto for ONE user? I think it's just one user responsible for those 35k copies because that first item is their access token.

Memory Usage

In my case I don't believe it has to do with queued messages. It looks like we have 132k allocated messages, but they are only taking up about 11MB of memory.

Okay last edit I hope... Here is the best view that I've found in dotMemory about the breakdown of memory usage.

[clarghi]

Hi,
we have the same issue: a Web Site send about 1K messages/minute, and the IIS service grows to 9GB in a few hours.
We have reduced the buffer without success,

`
public void Configuration(IAppBuilder app) {

   GlobalHost.Configuration.DefaultMessageBufferSize=32;
   app.MapSignalR();

}
`

[clarghi]

Hi,
I have download the source code and executed the example provided, changing in BackgroundThread.cs the frequence to 100ms (Thread.Sleep(100)) and changed the simple string sended to the client with an object.
The result is the same: the memory increase and after two minutes I must stop, otherwise crash with Out of memory error.

So, I think this is a SignalR bug.
I'm trying to find the bug, but the code is complex and only a few commented.

This is the modified code:

while (true)
{
try
{
var obj = new {
antani = "cappotto",
cuss = "cussato",
antarico="antani come di antaricato",
numerico=122.344393902,
Date=DateTime.Now
};
context.Connection.Broadcast(obj);
// hubContext.Clients.All.fromArbitraryCode(obj);
// hubContext.Clients.All.fromArbitraryCode(DateTime.Now.ToString());
} catch (Exception ex)
{
System.Diagnostics.Trace.TraceError("SignalR error thrown in Streaming broadcast: {0}", ex);
}
Thread.Sleep(100);
}

[penguinawesome]

Hi, I have download the source code and executed the example provided, changing in BackgroundThread.cs the frequence to 100ms (Thread.Sleep(100)) and changed the simple string sended to the client with an object. The result is the same: the memory increase and after two minutes I must stop, otherwise crash with Out of memory error.

So, I think this is a SignalR bug. I'm trying to find the bug, but the code is complex and only a few commented.

This is the modified code:

while (true) { try { var obj = new { antani = "cappotto", cuss = "cussato", antarico="antani come di antaricato", numerico=122.344393902, Date=DateTime.Now }; context.Connection.Broadcast(obj); // hubContext.Clients.All.fromArbitraryCode(obj); // hubContext.Clients.All.fromArbitraryCode(DateTime.Now.ToString()); } catch (Exception ex) { System.Diagnostics.Trace.TraceError("SignalR error thrown in Streaming broadcast: {0}", ex); } Thread.Sleep(100); }

did you find any solution? Let us know i'm also looking for it @clarghi

[clarghi]

Hi, I have download the source code and executed the example provided, changing in BackgroundThread.cs the frequence to 100ms (Thread.Sleep(100)) and changed the simple string sended to the client with an object. The result is the same: the memory increase and after two minutes I must stop, otherwise crash with Out of memory error.
So, I think this is a SignalR bug. I'm trying to find the bug, but the code is complex and only a few commented.
This is the modified code:
while (true) { try { var obj = new { antani = "cappotto", cuss = "cussato", antarico="antani come di antaricato", numerico=122.344393902, Date=DateTime.Now }; context.Connection.Broadcast(obj); // hubContext.Clients.All.fromArbitraryCode(obj); // hubContext.Clients.All.fromArbitraryCode(DateTime.Now.ToString()); } catch (Exception ex) { System.Diagnostics.Trace.TraceError("SignalR error thrown in Streaming broadcast: {0}", ex); } Thread.Sleep(100); }

did you find any solution? Let us know i'm also looking for it @clarghi

Hi, no. The code is complex and almost not commented :-(

[fdmarc]

We had a memory leak in our SignalR code at Flipdish.

Some snippets from our Slack thread about this:

We observed the G2 Heap size slowly increasing over time.

Analyzing a memory dump showed that we had thousands (34k) of DbContexts attached to ILifetimeScopes that are not being disposed.

Autofac documentation mentions that resolving dependencies from the root lifetime scope can cause memory leaks and their SignalR documentation mentions dependencies for SignalR hubs are resolved from the root lifetime scope and there are some suggestions about how to prevent the memory leak and dispose dependencies.
Also I've found a library that's supposed to help with this https://github.com/lethek/SignalR.Extras.Autofac/
It's possible that we have had this issue always but given the amount of deployments that we do and that SignalR hub requests were spread among all instances we did not realize?

We made a change to our system to use the SignalR.Extras.Autofac library, and this fixed our leak: