Video lags after few minutes and high memory consumption

[nodminger]

I am using FFmpeg File and Device Test to share my screen. It works perfectly fine initially, but after a few minutes, there is some lag—between 1-3 seconds, and very rarely, it becomes 5 seconds. Also, the application consumes a significant amount of memory, always hovering around 180 MB or more(200 MB+ some times). I am using .NET v6. Is there a way to solve these issues? Is anyone else facing the same issues as me?

[sbarrac]

I've had a very similar experience with both issues.

The memory usage is due to some non ideal memory management in the RTPPeerconnection.SendView & SendAudio methods. The byte arrays are being copied many times per call, due to byte arrays not being resizable.

I'm having the same issue with latency in video streams. As far as I can tell, it's due to the chrome jitter buffer detecting that the video stream is unstable so it starts to buffer it, however this is even happening on Localhost. I believe this is due to some issues with exceptions being thrown on a background thread somehow causing some jitter in the stream. When the latency starts increasing, check the output window in visual studio, look for either an exception being thrown or a thread exiting.

[nodminger]

I've had a very similar experience with both issues.

The memory usage is due to some non ideal memory management in the RTPPeerconnection.SendView & SendAudio methods. The byte arrays are being copied many times per call, due to byte arrays not being resizable.

I'm having the same issue with latency in video streams. As far as I can tell, it's due to the chrome jitter buffer detecting that the video stream is unstable so it starts to buffer it, however this is even happening on Localhost. I believe this is due to some issues with exceptions being thrown on a background thread somehow causing some jitter in the stream. When the latency starts increasing, check the output window in visual studio, look for either an exception being thrown or a thread exiting.

Yes, I am getting "Exceptions" and some "threads are exiting"

Exception thrown: 'Microsoft.CSharp.RuntimeBinder.RuntimeBinderException' in Microsoft.CSharp.dll
Exception thrown: 'System.ApplicationException' in SIPSorcery.dll
The thread 0x2434 has exited with code 0 (0x0).
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
The thread 0x2cbc has exited with code 0 (0x0).
The thread 0x1d64 has exited with code 0 (0x0).
Exception thrown: 'System.ApplicationException' in SIPSorcery.dll
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
Exception thrown: 'System.ApplicationException' in SIPSorcery.dll
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
The thread 0x5a54 has exited with code 0 (0x0).
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
The thread 0x6344 has exited with code 0 (0x0).
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
The thread 0x5944 has exited with code 0 (0x0).
Exception thrown: 'System.ApplicationException' in SIPSorcery.dll
The thread 0x3da0 has exited with code 0 (0x0).
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
The thread 0x4af0 has exited with code 0 (0x0).
Exception thrown: 'System.ApplicationException' in SIPSorcery.dll
The thread 0x4d7c has exited with code 0 (0x0).
Exception thrown: 'System.ArgumentException' in System.Private.CoreLib.dll
The thread 0x5410 has exited with code 0 (0x0).
Exception thrown: 'System.ApplicationException' in SIPSorcery.dll
Exception thrown: 'System.ApplicationException' in SIPSorcery.dll
The thread 0x1b00 has exited with code 0 (0x0).

[sipsorcery]

Optimization is possible in the video frame buffers, but this alone may not meet the required throughput levels. A significant performance challenge arises when crossing the native-to-managed boundary, such as from Win32 to .NET. This is evident in the sipsorcery library, which aims to implement as much of the WebRTC stack in .NET as possible. However, there's currently no direct .NET API for video/audio devices on Windows, and based on the last 23 years of the .NET Framework's evolution, it seems unlikely that Microsoft will introduce this feature.

In summary, for high performance, the best approach is to use C++ and libwebrtc. An alternative is to find a .NET wrapper for libwebrtc, ideally one that maintains a fully native media-to-SRTP pipeline. However, the existence of such a wrapper is uncertain.