Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thread contention in backend #12901

Open
lontivero opened this issue Apr 18, 2024 · 2 comments
Open

Thread contention in backend #12901

lontivero opened this issue Apr 18, 2024 · 2 comments

Comments

@lontivero
Copy link
Collaborator

lontivero commented Apr 18, 2024 •
edited

Threads callstack 
OS Thread Id: 0x2892e9
        Child SP               IP Call Site
00007EDD1B7FD970 00007f3e34286c96 [InlinedCallFrame: 00007edd1b7fd970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDD1B7FD970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edd1b7fd970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDD1B7FD960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDD1B7FDA00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDD1B7FDA40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDD1B7FDAE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDD1B7FDCD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edd1b7fdcd0]
OS Thread Id: 0x28dedc
        Child SP               IP Call Site
00007EDCF67FB970 00007f3e34286c96 [InlinedCallFrame: 00007edcf67fb970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDCF67FB970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edcf67fb970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDCF67FB960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDCF67FBA00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDCF67FBA40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDCF67FBAE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDCF67FBCD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edcf67fbcd0]
OS Thread Id: 0x28fa58
        Child SP               IP Call Site
00007EDC9CFF8970 00007f3e34286c96 [InlinedCallFrame: 00007edc9cff8970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDC9CFF8970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edc9cff8970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDC9CFF8960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDC9CFF8A00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDC9CFF8A40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDC9CFF8AE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDC9CFF8CD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edc9cff8cd0]
OS Thread Id: 0x28fa5b
        Child SP               IP Call Site
00007EDD41BFC970 00007f3e34286c96 [InlinedCallFrame: 00007edd41bfc970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDD41BFC970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edd41bfc970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDD41BFC960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDD41BFCA00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDD41BFCA40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDD41BFCAE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDD41BFCCD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edd41bfccd0]
OS Thread Id: 0x28fa5c
        Child SP               IP Call Site
00007EDCD4FF8970 00007f3e34286c96 [InlinedCallFrame: 00007edcd4ff8970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDCD4FF8970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edcd4ff8970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDCD4FF8960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDCD4FF8A00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDCD4FF8A40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDCD4FF8AE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDCD4FF8CD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edcd4ff8cd0]
OS Thread Id: 0x28fa5d
        Child SP               IP Call Site
00007EDD1BFFE970 00007f3e34286c96 [InlinedCallFrame: 00007edd1bffe970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDD1BFFE970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edd1bffe970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDD1BFFE960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDD1BFFEA00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDD1BFFEA40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDD1BFFEAE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDD1BFFECD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edd1bffecd0]
OS Thread Id: 0x28fa5e
        Child SP               IP Call Site
00007EDD18FF8970 00007f3e34286c96 [InlinedCallFrame: 00007edd18ff8970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDD18FF8970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edd18ff8970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDD18FF8960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDD18FF8A00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDD18FF8A40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDD18FF8AE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDD18FF8CD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edd18ff8cd0]
OS Thread Id: 0x28fa60
        Child SP               IP Call Site
00007EDCF77FD970 00007f3e34286c96 [InlinedCallFrame: 00007edcf77fd970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDCF77FD970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edcf77fd970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDCF77FD960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDCF77FDA00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDCF77FDA40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDCF77FDAE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDCF77FDCD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edcf77fdcd0]
OS Thread Id: 0x28fa62
        Child SP               IP Call Site
00007EDCF6FFC970 00007f3e34286c96 [InlinedCallFrame: 00007edcf6ffc970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDCF6FFC970 00007f3dbb6bcc8c [InlinedCallFrame: 00007edcf6ffc970] System.Threading.LowLevelLifoSemaphore.<WaitNative>g____PInvoke|3_0(IntPtr, Int32)
00007EDCF6FFC960 00007F3DBB6BCC8C System.Threading.LowLevelLifoSemaphore.WaitNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, Int32) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 2538]
00007EDCF6FFCA00 00007F3DBB6BF03D System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 140]
00007EDCF6FFCA40 00007F3DBB6BCB95 System.Threading.LowLevelLifoSemaphore.Wait(Int32, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.cs @ 65]
00007EDCF6FFCAE0 00007F3DBDF0A0E8 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 100]
00007EDCF6FFCCD0 00007f3e33f0aa07 [DebuggerU2MCatchHandlerFrame: 00007edcf6ffccd0]

This looks pretty much like the followings:

This makes the CPU usage % to go up because LowLevelLifoSemaphore methods spin using while (true) loops.
@lontivero
Copy link
Collaborator Author

According to this issue the spinning can be reduced by setting COMPlus_ThreadPool_UnfairSemaphoreSpinLimit=<number> which by default is 0x70 https://github.com/dotnet/corert/blob/c6af4cfc8b625851b91823d9be746c4f7abdc667/src/System.Private.CoreLib/shared/System/Threading/PortableThreadPool.WorkerThread.cs#L23

That can reduce the CPU usage but it doesn't seem to be the right thing to do.

@molnard
Copy link
Collaborator

molnard commented May 13, 2024

Is this still relevant? It seems like your solution of removing the backup solved the issue on MainNet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Relevance Realization
Status: 🧩 Triaged
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lontivero @molnard