Failed to obtain cache lock for

[narbit]

Which version of Duende IdentityServer are you using?
6.3.0

Which version of .NET are you using?
7.0

Describe the bug
Same as reported in:
#191
#659
#720

Occasionally one of the pods gets stuck on "Failed to obtain cache lock for" exception. The lock never releases taking the pod down completely and requiring a restart. Meantime the database is not under pressure and performs well.

To Reproduce
This seems to be random. We haven't figured out how to repro. Pod's CPU/memory seem to be fine prior to event. DB is fine as well.

Log output/exception with stacktrace
Exception: System.Exception: Failed to obtain cache lock for: 'Duende.IdentityServer.Services.DefaultCache1[Duende.IdentityServer.Models.Resources]'
at Duende.IdentityServer.Services.DefaultCache1.GetOrAddAsync(String key, TimeSpan duration, Func1 get) in /_/src/IdentityServer/Services/Default/DefaultCache.cs:line 149`

Expected behavior / Question
Relaxing the cache lock behavior still appears to be in feature request - DuendeSoftware/IdentityServer#1286.

Are there any new recommendations/findings that might help address this, besides tuning up the lock timeout parameter and recommendations from previous submissions?

[josephdecock]

Thanks for opening this issue. We're still planning what will go into the next version of IdentityServer, but user feedback is quite helpful as we prioritize. For the short term, our advice remains the same: try turn up the timeouts on the lock option and in the database connection, and if that doesn't resolve the issue, consider a custom caching mechanism that is tuned for your environment.

[narbit]

Thank you for your response and suggestions! We are still trying to understand the root cause. It seems the lock issues are preceded by a number of System.OutOfMemoryException from EntityFrameworkCore on a few different threads but in the same container that is having trouble. So we are wondering if the Finally block in the GetOrAddAsync is somehow not getting executed due to this. We can close this for now.