Investigate this. According to EFC docs, calling DbContextOptionsBuilder.UseMemoryCache with more than two IMemoryCache instances over time can kill performance.
https://learn.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.dbcontextoptionsbuilder.usememorycache?view=efcore-9.0
Note that changing the memory cache can cause EF to build a new internal service provider, which may cause issues with performance. Generally it is expected that no more than one or two different instances will be used for a given application.