CPU utilization in IDS 7.0.3 is 3x what it was in IDS 6.3.8

[ysichrisdag]

Which version of Duende IdentityServer are you using?
7.0.3

Which version of .NET are you using?
8

Describe the bug
We have a basic load test for our application that just loades each page in the app. When the app is running .net8 and IDS 6.3.8 the avg cpu utilization on an 1100m pod with 1gb ram is 54.5. With no changes other than upgrading the IDS dependencies to 7.0.3 the avg cpu utilization jumps to 162. I should note that with IDS 7.0.0 the avg cpu was 390.

To Reproduce

Unsure, host a sample app under both configs with load tests.

Expected behavior

Cpu utilization should stay the same, or a moderate increase.

Additional context

Our app was originally set up by RSK and uses dynamic auth and Entity Framework.

[AndersAbel]

Could you please share some more details on the load test? Is it the IdentityServer UI pages that you load?

Do you have any profiling data available?

[RolandGuijt]

@ysichrisdag We didn't hear back from you, does this mean this issue is solved for you?

[ysichrisdag]

Sorry I was travelling and didn't see the responses. I resolved the issue for our application by downgrading to IDS6. Problem still exists in 7. The pages loaded by the load test are all custom as our app is multi-tenant. It is the login, forgot password, register for MFA, as well as post-login pages for managing profile and configuring clients, SAML configurations, permissions etc. It only does GETS on theses pages with the exception of the login page which it POSTS. I have no profiling data because I was on a tight timeline and didn't have much time to work on diagnosing the reason for the cpu usage.

[RolandGuijt]

Exactly what version of 7 are you using? There was a performance issue in our license validation code and it was fixed in 7.0.2.
Can you please update to the latest version and let us know if that helped?
This would only affect an unlicensed version. So I'm assuming you don't have a license file in the load test environment?

[ysichrisdag]

I do have a license loaded in our load test environment. I had initially tested with both 7.0.0 and then again with 7.0.3. There was a significant improvement between 7.0.0 and 7.0.3 but it is still 3x the cpu usage of IDS 6.3.8. With making no other changes other than updating dependencies required for either net6/8 or IDS 6.3.8/7.x the results were:

Version|Baseline CPU|Underload CPU
IDS4_net6|6.95|38.85
IDS6.3.8_net6|2.8|34.95
IDS6.3.8_net8|3.05|54.5
IDS7.0.0_net8|28.75|390.3
IDS7.0.3_net8|5|162.55

[RolandGuijt]

Our intent is to investigate this thoroughly. To do that profiling data is essential. Do you have the possibility to share that?

You mentioned you're mainly testing the loading of pages in the load test. Can you describe to what extend these are related to IdentityServer? E.g. loading the login page, forgot password etc. (GET) will probably have less to do with IdentityServer but more with MVC/Razor pages.

[RolandGuijt]

@ysichrisdag Any news? If you've solved this we'd love to hear the solution.

[ysichrisdag]

Sorry guys I have been extremely busy and haven't had time to come back to this. I have no solution, my app is downgraded back to 6.3.8 and it will be at least 6 months before I have time to attempt a 7.x upgrade again. Here's what I can tell you:

The load tests are not doing much IDS specific functionality as they are just loading pages in the app. The majority of the pages in the app require authentication and are for administering a customers SSO environment (adding/removing client oauth apps, adding/removing users, setting up MFA requirements for their portal etc. However, as you can see below the only real difference is the Duende.IdentityServer library reference causing 3x cpu usage, so there must be either some background processing or some middleware that is processing on every request.

Relevent libraries in our app while showing high cpu:

        <PackageReference Include="Duende.IdentityServer" Version="7.0.3" />
        <PackageReference Include="Duende.IdentityServer.AspNetIdentity" Version="7.0.3" />
        <PackageReference Include="Duende.IdentityServer.EntityFramework" Version="7.0.3" />
        <PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore" Version="8.0.1" />
        <PackageReference Include="Microsoft.AspNetCore.Identity.EntityFrameworkCore" Version="8.0.1" />
        <PackageReference Include="Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation" Version="8.0.1" />
        <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="8.0.1">
        <PackageReference Include="Pomelo.EntityFrameworkCore.MySql" Version="8.0.0" />
        <PackageReference Include="Rsk.AspNetCore.Fido" Version="3.0.0" />
        <PackageReference Include="Rsk.AspNetCore.Fido.EntityFramework" Version="3.0.0" />
        <PackageReference Include="Rsk.DynamicAuthenticationProviders" Version="2.0.0" />
        <PackageReference Include="Rsk.DynamicAuthenticationProviders.EntityFramework" Version="2.0.0" />
        <PackageReference Include="Rsk.DynamicAuthenticationProviders.Saml" Version="2.0.0" />
        <PackageReference Include="Rsk.Saml.DuendeIdentityServer" Version="9.0.1" />

No other changes besides changing to these libraries shows normal cpu:

        <PackageReference Include="Duende.IdentityServer" Version="6.3.8" />
        <PackageReference Include="Duende.IdentityServer.AspNetIdentity" Version="6.3.8" />
        <PackageReference Include="Duende.IdentityServer.EntityFramework" Version="6.3.8" />
        <PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore" Version="8.0.1" />
        <PackageReference Include="Microsoft.AspNetCore.Identity.EntityFrameworkCore" Version="8.0.1" />
        <PackageReference Include="Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation" Version="8.0.1" />
        <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="8.0.1">
        <PackageReference Include="Pomelo.EntityFrameworkCore.MySql" Version="8.0.0" />
        <PackageReference Include="Rsk.AspNetCore.Fido" Version="3.0.0" />
        <PackageReference Include="Rsk.AspNetCore.Fido.EntityFramework" Version="3.0.0" />
        <PackageReference Include="Rsk.DynamicAuthenticationProviders" Version="2.0.0" />
        <PackageReference Include="Rsk.DynamicAuthenticationProviders.EntityFramework" Version="2.0.0" />
        <PackageReference Include="Rsk.DynamicAuthenticationProviders.Saml" Version="2.0.0" />
        <PackageReference Include="Rsk.Saml.DuendeIdentityServer" Version="8.1.0" />

[RolandGuijt]

Are all the pages you're load testing on which you see performance degradation doing something with the IdentityServer package? If not then something else is the problem.

It's hard to determine where the problem comes from without profiling data. The combination of packages or .NET version with packages could also be an issue.
The only way to continue investigating this is when we have more detailed information in the form of profiling data I'm afraid.

[RolandGuijt]

Closing this for the moment, but feel free to reopen when you have more information.