sentry web: slow page loading time

[vanyakosmos]

Self-Hosted Version

23.6.1

CPU Architecture

x86_64

Docker Version

24.0.5

Docker Compose Version

1.29.2

Steps to Reproduce

Open any sentry page

Expected Result

Fast page load

Actual Result

Slow page load, and sometimes it can just stuck indefinitely. Everything that goes to web service is suffering: emails loading, static files loading, initial page html loading.

My server has 6 cpus and 32gb RAM. I configured sentry web to use 9 uwsgi workers. Traffic is pretty moderate, only 8gb of memory is used and cpus are not overloaded.

At the time of "testing" I was probably the only person in the company who accessed sentry web. Problems occur at somewhat random time, one moment everything is fine, and the next one everything is slow as hell. If I open 5 tabs at the same time then everything will open veeery slowly (if ever).

How to debug something like this? At first glance everything should be fine...
Postgres works fine and some endpoints that do not use it are also lagging (eg static files).

Event ID

No response

[hubertdeng123]

Hi there, thanks for writing in. Unfortunately it is hard to debug this problem without additional information. Is there any logs that may be of use in your web service?

[vanyakosmos]

mainly access logs and some pydantic warnings, no error:

in general page loading looks like this (requests are sorted by response time):

[azaslavsky]

Can you check CPU, memory, and network usage during these episodes? I would imagine some resource is being saturated.

[vanyakosmos]

this is what I see after opening 5 tabs at the same time:

• single uwgi worker is doing all the job without parallelization
• redis ate 20gb of memory
• relay workers are also underutilized (it is strange but unrelated to the current issue)

right now I have an even bigger machine (with enough memory, which might have been a problem in a previous case), but the lags are still the same:

[azaslavsky]

Have you tried setting a memory limit for redis, a la #2262?

[getsantry]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀