Pinging @elastic/apm-ui (Team:APM)

I'm not sure this is related to serverless or boost window. I have a hypothesis that this could be recreated pretty easily in any deployment.

@chrisdistasio I've just tried that on stateful deployment with similar data and reproduced it. Noted a version in the description.

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

looking at the video--i would suggest the first problem we need to look at is the fact that it took over a minute to render the inventory list of dependencies for 30 days.

I see that the current api /internal/apm/dependencies/top_dependencies, might return too much data. It returns current/previous stats along with the time series.

We should break it down as we are doing on the service inventory page. First fetch the top dependencies with the summary statistics, then on a follow-up request, we get the time series and only for the dependencies visible on the screen, as the user navigates we keep fetching only the time series from the visible items.

@cauemarcondes you probably want to look at a trace first. The underlying requests are a little different from other places. We need to jump through some hoops to be able to resolve a destination service name from an exit span metric.

looking at the video--i would suggest the first problem we need to look at is the fact that it took over a minute to render the inventory list of dependencies for 30 days.

There might be some improvement on that after #178979 is merged.

There is probably some optimization aspect that needs to be done on the frontend. The tab that loads the page is constantly crashing and becoming unresponsive. 5.2GB memory usage as shown on the screenshot is a huge number.

The trace sample component doesn't necessarily need to render data for a large time range to present performance issues.

I've tested it with a 2h-range, and despite the API responding relatively fast, the trace sample component takes too much time to render the data. While it does that, the page becomes unresponsive for a short while and the tab consumes ~2.9Gb of memory.

The profiler shows the evolution of memory consumption

Testing it with a 4h range crashes the tab

The performance seems to worsen when the trace object becomes more complex. After digging into the code, I found what might be the two main performance detractors:

1 - getWaterfallMaxLevel - recursive function that could be optimized with the usage of the BFS algorithm. It can consume a lot of heap memory depending on how complex the structure is.

2 - AccordionWaterfall - It's another recursive call, and it renders everything at once. Instead, it could render the trace data on demand, as the user scrolls down the page.

There might be other possibilities of micro-optimizations that will make the component render faster.

Parsing the json was fast enough. We can move forward with the fix by improving the performance on the frontend.

@chrisdistasio @paulb-elastic, I've managed to refactor the trace sample and optimize the performance, but another problem surfaced:

In large date ranges, the waterfall miscalculates the position of the elements in the row.

I'll open a new ticket for this, but it doesn't seem like something related to serverless performance

@crespocarlos to clarify, does this only happen because of the changed implementation?

If so, the concern of addressing this separately is that this new (incorrect) behaviour will impact users as soon as it's merged, until a subsequent fix is implemented/merged, in which case it would be better to fix in the same PR (have I understood it correctly?).

@paulb-elastic , it's an existing issue. It can be seen on the edge cluster:

https://edge-oblt.kb.us-west2.gcp.elastic-cloud.com/app/apm/dependencies/operation?comparisonEnabled=true&dependencyName=checkoutService&detailTab=timeline&environment=ENVIRONMENT_ALL&kuery=&offset=1d&rangeFrom=now-30m&rangeTo=now&showCriticalPath=false&spanName=POST%20%2Fnodejs%2FplaceOrder&spanId=0000000000104250&page=0

It seems like a completely different problem. It has to be addressed but IMO it'd be better to do so in a separate ticket not to increase the scope of this PR. It could also be a problem with the data on the edge cluster because I couldn't find any SDH complaining about the rendering problem.

it'd be better to do so in a separate ticket not to increase the scope of this PR

@crespocarlos thanks for confirming it's an existing issue and not related to this PR - fully agree it should be tacked separately. As on edge, it's also not specifically related to serverless, so would you mind creating a separate issue for it that goes onto the infra&services backlog (as opposed to this project), thanks.

Tested the fix in qa with 30-day range