unexpected database load with JpaEventStorageEngine::fetchEvents ramping up

[junkdog]

Basic information

• Axon Framework version: 4.7.1
• JDK version: Amazon Corretto 17

Description

Hi,

We have identified an issue with one of our applications that is built using Spring Boot 3, JPA, and Axon Framework 4.7.1. In a particular environment, we have noticed that despite low traffic, the database load is uneven.

(requests to RDS grouped by query)

Upon investigation, we have found that JpaEventStorageEngine::fetchEvents appeared to be ramping up over time, as indicated by the increasing blue bars. It is worth noting that during this time period, there was no scaling activity apart from a single deploy at 23 Feb 1500, which only caused a small bump. Interestingly, the UPDATE token_entry statement shown in the yellow bars was running at a constant rate.

Is this something you've seen in the past? Any ideas on how we could investigate this further?

Expected behavior

During no load, JpaEventStorageEngine::fetchEvents should be invoked at a more-or-less constant rate.

Actual behaviour

In one of our environments, it ramped up over time.

[gklijs]

What is the scale on the graph, and how many events are there? Since for a relational database all rows/events are considered the same, I would expect a fetch events query to be getting a bit slower over time. The tokens are limited in size, so indeed I would expect token-related operations to stay constant over time.

[smcvb]

Is this something you've seen in the past? Any ideas on how we could investigate this further?

Only when the Event Store was passing sizes could not effectively handle the RDBMS indices anymore.
Note that regular RDBMS solutions of an event store are expected to degrade in performance over time as the size grows.
Allard describes this in a presentation from 2019, explaining why AxonIQ constructed a dedicated Event Store (which you can see here).

However, whether this is the case yes or no isn't clear at the moment.

So, what's the amount of events currently in the RDBMS, @junkdog?
Furthermore, I'm also curious about the scale of the image you've shared.
Additionally, as you're post is already several weeks old, are you able to share whether this increase is still happening?

While waiting for your response, I have added the information required label, as well as replaced the bug label for a question label.
The bug-to-question label switch is mainly a hunch. If we proof something's amiss with AF, we'll obviously reintroduce it.

[smcvb]

Closing this issue due to inactivity.
Feel free to add comments at a later stage, @junkdog, when your time allows it!
Based on that, we should be able to reopen the issue and work on it if necessary.