New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When EmbeddedEventStore
enables optimizeEventConsumption
(default) it will cause a memory leak of Node
sometimes
#2954
Comments
EmbeddedEventStore
enables optimizeEventConsumption
(default)it will cause a memory leak of Node
sometimes
EmbeddedEventStore
enables optimizeEventConsumption
(default)it will cause a memory leak of Node
sometimesEmbeddedEventStore
enables optimizeEventConsumption
(default) it will cause a memory leak of Node
sometimes
First off, thanks for filing this concern with us, @yitizi. Secondly, perhaps an obvious pointer, but Axon Framework 4.4.5 is an old version (November 2020) of Axon Framework, as we are currently at release 4.9.1. Hence, it would be very helpful if you could show the same predicament under a more recent release of Axon Framework. Another point that interests me, is your specific mention of the Now let me comment a little to your explanation:
The
This number you're inserting is pretty identical to the default cache window of the This raises another question: how many
How are you exactly telling application X to release "a" Concluding: let's get to the bottom of this, @yitizi 💪 |
Thanks for the reply. You suggested that I update the Axon Framework to the newest version, but the application is already published in production, so the update cannot be done quickly. I also noticed that the EventProducer code hasn't changed between versions 4.4.5 and 4.9.1.
Yes, only when optimizeEventConsumption is set to true, will the EventProducer initiate, and the linked list-like usage will be activated. When the EventProducer opens an event stream with JdbcEventStorageEngine, it fetches to the end, regardless of the existence of trailing consumers. Therefore, when I insert numerous events, they get cached in memory, and the trimCache method resets the oldest. However, these cached nodes still need to wait for the JVM garbage collection to be triggered.
I have no doubt about the capability to switch from a standard EventConsumer to a custom EventConsumer. It's a fantastic feature that can significantly decrease the usage of database connections.
in debug environment is calling the org.axonframework.eventhandling.TrackingEventProcessor#releaseSegment(int) method. in production sometime just occur an error.but i search for the application i mention before,it seems to is just trigger the cleaner ,but just can't be done?im sure it is just two tracking.it has those log.
Through the EventProducer's fetching method and the caching of nodes using a linked list approach, it puts pressure on the garbage collector (GC). Eventually, this prevents the garbage collector from functioning properly. |
Although I am aware that we have practically made no changes to the Note that this choice is not something we've made lightly. With that said, I will reflect on your others comments.
Thanks for the investigation here, @yitizi :-)
It seems like your environment only has two Streaming Event Processors in place.
Although true, if you just have two Streaming Event Processors with a single token each, the Granted, if you would increase the number of segments for your TrackingEventProcessor, you would have an event stream (so database connection) per segment.
Furthermore, they're both sagas, which I base on the processor name. The log messages provided by Axon Framework are set to
If you would point the issue to be the linked list, that would mean references to Would you mind running a test with that value set to, e.g., 1_000? Just to see whether the issue remains by the way! By the way, I have spent quite some time in the The most reasonable culprits of this, would be the So, let me conclude this comment:
Thanks again for opening this issue, @yitizi! :-) |
Thanks for your advice. We are in the process of updating Axon to version 4.7.6 to see if it resolves the issue. The problem has occurred, and we are working to identify the cause. Given that the code appears to have undergone minimal changes, we suspect it might recur. Additionally, this problem is not 100% reproducible. |
Basic information
Steps to reproduce
use the JdbcEventStorageEngine and tracking mode to consumer events from eventbus.and startup 2 application.
firstly,the eventstore has some event ,and then the trackingprocessor will process those events.
when consume to the end,it will start a EventProducer to fetch events instead of the privateStream.
secondly,insert a lot of events to the event store,about 10000,which can't be done immediately.
thirdly,release the trackingtoken which hold it,so another one will take it.
it happens occasionally in production environment.i repeat it once,but when i call heapdump i has bean recycle by full gc.
the production environment is arm64 by the way.
Expected behaviour
the application can be stable running.
Actual behaviour
the oldest node(not the EmbeddedEventStore hold,but exist in G1 Old Gen) is no gc root ,but can't be recycle,and the survivor space memory usage is keep growing until it cause oom.
mat images
The text was updated successfully, but these errors were encountered: