Remove stream managers when decommissioning partitions #4214
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dominiklohmann
added
bug
dominiklohmann
force-pushed
the
topic/stale-strema-managers
branch
from
81f09b2
to
622d5d0
Compare
tobim
approved these changes
May 13, 2024
We noticed that in some deployments, the CPU usage of the `tenzir.index` thread would rise over time. Most time was spent in `std::_Rb_tree_increment`—which is an internal of `std::map` or `std::set`. But we didn't use that anywhere in the index actor, or so we thought: it turned out that CAF itself uses a `std::map` for its stream managers, creating a new one for every active partition. When decommissioning an active partition into a passive partition, that entry in the map was not removed, making further interactions with the map increasingly expensive. This change makes it so we manually erase stale stream managers when decommissioning an active partition. This really should be considered a bug in CAF, but since upstream has replaced the streaming functionality entirely it doesn't make much sense to report or fix it there.
tobim
force-pushed
the
topic/stale-strema-managers
branch
from
622d5d0
to
7a5efd9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We noticed that in some deployments, the CPU usage of the
tenzir.indexthread would rise over time. Most time was spent instd::_Rb_tree_increment—which is an internal ofstd::maporstd::set. But we didn't use that anywhere in the index actor, or so we thought: it turned out that CAF itself uses astd::mapfor its stream managers, creating a new one for every active partition. When decommissioning an active partition into a passive partition, that entry in the map was not removed, making further interactions with the map increasingly expensive.This change makes it so we manually erase stale stream managers when decommissioning an active partition. This really should be considered a bug in CAF, but since upstream has replaced the streaming functionality entirely it doesn't make much sense to report or fix it there.