New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[improve][meta] Fix invalid use of drain API and race condition in closing metadata store #22585
base: master
Are you sure you want to change the base?
Conversation
It seems that the OOME is another issue. #22586 |
} | ||
while ((op = writeOps.poll()) != null) { | ||
op.getFuture().completeExceptionally(ex); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
@@ -99,7 +103,13 @@ public void close() throws Exception { | |||
private void flush() { | |||
while (!readOps.isEmpty()) { | |||
List<MetadataOp> ops = new ArrayList<>(); | |||
readOps.drain(ops::add, maxOperations); | |||
for (int i = 0; i < maxOperations; i++) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one should be ok, since it's already done in a loop: while (!readOps.isEmpty()) {...}
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #22585 +/- ##
============================================
- Coverage 73.57% 71.02% -2.55%
+ Complexity 32624 5630 -26994
============================================
Files 1877 1891 +14
Lines 139502 152144 +12642
Branches 15299 18133 +2834
============================================
+ Hits 102638 108061 +5423
- Misses 28908 35445 +6537
- Partials 7956 8638 +682
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Motivation
There's currently some memory leaks in tests and while investigating the issue, I found out that there's a large number of uncompleted CompletableFutures in the heap dump.
Currently the metadata store doesn't complete all pending operations when it is closed.
There are multiple problems:
Modifications
Additional Context
CompletableFutures in the heap dump:
The instances are related to
org.apache.pulsar.broker.resources.NamespaceResources$PartitionedTopicResources$$Lambda$1819+0x00007f08a8b65ee8
:org.apache.pulsar.broker.resources.NamespaceResources$PartitionedTopicResources$$Lambda$1819+0x00007f08a8b65ee8
seems to be this code block:pulsar/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/NamespaceResources.java
Lines 345 to 374 in bbdc173
Documentation
doc
doc-required
doc-not-needed
doc-complete