New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] consumers stops receiving new messages due to invalid blockedConsumerOnUnackedMsgs state #22657
Comments
Thanks for the great issue report @180254. @poorbarcode or @Technoboy- do you have a chance to take a look at this issue report? |
@180254 in your case, can you detect the issue from topic stats? for example, does it tell |
sorry, noticed this now. I guess topics stats wouldn't have |
Here are the statistics for a "broken topic", collected them after a test.
There are 4 consumers, each of them is |
@AdrianPedziwiatr-TomTom thanks for sharing. This is an interesting detail. |
A little progress in reproducing the problem in the unit test: 180254@5822ab6 (test22657_1_parameterized) v3.0.1: fails when [ |
I reproduced the problem also for larger values of maxUnackedMsgPerConsumer. Please see: Test results:
Some log from the failed case:
(in summary)
At branch-3.0...180254:pulsar-issue-22657:branch-3.0 you can find all my tests and the restored old version of the individualAckNormal method for testing/comparison. |
Getting the same issue, consumer stops receiving new messages { pulsar version 3.2.1 |
@prasathsekar You might be facing another bug that is already fixed in 3.2.3 with #22454. Please upgrade to Pulsar 3.2.3 and then comment whether the problem is resolved. |
Search before asking
Read release policy
Version
pulsar server: docker image apachepulsar/pulsar:3.0.4 + helm chart pulsar-helm-chart
pulsar client: java client org.apache.pulsar:pulsar-client:3.0.4
Minimal reproduce step
After updating Apache Pulsar, we noticed that one of the consumers sometimes stops receiving new messages for some topics.
The last fully working version for us is 3.0.1. I have tested all later versions released so far and also built a branch-3.0.
I looked through the commits and determined when our service stops working:
I performed a test using the last commit from branch 3.0 (fd823f6) and reverting the individualAckNormal method to the last version before the "commit which breaks." The change looks as follows: 180254@6dac4bf. I have no problem with the modified code.
I found nothing in the logs that would inform me about the consumer suspension, etc. There are no unusual logs at all. Restarting the Kubernetes pod with consumers has helped for some time.
What did you expect to see?
consumer retrieves all messages
What did you see instead?
consumers stops receiving new messages for some topics
Anything else?
The configuration we use:
We can reproduce it on our service. Test scenario: serviced approximately 20 customers (== 20 topics), each with about 20 messages per second. 1 message is processed in approximately 200ms. The problem occurs for a certain number of topics in the test, not for all
When a problem occurs:
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: