how to apply backpressure #1007
Labels
Design
Problem hard to fix, affects design.
Developer
not a problem, more of a note to self for devs about work to do.
Discussion_Needed
developers should discuss this issue.
enhancement
New feature or request
NextRelease
Feature Targeted for Next Release
Priority 3 - Important
worrisome impact... should study carefully
Refactor
change implementation of existing functionality.
ReliabilityRecovery
improve behaviour in failure situations.
UserStory
interesting to read to consider improving
v3only
Only affects v3 branches.
was discussing with @reidsunderland a situation where if a node in a cluster fails, we want the sender to it to stop consuming, letting other consumers of a shared queue take over the entire load if this sender's downstream is broken.
https://en.wikipedia.org/wiki/Backpressure_routing
In v2, backpressure applied naturally, since we processed one message at a time, and simply tried to deliver or download the same item forever. If a delivery was failing, we would never loop back to consume more from the queue.
with sr3, we have both download_retry, and post_retry queues, so that if individual transfers fail, we can keep going. The problem is one can put millions of files in those retry queues, and a failing node may even consume from the queue faster because the failures may be quicker to process than successful deliveries.
So... there need to be some criteria when processings is going badly to stop consuming... that is, to apply backpressure to the upstream side.
The text was updated successfully, but these errors were encountered: