Limit/Thottle/Backpressure

[ctrlplusb]

Awesome lib thanks!

Are there any mechanisms built into the library to support rate limiting? Or is it the responsibility of each worker node?

[LewisJEllis]

What specific operation are you interested in rate limiting? Are you looking to enforce a policy like "one job producer may not produce more than X jobs/second on a given queue" or "all job producers together cannot produce more than X jobs/second on a given queue" or something else?

We don't have any sort of built-in mechanisms for this, especially for back-pressure. The general thinking has been something like "process jobs as fast as possible and let jobs queue up and wait when necessary, and if the queue gets too backed up add more workers". No use for back pressure because every job is important, so we'd rather just put the jobs in the queue and let them sit there rather than tell the producers to slow it down. This makes me curious about your use case.

[grantcarthew]

@LewisJEllis Have a look here mate. I have been documenting this feature for implementation. Not going to be done on my queue for a while however the links will help out.

grantcarthew/node-rethinkdb-job-queue#61