Discussion: Resque-like priority queues

[h3nnn4n]

Situation

The default behavior of Dramatiq when multiple queues are present (either all of them or a subset), is to fetch data from them using the consumer threads. This in practice means that all queues are drained equally (assuming they all have messages in them).

In contrast, Resque, grabs messages in the order the queues are set when starting the workers. For example, given the queues low, high and default, one could start a worker with --queues=high,default,low and the queues would be drained in order. This ofc could lead to the low queue being starved if high or default always had a constant influx of new messages. To overcome this, often there would be multiple workers with different settings like --queues=low,default,high to ensure that all queues had at least one workers dedicated to them.

When compared to dramatiq, resque allows creating dedicated workers for a given queue, that when idle will try to process jobs from other queues. Dramatiq can have dedicated workers for a queue, however, they will stay idle if that queue is empty.

Potential Solutions

Middleware

Since dramatiq has good middleware support, one could write a new middleware to implement this behavior. However, it seems that there aren't any available hooks that could be used to handle this in an efficient way.

Ideally, no tasks should be waiting in a worker, while a potential another worker could be handling it. We should also avoid taking tasks off a queue, and then adding it back, repeatedly.

Builtin

Changing how dramatiq works probably isn't the best approach, since it could break setups that are functional with the current behavior. We could have a setting to toggle this behavior. Something like --drain-queues-in-order. IMO this solution is less preferred, since it adds more complexity and a somewhat opinionated way of how things should work.

Questions

0. Is anyone else interested in this? Is this even useful or desired?
1. Can any of the current hooks provided by the middleware architecture allow this to be implemented in a reasonable manner? Preferentially without changing dramatiq itself
2. If not, what do we need in order do implement it the "middleware way"?

I am trying to get an MVP working using the middleware approach, but so far nothing. Still getting familiar with dramatiq's internals

Note: I am starting the discussion here because the reddit seems to be abandoned

[davidt99]

Not sure if this is relevant but I implemented something similar now in #584