Support :backend as a queue_attribute

[johnnyshields]

The Problem

I would like to sub-divide my queues into various database tables. The reason is that I have many types of worker machines each scanning different queues. When Worker A's queue is full (e.g. 100,000 pending jobs) and Worker B's queue is empty, this can happen in my database due to Worker B scanning the table for jobs. (I'm using MongoDB, not SQL)

The Fix

DelayedJob currently supports backend and queue_attributes configurations:

# This is currently possible
Delayed::Worker.backend = MyBackendClass

# This is also currently possible
Delayed::Worker.queue_attributes = {
  payments:          { priority: -10 },
  sms:               { priority: -10 },
  push:              { priority: -5 },
  mailers:           { priority: -5, run_delay: 15 },
  ...
}

I would like to support :backend as a queue attribute, like this:

Delayed::Worker.queue_attributes = {
  payments:          { priority: -10 },
  sms:               { priority: -10 },
  push:              { backend: MyBackendClass, priority: -5 },
  mailers:           { backend: MyOtherBackendClass, priority: -5, run_delay: 15 },
  ...
}

(If not specified, it falls back to the base backend configuration)

This way, I can put Worker A and Worker B on separate backends (i.e. separate database tables) and avoid excessive table scanning. This can be thought of as a form of sharding / partitioning.