Add support for worker pools that exclude specific queues #1148
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Delayed Job already supports worker pools that draw from specific queues. This is useful to ensure those tasks are processed in a timely manner and do not starve. Coupling that with a pool that processes all jobs is a common practice. However, it is possible to create so many jobs for a specific that other jobs are delayed to an unacceptable degree. In this case we want to have a worker pool that only processes jobs that are not from that queue (or queues). There is currently no mechanism for this, hence this PR.
The changes allow worker pools to be named with a
!
prefix which would indicate that we want this pool to process any jobs except those in the given queues. It is intended to be coupled with a change at the adapter level which would read theexclude_specified_queues
flag and return the appropriate jobs based on whether the list of queues is intended as an inclusion or exclusion filter.See this PR for changes made to DJ active_record adapter to support this new feature collectiveidea/delayed_job_active_record#194.