FEATURE: Adding Recurring Job Support to Delayed_Job
Delayed_job provides an ideal platform for providing support for distributed job scheduling, however it currently lacks a way for repeat or recurring jobs to run. Adding support for recurring jobs will provide developers with several benefits:
- Move ‘cron’ type operational configs into application code
- Ensure redundancy by allowing recurring jobs to run anywhere workers are running
- Ensure recurring jobs don’t run in multiple locations simultaneously (overlap)
While I’m sure some cron hackers have solved this problem in a variety of ways, it’s a non trivial challenge to ensure that regular, scheduled jobs in a redundant cluster run only once (on schedule), and still run if the cron server(s) is down. Recurring job support in delayed_job can solve both of these issues (redundancy and Singleton execution) in clustered environments cleanly.
Other distributed scheduling implementations
- Quartz Scheduler (Java): http://www.quartz-scheduler.org/
There are at least two forks that add recurring job scheduling to delayed_job:
- http://github.com/briandemant/delayed_job (not actively developed)
- http://github.com/terrbear/delayed_job (not actively developed)
- easy to understand API for usage
- continued support for run_at semantics
- addition of recurring semantics: 5.minutes, 1.day, 1.week, etc.
- ability to ensure that a set of scheduled, recurring jobs are properly configured and running in the queue
- error handling and recovery, compatible with the delayed_job design
- Run single job right now
- Run single job at specific time in the future
- Schedule recurring job beginning first run immediately
- Schedule recurring job beginning first run at specific time in the future
- More variations??
Any job that has a non-nil :run_every entry is by default a recurring job. :run_every should be applied when jobs are run to ensure there is no time drift. For example:
:run_at => Time(8.AM), :run_every => 1.day
on execution, would ensure that the next run_at time is set for 8AM the next day, not 8AM+job_execution_time – that would be considered drift
Implement recurring functionality as a Mixin/Extension, so code can be clearly delinated from the mainline delayed_job code
Notes
Notes
- Successful job requeues
- Unsuccessful job doesn’t requeue
- Failed job (after N attempts) does requeue
- dj_remixes http://github.com/markbates/dj_remixes While the documentation still needs a little work, this gem adds recurring job support to DJ in a really clean way. Plus, since it defines the ‘run every’ in the worker class, it can be adjusted to fit the situation programmatically, instead of just a rigid time interval. Currently though, the gem only supports ActiveRecord backends.
- Delayed::Cron::Job https://github.com/codez/delayed_cron_job is an extension to Delayed::Job that allows you to set cron expressions for your jobs to run repeatedly.
- delayed_job_recurring https://github.com/amitree/delayed_job_recurring is an extension to Delayed::Job that allows you to define recurring jobs.