Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add random offset capabilities to desynchronize parallel workers #615

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

HonoreDB
Copy link

Hi, this is a feature request plus code to implement it--I'd be happy to adjust the DSL or implementation if you had different ideas for how to do this. The idea is to implement the pattern described here where when several nodes are running the same code, we avoid having their crons all fire at the same time by starting each job with a random sleep.

My thought was to make the sleep distribution configurable on a per-list level, so that you can have some jobs be fuzzier than others (have a daily job have an hourly offset, an hourly job have a 2 minute offset). So the schedule.rb could look like

every(1.hour).and_about(2.minutes) do
...
end

every(:sunday, :at => '12pm').and_about(1.hour) do
end

every '0 0 27-31 * *', random_offset: <% server_offset_seconds %> do
end

Jobs in the first list would fire anywhere in the first four minutes of the hour, while jobs in the second list will fire anywhere between 12:00 and 2:00 PM, with any second being equally likely.

@evmorov
Copy link

evmorov commented May 10, 2018

While it's not merged you can use:

every 1.day, at: (Time.zone.parse('10:00') + rand(0..600).seconds).localtime do

@jjb
Copy link

jjb commented Feb 1, 2020

the randomness will only happy whenever the crontab is generated - is that what you're going for?

a better approach might be to specify a list of jobs that should never run together and then using some sort of simple wrapper script, execute them serially.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants