Prevent jobs from stalling over-and-over

[bradvogel]

We ran into a situation where a single job kept stalling over-and-over because the work it happened to be doing was crashing the instance.

Although it's not the queue's fault, in thinking about this, it's probably quite common that jobs stall because of the work that particular job instance is doing (instead of outside factors such as random server restarts). If that work crashes the instance 100% of the time, it'll get stuck in a "stall loop".

I propose adding stallLimit as a parameter. If a job stalls more than this many times, it'll be moved to failed with the failure reason Job stalled more than stallLimit.

What do you think?

[bradvogel]

cc @skeggse

[LewisJEllis]

Yea, definitely want to do something along these lines; I've run into similar issues in certain situations.

Should stallLimit be a property of the queue, or of the job? I suppose we could have it be a property of both, where a job inherits its queue's stallLimit by default but that can be overridden.

[bradvogel]

I'm thinking the job since it's somewhat analogous to retries. However it makes the implementation a bit more complex since we have to store the stallLimit with each job alongside stallCount.

[LewisJEllis]

Observation/thought I had tonight - if we make the stallLimit a property of the queue, and we let absence of stallCount implicitly mean 0, then we can actually implement this without any extra data on typical jobs from what we have now. We would only need to add a stallCount property to jobs that have stalled and been retried at least once. Will complicate the checkStalledJobs Lua script, but probably worthwhile for the average case upside.