How to skip concurrent executions without failing the job

[dgaspar]

I've got a minutely recurring job that uses DisableConcurrentExecution(timeout:0) (0 retries), as only one worker should be processing it at a given time, specially if the job takes more than a minute to complete (it varies).

I don't want the "skipped" jobs to be flagged as failed on the dashboard, so I wanted to add a SkipConcurrentExecutionAttribute that would be almost identical to DisableConcurrentExecutionAttribute, except that it would try/catch around the lock acquisition, and swallow DistributedLockTimeoutException and set the job to DeletedState with the reason why.

But I can't do this properly, because GetResource() is private, and its implementation refers to other methods that are internal to the library.
$"{job.Type.ToGenericTypeString()}.{job.Method.Name}" seems to be the safe way of producing non-conflicting distributed lock names, but I can't refer to it.
Can we have in the lib a public method that generates the job's distributed lock name?

cf., https://discuss.hangfire.io/t/disableconcurentexecution-for-job-groups/1389/4

[dgaspar]

I just noticed that filterContext.BackgroundJob.Job.ToString() would give the same string as the internal TypeExtensions.cs and DisableConcurrentExecutionAttribute.cs (re)implementations would... problem solved.

[odinserj]

@dgaspar, you can tell recurring job scheduler to skip creation of the next recurring job, when previous one is still running, please see this gist: https://gist.github.com/odinserj/a6ad7ba6686076c9b9b2e03fcf6bf74e.

[dgaspar]

Thanks for the suggestion @odinserj. I noticed that if the worker crashes during execution, that "Running" row will remain in the Hash table. Will it prevent the job from being re-scheduled, until I manually delete that row? Or is there an expiration/cleanup that I'm missing? (btw, I'm using new SqlServerStorageOptions { SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5) }, if that matters)

[odinserj]

When a worker is terminated during the execution of a background job, that background job will be rescheduled automatically, because all message queues are transactional (at least the official ones, other storage implementations should act in the same way).