Configure queue per task

[adamzapasnik]

I don't think it's possible right now, is it? My reasoning for such option is:

1. some tasks run data migrations that can take hours/days and we're fine with them running in the lowest priority queue.
2. but some tasks are "crucial" and they run within a few minutes and we expect them to run immediately after being ran/enqueued via UI, thus we need to run them in the highest priority queue. An example: some data from a 3rd party API is out of sync, we'd like to sync it immediately via a maintenance task.

Does it make sense? Or maintenance_tasks wasn't really meant for such cases?

Tasks

Beta Give feedback

No tasks being tracked yet.

[nvasilevski]

Does it make sense?

It does! But one thing I would do is to look on the issue at a higher level and consider allowing MaintenanceTasks.job to be configured on per-task basis. The reason for this is that queue is a concept from background processing and not from maintenance tasks itself so it would be better for the gem not to know about queues as nothing really enforces the job that executes tasks to even have a queue.
Despite of the gem heavily mentioning Job concept and providing TaskJob as basically the only native way to execute a task, I still see it as an "executor" abstraction. By that I mean that technically we still can configure MaintenanceTasks.job = to be "MyCustomClassThatIsTotallyNotAJob" and the gem will work as long as MyCustomClassThatIsTotallyNotAJob string can be constantized, respond to new, instances of the class respond to enqueue(run) and the class knows how to handle run. Using a non-job based executor sounds unnecessary but still possible design-wise as long as the executor is abstracted well enough.

To summarize, to me it makes total sense if we look on it from the per-task executor point of view. If a single task can specify custom executor (a job, for most cases) then this task can be executed within a custom context, whether it's a separate queue, or specific parallel execution constraints, or custom retries policy, the maintenance_task gem itself won't need to know about these background-execution concepts

[adamzapasnik]

@nvasilevski hey, thanks. Definitely agree with you, this would be the best solution and personally preferred. I think the only reason I didn't mention it was because I saw this #504 and it seems that it such option was available but then reverted?

[nvasilevski]

because I saw this #504 and it seems that it such option was available but then reverted?

Oh, to be honest I didn't know we have explored it. As far as I can see we decided not to go this way as it obligates maintenance tasks user to always explicitly specify a job which is indeed not the best UX. However I think we can revise the approach if we keep the MaintenanceTasks.job as the default and only allow specifying with_job_class if necessary.

Update:
We actually had the global value as the default so perhaps we should just revisit the decision.

maintenance_tasks/app/models/maintenance_tasks/task.rb

Line 23 in 1e0989c

class_attribute :job_class, default: -> { MaintenanceTasks.job.constantize }

[adamzapasnik]

🤞 I'd definitely be interested, otherwise I may end up with my own job definition and a dynamic queue_as

[github-actions]

This issue has been marked as stale because it has not been commented on in two months.
Please reply in order to keep the issue open. Otherwise, it will close in 14 days.
Thank you for contributing!