Background workers RFC

[zerolab]

Hello fine Wagtail community,

The Wagtail performance sub-team has an RFC about introducing a background workers mechanism into core - wagtail/rfcs#72
The aim is to allow deferring expensive tasks without making assumptions about anyone's setup.

We welcome all feedback

[jams2]

Hi,

great proposal.

Regarding the example API of the BaseJobBackend, I find it curious to have scheduled/repeated tasks defined in terms of cron. I understand that this is how similar functionality is defined in some (many?) other libraries, but I think it's worth considering that not all Wagtail developers will be familiar with the concept of cronjobs and the crontab syntax.

From that perspective, this kind of code -
background.cron(do_a_task, args=[], kwargs={}, schedule="* 5 * * *")
is pretty unintuitive. You could possibly rename the cron method to something more descriptive.

In the spirit of making things explicit, I would also suggest you consider only allowing a human readable format for the execution schedule. It seems odd to me to use the crontab syntax when the cron daemon isn't involved at all (unless I've misread the proposal), and where other language constructs - that will also allow for better support in IDEs/editors - exist. Some examples:

background.schedule(do_a_task, args=[], kwargs={}, when={"hour": 5})
background.repeat(do_a_task, args=[], kwargs={}, hour=5)
background.repeat(task, task_args=[], task_kwargs={}, schedule=TaskSchedule(hour=5))

I appreciate that the proposal is in the early stages so apologies if I'm putting the horse before the cart :)

[RealOrangeOne]

Looking back on the proposal, I strongly agree with your idea. The cron syntax is more concise, but that's not really a useful metric here. People also are more likely to already understand a cron expression, but in this case if it's human-readable enough that won't matter!

However, this would lose the ability to run tasks at a given interval (.repeat in your above example). I wonder how this should work? Should users instead be forced to manually schedule tasks as needed (eg to run a task every 10 minutes, they schedule 6 tasks for minutes 0, 10, 20, 30, 40 and 50 minutes past the hour? This API makes that incredibly simple). It's explicit, however feels rather verbose. Tasks could instead automatically schedule their next task, however if a task fails in an unexpected way, the schedule may be lostl

I'll admit, I hadn't considered how interval-based tasks would work, which are a part of the cron syntax I had forgotten about.

[tbrlpld]

Not sure how you want to invoke the tasks, but maybe this would be helpful:

https://pypi.org/project/schedule/

I guess this would require a long running background process. The syntax is nice though 😬

[jams2]

I'll admit, I hadn't considered how interval-based tasks would work, which are a part of the cron syntax I had forgotten about.

I had forgotten about interval tasks as well. I think my suggestion would still work - you could use .schedule for scheduled regular execution, and .repeat for interval based execution. It may depend on how you're scheduling the tasks behind the scenes.

Another way would be to have the .schedule method take a TaskSchedule-like object, which could provide a method for doing the actual scheduling/generating. Loose example follows:

class TaskSchedule:
    def get_schedule(self):
        ... # Non-interval task scheduling

class IntervallicTaskSchedule:
    def get_schedule(self):
        ... # e.g. every 10 minutes

class MyBackend(BaseJobBackend):
    def schedule(self, task, task_args, task_kwargs, when: TaskSchedule):
        ... # Do something with when.get_schedule

But having said all that, maybe leaning into cron-isms is the simplest way to go about it :)

[salty-ivy]

pardon me for the silly questions though, bit unclear about whether we would creating a backend API interface to connect to pre-existing job scheduler modules like celery, rq, huey etc. we would also be creating our own schedulers as well (inspired by other schedulers)?

EDIT: as i understand, ImmediateBackend will be wagtails default backend. that will use current database and ORM.
but background jobs runner would be our own simple multithreading implementation or would be using some other third party package behind the back.

[RealOrangeOne]

ImmediateBackend would be the default, as it means that once implemented, functionality wouldn't change. ImmediateBackend wouldn't have a need for job storage, so wouldn't touch the ORM - it'd just execute things immediately in the same thread.

The main work behind this is simply creating an interface for external task processors (Celery, RQ etc). I doubt we'd necessary ship one built-in that's fully custom, but have some first-party maintained ones for the common task processors is definitely likely. But then if someone wants to write one for their favourite processor, they absolutely can.

Once there's a first-party interface, it can start being used both downstream and inside Wagtail for intensive tasks

[salty-ivy]

alright, thanks a lot for the clarity. got the whole architecture clear.

[salty-ivy]

Is this going to be contrib entirely?

[salty-ivy]

if so, I have a doubt, as far as I understand we will be permanently using the scheduling interface in wagtail internally to shift intensive actions in the background, backword compatibility will be maintained by ImmediateBackend.

suppose we are using our background object to do that.
would it be possible to use this object application wide? as we would not be sure if the contrib module will be added in INSTALLED_APPS or not.

is it possible to directly use the contrib code in core internally?

[thibaudcolas]

It’s technically possible, but generally the point of marking something as "contrib" is to convey it’s optional.

The background tasks support will become quite a fundamental place of infrastructure for existing core apps, so I’d expect it won’t be a contrib module.

[RealOrangeOne]

On reflection, I think you're right. contrib implies it's an optional extra. Task functionality actually isn't optional, the optionality comes from which backend you choose. Having components of core Wagtail call out to contrib packages feels like an anti-pattern to me. Given that, I suspect it would need to be in core (wagtail.tasks in the post-3.0 vernacular).

[salty-ivy]

That's what I was having a doubt about!