Feature request: Immediate executing of new jobs

[ukolovda]

Is your feature request related to a problem? Please describe.
Sometimes I need run job "as soon as possible", not waiting up to 60 seconds to refetch chains.

Describe the solution you'd like
Scheduler should load new chain and tasks as soon as it was append to table.
My be, we can use notifications from table triggers (after commit of transaction).

Describe alternatives you've considered
Decrease refetch interval in source code, but this will increase database loading.

[pashagolub]

Hello.

Have you tried timetable.notify_chain_start(chain_id BIGINT, worker_name TEXT) function? I believe it's fast enough for the most of requirements.

I'm not sure we should load as soon as possible. pg_timetable is a scheduler not a real-time queue processing software. For async processing, PGQ is the first what I'm thinking about. But there are a lot more nowadays. Should check the situation.

[msocias]

A possible expansion of pg_timetable beyond scheduling would be to add a "staging" table, to which chains get added via the scheduler. This new table could act as a queue, with priority settings, minimum requirements of workers, etc.

The drawback is to try to make pg_timetable into something more than a scheduler (adding queue capabilities). But the advantage is that pretty much everything else is in place for having a PostgreSQL-based queue for async tasks (including shell scripts), logging, etc.

[pashagolub]

Do you have in mind the definition for such a table?

[msocias]

Pavlo, I think you already have this queue table in the form of the timetable.active_chain table. Just two thoughts:

1. About Dmitry's comment, instead of "forcing" a scheduled chain to start now (via the timetable.notify_chain_start function), it is probably cleaner (only in my opinion) to create a new function like timetable.add_active_chain, which would add a chain directly to the active_chains table.

2. If timetable.add_active_chain were implemented, it could be interesting to flesh out the timetable.active_chain table a little bit more by adding the created_at column. This would allow knowing how long chains have been waiting for execution.

queue_id       BIGSERIAL     PRIMARY KEY,
created_at     TIMESTAMPTZ   DEFAULT now(),
started_at     TIMESTAMPTZ,
chain_id       BIGINT        timetable.chain(chain_id),
chain_name     TEXT          NOT NULL UNIQUE,

[pashagolub]

Sorry for delay in asnswer. timetable.active_chain table used only for monitoring what chains are active at the moment. One shouldn't add anything there. It will just produce trash records. Only pg_timetable instances suppose to add records there

[msocias]

Thank you Pavlo. Makes sense.