Add data integrity check for cron task

[tintinthong]

This adds a data integrity check that checks when a graphile worker task was last fired. If the task runs every 5 minutes, we give it an allowances of 5*3 = 15 minutes. Right now every task uses 3x multiplier but this tolerance can be adjusted for a specific task if needed

I didn't use prisma. Well because graphile_worker tables exist in the graphile_worker namespace and typically prisma schema file only allow to specify only one namespace

[jurgenwerk]

Overall looks good but would be really good to explore the possibilities of getting the cron job frequency dynamically so that we reduce the chance of getting out of sync with checks in case someone adjusts the tasks frequency.

Also is there a better word to label a task as a task that didn't run but it was supposed to? You used "stopped" but that can also indicate someone stopped it intentionally. I'd prefer maybe "lagging"

[tintinthong]

@jurgenwerk this was perhaps more involved than I wanted to. I had to parse the way the graphile worker encoded ParsedCronItem. Which is their object encoding of the original crontab configuration. But in any case I created classes that would correspond to a conservative understanding of what each ParsedCronItem meant

export const CRON_TAB_STRING: string = [
  '0 5 * * * remove-old-sent-notifications ?max=5',
  '*/5 * * * * print-queued-jobs',
  '*/5 * * * * execute-scheduled-payments',
  ...(config.get('rewardsIndexer.enabled') ? ['*/10 * * * * check-reward-roots'] : []),
].join('\n');

-> 

ParsedCronItem[]

This exercise also made me realise that I was perceiving remove-old-sent-notifications wrongly. The task is actually sent daily at 5am UTC. Previously, I thought it was just an interval type configuration -- which is wrong. I had to make adjustments

[jurgenwerk]

All the logic in this function is a bit hard to follow for me. I think it went a bit too far in complexity

Have you considered parsing the cron string, get the interval for each entry, and simply compare whether it has been executed in the last interval + some padding? Or am I missing something?

Looks like ChatGPT can help to spit out the function for that:

function calculateCronInterval(expression) {
    const fields = expression.split(' ');
    const [minute, hour, dayOfMonth, month, dayOfWeek] = fields;

    function processField(field, min, max) {
        if (field === '*') {
            return max - min + 1;
        }
        const values = field.split(',').map(value => parseInt(value));
        return Math.min(...values.map(value => max - min + 1));
    }

    const minutesInterval = processField(minute, 0, 59);
    const hoursInterval = processField(hour, 0, 23);
    const daysInterval = processField(dayOfMonth, 1, 31);
    const monthsInterval = processField(month, 1, 12);
    const weekdaysInterval = processField(dayOfWeek, 0, 6);

    // Calculate the overall interval in minutes
    const overallInterval = minutesInterval * hoursInterval * daysInterval * monthsInterval * weekdaysInterval;

    return overallInterval;
}

const cronExpression = "*/15 * * * *"; // Example cron expression
const intervalInMinutes = calculateCronInterval(cronExpression);
console.log(`Interval in minutes: ${intervalInMinutes}`);

[tintinthong]

Noted. I have over-abstracted at the expense of clarity. I can just parse the crontab string and simplify.

[tintinthong]

I have simplified everything. This function above is wrong.

I think we prob don't want to model ENTIRE graphile workers crontab string.

The main case we consider is just the difference */15 * * * * and 0 5 * * *. Former is every 15 minutes. Latter is every day at 5am UTC. For our purposes, we just perceive these two cases as every 15 minutes and every 24*60 minutes (1 day) from last execution.

That means that I write the code CONSERVATIVELY in that if you write something */15 15 * * *. Which means every 15 minutes within the 15th hour of the day. I expect to just not parse this and error out. This keeps our code exact but incomplete.

[tintinthong]

I wrote test here