Monitor latest task runs per CLI

[nhoening]

New CLI function to monitor if tasks have run successfully and their latest run time is acceptable.

Alerts are sent to sentry and configurable email addresses.

[nhoening]

I decided to have one CLI task for multiple task checks. We also know how often we call these tasks in the cron environment, so it's not a bad choice to also pass this info (how long ago is acceptable) in the same environment.

To test:

• set your email address so you get the alert: MAIL_MONITORING_RECIPIENTS = "you@something.com"
• check if your table latest_task_run contains any entries you might want to test, add one if you want
• Call the CLI task, e.g.: flexmeasures monitor tasks --task get_historic_meter_data 5 --task run_forecasting_jobs 15 (where 5 and 15 are the amounts in minutes that are acceptable)

[Flix6x]

Missing changelog entry.

[Flix6x]

What's the now + acceptable_interval supposed to do? Doesn't seem documented.

[nhoening]

Good point. I got this from Bobby's code. It checks if the latest run time isn't too far from the monitor's now perspective.

Well, re-thinking it, I am actually not sure this makes sense. This date can practically happen to be in the future from the monitor's perspective, but if the clocks of monitoring server and task-executing server are out of sync, that should become a different kind of warning I guess. Then the allowed interval isn't a good measurement, as we don't know how much we are out of bounds.

AAAnyway ― in our case the same server is executing and monitoring so I'll not do that extra warning. I'll simply remove the future check.