Exports generation takes too much time and sometimes fails

[CharlesNepote]

1. gen_feeds_daily_off.sh is taking too much time (more than 4:30). Mirabelle updates (launched at 4:30 UTC) and Data quality Daily updates (launched at 7:00 UTC) have to wait hours to be launched.

2. Sometimes gen_feeds_daily_off.sh fails. In this case we have to relaunch it manually. Then I have also to relaunch mirabelle and Data quality Daily updates manually. With all those manual operations, sometimes the Data Quality Daily email can't be sent in time and data quality stats can't be updated (we're loosing stats for this day).

[alexgarel]

For point 2 I don't think we can do anything ! Things fails from time to time… that's normal !

For point 1 I proposed to test using the jsonl file which is generated some time before.

[CharlesNepote]

For point 2 I don't think we can do anything ! Things fails from time to time… that's normal !

We could test the exit code and at least send an email in case of failure.

For point 1 I proposed to test using the jsonl file which is generated some time before.

It would be great. But don't forget export_database.pl is also filtering some things such as bad chars, etc.

[alexgarel]

We could test the exit code and at least send an email in case of failure.

It should send an email if the service fail, what we should do is keep error returns and exit with them if one is non 0.