BUG: API reset can cause evaluations to get stuck with RUNNING status.

[czaloom]

valor version checks

• I have confirmed this bug exists on the latest version of valor.

Reproducible Example

Restart server mid-computation but retain the database.

Issue Description

Evaluations in Valor are performed using a combination of python code in the backend and some postgresql queries. If the API is restarted mid-computation a desynchronization occurs between the API and the database.

Expected Behavior

Any evaluation stuck in a RUNNING status after a timeout threshold should be set to the FAILED status.

[ntlind]

Moving this one to backlog. The best fix here might be implementing a message broker.