try foreasting and scheduling jobs for up to one day (#198)

* try foreasting and scheduling for up to one day, after that let Redis remove the job if it needs memory * changelog entry * make it possible to control job's time to live in their queue
FlexMeasures · Oct 1, 2021 · cf3e260 · cf3e260
1 parent 664c958
commit cf3e260
Show file tree

Hide file tree

Showing 5 changed files with 22 additions and 0 deletions.
diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -24,6 +24,7 @@ Infrastructure / Support
 * To avoid databases from filling up with irrelevant information, only beliefs data representing *changed beliefs are saved*, and *unchanged beliefs are dropped* [see `PR #194 <http://www.github.com/SeitaBV/flexmeasures/pull/194>`_]
 * Monitored CLI tasks can get better names for identification [see `PR #193 <http://www.github.com/SeitaBV/flexmeasures/pull/193>`_]
 * Less custom logfile location, document logging for devs [see `PR #196 <http://www.github.com/SeitaBV/flexmeasures/pull/196>`_]
+* Keep forecasting and scheduling jobs in the queues for only up to one day [see `PR #198 <http://www.github.com/SeitaBV/flexmeasures/pull/198>`_]
 
 
 v0.6.1 | September XX, 2021

diff --git a/documentation/configuration.rst b/documentation/configuration.rst
@@ -207,6 +207,16 @@ Timezone in which the platform operates. This is useful when datetimes are being
 
 Default: ``"Asia/Seoul"``
 
+
+FLEXMEASURES_JOB_TTL
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Time to live for jobs (e.g. forecasting, scheduling) in their respective queue.
+
+A job that is passed this time to live might get cleaned out by Redis' memory manager.
+
+Default: ``timedelta(days=1)``
+
 FLEXMEASURES_PLANNING_TTL
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/flexmeasures/data/services/forecasting.py b/flexmeasures/data/services/forecasting.py
@@ -108,6 +108,11 @@ def create_forecasting_jobs(
                 custom_model_params=custom_model_params,
             ),
             connection=current_app.queues["forecasting"].connection,
+            ttl=int(
+                current_app.config.get(
+                    "FLEXMEASURES_JOB_TTL", timedelta(-1)
+                ).total_seconds()
+            ),
         )
         job.meta["model_search_term"] = model_search_term
         job.save_meta()

diff --git a/flexmeasures/data/services/scheduling.py b/flexmeasures/data/services/scheduling.py
@@ -64,6 +64,11 @@ def create_scheduling_job(
         ),
         id=udi_event_ea,
         connection=current_app.queues["scheduling"].connection,
+        ttl=int(
+            current_app.config.get(
+                "FLEXMEASURES_JOB_TTL", timedelta(-1)
+            ).total_seconds()
+        ),
         result_ttl=int(
             current_app.config.get(
                 "FLEXMEASURES_PLANNING_TTL", timedelta(-1)

diff --git a/flexmeasures/utils/config_defaults.py b/flexmeasures/utils/config_defaults.py
@@ -110,6 +110,7 @@ class Config(object):
     FLEXMEASURES_MENU_LISTED_VIEW_ICONS: Dict[str, str] = {}
     FLEXMEASURES_MENU_LISTED_VIEW_TITLES: Dict[str, str] = {}
     FLEXMEASURES_LP_SOLVER: str = "cbc"
+    FLEXMEASURES_JOB_TTL: timedelta = timedelta(days=1)
     FLEXMEASURES_PLANNING_HORIZON: timedelta = timedelta(hours=2 * 24)
     FLEXMEASURES_PLANNING_TTL: timedelta = timedelta(
         days=7