Docker compose redis worker (#455)

* also allow warnable config settigs (mail, redis) to be set as env vars Signed-off-by: Nicolas Höning <nicolas@seita.nl> * allow the CLI command for schedule creation to also simply add jobs to the queue (--as-jobs) Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add redis and worker node to docker compose setup Signed-off-by: Nicolas Höning <nicolas@seita.nl> * update pre-commit tooling to Python 3.9 Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Document worker queues in docker-compose; various improvements in other parts, e.g. update and correct usage of CLI forecasting, redis as independent page under hosting Signed-off-by: Nicolas Höning <nicolas@seita.nl> * run black Signed-off-by: Nicolas Höning <nicolas@seita.nl> * changelog entry Signed-off-by: Nicolas Höning <nicolas@seita.nl> * changelog entry Signed-off-by: Nicolas Höning <nicolas@seita.nl> * small doc corrections from review Signed-off-by: Nicolas Höning <nicolas@seita.nl> * fix parameter names in example Signed-off-by: Nicolas Höning <nicolas@seita.nl> * improve handling of new worker names - random if not given, add iterated suffix number if taken Signed-off-by: Nicolas Höning <nicolas@seita.nl> * allow to clear failed queue and add show-queues command Signed-off-by: Nicolas Höning <nicolas@seita.nl> * improve reporting on added jobs in CLI Signed-off-by: Nicolas Höning <nicolas@seita.nl> * fix clearing the registry of failed jobs; improve a variable name and ordering of CLI args Signed-off-by: Nicolas Höning <nicolas@seita.nl> * small improvement to UX of jobqueue-handling CLI Signed-off-by: Nicolas Höning <nicolas@seita.nl>
FlexMeasures · Jul 15, 2022 · d428472 · d428472
1 parent 55589f3
commit d428472
Show file tree

Hide file tree

Showing 17 changed files with 422 additions and 163 deletions.
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -18,6 +18,16 @@ services:
       POSTGRES_PASSWORD: fm-dev-db-pass
     volumes:
       - ./ci/load-psql-extensions.sql:/docker-entrypoint-initdb.d/load-psql-extensions.sql
+  queue-db:
+    image: redis
+    restart: always
+    command: redis-server --loglevel warning --requirepass fm-redis-pass
+    expose:
+      - 6379
+    volumes:
+      - redis-cache:/data
+    environment:
+     - REDIS_REPLICATION_MODE=master
   server:
     build:
       context: .
@@ -47,6 +57,25 @@ services:
       bash -c "flexmeasures db upgrade
       && flexmeasures add toy-account --name 'Docker Toy Account'
       && gunicorn --bind 0.0.0.0:5000 --worker-tmp-dir /dev/shm --workers 2 --threads 4 wsgi:application"
+  worker:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    depends_on:
+      - dev-db
+      - queue-db
+    restart: on-failure
+    environment:
+      SQLALCHEMY_DATABASE_URI: "postgresql://fm-dev-db-user:fm-dev-db-pass@dev-db:5432/fm-dev-db"
+      FLEXMEASURES_REDIS_URL: queue-db
+      FLEXMEASURES_REDIS_PASSWORD: fm-redis-pass
+      SECRET_KEY: notsecret
+      FLASK_ENV: development
+      LOGGING_LEVEL: INFO 
+    volumes:
+      # a place for config and plugin code
+      - ./flexmeasures-instance/:/usr/var/flexmeasures-instance/:ro
+    command: flexmeasures jobs run-worker --name flexmeasures-worker --queue forecasting\|scheduling
   test-db:
     image: postgres
     expose:
@@ -60,4 +89,6 @@ services:
       - ./ci/load-psql-extensions.sql:/docker-entrypoint-initdb.d/load-psql-extensions.sql
 
 volumes:
-  flexmeasures-instance:
+  redis-cache:
+    driver: local
+  flexmeasures-instance:
diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -9,7 +9,8 @@ New features
 -------------
 * The asset page now shows the most relevant sensor data for the asset [see `PR #449 <http://www.github.com/FlexMeasures/flexmeasures/pull/449>`_]
 * Individual sensor charts show available annotations [see `PR #428 <http://www.github.com/FlexMeasures/flexmeasures/pull/428>`_]
-* Collapsible sidepanel (hover/swipe) used for date selection on sensor charts, and various styling improvements [see `PR #447 <http://www.github.com/FlexMeasures/flexmeasures/pull/447>`_ and `PR #448 <http://www.github.com/FlexMeasures/flexmeasures/pull/448>`_]
+* Add CLI command ``flexmeasures jobs show-queues`` [see `PR #455 <http://www.github.com/FlexMeasures/flexmeasures/pull/455>`_]
+* Collapsible side-panel (hover/swipe) used for date selection on sensor charts, and various styling improvements [see `PR #447 <http://www.github.com/FlexMeasures/flexmeasures/pull/447>`_ and `PR #448 <http://www.github.com/FlexMeasures/flexmeasures/pull/448>`_]
 * Switched from 12-hour AM/PM to 24-hour clock notation for time series chart axis labels [see `PR #446 <http://www.github.com/FlexMeasures/flexmeasures/pull/446>`_]
 * Get data in a given resolution [see `PR #458 <http://www.github.com/FlexMeasures/flexmeasures/pull/458>`_]
 
@@ -20,7 +21,9 @@ Bugfixes
 
 Infrastructure / Support
 ----------------------
+* Docker compose stack now with Redis worker queue [see `PR #455 <http://www.github.com/FlexMeasures/flexmeasures/pull/455>`_]
 * Allow access tokens to be passed as env vars as well [see `PR #443 <http://www.github.com/FlexMeasures/flexmeasures/pull/443>`_]
+* Queue workers can get initialised without a custom name and name collisions are handled [see `PR #455 <http://www.github.com/FlexMeasures/flexmeasures/pull/455>`_]
 
 
 v0.10.1 | June XX, 2022
@@ -100,7 +103,7 @@ New features
 * Add CLI option to specify custom strings that should be interpreted as NaN values when reading in time series data from CSV [see `PR #357 <http://www.github.com/FlexMeasures/flexmeasures/pull/357>`_]
 * Add CLI commands ``flexmeasures add sensor``, ``flexmeasures add asset-type``, ``flexmeasures add beliefs`` (which were experimental features before) [see `PR #337 <http://www.github.com/FlexMeasures/flexmeasures/pull/337>`_]
 * Add CLI commands for showing organisational structure [see `PR #339 <http://www.github.com/FlexMeasures/flexmeasures/pull/339>`_]
-* Add a CLI command for showing time series data [see `PR #379 <http://www.github.com/FlexMeasures/flexmeasures/pull/379>`_]
+* Add CLI command for showing time series data [see `PR #379 <http://www.github.com/FlexMeasures/flexmeasures/pull/379>`_]
 * Add CLI command for attaching annotations to assets: ``flexmeasures add holidays`` adds public holidays [see `PR #343 <http://www.github.com/FlexMeasures/flexmeasures/pull/343>`_]
 * Add CLI command for resampling existing sensor data to new resolution [see `PR #360 <http://www.github.com/FlexMeasures/flexmeasures/pull/360>`_]
 * Add CLI command to delete an asset, with its sensors and data. [see `PR #395 <http://www.github.com/FlexMeasures/flexmeasures/pull/395>`_]

diff --git a/documentation/cli/commands.rst b/documentation/cli/commands.rst
@@ -86,6 +86,7 @@ of which some are referred to in this documentation.
 
 ================================================= =======================================
 ``flexmeasures jobs run-worker``                  Start a worker process for forecasting and/or scheduling jobs.
+``flexmeasures jobs show queues``                 List job queues.
 ``flexmeasures jobs clear-queue``                 Clear a job queue.
 ================================================= =======================================
 

diff --git a/documentation/dev/docker-compose.rst b/documentation/dev/docker-compose.rst
@@ -10,6 +10,8 @@ For this, we assume you are in the directory housing ``docker-compose.yml``.
 
 .. note:: The minimum Docker version is 17.09 and for docker-compose we tested successfully at version 1.25. You can check your versions with ``docker[-compose] --version``.
 
+.. note:: The command might also be ``docker compose`` (no dash), for instance if you are using `Docker Desktop <https://docs.docker.com/desktop>`_.
+
 Build the compose stack
 ------------------------
 
@@ -19,11 +21,12 @@ Run this:
 
     docker-compose build
 
-This pulls the images you need, and re-builds the FlexMeasures one from code. If you change code, re-running this will re-build that image.
+This pulls the images you need, and re-builds the FlexMeasures ones from code. If you change code, re-running this will re-build that image.
 
 This compose script can also serve as an inspiration for using FlexMeasures in modern cloud environments (like Kubernetes). For instance, you might want to not build the FlexMeasures image from code, but simply pull the image from DockerHub.
 
-.. todo:: This stack runs FlexMeasures, but misses the background worker aspect. For this, we'll add a redis node and one additional FlexMeasures node, which runs a worker as entry point instead (see `issue 418 <https://github.com/FlexMeasures/flexmeasures/issues/418>`_).
+If you wanted, you could stop building from source, and directly use the official flexmeasures image for the server and worker container
+(set ``image: lfenergy/flexmeasures`` in the file ``docker-compose.yml``).
 
 
 Run the compose stack
@@ -35,38 +38,121 @@ Start the stack like this:
 
     docker-compose up
 
-You can see log output in the terminal, but ``docker-compose logs`` is also available to you.
+.. warning:: This might fail if ports 5000 (Flask) or 6379 (Redis) are in use on your system. Stop these processes before you continue.
 
 Check ``docker ps`` or ``docker-compose ps`` to see if your containers are running:
 
 
 .. code-block:: console
 
     ± docker ps
-    CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                    PORTS                    NAMES
-    dda1a8606926        flexmeasures_server   "bash -c 'flexmeasur…"   43 seconds ago      Up 41 seconds (healthy)   0.0.0.0:5000->5000/tcp   flexmeasures-server-1
-    27ed9eef1b04        postgres              "docker-entrypoint.s…"   2 days ago          Up 42 seconds             5432/tcp                 flexmeasures-dev-db-1
-    90df2065e08d        postgres              "docker-entrypoint.s…"   2 days ago          Up 42 seconds             5432/tcp                 flexmeasures-test-db-1
-
+    CONTAINER ID   IMAGE                 COMMAND                  CREATED          STATUS                             PORTS                    NAMES
+    beb9bf567303   flexmeasures_server   "bash -c 'flexmeasur…"   44 seconds ago   Up 38 seconds (health: starting)   0.0.0.0:5000->5000/tcp   flexmeasures-server-1
+    e36cd54a7fd5   flexmeasures_worker   "flexmeasures jobs r…"   44 seconds ago   Up 5 seconds                       5000/tcp                 flexmeasures-worker-1
+    c9985de27f68   postgres              "docker-entrypoint.s…"   45 seconds ago   Up 40 seconds                      5432/tcp                 flexmeasures-test-db-1
+    03582d37230e   postgres              "docker-entrypoint.s…"   45 seconds ago   Up 40 seconds                      5432/tcp                 flexmeasures-dev-db-1
+    792ec3d86e71   redis                 "docker-entrypoint.s…"   45 seconds ago   Up 40 seconds                      0.0.0.0:6379->6379/tcp   flexmeasures-queue-db-1
 
-The FlexMeasures container has a health check implemented, which is reflected in this output and you can see which ports are available on your machine to interact.
 
-You can use ``docker-compose logs`` to look at output. ``docker inspect <container>`` and ``docker exec -it <container> bash`` can be quite useful to dive into details. 
+The FlexMeasures server container has a health check implemented, which is reflected in this output and you can see which ports are available on your machine to interact.
 
-.. todo:: We should provide a way to test that this is working, e.g. a list of steps. Document this, but also include that in our tsc/Release list (as a test step to see if Dockerization still works, plus a publish step for the released version).
+You can use the terminal or ``docker-compose logs`` to look at output. ``docker inspect <container>`` and ``docker exec -it <container> bash`` can be quite useful to dive into details. 
+We'll see the latter more in this tutorial.
 
 
 Configuration
 ---------------
 
-You can pass in your own configuration (e.g. for MapBox access token, or db URI, see below) like we described above for running a container: put a file ``flexmeasures.cfg`` into a local folder called ``flexmeasures-instance``.
+You can pass in your own configuration (e.g. for MapBox access token, or db URI, see below) like we described in :ref:`docker_configuration` ― put a file ``flexmeasures.cfg`` into a local folder called ``flexmeasures-instance`` (the volume should be already mapped).
 
 
 Data
 -------
 
 The postgres database is a test database with toy data filled in when the flexmeasures container starts.
-You could also connect it to some other database, by setting a different ``SQLALCHEMY_DATABASE_URI`` in the config. 
+You could also connect it to some other database (on your PC, in the cloud), by setting a different ``SQLALCHEMY_DATABASE_URI`` in the config. 
+
+
+Seeing it work: Running the toy tutorial
+--------------------------------------
+
+A good way to see if these containers work well together, and maybe to inspire how to use them for your own purposes, is the :ref:`tut_toy_schedule`.
+The server container already creates the toy account when it starts. We'll now run the rest of that tutorial, with one twist at the end, when we create the battery schedule.
+
+Let's go into the worker container:
+
+.. code-block:: console
+
+    docker exec -it flexmeasures-worker-1 bash
+
+There, we add the price data, as described in :ref:`tut_toy_schedule_price_data`. Create the prices and add them to the FlexMeasures DB in the container's bash session.
+
+Next, we put a scheduling job in the worker's queue. This uses the Redis container ― the toy tutorial isn't doing that (the difference is ``--as-job``).
+
+.. code-block:: console
+
+    flexmeasures add schedule --sensor-id 2 --optimization-context-id 3 \
+        --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \
+        --roundtrip-efficiency 90% --as-job
+
+We should now see in the output of ``docker logs flexmeasures-worker-1`` something like the following:
+
+.. code-block:: console
+
+    Running Scheduling Job d3e10f6d-31d2-46c6-8308-01ede48f8fdd: <Sensor 2: charging, unit: MW res.: 0:15:00>, from 2022-07-06 07:00:00+01:00 to 2022-07-06 19:00:00+01:00
+
+So the job had been queued in Redis, was then picked up by the worker process, and the result should be in our SQL database container. Let's check!
+
+We'll not go into the server container this time, but simply send a command:
+
+.. code-block:: console
+
+    TOMORROW=$(date --date="next day" '+%Y-%m-%d')
+    docker exec -it flexmeasures-server-1 bash -c "flexmeasures show beliefs --sensor-id 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H"
+
+The charging/discharging schedule should be there:
+
+.. code-block:: console
+
+    ┌────────────────────────────────────────────────────────────┐
+    │   ▐                      ▐▀▀▌                           ▛▀▀│ 
+    │   ▞▌                     ▞  ▐                           ▌  │ 0.4MW
+    │   ▌▌                     ▌  ▐                          ▐   │ 
+    │  ▗▘▌                     ▌  ▐                          ▐   │ 
+    │  ▐ ▐                    ▗▘  ▝▖                         ▐   │ 
+    │  ▞ ▐                    ▐    ▌                         ▌   │ 0.2MW
+    │ ▗▘ ▐                    ▐    ▌                         ▌   │ 
+    │ ▐  ▝▖                   ▌    ▚                        ▞    │ 
+    │▀▘───▀▀▀▀▀▀▀▀▀▀▀▀▀▀▌────▐─────▝▀▀▀▀▀▀▀▀▜─────▐▀▀▀▀▀▀▀▀▀─────│ 0MW
+    │                   ▌    ▞              ▐    ▗▘              │ 
+    │                   ▚    ▌              ▐    ▐               │ 
+    │                   ▐   ▗▘              ▝▖   ▌               │ -0.2MW
+    │                   ▐   ▐                ▌   ▌               │ 
+    │                   ▐   ▐                ▌  ▗▘               │ 
+    │                    ▌  ▞                ▌  ▐                │ 
+    │                    ▌  ▌                ▐  ▐                │ -0.4MW
+    │                    ▙▄▄▌                ▐▄▄▞                │ 
+    └────────────────────────────────────────────────────────────┘
+            10           20           30          40
+                         ██ charging
+
+Like in the original toy tutorial, we can also check in the server container's `web UI <http://localhost:5000/sensors/2/>`_ (username is "toy-user@flexmeasures.io", password is "toy-password"):
+
+.. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/sensor-data-charging.png
+    :align: center
+
+
+Scripting with the Docker stack
+----------------------------------
+
+A very important aspect of this stack is if it can be put to interesting use.
+For this, developers need to be able to script things ― like we just did with the toy tutorial.
+
+Note that instead of starting a console in the containers, we can also send commands to them right away.
+For instance, we sent the complete ``flexmeasures show beliefs`` command and then viewed the output on our own machine.
+Likewise, we send the ``pytest`` command to run the unit tests (see below).
+
+Used this way, and in combination with the powerful list of :ref:`cli`, this FlexMeasures Docker stack is scriptable for interesting applications and simulations!
 
 
 Running tests