Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
First version of Dockerfile & docker-compose (#416)
* First version of Dockerfile & docker-compose, including a health check in the FM API Signed-off-by: Nicolas Höning <nicolas@seita.nl> * fix inability to call flexmeasures command in container (had to use flask), by not installing in editable mode Signed-off-by: Nicolas Höning <nicolas@seita.nl> * load the instance directory as volume, which allows to pass a configuration file; use flexmeasures run as default command for now Signed-off-by: Nicolas Höning <nicolas@seita.nl> * document better & correctly how to mount config file into a FlexMeasures container, clean up docs Signed-off-by: Nicolas Höning <nicolas@seita.nl> * copy less unneeded files into FlexMeasures image Signed-off-by: Nicolas Höning <nicolas@seita.nl> * run the FlexMeasures image via Gunicorn Signed-off-by: Nicolas Höning <nicolas@seita.nl> * install gunicorn early on Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add changelog entries Signed-off-by: Nicolas Höning <nicolas@seita.nl> * mention the Docker image on index and in toy tutorial section Signed-off-by: Nicolas Höning <nicolas@seita.nl> * move the note about using Docker for the tutorial a little lower Signed-off-by: Nicolas Höning <nicolas@seita.nl> * install solver as well Signed-off-by: Nicolas Höning <nicolas@seita.nl> * document how at this point one can run tests inside a FlexMeasures container Signed-off-by: Nicolas Höning <nicolas@seita.nl> * load sql extensions into postgres service Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add another note to the preliminary testing setup Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Run tests easily in Docker: add test-db service in compose stack, and a way to tell FlexMeasures to use a different database URI than the standard one when testing Signed-off-by: Nicolas Höning <nicolas@seita.nl> * improve documentation, from review comments Signed-off-by: Nicolas Höning <nicolas@seita.nl> * fix typo Signed-off-by: Nicolas Höning <nicolas@seita.nl> * lazy loading in SensorIdField (fixes flexmeasures add schedule) Signed-off-by: Nicolas Höning <nicolas@seita.nl> * docs: add FLASK_ENV=development to docker run call Signed-off-by: Nicolas Höning <nicolas@seita.nl> * split docker and docker-compose docs (first is relevant for installing/deploying), the latter more for development. Also fix small errors and give better info on WSGI scripts. Signed-off-by: Nicolas Höning <nicolas@seita.nl> * fix regression in not attaching account to assets correctly in toy tutorial Signed-off-by: Nicolas Höning <nicolas@seita.nl> * separate installation instructions for toy tutorial into Docker and non-Docker Signed-off-by: Nicolas Höning <nicolas@seita.nl> * some text improvements Signed-off-by: Nicolas Höning <nicolas@seita.nl> * correct the link in coverage badge, to report on main branch Signed-off-by: Nicolas Höning <nicolas@seita.nl> * switch to lfenergy/flexmeasures Signed-off-by: Nicolas Höning <nicolas@seita.nl> * small comments from review Signed-off-by: Nicolas Höning <nicolas@seita.nl> * demanding the lowest docker compose version we know we need (we use start_period for healthchecks) Signed-off-by: Nicolas Höning <nicolas@seita.nl> * more attention-grabbing note about the tutorial happening inside the docker container Signed-off-by: Nicolas Höning <nicolas@seita.nl> * not only the API runs in the container Signed-off-by: Nicolas Höning <nicolas@seita.nl> * better name for the Docker container in the compose tutorial & smaller doc fixes Signed-off-by: Nicolas Höning <nicolas@seita.nl>
- Loading branch information
Showing
27 changed files
with
451 additions
and
42 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,6 +16,7 @@ venv | |
.env-flexmeasures/ | ||
flexmeasures.egg-info | ||
flexmeasures.log* | ||
*.csv | ||
|
||
.cache | ||
__pycache__ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
FROM ubuntu:focal | ||
|
||
# TODO: Cbc solver | ||
# TODO: run gunicorn as entry command | ||
|
||
ENV DEBIAN_FRONTEND noninteractive | ||
ENV LC_ALL C.UTF-8 | ||
ENV LANG C.UTF-8 | ||
|
||
# pre-requisites | ||
RUN apt-get update && apt-get install -y --upgrade python3 python3-pip git curl gunicorn coinor-cbc | ||
|
||
WORKDIR /app | ||
# requirements - doing this earlier, so we don't install them each time. Use --no-cache to refresh them. | ||
COPY requirements /app/requirements | ||
|
||
# py dev tooling | ||
RUN python3 -m pip install --upgrade pip && python3 --version | ||
RUN pip3 install --upgrade setuptools | ||
RUN pip3 install -r requirements/app.txt -r requirements/dev.txt -r requirements/test.txt | ||
|
||
# Copy code and meta/config data | ||
COPY setup.* .flaskenv wsgi.py /app/ | ||
COPY flexmeasures/ /app/flexmeasures | ||
RUN find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf | ||
COPY .git/ /app/.git | ||
|
||
RUN pip3 install . | ||
|
||
EXPOSE 5000 | ||
|
||
CMD [ \ | ||
"gunicorn", \ | ||
"--bind", "0.0.0.0:5000", \ | ||
# This is set to /tmp by default, but this is part of the Docker overlay filesystem, and can cause stalls. | ||
# http://docs.gunicorn.org/en/latest/faq.html#how-do-i-avoid-gunicorn-excessively-blocking-in-os-fchmod | ||
"--worker-tmp-dir", "/dev/shm", \ | ||
# Ideally you'd want one worker per container, but we don't want to risk the health check timing out because | ||
# another request is taking a long time to complete. | ||
"--workers", "2", "--threads", "4", \ | ||
"wsgi:application" \ | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
CREATE EXTENSION IF NOT EXISTS cube; | ||
CREATE EXTENSION IF NOT EXISTS earthdistance; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
version: "3.4" | ||
|
||
# ----------------------------------------------- | ||
# TODO: | ||
# * Add redis service | ||
# * Add a FlexMeasures worker service, maybe using https://docs.docker.com/compose/compose-file/#entrypoint | ||
# ----------------------------------------------- | ||
|
||
services: | ||
dev-db: | ||
image: postgres | ||
expose: | ||
- 5432 | ||
restart: always | ||
environment: | ||
POSTGRES_DB: fm-dev-db | ||
POSTGRES_USER: fm-dev-db-user | ||
POSTGRES_PASSWORD: fm-dev-db-pass | ||
volumes: | ||
- ./ci/load-psql-extensions.sql:/docker-entrypoint-initdb.d/load-psql-extensions.sql | ||
server: | ||
build: | ||
context: . | ||
dockerfile: Dockerfile | ||
ports: | ||
- 5000:5000 | ||
depends_on: | ||
- dev-db | ||
- test-db # use -e SQLALCHEMY_TEST_DATABASE_URI=... to exec pytest | ||
restart: on-failure | ||
healthcheck: | ||
test: ["CMD", "curl", "-f", "http://localhost:5000/api/v3_0/health/ready"] | ||
start_period: 10s | ||
interval: 20s | ||
timeout: 10s | ||
retries: 6 | ||
environment: | ||
SQLALCHEMY_DATABASE_URI: "postgresql://fm-dev-db-user:fm-dev-db-pass@dev-db:5432/fm-dev-db" | ||
SECRET_KEY: notsecret | ||
FLASK_ENV: development | ||
LOGGING_LEVEL: INFO | ||
volumes: | ||
# a place for config and plugin code - the mount point is for running the FlexMeasures CLI, the 2nd for gunicorn | ||
- ./flexmeasures-instance/:/usr/var/flexmeasures-instance/:ro | ||
- ./flexmeasures-instance/:/app/instance/:ro | ||
command: > | ||
bash -c "flexmeasures db upgrade | ||
&& flexmeasures add toy-account --name 'Docker Toy Account' | ||
&& gunicorn --bind 0.0.0.0:5000 --worker-tmp-dir /dev/shm --workers 2 --threads 4 wsgi:application" | ||
test-db: | ||
image: postgres | ||
expose: | ||
- 5432 | ||
restart: always | ||
environment: | ||
POSTGRES_DB: fm-test-db | ||
POSTGRES_USER: fm-test-db-user | ||
POSTGRES_PASSWORD: fm-test-db-pass | ||
volumes: | ||
- ./ci/load-psql-extensions.sql:/docker-entrypoint-initdb.d/load-psql-extensions.sql | ||
|
||
volumes: | ||
flexmeasures-instance: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
.. _docker-compose: | ||
|
||
Running a complete stack with docker-compose | ||
============================================= | ||
|
||
To install FlexMeasures, plus the libraries and databases it depends on, on your computer is some work, and can have unexpected hurdles, e.g. depending on the operating system. A nice alternative is to let that happen within Docker. The whole stack can be run via `Docker compose <https://docs.docker.com/compose/>`_, saving the developer much time. | ||
|
||
For this, we assume you are in the directory housing ``docker-compose.yml``. | ||
|
||
|
||
.. note:: The minimum Docker version is 17.09 and for docker-compose we tested successfully at version 1.25. You can check your versions with ``docker[-compose] --version``. | ||
|
||
Build the compose stack | ||
------------------------ | ||
|
||
Run this: | ||
|
||
.. code-block:: bash | ||
docker-compose build | ||
This pulls the images you need, and re-builds the FlexMeasures one from code. If you change code, re-running this will re-build that image. | ||
|
||
This compose script can also serve as an inspiration for using FlexMeasures in modern cloud environments (like Kubernetes). For instance, you might want to not build the FlexMeasures image from code, but simply pull the image from DockerHub. | ||
|
||
.. todo:: This stack runs FlexMeasures, but misses the background worker aspect. For this, we'll add a redis node and one additional FlexMeasures node, which runs a worker as entry point instead (see `issue 418 <https://github.com/FlexMeasures/flexmeasures/issues/418>`_). | ||
|
||
|
||
Run the compose stack | ||
---------------------- | ||
|
||
Start the stack like this: | ||
|
||
.. code-block:: bash | ||
docker-compose up | ||
You can see log output in the terminal, but ``docker-compose logs`` is also available to you. | ||
|
||
Check ``docker ps`` or ``docker-compose ps`` to see if your containers are running: | ||
|
||
|
||
.. code-block:: console | ||
± docker ps | ||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES | ||
dda1a8606926 flexmeasures_server "bash -c 'flexmeasur…" 43 seconds ago Up 41 seconds (healthy) 0.0.0.0:5000->5000/tcp flexmeasures_server_1 | ||
27ed9eef1b04 postgres "docker-entrypoint.s…" 2 days ago Up 42 seconds 5432/tcp flexmeasures_dev-db_1 | ||
90df2065e08d postgres "docker-entrypoint.s…" 2 days ago Up 42 seconds 5432/tcp flexmeasures_test-db_1 | ||
The FlexMeasures container has a health check implemented, which is reflected in this output and you can see which ports are available on your machine to interact. | ||
|
||
You can use ``docker-compose logs`` to look at output. ``docker inspect <container>`` and ``docker exec -it <container> bash`` can be quite useful to dive into details. | ||
|
||
.. todo:: We should provide a way to test that this is working, e.g. a list of steps. Document this, but also include that in our tsc/Release list (as a test step to see if Dockerization still works, plus a publish step for the released version). | ||
|
||
|
||
Configuration | ||
--------------- | ||
|
||
You can pass in your own configuration (e.g. for MapBox access token, or db URI, see below) like we described above for running a container: put a file ``flexmeasures.cfg`` into a local folder called ``flexmeasures-instance``. | ||
|
||
|
||
Data | ||
------- | ||
|
||
The postgres database is a test database with toy data filled in when the flexmeasures container starts. | ||
You could also connect it to some other database, by setting a different ``SQLALCHEMY_DATABASE_URI`` in the config. | ||
|
||
|
||
Running tests | ||
--------------- | ||
|
||
You can run tests in the flexmeasures docker container, using the database service ``test-db`` in the compose file (per default, we are using the ``dev-db`` database service). | ||
|
||
After you've started the compose stack with ``docker-compose up``, run: | ||
|
||
.. code-block:: console | ||
docker exec -it -e SQLALCHEMY_TEST_DATABASE_URI="postgresql://fm-test-db-user:fm-test-db-pass@test-db:5432/fm-test-db" flexmeasures_server_1 pytest | ||
This rounds up the dev experience offered by running FlexMeasures in Docker. Now you can develop FlexMeasures and also run your tests. If you develop plugins, you could extend the command being used, e.g. ``bash -c "cd /path/to/my/plugin && pytest"``. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.