first version of the toy scheduling tutorial (#381)

* first version of the toy scheduling tutorial

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* allow logging settings to be set via env vars, makes for better tutorials.

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* make the battery bigger (in MW), also use the upcoming roundtrip-efficiency encoding in percent

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* Clearer index, update small things in tutorial and getting started

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* small improvements on db handling tutorial

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* Update tutorial against revisions of scheduling CLI command

Signed-off-by: F.N. Claessen <felix@seita.nl>

* typo

Signed-off-by: F.N. Claessen <felix@seita.nl>

* also replace factor-id parameter on docs index

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* also add percentages to schedule command in doc index

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* Switch code-blocks from BashLexer to BashSessionLexer to get more appropriate code highlighting

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update code-block lexers

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix command for price input (at least, with respect to how it's displayed in the tutorial)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add copybutton to documentation code-blocks

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Set up the toy day-ahead market with an appropriate knowledge horizon set to a gate closure at noon

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Adjust tutorial note on price forecasts

Signed-off-by: F.N. Claessen <felix@seita.nl>

* add copybutton to doc requirements and also update vesions

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* also use console lexing for main page tutorial version, update use of TOMORROW variable

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

Co-authored-by: F.N. Claessen <felix@seita.nl>

documentation/changelog.rst

@@ -14,7 +14,6 @@ New features
 * Add CLI option to specify custom strings that should be interpreted as NaN values when reading in time series data from CSV [see `PR #357 <http://www.github.com/FlexMeasures/flexmeasures/pull/357>`_]
 * Add CLI commands ``flexmeasures add sensor``, ``flexmeasures add asset-type``, ``flexmeasures add beliefs`` (which were experimental features before) [see `PR #337 <http://www.github.com/FlexMeasures/flexmeasures/pull/337>`_]
 * Add CLI commands for showing data [see `PR #339 <http://www.github.com/FlexMeasures/flexmeasures/pull/339>`_]
-
 * Add CLI command for attaching annotations to assets: ``flexmeasures add holidays`` adds public holidays [see `PR #343 <http://www.github.com/FlexMeasures/flexmeasures/pull/343>`_]
 * Add CLI command for resampling existing sensor data to new resolution [see `PR #360 <http://www.github.com/FlexMeasures/flexmeasures/pull/360>`_]
 * Add CLI command to edit/add an attribute on an asset or sensor. [see `PR #380 <http://www.github.com/FlexMeasures/flexmeasures/pull/380>`_]

documentation/cli/commands.rst

@@ -11,7 +11,7 @@ Each command has more extensive documentation if you call it with ``--help``.
 We keep track of changes to these commands in :ref:`cli-changelog`.
 You can also get the current overview over the commands you have available by:
 
-.. code-block::
+.. code-block:: console
 
     flexmeasures --help
 

documentation/conf.py

@@ -47,6 +47,7 @@
     "sphinx.ext.imgmath",
     "sphinx.ext.ifconfig",
     "sphinx.ext.todo",
+    "sphinx_copybutton",
     "sphinx_fontawesome",
     "sphinxcontrib.autohttp.flask",
     "sphinxcontrib.autohttp.flaskqref",
@@ -191,6 +192,11 @@
 # Example configuration for intersphinx: refer to the Python standard library.
 intersphinx_mapping = {"https://docs.python.org/": None}
 
+# -- Options for copybytton extension ---------------------------------------
+copybutton_prompt_is_regexp = True
+copybutton_prompt_text = r">>> |\.\.\. |\$ "  # Python Repl + continuation + Bash
+copybutton_line_continuation_character = "\\"
+
 # -- Options for ifconfig extension ---------------------------------------
 
 

documentation/configuration.rst

@@ -293,7 +293,7 @@ Configuration of the SQLAlchemy engine.
 
 Default: 
 
-.. code-block::
+.. code-block:: python
 
        {
            "pool_recycle": 299,
@@ -427,7 +427,7 @@ Tuple of shown name of sender and their email address.
 
 Default:
 
-.. code-block::
+.. code-block:: python
 
    (
        "FlexMeasures",

documentation/dev/ci.rst

@@ -44,7 +44,7 @@ Cbc needs to be present on the server where FlexMeasures runs, under the ``cbc``
 
 You can install it on Debian like this:
 
-.. code-block:: bash
+.. code-block:: console
 
    apt-get install coinor-cbc
 

documentation/dev/data.rst

@@ -1,6 +1,6 @@
 .. _dev-data:
 
-Handling data
+Handling databases
 =============================
 
 This document describes how to get the postgres database ready to use and maintain it (do migrations / changes to the structure).
@@ -29,7 +29,7 @@ Install
 
 On Unix:
 
-.. code-block:: bash
+.. code-block:: console
 
    sudo apt-get install postgresql
    pip install psycopg2-binary
@@ -61,7 +61,7 @@ Find the ``timezone`` setting and set it to 'UTC'.
 
 Then restart the postgres server.
 
-.. code-block:: bash
+.. code-block:: console
 
     service postgresql restart
 
@@ -71,7 +71,7 @@ Setup the "flexmeasures" Unix user
 
 This may in fact not be needed:
 
-.. code-block:: bash
+.. code-block:: console
 
    sudo /usr/sbin/adduser flexmeasures
 
@@ -84,7 +84,7 @@ From the terminal:
 Open a console (use your Windows key and type ``cmd``\ ).
 Proceed to create a database as the postgres superuser (using your postgres user password):
 
-.. code-block:: bash
+.. code-block:: console
 
    sudo -i -u postgres
    createdb -U postgres flexmeasures
@@ -106,7 +106,7 @@ Or, from within Postgres console:
 
 Finally, test if you can log in as the flexmeasures user:
 
-.. code-block:: bash
+.. code-block:: console
 
    psql -U flexmeasures --password -h 127.0.0.1 -d flexmeasures
 
@@ -121,7 +121,7 @@ Add Postgres Extensions to your database(s)
 To find the nearest sensors, FlexMeasures needs some extra Postgres support.
 Add the following extensions while logged in as the postgres superuser:
 
-.. code-block:: bash
+.. code-block:: console
 
    sudo -u postgres psql
 
@@ -161,7 +161,7 @@ Here is a short recipe to import data from a FlexMeasures database (e.g. a demo
 
 On the to-be-exported database:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db-ops dump
 
@@ -170,14 +170,14 @@ On the to-be-exported database:
 
 Then, we create the structure in our database anew, based on the data model given by the local codebase:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db-ops reset
 
 
 Then we import the data dump we made earlier:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db-ops restore <DATABASE DUMP FILENAME>
 
@@ -192,7 +192,7 @@ Create data manually
 
 First, you can get the database structure with:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db upgrade
 
@@ -201,42 +201,41 @@ First, you can get the database structure with:
 
 You can create users with the ``new-user`` command. Check it out:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures add user --help
 
 
 You can create some pre-determined asset types and data sources with this command:
 
-.. code-block:: bash
+.. code-block:: console
 
-   flexmeasures add structure
+   flexmeasures add initial-structure
 
+You can also create assets in the FlexMeasures UI.
 
-.. todo:: We should instead offer CLI commands to be able to create asset types as needed.
+On the command line, you can add many things. Check what data you can add yourself:
 
-You can create assets in the FlexMeasures UI.
+.. code-block:: console
 
-.. todo:: Maybe a CLI command would help to script all data creation.
-.. todo:: We still need a decent way to load in metering data, e.g. from CSV - often, a custom loading script will be necessary anyways.
+   flexmeasures add --help
 
-You can create forecasts for your existing metered data with this command:
 
-.. code-block:: bash
+For instance, you can create forecasts for your existing metered data with this command:
+
+.. code-block:: console
 
    flexmeasures add forecasts
 
 
 Check out it's ``--help`` content to learn more. You can set which assets and which time window you want to forecast. Of course, making forecasts takes a while for a larger dataset.
 You can also simply queue a job with this command (and run a worker to process the :ref:`redis-queue`).
 
-Just to note, there are also commands to get rid of data, such as:
+Just to note, there are also commands to get rid of data. Check:
 
-.. code-block:: bash
+.. code-block:: console
 
-   flexmeasures delete structure
-   flexmeasures delete measurements
-   flexmeasures delete forecasts
+   flexmeasures delete --help
 
 Check out the :ref:`cli` documentation for more details.
 
@@ -247,7 +246,7 @@ Visualize the data model
 
 You can visualise the data model like this:
 
-.. code-block:: bash
+.. code-block:: console
 
    make show-data-model
 
@@ -268,7 +267,7 @@ Make first migration
 
 Run these commands from the repository root directory (read below comments first):
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db init
    flexmeasures db migrate
@@ -288,7 +287,7 @@ Make another migration
 
 Just to be clear that the ``db init`` command is needed only at the beginning - you usually do, if your model changed:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db migrate --message "Please explain what you did, it helps for later"
    flexmeasures db upgrade
@@ -299,7 +298,7 @@ Get database structure updated
 
 The goal is that on any other computer, you can always execute
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db upgrade
 
@@ -311,15 +310,15 @@ Working with the migration history
 
 The history of migrations is at your fingertips:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db current
    flexmeasures db history
 
 
 You can move back and forth through the history:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures db downgrade
    flexmeasures db upgrade
@@ -332,7 +331,7 @@ Check out database status
 
 Log in into the database:
 
-.. code-block:: bash
+.. code-block:: console
 
    psql -U flexmeasures --password -h 127.0.0.1 -d flexmeasures
 
@@ -371,7 +370,7 @@ It relies on a Redis server, which is has to be installed locally, or used on a
 
 Forecasting jobs are usually created (and enqueued) when new data comes in via the API. To asynchronously work on these forecasting jobs, run this in a console:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures jobs run_worker --queue forecasting
 
@@ -386,7 +385,7 @@ Inspect the queue and jobs
 
 The first option to inspect the state of the ``forecasting`` queue should be via the formidable `RQ dashboard <https://github.com/Parallels/rq-dashboard>`_. If you have admin rights, you can access it at ``your-flexmeasures-url/rq/``\ , so for instance ``http://localhost:5000/rq/``. You can also start RQ dashboard yourself (but you need to know the redis server credentials):
 
-.. code-block:: bash
+.. code-block:: console
 
    pip install rq-dashboard
    rq-dashboard --redis-host my.ip.addr.ess --redis-password secret --redis-database 0

documentation/dev/error-monitoring.rst

@@ -10,7 +10,7 @@ When you run a FlexMeasures server, you want to stay on top of things going wron
 
 For illustration of the latter monitoring, here is one example of how we monitor tasks on a server ― the below is run in a cron script every hour and checks if every listed task ran 60, 6 or 1440 minutes ago, respectively:
 
-.. code-block:: bash
+.. code-block:: console
 
     flexmeasures monitor tasks --task get_weather_forecasts 60 --task get_recent_meter_data 6  --task import_epex_prices 1440
 

documentation/dev/introduction.rst

@@ -30,7 +30,7 @@ Dependencies
 
 Install all dependencies including the ones needed for development:
 
-.. code-block:: bash
+.. code-block:: console
 
    make install-for-dev
 
@@ -46,7 +46,7 @@ Loading data
 
 If you have a SQL Dump file, you can load that:
 
-.. code-block:: bash
+.. code-block:: console
 
    psql -U {user_name} -h {host_name} -d {database_name} -f {file_path}
 
@@ -56,14 +56,14 @@ Run locally
 
 Now, to start the web application, you can run:
 
-.. code-block:: bash
+.. code-block:: console
 
    flexmeasures run
 
 
 Or:
 
-.. code-block:: bash
+.. code-block:: console
 
    python run-local.py
 
@@ -86,7 +86,7 @@ Tests
 
 You can run automated tests with:
 
-.. code-block:: bash
+.. code-block:: console
 
    make test
 
@@ -95,7 +95,7 @@ which behind the curtains installs dependencies and calls pytest.
 
 A coverage report can be created like this:
 
-.. code-block:: bash
+.. code-block:: console
 
    pytest --cov=flexmeasures --cov-config .coveragerc
 
@@ -104,7 +104,7 @@ You can add --cov-report=html after which a htmlcov/index.html is generated.
 
 It's also possible to use:
 
-.. code-block:: bash
+.. code-block:: console
 
    python setup.py test
 
@@ -133,7 +133,7 @@ We also run `mypy <http://mypy-lang.org/>`_ on many files to do some static type
 We do this so real problems are found faster and the discussion about formatting is limited.
 All of these can be installed by using ``pip``, but we recommend using them as a pre-commit hook. To activate that behaviour, do:
 
-.. code-block:: bash
+.. code-block:: console
 
    pip install pre-commit
    pre-commit install
@@ -156,7 +156,7 @@ A hint about using notebooks
 
 If you edit notebooks, make sure results do not end up in git:
 
-.. code-block:: bash
+.. code-block:: console
 
    conda install -c conda-forge nbstripout
    nbstripout --install

documentation/dev/plugins.rst

@@ -364,6 +364,6 @@ Place this template file in the template folder of your plugin blueprint (see ab
 
 Finally, add this config setting to your FlexMeasures config file (using the template filename you chose, obviously):
 
- .. code-block:: bash
+ .. code-block:: python
 
     SECURITY_LOGIN_USER_TEMPLATE = "my_user_login.html"