CLI function for adding assets & weather sensors; add dummy market (#74)

* add new-asset CLI function; move Asset schema to data package and enrich its validations * group CLI commands (in the data package) into add, delete, db-ops and jobs groups * add weather sensor CLI function; starting a unified Sensor schema in the process * protect analytics view against crashing if no relevant weather sensor is found * add default weather sensor types and a dummy TOU market as structure * update documentation with new CLI commands; add complete CLI reference * prepare add weather sensor command for printing out the EA address
FlexMeasures · Apr 1, 2021 · f68d014 · f68d014
1 parent aef5f8e
commit f68d014
Show file tree

Hide file tree

Showing 29 changed files with 1,089 additions and 589 deletions.
diff --git a/Readme.md b/Readme.md
@@ -3,6 +3,7 @@
 ![lint-and-test](https://github.com/SeitaBV/flexmeasures/workflows/lint-and-test/badge.svg)
 [![](https://img.shields.io/badge/python-3.6+-blue.svg)](https://www.python.org/downloads/)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Documentation Status](https://readthedocs.org/projects/flexmeasures/badge/?version=latest)](https://flexmeasures.readthedocs.io/en/latest/?badge=latest)
 
 The *FlexMeasures Platform* is a tool for scheduling flexible actions for energy assets.
 For this purpose, it performs monitoring, forecasting and scheduling services.

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -8,8 +8,9 @@ v0.2.4 | March XX, 2021
 
 New features
 -----------
-* FlexMeasures can be installed with `pip` and its CLI commands can be run with `flexmeasures` [see `PR #54 <http://www.github.com/SeitaBV/flexmeasures/pull/54>`_]
+* FlexMeasures can be installed with ``pip`` and its CLI commands can be run with ``flexmeasures`` [see `PR #54 <http://www.github.com/SeitaBV/flexmeasures/pull/54>`_]
 * Optionally setting recording time when posting data [see `PR #41 <http://www.github.com/SeitaBV/flexmeasures/pull/41>`_]
+* Add assets and weather sensors with CLI commands [see `PR #74 <https://github.com/SeitaBV/flexmeasures/pull/74>`_]
 
 Bugfixes
 --------

diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst
@@ -0,0 +1,13 @@
+.. _cli-changelog:
+
+**********************
+FlexMeasures CLI Changelog
+**********************
+
+
+since v0.2.4 | March XX, 2021
+=====================
+
+* Refactor CLI into the main groups ``add``, ``delete``, ``jobs`` and ``db-ops``
+* Add ``flexmeasures add asset``,  ``flexmeasures add user`` and ``flexmeasures add weather-sensor``
+* split the ``populate-db`` command into ``flexmeasures add structure`` and ``flexmeasures add forecasts``
diff --git a/documentation/cli/commands.rst b/documentation/cli/commands.rst
@@ -0,0 +1,66 @@
+.. _cli:
+
+Command Line Interface (CLI)
+=============================
+
+FlexMeasures comes with a command-line utility, which helps to manage data.
+Below, we list all available commands.
+
+Each command has more extensive documentation if you call it with ``--help``.
+
+We keep track of changes to these commands in :ref:`cli-changelog`.
+You can also get the current overview over the commands you have available by:
+
+.. code-block::
+
+    flexmeasures --help
+
+This also shows admin commands made available through Flask and installed extensions (such as `Flask-Security <https://flask-security-too.readthedocs.io>`_ and `Flask-Migrate <https://flask-migrate.readthedocs.io>`_),
+of which some are referred to in this documentation.
+
+
+``add`` - Add data
+--------------
+
+================================================= =======================================
+``flexmeasures add structure``                    Initialize structural data like asset types, 
+                                                  market types and weather sensor types.
+``flexmeasures add user``                         Create a FlexMeasures user.
+``flexmeasures add asset``                        Create a new asset.
+``flexmeasures add weather-sensor``               Add a weather sensor.
+``flexmeasures add external-weather-forecasts``   Collect weather forecasts from the DarkSky API.
+``flexmeasures add forecasts``                    Create forecasts.
+================================================= =======================================
+
+
+``delete`` - Delete data
+--------------
+
+================================================= =======================================
+``flexmeasures delete structure``                 Delete all structural (non time-series) data like assets (types), 
+                                                  markets (types) and weather sensors (types) and users.
+``flexmeasures delete user``                      Delete a user & also their assets and power measurements.
+``flexmeasures delete measurements``              Delete measurements (with horizon <= 0).
+``flexmeasures delete prognoses``                 Delete forecasts and schedules (forecasts > 0).
+================================================= =======================================
+
+
+``jobs`` - Job queueing
+--------------
+
+================================================= =======================================
+``flexmeasures jobs run-worker``                  Start a worker process for forecasting and/or scheduling jobs.
+``flexmeasures jobs clear-queue``                 Clear a job queue.
+================================================= =======================================
+
+
+``db-ops`` - Operations on the whole database
+--------------
+
+================================================= =======================================
+``flexmeasures db-ops dump``                      Create a dump of all current data (using `pg_dump`).
+``flexmeasures db-ops load``                      Load backed-up contents (see `db-ops save`), run `reset` first.
+``flexmeasures db-ops reset``                     Reset database data and re-create tables from data model.
+``flexmeasures db-ops restore``                   Restore the dump file, see `db-ops dump` (run `reset` first).
+``flexmeasures db-ops save``                      Backup db content to files.
+================================================= =======================================
diff --git a/documentation/configuration.rst b/documentation/configuration.rst
@@ -13,7 +13,7 @@ Recommended settings (e.g. mail, redis) are marked by one star (*).
 
 
 * in the user's home directory (e.g. ``~/.flexmeasures.cfg`` on Unix). In this case, note the dot at the beginning of the filename!
-* in the apps's instance directory (e.g. ``/path/to/your/flexmeasures/code/instance/flexmeasures.cfg``\ ). The path to that instance directory is shown to you by running flexmeasures (e.g. ``flexmeasures run``\ ) with required settings missing or otherwise by running ``flexmeasures shell``.
+* in the app's instance directory (e.g. ``/path/to/your/flexmeasures/code/instance/flexmeasures.cfg``\ ). The path to that instance directory is shown to you by running flexmeasures (e.g. ``flexmeasures run``\ ) with required settings missing or otherwise by running ``flexmeasures shell``.
 
 Basic functionality
 -------------------
@@ -85,7 +85,7 @@ Default: ``False``
 RQ_DASHBOARD_POLL_INTERVAL
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Interval in which viewing the queues dashboard refreshes itself, in miliseconds.
+Interval in which viewing the queues dashboard refreshes itself, in milliseconds.
 
 Default: ``3000`` (3 seconds) 
 
@@ -121,9 +121,10 @@ DARK_SKY_API_KEY
 
 Token for accessing the DarkSky weather forecasting service.
 
-.. note:: DarkSky will be soon (Aug 1, 2021) become non-public, so thay are not giving out new tokens. We'll use another service soon, `see this issue <https://github.com/SeitaBV/flexmeasures/issues/3>`_.
-
-This is unfortunate. In the meantime, if you can't find anybody lending their token, you can add weather forecasts to the FlexMeasures db yourself. 
+.. note:: DarkSky will soon become non-public (Aug 1, 2021), so they are not giving out new tokens.
+          We'll use another service soon (`see this issue <https://github.com/SeitaBV/flexmeasures/issues/3>`_).
+          This is unfortunate.
+          In the meantime, if you can't find anybody lending their token, consider posting weather forecasts to the FlexMeasures database yourself.
 
 Default: ``None``
 

diff --git a/documentation/dev/data.rst b/documentation/dev/data.rst
@@ -43,6 +43,7 @@ On Windows:
 * Add the lib and bin directories to your Windows path: http://bobbyong.com/blog/installing-postgresql-on-windoes/
 * ``conda install psycopg2``
 
+
 Make sure postgres represents datetimes in UTC timezone
 ^^^^^^^^^^^^^
 
@@ -60,6 +61,7 @@ Find the ``timezone`` setting and set it to 'UTC'.
 
 Then restart the postgres server.
 
+
 Setup the "flexmeasures" Unix user
 ^^^^^^^^^^^^^
 
@@ -142,11 +144,13 @@ Write:
 
 into the config file you are using, e.g. ~/flexmeasures.cfg
 
+
 Get structure (and some data) into place
 ^^^^^^^^^^^^^
 
 You need data to enjoy the benefits of FlexMeasures or to develop features for it. In this section, there are some ways to get started.
 
+
 Import from another database
 """"""""""""""""""""""""""""""
 
@@ -156,7 +160,7 @@ On the to-be-exported database:
 
 .. code-block:: bash
 
-   flask db-dump
+   flexmeasures db-ops dump
 
 
 .. note:: Only the data gets dumped here.
@@ -165,14 +169,14 @@ Then, we create the structure in our database anew, based on the data model give
 
 .. code-block:: bash
 
-   flexmeasures db-reset
+   flexmeasures db-ops reset
 
 
 Then we import the data dump we made earlier:
 
 .. code-block:: bash
 
-   flask db-restore <DATABASE DUMP FILENAME>
+   flask db-ops restore <DATABASE DUMP FILENAME>
 
 
 A potential ``alembic_version`` error should not prevent other data tables from being restored.
@@ -196,14 +200,14 @@ You can create users with the ``new-user`` command. Check it out:
 
 .. code-block:: bash
 
-   flexmeasures new-user --help
+   flexmeasures add user --help
 
 
 You can create some pre-determined asset types and data sources with this command:
 
 .. code-block:: bash
 
-   flexmeasures db-populate --structure
+   flexmeasures add structure
 
 
 .. todo:: We should instead offer CLI commands to be able to create asset types as needed.
@@ -217,16 +221,22 @@ You can create forecasts for your existing metered data with this command:
 
 .. code-block:: bash
 
-   flexmeasures db-populate --forecasts
+   flexmeasures add forecasts
 
 
-Check out it's ``--help`` content to learn more. You can set which assets and which time window you want to forecast. At the time of writing, the forecasts horizons are fixed to 1, 6, 24 and 48 hours. Of course, making forecasts takes a while for a larger dataset.
+Check out it's ``--help`` content to learn more. You can set which assets and which time window you want to forecast. Of course, making forecasts takes a while for a larger dataset.
+You can also simply queue a job with this command (and run a worker to process the :ref:`redis-queue`).
 
-Just to note: There is also a command to get rid of data:
+Just to note, there are also commands to get rid of data, such as:
 
 .. code-block:: bash
 
-   flexmeasures db-depopulate --structure --data --forecasts
+   flexmeasures delete structure
+   flexmeasures delete measurements
+   flexmeasures delete forecasts
+
+Check out the :ref:`cli` documentation for more details.
+
 
 
 Visualize the data model
@@ -247,7 +257,8 @@ Maintenance
 
 Maintenance is supported with the alembic tool. It reacts automatically
 to almost all changes in the SQLAlchemy code. With alembic, multiple databases,
-e.g. dev, staging and production can be kept in sync.
+such as development, staging and production databases can be kept in sync.
+
 
 Make first migration
 ^^^^^^^^^^^^^^^^^^^^^^^
@@ -345,7 +356,10 @@ It is really useful (and therefore an industry standard) to bundle certain datab
 Please see the package ``flexmeasures.data.transactional`` for details on how a FlexMeasures developer should make use of this concept.
 If you are writing a script or a view, you will find there the necessary structural help to bundle your work in a transaction.
 
-Redis and redis queue
+
+.. _redis-queue:
+
+Redis queue
 -----------------------
 
 FlexMeasures supports jobs (e.g. forecasting) running asynchronously to the main FlexMeasures application using `Redis Queue <http://python-rq.org/>`_.
@@ -356,17 +370,18 @@ Forecasting jobs are usually created (and enqueued) when new data comes in via t
 
 .. code-block:: bash
 
-   flexmeasures run_worker --queue forecasting
+   flexmeasures jobs run_worker --queue forecasting
 
 
 You should be able to run multiple workers in parallel, if necessary. You can add the ``--name`` argument to keep them a bit more organized.
 
 The FlexMeasures unit tests use fakeredis to simulate this task queueing, with no configuration required.
 
+
 Inspect the queue and jobs
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The first option to inspect the state of the ``forecasting`` queue should be via the formiddable `RQ dashboard <https://github.com/Parallels/rq-dashboard>`_. If you have admin rights, you can access it at ``your-flexmeasures-url/rq/``\ , so for instance ``http://localhost:5000/rq/``. You can also start RQ dashboard yourself (but you need to know the redis server credentials):
+The first option to inspect the state of the ``forecasting`` queue should be via the formidable `RQ dashboard <https://github.com/Parallels/rq-dashboard>`_. If you have admin rights, you can access it at ``your-flexmeasures-url/rq/``\ , so for instance ``http://localhost:5000/rq/``. You can also start RQ dashboard yourself (but you need to know the redis server credentials):
 
 .. code-block:: bash
 

diff --git a/documentation/getting-started.rst b/documentation/getting-started.rst
@@ -99,20 +99,53 @@ FlexMeasures is a web-based platform, so we need a user account:
 
 .. code-block::
 
-   flexmeasures new-user --username <your-username> --email <your-email-address> --roles=admin
+   flexmeasures add user --username <your-username> --email <your-email-address> --roles=admin
 
 
 * This will ask you to set a password for the user.
 * Giving the first user the ``admin`` role is probably what you want.
 
+
 Add structure
 ^^^^^^^^^^^^^
 
-Populate the database with some standard energy asset types:
+Populate the database with some standard energy asset types, weather sensor types and a dummy market:
+
+.. code-block::
+
+   flexmeasures add structure
+
+
+Add your first weather sensor
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Weather plays a role for almost all use cases.
+FlexMeasures supports a few weather sensor types out of the box ("temperature", "wind_speed" and "radiation"), but you need to decide which ones you need and where they are located.
+Let's use the ``flexmeasures`` :ref:`cli` to add one:
 
 .. code-block::
 
-   flexmeasures db-populate --structure
+   flexmeasures add weather-sensor --name "my rooftop thermometer" --weather-sensor-type-name temperature --unit °C --event-resolution 15 --latitude 33 --longitude 2.4
+
+
+Add your first asset
+^^^^^^^^^^^^^^^^^^^^
+
+There are three ways to add assets:
+
+Head over to ``http://localhost:5000/assets`` and add a new asset there.
+
+Or, use the ``flexmeasures`` :ref:`cli`:
+
+.. code-block::
+
+    flexmeasures add asset --name "my basement battery pack" --asset-type-name battery --capacity-in-MW 30 --event-resolution 2 --latitude 65 --longitude 123.76 --owner-id 1
+
+Here, I left out the ``--market-id`` parameter, because in this quickstart scenario I'm fine with the dummy market created with ``flexmeasures add structure`` above.
+For the ownership, I got my user ID from the output of ``flexmeasures add user`` above, or I can browse to `FlexMeasures' user listing <http://localhost:5000/users>`_ and hover over my username.
+
+Finally, you can also use the `POST /api/v2_0/assets <api/v2_0.html#post--api-v2_0-assets>`_ endpoint in the FlexMeasures API to create an asset.
+
 
 Run FlexMeasures
 ^^^^^^^^^^^^^^^^
@@ -126,19 +159,11 @@ It's finally time to start running FlexMeasures:
 (This might print some warnings, see the next section where we go into more detail)
 
 .. note:: In a production context, you shouldn't run a script - hand the ``app`` object to a WSGI process, as your platform of choice describes.
-Often, that requires a WSGI script. We provide an example WSGI script in :ref:`continuous_integration`.
+          Often, that requires a WSGI script. We provide an example WSGI script in :ref:`continuous_integration`.
 
 You can visit ``http://localhost:5000`` now to see if the app's UI works.
 When you see the dashboard, the map will not work. For that, you'll need to get your :ref:`mapbox_access_token` and add it to your config file.
 
-Add your first asset
-^^^^^^^^^^^^^^^^^^^^
-
-Head over to ``http://localhost:5000/assets`` and add a new asset there.
-
-.. note:: `issue 57 <https://github.com/SeitaBV/flexmeasures/issues/57>`_ should create a CLI function for this.
-
-.. note:: You can also use the `POST /api/v2_0/assets <api/v2_0.html#post--api-v2_0-assets>`_ endpoint in the FlexMeasures API to create an asset.
 
 Add data
 ^^^^^^^^
@@ -147,11 +172,11 @@ You can use the `POST /api/v2_0/postMeterData <api/v2_0.html#post--api-v2_0-post
 
 .. note::  `issue 56 <https://github.com/SeitaBV/flexmeasures/issues/56>`_ should create a CLI function for adding a lot of data at once, from a CSV dataset.
 
-Also, you can add forecasts for your meter data with the ``db_populate`` command, here is an example:
+Also, you can add forecasts for your meter data with the ``flexmeasures add`` command, here is an example:
 
 .. code-block::
 
-   flexmeasures db-populate --forecasts --from-date 2020-03-08 --to-date 2020-04-08 --asset-type Asset --asset my-solar-panel
+   flexmeasures add forecasts --from-date 2020-03-08 --to-date 2020-04-08 --asset-type Asset --asset my-solar-panel
 
 .. note:: You can also use the API to send forecast data.
 
@@ -186,12 +211,13 @@ More information (e.g. for installing on Windows) on `the Cbc website <https://p
 Start collecting weather data
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-To collect weather measurements and forecasts, there is a task you could run periodically, probably once per hour. Here is an example:
+To collect weather measurements and forecasts from the DarkSky API, there is a task you could run periodically, probably once per hour. Here is an example:
 
 .. code-block::
 
-   flexmeasures collect-weather-data--location 33.4366,126.5269 --store-in-db 
+   flexmeasures add external-weather-forecasts --location 33.4366,126.5269 --store-in-db 
 
+.. note::  DarkSky is not handing out tokens anymore, as they have been bought by Apple (see `issue 3 <https://github.com/SeitaBV/flexmeasures/issues/56>`_).
 
 
 Preparing the job queue database and start workers

diff --git a/documentation/index.rst b/documentation/index.rst
@@ -72,6 +72,13 @@ The platform operator of FlexMeasures can be an Aggregator.
     api/v1
     api/change_log
 
+.. toctree::
+    :caption: The CLI 
+    :maxdepth: 1
+
+    cli/commands
+    cli/change_log
+
 
 .. toctree::
     :caption: Developers