CLI function for adding assets, weather sensors & add dummy market

Readme.md

@@ -3,6 +3,7 @@
 ![lint-and-test](https://github.com/SeitaBV/flexmeasures/workflows/lint-and-test/badge.svg)
 [![](https://img.shields.io/badge/python-3.6+-blue.svg)](https://www.python.org/downloads/)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Documentation Status](https://readthedocs.org/projects/flexmeasures/badge/?version=latest)](https://flexmeasures.readthedocs.io/en/latest/?badge=latest)
 
 The *FlexMeasures Platform* is a tool for scheduling flexible actions for energy assets.
 For this purpose, it performs monitoring, forecasting and scheduling services.

documentation/changelog.rst

@@ -8,8 +8,9 @@ v0.2.4 | March XX, 2021
 
 New features
 -----------
-* FlexMeasures can be installed with `pip` and its CLI commands can be run with `flexmeasures` [see `PR #54 <http://www.github.com/SeitaBV/flexmeasures/pull/54>`_]
+* FlexMeasures can be installed with ``pip`` and its CLI commands can be run with ``flexmeasures`` [see `PR #54 <http://www.github.com/SeitaBV/flexmeasures/pull/54>`_]
 * Optionally setting recording time when posting data [see `PR #41 <http://www.github.com/SeitaBV/flexmeasures/pull/41>`_]
+* Add assets and weather sensors with CLI commands [see `PR #74 <https://github.com/SeitaBV/flexmeasures/pull/74>`_]
 
 Bugfixes
 --------

documentation/cli/change_log.rst

@@ -0,0 +1,13 @@
+.. _cli-changelog:
+
+**********************
+FlexMeasures CLI Changelog
+**********************
+
+
+since v0.2.4 | March XX, 2021
+=====================
+
+* Refactor CLI into the main groups ``add``, ``delete``, ``jobs`` and ``db-ops``
+* Add ``flexmeasures add asset``,  ``flexmeasures add user`` and ``flexmeasures add weather-sensor``
+* split the ``populate-db`` command into ``flexmeasures add structure`` and ``flexmeasures add forecasts``

documentation/cli/commands.rst

@@ -0,0 +1,65 @@
+.. _cli:
+
+Command Line Interface (CLI)
+=============================
+
+FlexMeasures comes with a command-line utility, which helps to to manage data.
+Below, we list all available commands.
+
+Each command has more extensive documentation if you call it with ``--help``.
+
+We keep track of changes to these commands in :ref:`cli-changelog`.
+You can also get the current overview over the commands yiu have available by:
+
+.. code-block::
+
+    flexmeasures --help
+
+This will also show commands made available through Flask and the installed extensions (e.g. `Flask-Security <https://flask-security-too.readthedocs.io>`_, or `Flask-Migrate <https://flask-migrate.readthedocs.io>`_). These are also very interesting for admins (and partially come up in this documentation).
+
+
+``add`` : Add data
+--------------
+
+================================================= =======================================
+``flexmeasures add structure``                    Initialize the database with static values.
+``flexmeasures add user``                         Create a FlexMeasures user.
+``flexmeasures add asset``                        Create a new asset.
+``flexmeasures add weather-sensor``               Add a weather sensor.
+``flexmeasures add external-weather-forecasts``   Collect weather forecasts from the DarkSky API.
+``flexmeasures add forecasts``                    Create forecasts.
+================================================= =======================================
+
+
+``delete`` : Delete data
+--------------
+
+================================================= =======================================
+``flexmeasures delete structure``                 Delete structural data like asset (types), 
+                                                  market (types), weather (sensors), users, roles.
+``flexmeasures delete user``                      Delete a user & also their data.
+``flexmeasures delete measurements``              Delete measurements (with horizon <= 0).
+``flexmeasures delete prognoses``                 Delete forecasts and schedules (forecasts > 0).
+================================================= =======================================
+
+
+``jobs`` : Job queueing
+--------------
+
+================================================= =======================================
+``flexmeasures jobs run-worker``                  Start a worker process for forecasting and/or scheduling jobs.
+``flexmeasures jobs clear-queue``                 Clear a job queue.
+================================================= =======================================
+
+
+``db-ops`` : Operations on the whole database
+--------------
+
+================================================= =======================================
+``flexmeasures db-ops dump``                      Create a database dump of the database used by the app.
+``flexmeasures db-ops load``                      Load structure and/or data for the database from a backup file.
+``flexmeasures db-ops reset``                     Reset database, with options to load fresh data.
+``flexmeasures db-ops restore``                   Restore the database used by the app, from a given database 
+                                                  dump file, after you've reset the database.
+``flexmeasures db-ops save``                      Save structure of the database to a backup file.
+================================================= =======================================

documentation/configuration.rst

@@ -121,9 +121,7 @@ DARK_SKY_API_KEY
 
 Token for accessing the DarkSky weather forecasting service.
 
-.. note:: DarkSky will be soon (Aug 1, 2021) become non-public, so thay are not giving out new tokens. We'll use another service soon, `see this issue <https://github.com/SeitaBV/flexmeasures/issues/3>`_.
-
-This is unfortunate. In the meantime, if you can't find anybody lending their token, you can add weather forecasts to the FlexMeasures db yourself. 
+.. note:: DarkSky will be soon (Aug 1, 2021) become non-public, so thay are not giving out new tokens. We'll use another service soon, `see this issue <https://github.com/SeitaBV/flexmeasures/issues/3>`_. This is unfortunate. In the meantime, if you can't find anybody lending their token, you can add weather forecasts to the FlexMeasures database yourself. 
 
 Default: ``None``
 

documentation/dev/data.rst

@@ -43,6 +43,7 @@ On Windows:
 * Add the lib and bin directories to your Windows path: http://bobbyong.com/blog/installing-postgresql-on-windoes/
 * ``conda install psycopg2``
 
+
 Make sure postgres represents datetimes in UTC timezone
 ^^^^^^^^^^^^^
 
@@ -60,6 +61,7 @@ Find the ``timezone`` setting and set it to 'UTC'.
 
 Then restart the postgres server.
 
+
 Setup the "flexmeasures" Unix user
 ^^^^^^^^^^^^^
 
@@ -142,11 +144,13 @@ Write:
 
 into the config file you are using, e.g. ~/flexmeasures.cfg
 
+
 Get structure (and some data) into place
 ^^^^^^^^^^^^^
 
 You need data to enjoy the benefits of FlexMeasures or to develop features for it. In this section, there are some ways to get started.
 
+
 Import from another database
 """"""""""""""""""""""""""""""
 
@@ -156,7 +160,7 @@ On the to-be-exported database:
 
 .. code-block:: bash
 
-   flask db-dump
+   flexmeasures db-ops dump
 
 
 .. note:: Only the data gets dumped here.
@@ -165,14 +169,14 @@ Then, we create the structure in our database anew, based on the data model give
 
 .. code-block:: bash
 
-   flexmeasures db-reset
+   flexmeasures db-ops reset
 
 
 Then we import the data dump we made earlier:
 
 .. code-block:: bash
 
-   flask db-restore <DATABASE DUMP FILENAME>
+   flask db-ops restore <DATABASE DUMP FILENAME>
 
 
 A potential ``alembic_version`` error should not prevent other data tables from being restored.
@@ -196,14 +200,14 @@ You can create users with the ``new-user`` command. Check it out:
 
 .. code-block:: bash
 
-   flexmeasures new-user --help
+   flexmeasures add user --help
 
 
 You can create some pre-determined asset types and data sources with this command:
 
 .. code-block:: bash
 
-   flexmeasures db-populate --structure
+   flexmeasures add structure
 
 
 .. todo:: We should instead offer CLI commands to be able to create asset types as needed.
@@ -217,16 +221,21 @@ You can create forecasts for your existing metered data with this command:
 
 .. code-block:: bash
 
-   flexmeasures db-populate --forecasts
+   flexmeasures add forecasts
 
 
-Check out it's ``--help`` content to learn more. You can set which assets and which time window you want to forecast. At the time of writing, the forecasts horizons are fixed to 1, 6, 24 and 48 hours. Of course, making forecasts takes a while for a larger dataset.
+Check out it's ``--help`` content to learn more. You can set which assets and which time window you want to forecast. Of course, making forecasts takes a while for a larger dataset. You can also simply queue a job with this command.
 
-Just to note: There is also a command to get rid of data:
+Just to note: There are also commands to get rid of data, e.g.:
 
 .. code-block:: bash
 
-   flexmeasures db-depopulate --structure --data --forecasts
+   flexmeasures delete structure
+   flexmeasures delete measurements
+   flexmeasures delete forecasts
+
+Check out the :ref:`cli` documentation for more details.
+
 
 
 Visualize the data model
@@ -249,6 +258,7 @@ Maintenance is supported with the alembic tool. It reacts automatically
 to almost all changes in the SQLAlchemy code. With alembic, multiple databases,
 e.g. dev, staging and production can be kept in sync.
 
+
 Make first migration
 ^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -345,6 +355,7 @@ It is really useful (and therefore an industry standard) to bundle certain datab
 Please see the package ``flexmeasures.data.transactional`` for details on how a FlexMeasures developer should make use of this concept.
 If you are writing a script or a view, you will find there the necessary structural help to bundle your work in a transaction.
 
+
 Redis and redis queue
 -----------------------
 
@@ -356,13 +367,14 @@ Forecasting jobs are usually created (and enqueued) when new data comes in via t
 
 .. code-block:: bash
 
-   flexmeasures run_worker --queue forecasting
+   flexmeasures jobs run_worker --queue forecasting
 
 
 You should be able to run multiple workers in parallel, if necessary. You can add the ``--name`` argument to keep them a bit more organized.
 
 The FlexMeasures unit tests use fakeredis to simulate this task queueing, with no configuration required.
 
+
 Inspect the queue and jobs
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

documentation/getting-started.rst

@@ -99,20 +99,50 @@ FlexMeasures is a web-based platform, so we need a user account:
 
 .. code-block::
 
-   flexmeasures new-user --username <your-username> --email <your-email-address> --roles=admin
+   flexmeasures add user --username <your-username> --email <your-email-address> --roles=admin
 
 
 * This will ask you to set a password for the user.
 * Giving the first user the ``admin`` role is probably what you want.
 
+
 Add structure
 ^^^^^^^^^^^^^
 
-Populate the database with some standard energy asset types:
+Populate the database with some standard energy asset types, weather sensor types and a dummy market:
+
+.. code-block::
+
+   flexmeasures add structure
+
+
+Add your first weather sensor
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Weather plays a role for almost all use cases. FlexMeasure supports a few weather sensor types out of the box ("temperature", "wind_speed" and "radiation"), but you you need to decide which ones you need and where they are located. Let's use the ``flexmeasures`` :ref:`cli` to add one:
 
 .. code-block::
 
-   flexmeasures db-populate --structure
+   flexmeasures add weather-sensor --name a-temperature-sensor --weather-sensor-type-name temperature --unit C --event-resolution 15 --latitude 33 --longitude 2.4
+
+
+Add your first asset
+^^^^^^^^^^^^^^^^^^^^
+
+There are three ways to add assets:
+
+Head over to ``http://localhost:5000/assets`` and add a new asset there.
+
+Or, use the ``flexmeasures`` :ref:`cli`:
+
+.. code-block::
+
+    flexmeasures add asset --name TestA --asset-type-name battery --capacity-in-MW 30 --event-resolution 2 --latitude 65 --longitude 123.76 --owner-id 1
+
+Here, I left off the ``--market-id`` parameter, because in this quickstart scenario I'm fine with the dummy market created with ``flexmeasures add structure`` above. For the ownership, I got my user ID from the output of ``flexmeasures add user`` above, or I can browse to `FlexMeasure's user listing <http://localhost:5000/users>`_ and hover over my username.
+
+Finally, you can also use the `POST /api/v2_0/assets <api/v2_0.html#post--api-v2_0-assets>`_ endpoint in the FlexMeasures API to create an asset.
+
 
 Run FlexMeasures
 ^^^^^^^^^^^^^^^^
@@ -125,20 +155,11 @@ It's finally time to start running FlexMeasures:
 
 (This might print some warnings, see the next section where we go into more detail)
 
-.. note:: In a production context, you shouldn't run a script - hand the ``app`` object to a WSGI process, as your platform of choice describes.
-Often, that requires a WSGI script. We provide an example WSGI script in :ref:`continuous_integration`.
+.. note:: In a production context, you shouldn't run a script - hand the ``app`` object to a WSGI process, as your platform of choice describes. Often, that requires a WSGI script. We provide an example WSGI script in :ref:`continuous_integration`.
 
 You can visit ``http://localhost:5000`` now to see if the app's UI works.
 When you see the dashboard, the map will not work. For that, you'll need to get your :ref:`mapbox_access_token` and add it to your config file.
 
-Add your first asset
-^^^^^^^^^^^^^^^^^^^^
-
-Head over to ``http://localhost:5000/assets`` and add a new asset there.
-
-.. note:: `issue 57 <https://github.com/SeitaBV/flexmeasures/issues/57>`_ should create a CLI function for this.
-
-.. note:: You can also use the `POST /api/v2_0/assets <api/v2_0.html#post--api-v2_0-assets>`_ endpoint in the FlexMeasures API to create an asset.
 
 Add data
 ^^^^^^^^
@@ -147,11 +168,11 @@ You can use the `POST /api/v2_0/postMeterData <api/v2_0.html#post--api-v2_0-post
 
 .. note::  `issue 56 <https://github.com/SeitaBV/flexmeasures/issues/56>`_ should create a CLI function for adding a lot of data at once, from a CSV dataset.
 
-Also, you can add forecasts for your meter data with the ``db_populate`` command, here is an example:
+Also, you can add forecasts for your meter data with the ``flexmeasures add`` command, here is an example:
 
 .. code-block::
 
-   flexmeasures db-populate --forecasts --from-date 2020-03-08 --to-date 2020-04-08 --asset-type Asset --asset my-solar-panel
+   flexmeasures add forecasts --from-date 2020-03-08 --to-date 2020-04-08 --asset-type Asset --asset my-solar-panel
 
 .. note:: You can also use the API to send forecast data.
 
@@ -186,12 +207,13 @@ More information (e.g. for installing on Windows) on `the Cbc website <https://p
 Start collecting weather data
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-To collect weather measurements and forecasts, there is a task you could run periodically, probably once per hour. Here is an example:
+To collect weather measurements and forecasts from the DarkSky API, there is a task you could run periodically, probably once per hour. Here is an example:
 
 .. code-block::
 
-   flexmeasures collect-weather-data--location 33.4366,126.5269 --store-in-db 
+   flexmeasures add external-weather-forecasts --location 33.4366,126.5269 --store-in-db 
 
+.. note::  DarkSky is not handing out tokens anymore, as they have been bought by Apple, see `issue 3 <https://github.com/SeitaBV/flexmeasures/issues/56>`_ 
 
 
 Preparing the job queue database and start workers

documentation/index.rst

@@ -72,6 +72,13 @@ The platform operator of FlexMeasures can be an Aggregator.
     api/v1
     api/change_log
 
+.. toctree::
+    :caption: The CLI 
+    :maxdepth: 1
+
+    cli/commands
+    cli/change_log
+
 
 .. toctree::
     :caption: Developers

flexmeasures/api/__init__.py

@@ -1,5 +1,4 @@
 from flask import Flask, Blueprint, request
-from flask_marshmallow import Marshmallow
 from flask_security.utils import verify_password
 from flask_json import as_json
 from flask_login import current_user
@@ -14,8 +13,6 @@
 # The api blueprint. It is registered with the Flask app (see app.py)
 flexmeasures_api = Blueprint("flexmeasures_api", __name__)
 
-ma: Marshmallow = Marshmallow()
-
 
 @flexmeasures_api.route("/requestAuthToken", methods=["POST"])
 @as_json
@@ -83,8 +80,6 @@ def get_versions() -> dict:
 
 def register_at(app: Flask):
     """This can be used to register this blueprint together with other api-related things"""
-    global ma
-    ma.init_app(app)
 
     # handle API specific errors
     app.register_error_handler(FMValidationError, validation_error_handler)