CLI: move dev-add commands into main add group; remove add asset command

documentation/changelog.rst

@@ -9,6 +9,8 @@ New features
 -----------
 * Three new CLI commands for cleaning up your database: delete 1) unchanged beliefs, 2) NaN values or 3) a sensor and all of its time series data [see `PR #328 <http://www.github.com/FlexMeasures/flexmeasures/pull/328>`_]
 
+* Add CLI-commands ``flexmeasures add sensor``, ``flexmeasures add asset-type``, ``flexmeasures add beliefs`` (which were experimental features before). [see `PR #337 <http://www.github.com/FlexMeasures/flexmeasures/pull/337>`_]
+
 Bugfixes
 -----------
 

documentation/cli/change_log.rst

@@ -5,6 +5,14 @@ FlexMeasures CLI Changelog
 **********************
 
 
+since v0.9.0 | January 26, 2022
+=====================
+
+* add ``flexmeasures add sensor``, ''flexmeasures add asset-type``, ```flexmeasures add beliefs``. These were previously experimental features (under the `dev-add` command group).
+* ``flexmeasures add asset`` now directly creates an asset in the new data model.
+* add ``flexmeasures delete sensor``, ``flexmeasures delete nan-beliefs`` and ``flexmeasures delete unchanged-beliefs``. 
+
+
 since v0.6.0 | April 2, 2021
 =====================
 

documentation/cli/commands.rst

@@ -28,9 +28,12 @@ of which some are referred to in this documentation.
 ``flexmeasures add account-role``                 Create a FlexMeasures tenant account role.
 ``flexmeasures add account``                      Create a FlexMeasures tenant account.
 ``flexmeasures add user``                         Create a FlexMeasures user.
+``flexmeasures add asset-type``                   Create a new asset type.
 ``flexmeasures add asset``                        Create a new asset.
+``flexmeasures add sensor``                       Add a new sensor.
 ``flexmeasures add weather-sensor``               Add a weather sensor.
 ``flexmeasures add external-weather-forecasts``   Collect weather forecasts from the DarkSky API.
+``flexmeasures add beliefs``                      Load beliefs from file.
 ``flexmeasures add forecasts``                    Create forecasts.
 ================================================= =======================================
 
@@ -44,8 +47,11 @@ of which some are referred to in this documentation.
 ``flexmeasures delete account-role``              Delete a tenant account role.
 ``flexmeasures delete account``                   Delete a tenant account & also their users (with assets and power measurements).
 ``flexmeasures delete user``                      Delete a user & also their assets and power measurements.
+``flexmeasures delete sensor``                    Delete a sensor and all beliefs about it.
 ``flexmeasures delete measurements``              Delete measurements (with horizon <= 0).
 ``flexmeasures delete prognoses``                 Delete forecasts and schedules (forecasts > 0).
+``flexmeasures delete unchanged-beliefs``         Delete unchanged beliefs.
+``flexmeasures delete nan-beliefs``               Delete NaN beliefs.
 ================================================= =======================================
 
 
@@ -67,4 +73,4 @@ of which some are referred to in this documentation.
 ``flexmeasures db-ops reset``                     Reset database data and re-create tables from data model.
 ``flexmeasures db-ops restore``                   Restore the dump file, see `db-ops dump` (run `reset` first).
 ``flexmeasures db-ops save``                      Backup db content to files.
-================================================= =======================================
+================================================= =======================================

documentation/getting-started.rst

@@ -147,20 +147,35 @@ Add your first asset
 
 There are three ways to add assets:
 
-Use the ``flexmeasures`` :ref:`cli`:
+First, you can use the ``flexmeasures`` :ref:`cli`:
 
 .. code-block::
 
-    flexmeasures add asset --name "my basement battery pack" --asset-type-name battery --capacity-in-MW 30 --event-resolution 2 --latitude 65 --longitude 123.76 --owner-id 1
+    flexmeasures add asset --name "my basement battery pack" --asset-type-id 3 --latitude 65 --longitude 123.76 --account-id 2
 
-Here, I left out the ``--market-id`` parameter, because in this quickstart scenario I'm fine with the dummy market created with ``flexmeasures add structure`` above.
-For the ownership, I got my user ID from the output of ``flexmeasures add user`` above, or I can browse to `FlexMeasures' user listing <http://localhost:5000/users>`_ and hover over my username.
+For the asset type ID, I consult ``flexmeasures show asset-types``.
 
-Or, you could head over to ``http://localhost:5000/assets`` (after you started FlexMeasures, see next step) and add a new asset there in a web form.
+For the account ID, I looked at the output of ``flexmeasures add account`` (the command we issued above) ― I could also have consulted ``flexmeasures show accounts``.
+
+The second way to add an asset is the UI ― head over to ``https://localhost:5000/assets`` (after you started FlexMeasures, see step "Run FlexMeasures" further down) and add a new asset there in a web form.
 
 Finally, you can also use the `POST /api/v2_0/assets <api/v2_0.html#post--api-v2_0-assets>`_ endpoint in the FlexMeasures API to create an asset.
 
 
+Add your first sensor
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Usually, we are here because we want to measure something with respect to our assets. Each assets can have sensors for that, so let's add a power sensor to our new battery asset, using the ``flexmeasures`` :ref:`cli`:
+
+.. code-block::
+
+   flexmeasures add sensor --name power --unit MW --event-resolution 5 --timezone Europe/Amsterdam --asset-id 1 --attributes '{"capacity_in_mw": 7}'
+
+The asset ID I got from the last CLI command, or I could consult ``flexmeasures show account --account-id <my-account-id>``.
+
+.. note: The event resolution is given in minutes. Capacity is something unique to power sensors, so it is added as an attribute.
+
+
 Run FlexMeasures
 ^^^^^^^^^^^^^^^^
 
@@ -182,11 +197,19 @@ When you see the dashboard, the map will not work. For that, you'll need to get
 Add data
 ^^^^^^^^
 
-You can use the `POST /api/v2_0/postMeterData <api/v2_0.html#post--api-v2_0-postMeterData>`_ endpoint in the FlexMeasures API to send meter data.
+There are three ways to add data:
 
-.. note::  `issue 56 <https://github.com/FlexMeasures/flexmeasures/issues/56>`_ should create a CLI function for adding a lot of data at once, from a CSV dataset.
+First, you can load in data from a file (CSV or Excel) via the ``flexmeasures`` :ref:`cli`:
 
-Also, you can add forecasts for your meter data with the ``flexmeasures add`` command, here is an example:
+.. code-block::
+
+   flexmeasures add beliefs --file my-data.csv --skiprows 2 --delimiter ";" --source OurLegacyDatabase --sensor-id 1
+
+This assumes you have a file `my-data.csv` with measurements, which was exported from some legacy database, and that the data is about our sensor with ID 1. This command has many options, so do use its ``--help`` function.
+
+Second, you can use the `POST /api/v2_0/postMeterData <api/v2_0.html#post--api-v2_0-postMeterData>`_ endpoint in the FlexMeasures API to send meter data.
+
+Finally, you can tell FlexMeasures to create forecasts for your meter data with the ``flexmeasures add forecasts`` command, here is an example:
 
 .. code-block::
 
@@ -226,3 +249,11 @@ Install and configure Redis
 ^^^^^^^^^^^^^^^^^^^^^^^
 
 To let FlexMeasures queue forecasting and scheduling jobs, install a `Redis <https://redis.io/>`_ server (or rent one) and configure access to it within FlexMeasures' config file (see above). You can find the necessary settings in :ref:`redis-config`.
+
+
+Where to go from here?
+------------------------
+
+If your data structure is good, you should think about (continually) adding measurement data. This tutorial mentioned how to add data, but :ref:`_tut_posting_data` goes deeper with examples and terms & definitions.
+
+Then, you probably want to use FlexMeasures to generate forecasts and schedules! For this, read further in :ref:`_tut_forecasting_scheduling`. 

flexmeasures/cli/data_add.py

@@ -23,10 +23,7 @@
     GenericAssetSchema,
     GenericAssetTypeSchema,
 )
-from flexmeasures.data.models.assets import Asset
-from flexmeasures.data.schemas.assets import AssetSchema
 from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
-from flexmeasures.data.models.markets import Market
 from flexmeasures.data.models.weather import WeatherSensor
 from flexmeasures.data.schemas.weather import WeatherSensorSchema
 from flexmeasures.data.models.data_sources import (
@@ -41,11 +38,6 @@ def fm_add_data():
     """FlexMeasures: Add data."""
 
 
-@click.group("dev-add")
-def fm_dev_add_data():
-    """Developer CLI commands not yet meant for users: Add data."""
-
-
 @fm_add_data.command("account-role")
 @with_appcontext
 @click.option("--name", required=True)
@@ -149,7 +141,7 @@ def new_user(
     print(f"Successfully created user {created_user}")
 
 
-@fm_dev_add_data.command("sensor")
+@fm_add_data.command("sensor")
 @with_appcontext
 @click.option("--name", required=True)
 @click.option("--unit", required=True, help="e.g. °C, m/s, kW/m²")
@@ -165,7 +157,8 @@ def new_user(
     help="timezone as string, e.g. 'UTC' or 'Europe/Amsterdam'",
 )
 @click.option(
-    "--generic-asset-id",
+    "--asset-id",
+    "generic_asset_id",
     required=True,
     type=int,
     help="Generic asset to assign this sensor to",
@@ -203,25 +196,25 @@ def add_sensor(**args):
     print(f"You can access it at its entity address {sensor.entity_address}")
 
 
-@fm_dev_add_data.command("generic-asset-type")
+@fm_add_data.command("asset-type")
 @with_appcontext
 @click.option("--name", required=True)
 @click.option(
     "--description",
     type=str,
     help="Description (useful to explain acronyms, for example).",
 )
-def add_generic_asset_type(**args):
-    """Add a generic asset type."""
+def add_asset_type(**args):
+    """Add an asset type."""
     check_errors(GenericAssetTypeSchema().validate(args))
     generic_asset_type = GenericAssetType(**args)
     db.session.add(generic_asset_type)
     db.session.commit()
-    print(f"Successfully created generic asset type with ID {generic_asset_type.id}")
-    print("You can now assign generic assets to it")
+    print(f"Successfully created asset type with ID {generic_asset_type.id}.")
+    print("You can now assign assets to it.")
 
 
-@fm_dev_add_data.command("generic-asset")
+@fm_add_data.command("asset")
 @with_appcontext
 @click.option("--name", required=True)
 @click.option(
@@ -236,90 +229,20 @@ def add_generic_asset_type(**args):
 )
 @click.option("--account-id", type=int, required=True)
 @click.option(
-    "--generic-asset-type-id",
+    "--asset-type-id",
+    "generic_asset_type_id",
     required=True,
     type=int,
-    help="Generic asset type to assign to this asset",
+    help="Asset type to assign to this asset",
 )
-def add_generic_asset(**args):
-    """Add a generic asset."""
+def add_asset(**args):
+    """Add an asset."""
     check_errors(GenericAssetSchema().validate(args))
     generic_asset = GenericAsset(**args)
     db.session.add(generic_asset)
     db.session.commit()
-    print(f"Successfully created generic asset with ID {generic_asset.id}")
-    print("You can now assign sensors to it")
-
-
-@fm_add_data.command("asset")
-@with_appcontext
-@click.option("--name", required=True)
-@click.option("--asset-type-name", required=True)
-@click.option(
-    "--unit",
-    help="unit of rate, just MW (default) for now",
-    type=click.Choice(["MW"]),
-    default="MW",
-)  # TODO: enable others
-@click.option(
-    "--capacity-in-MW",
-    required=True,
-    type=float,
-    help="Maximum rate of this asset in MW",
-)
-@click.option(
-    "--event-resolution",
-    required=True,
-    type=int,
-    help="Expected resolution of the data in minutes",
-)
-@click.option(
-    "--latitude",
-    required=True,
-    type=float,
-    help="Latitude of the asset's location",
-)
-@click.option(
-    "--longitude",
-    required=True,
-    type=float,
-    help="Longitude of the asset's location",
-)
-@click.option(
-    "--owner-id", required=True, type=int, help="Id of the user who owns this asset."
-)
-@click.option(
-    "--market-id",
-    type=int,
-    help="Id of the market used to price this asset. Defaults to a dummy TOU market.",
-)
-@click.option(
-    "--timezone",
-    default="UTC",
-    help="timezone as string, e.g. 'UTC' (default) or 'Europe/Amsterdam'.",
-)
-def new_asset(**args):
-    """
-    Create a new asset.
-    This is legacy, with the new data model we only want to add GenericAssets.
-    """
-    check_timezone(args["timezone"])
-    # if no market given, select dummy market
-    if args["market_id"] is None:
-        dummy_market = Market.query.filter(Market.name == "dummy-tou").one_or_none()
-        if not dummy_market:
-            print(
-                "No market ID given and also no dummy TOU market available. Maybe add structure first."
-            )
-            raise click.Abort()
-        args["market_id"] = dummy_market.id
-    check_errors(AssetSchema().validate(args))
-    args["event_resolution"] = timedelta(minutes=args["event_resolution"])
-    asset = Asset(**args)
-    db.session.add(asset)
-    db.session.commit()
-    print(f"Successfully created asset with ID {asset.id}")
-    print(f"You can access it at its entity address {asset.entity_address}")
+    print(f"Successfully created asset with ID {generic_asset.id}.")
+    print("You can now assign sensors to it.")
 
 
 @fm_add_data.command("weather-sensor")
@@ -375,7 +298,7 @@ def add_initial_structure():
     populate_structure(db)
 
 
-@fm_dev_add_data.command("beliefs")
+@fm_add_data.command("beliefs")
 @with_appcontext
 @click.argument("file", type=click.Path(exists=True))
 @click.option(
@@ -711,7 +634,6 @@ def collect_weather_data(region, location, num_cells, method, store_in_db):
 
 
 app.cli.add_command(fm_add_data)
-app.cli.add_command(fm_dev_add_data)
 
 
 def check_timezone(timezone):

flexmeasures/cli/data_delete.py

@@ -197,7 +197,7 @@ def delete_prognoses(
     depopulate_prognoses(app.db, sensor_id)
 
 
-@fm_delete_data.command("unchanged_beliefs")
+@fm_delete_data.command("unchanged-beliefs")
 @with_appcontext
 @click.option(
     "--sensor-id",
@@ -275,7 +275,7 @@ def delete_unchanged_beliefs(
     print(f"Done! {num_beliefs_after} beliefs left")
 
 
-@fm_delete_data.command("nan_beliefs")
+@fm_delete_data.command("nan-beliefs")
 @with_appcontext
 @click.option(
     "--sensor-id",