Skip to content

Commit

Permalink
CLI: move dev-add commands into main add group; remove add asset comm…
Browse files Browse the repository at this point in the history 
…and (#337)

* move dev-add commands into main add group; remove add asset command

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* Update getting-started tutorial to use our new data model

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* add CLI changelog entry

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* do not communicate the 'generic-' part to the user; add commands from #338 work to getting-stated guide; add commands to CLI reference

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* add missing CLI documentation from PR#328

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* resolve local merge conflict

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* udpate CLI commands list

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* add changelog entry

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* fix quotes in changelog

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* rename commands for deleting beliefs for consistency

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* fix two small things from review

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* Add missing period

Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
  • Loading branch information
@nhoening @Flix6x
nhoening and Flix6x committed Jan 29, 2022
1 parent 1c7c760 commit 3c6d3a7
Show file tree
Hide file tree
Showing 6 changed files with 75 additions and 106 deletions.
2 changes: 2 additions & 0 deletions documentation/changelog.rst
View file
Expand Up @@ -9,6 +9,8 @@ New features
-----------
* Three new CLI commands for cleaning up your database: delete 1) unchanged beliefs, 2) NaN values or 3) a sensor and all of its time series data [see `PR #328 <http://www.github.com/FlexMeasures/flexmeasures/pull/328>`_]

* Add CLI-commands ``flexmeasures add sensor``, ``flexmeasures add asset-type``, ``flexmeasures add beliefs`` (which were experimental features before). [see `PR #337 <http://www.github.com/FlexMeasures/flexmeasures/pull/337>`_]

Bugfixes
-----------

Expand Down
8 changes: 8 additions & 0 deletions documentation/cli/change_log.rst
View file
Expand Up @@ -5,6 +5,14 @@ FlexMeasures CLI Changelog
**********************


since v0.9.0 | January 26, 2022
=====================

* add ``flexmeasures add sensor``, ''flexmeasures add asset-type``, ```flexmeasures add beliefs``. These were previously experimental features (under the `dev-add` command group).
* ``flexmeasures add asset`` now directly creates an asset in the new data model.
* add ``flexmeasures delete sensor``, ``flexmeasures delete nan-beliefs`` and ``flexmeasures delete unchanged-beliefs``.


since v0.6.0 | April 2, 2021
=====================

Expand Down
8 changes: 7 additions & 1 deletion documentation/cli/commands.rst
View file
Expand Up @@ -28,9 +28,12 @@ of which some are referred to in this documentation.
``flexmeasures add account-role`` Create a FlexMeasures tenant account role.
``flexmeasures add account`` Create a FlexMeasures tenant account.
``flexmeasures add user`` Create a FlexMeasures user.
``flexmeasures add asset-type`` Create a new asset type.
``flexmeasures add asset`` Create a new asset.
``flexmeasures add sensor`` Add a new sensor.
``flexmeasures add weather-sensor`` Add a weather sensor.
``flexmeasures add external-weather-forecasts`` Collect weather forecasts from the DarkSky API.
``flexmeasures add beliefs`` Load beliefs from file.
``flexmeasures add forecasts`` Create forecasts.
================================================= =======================================

Expand All @@ -44,8 +47,11 @@ of which some are referred to in this documentation.
``flexmeasures delete account-role`` Delete a tenant account role.
``flexmeasures delete account`` Delete a tenant account & also their users (with assets and power measurements).
``flexmeasures delete user`` Delete a user & also their assets and power measurements.
``flexmeasures delete sensor`` Delete a sensor and all beliefs about it.
``flexmeasures delete measurements`` Delete measurements (with horizon <= 0).
``flexmeasures delete prognoses`` Delete forecasts and schedules (forecasts > 0).
``flexmeasures delete unchanged-beliefs`` Delete unchanged beliefs.
``flexmeasures delete nan-beliefs`` Delete NaN beliefs.
================================================= =======================================


Expand All @@ -67,4 +73,4 @@ of which some are referred to in this documentation.
``flexmeasures db-ops reset`` Reset database data and re-create tables from data model.
``flexmeasures db-ops restore`` Restore the dump file, see `db-ops dump` (run `reset` first).
``flexmeasures db-ops save`` Backup db content to files.
================================================= =======================================
================================================= =======================================
47 changes: 39 additions & 8 deletions documentation/getting-started.rst
View file
Expand Up @@ -147,20 +147,35 @@ Add your first asset

There are three ways to add assets:

Use the ``flexmeasures`` :ref:`cli`:
First, you can use the ``flexmeasures`` :ref:`cli`:

.. code-block::
flexmeasures add asset --name "my basement battery pack" --asset-type-name battery --capacity-in-MW 30 --event-resolution 2 --latitude 65 --longitude 123.76 --owner-id 1
flexmeasures add asset --name "my basement battery pack" --asset-type-id 3 --latitude 65 --longitude 123.76 --account-id 2
Here, I left out the ``--market-id`` parameter, because in this quickstart scenario I'm fine with the dummy market created with ``flexmeasures add structure`` above.
For the ownership, I got my user ID from the output of ``flexmeasures add user`` above, or I can browse to `FlexMeasures' user listing <http://localhost:5000/users>`_ and hover over my username.
For the asset type ID, I consult ``flexmeasures show asset-types``.

Or, you could head over to ``http://localhost:5000/assets`` (after you started FlexMeasures, see next step) and add a new asset there in a web form.
For the account ID, I looked at the output of ``flexmeasures add account`` (the command we issued above) ― I could also have consulted ``flexmeasures show accounts``.

The second way to add an asset is the UI ― head over to ``https://localhost:5000/assets`` (after you started FlexMeasures, see step "Run FlexMeasures" further down) and add a new asset there in a web form.

Finally, you can also use the `POST /api/v2_0/assets <api/v2_0.html#post--api-v2_0-assets>`_ endpoint in the FlexMeasures API to create an asset.


Add your first sensor
^^^^^^^^^^^^^^^^^^^^^^^^

Usually, we are here because we want to measure something with respect to our assets. Each assets can have sensors for that, so let's add a power sensor to our new battery asset, using the ``flexmeasures`` :ref:`cli`:

.. code-block::
flexmeasures add sensor --name power --unit MW --event-resolution 5 --timezone Europe/Amsterdam --asset-id 1 --attributes '{"capacity_in_mw": 7}'
The asset ID I got from the last CLI command, or I could consult ``flexmeasures show account --account-id <my-account-id>``.

.. note: The event resolution is given in minutes. Capacity is something unique to power sensors, so it is added as an attribute.
Run FlexMeasures
^^^^^^^^^^^^^^^^

Expand All @@ -182,11 +197,19 @@ When you see the dashboard, the map will not work. For that, you'll need to get
Add data
^^^^^^^^

You can use the `POST /api/v2_0/postMeterData <api/v2_0.html#post--api-v2_0-postMeterData>`_ endpoint in the FlexMeasures API to send meter data.
There are three ways to add data:

.. note:: `issue 56 <https://github.com/FlexMeasures/flexmeasures/issues/56>`_ should create a CLI function for adding a lot of data at once, from a CSV dataset.
First, you can load in data from a file (CSV or Excel) via the ``flexmeasures`` :ref:`cli`:

Also, you can add forecasts for your meter data with the ``flexmeasures add`` command, here is an example:
.. code-block::
flexmeasures add beliefs --file my-data.csv --skiprows 2 --delimiter ";" --source OurLegacyDatabase --sensor-id 1
This assumes you have a file `my-data.csv` with measurements, which was exported from some legacy database, and that the data is about our sensor with ID 1. This command has many options, so do use its ``--help`` function.

Second, you can use the `POST /api/v2_0/postMeterData <api/v2_0.html#post--api-v2_0-postMeterData>`_ endpoint in the FlexMeasures API to send meter data.

Finally, you can tell FlexMeasures to create forecasts for your meter data with the ``flexmeasures add forecasts`` command, here is an example:

.. code-block::
Expand Down Expand Up @@ -226,3 +249,11 @@ Install and configure Redis
^^^^^^^^^^^^^^^^^^^^^^^

To let FlexMeasures queue forecasting and scheduling jobs, install a `Redis <https://redis.io/>`_ server (or rent one) and configure access to it within FlexMeasures' config file (see above). You can find the necessary settings in :ref:`redis-config`.


Where to go from here?
------------------------

If your data structure is good, you should think about (continually) adding measurement data. This tutorial mentioned how to add data, but :ref:`_tut_posting_data` goes deeper with examples and terms & definitions.

Then, you probably want to use FlexMeasures to generate forecasts and schedules! For this, read further in :ref:`_tut_forecasting_scheduling`.
112 changes: 17 additions & 95 deletions flexmeasures/cli/data_add.py
View file
Expand Up @@ -23,10 +23,7 @@
GenericAssetSchema,
GenericAssetTypeSchema,
)
from flexmeasures.data.models.assets import Asset
from flexmeasures.data.schemas.assets import AssetSchema
from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
from flexmeasures.data.models.markets import Market
from flexmeasures.data.models.weather import WeatherSensor
from flexmeasures.data.schemas.weather import WeatherSensorSchema
from flexmeasures.data.models.data_sources import (
Expand All @@ -41,11 +38,6 @@ def fm_add_data():
"""FlexMeasures: Add data."""


@click.group("dev-add")
def fm_dev_add_data():
"""Developer CLI commands not yet meant for users: Add data."""


@fm_add_data.command("account-role")
@with_appcontext
@click.option("--name", required=True)
Expand Down Expand Up @@ -149,7 +141,7 @@ def new_user(
print(f"Successfully created user {created_user}")


@fm_dev_add_data.command("sensor")
@fm_add_data.command("sensor")
@with_appcontext
@click.option("--name", required=True)
@click.option("--unit", required=True, help="e.g. °C, m/s, kW/m²")
Expand All @@ -165,7 +157,8 @@ def new_user(
help="timezone as string, e.g. 'UTC' or 'Europe/Amsterdam'",
)
@click.option(
"--generic-asset-id",
"--asset-id",
"generic_asset_id",
required=True,
type=int,
help="Generic asset to assign this sensor to",
Expand Down Expand Up @@ -203,25 +196,25 @@ def add_sensor(**args):
print(f"You can access it at its entity address {sensor.entity_address}")


@fm_dev_add_data.command("generic-asset-type")
@fm_add_data.command("asset-type")
@with_appcontext
@click.option("--name", required=True)
@click.option(
"--description",
type=str,
help="Description (useful to explain acronyms, for example).",
)
def add_generic_asset_type(**args):
"""Add a generic asset type."""
def add_asset_type(**args):
"""Add an asset type."""
check_errors(GenericAssetTypeSchema().validate(args))
generic_asset_type = GenericAssetType(**args)
db.session.add(generic_asset_type)
db.session.commit()
print(f"Successfully created generic asset type with ID {generic_asset_type.id}")
print("You can now assign generic assets to it")
print(f"Successfully created asset type with ID {generic_asset_type.id}.")
print("You can now assign assets to it.")


@fm_dev_add_data.command("generic-asset")
@fm_add_data.command("asset")
@with_appcontext
@click.option("--name", required=True)
@click.option(
Expand All @@ -236,90 +229,20 @@ def add_generic_asset_type(**args):
)
@click.option("--account-id", type=int, required=True)
@click.option(
"--generic-asset-type-id",
"--asset-type-id",
"generic_asset_type_id",
required=True,
type=int,
help="Generic asset type to assign to this asset",
help="Asset type to assign to this asset",
)
def add_generic_asset(**args):
"""Add a generic asset."""
def add_asset(**args):
"""Add an asset."""
check_errors(GenericAssetSchema().validate(args))
generic_asset = GenericAsset(**args)
db.session.add(generic_asset)
db.session.commit()
print(f"Successfully created generic asset with ID {generic_asset.id}")
print("You can now assign sensors to it")


@fm_add_data.command("asset")
@with_appcontext
@click.option("--name", required=True)
@click.option("--asset-type-name", required=True)
@click.option(
"--unit",
help="unit of rate, just MW (default) for now",
type=click.Choice(["MW"]),
default="MW",
) # TODO: enable others
@click.option(
"--capacity-in-MW",
required=True,
type=float,
help="Maximum rate of this asset in MW",
)
@click.option(
"--event-resolution",
required=True,
type=int,
help="Expected resolution of the data in minutes",
)
@click.option(
"--latitude",
required=True,
type=float,
help="Latitude of the asset's location",
)
@click.option(
"--longitude",
required=True,
type=float,
help="Longitude of the asset's location",
)
@click.option(
"--owner-id", required=True, type=int, help="Id of the user who owns this asset."
)
@click.option(
"--market-id",
type=int,
help="Id of the market used to price this asset. Defaults to a dummy TOU market.",
)
@click.option(
"--timezone",
default="UTC",
help="timezone as string, e.g. 'UTC' (default) or 'Europe/Amsterdam'.",
)
def new_asset(**args):
"""
Create a new asset.
This is legacy, with the new data model we only want to add GenericAssets.
"""
check_timezone(args["timezone"])
# if no market given, select dummy market
if args["market_id"] is None:
dummy_market = Market.query.filter(Market.name == "dummy-tou").one_or_none()
if not dummy_market:
print(
"No market ID given and also no dummy TOU market available. Maybe add structure first."
)
raise click.Abort()
args["market_id"] = dummy_market.id
check_errors(AssetSchema().validate(args))
args["event_resolution"] = timedelta(minutes=args["event_resolution"])
asset = Asset(**args)
db.session.add(asset)
db.session.commit()
print(f"Successfully created asset with ID {asset.id}")
print(f"You can access it at its entity address {asset.entity_address}")
print(f"Successfully created asset with ID {generic_asset.id}.")
print("You can now assign sensors to it.")


@fm_add_data.command("weather-sensor")
Expand Down Expand Up @@ -375,7 +298,7 @@ def add_initial_structure():
populate_structure(db)


@fm_dev_add_data.command("beliefs")
@fm_add_data.command("beliefs")
@with_appcontext
@click.argument("file", type=click.Path(exists=True))
@click.option(
Expand Down Expand Up @@ -711,7 +634,6 @@ def collect_weather_data(region, location, num_cells, method, store_in_db):


app.cli.add_command(fm_add_data)
app.cli.add_command(fm_dev_add_data)


def check_timezone(timezone):
Expand Down
4 changes: 2 additions & 2 deletions flexmeasures/cli/data_delete.py
View file
Expand Up @@ -197,7 +197,7 @@ def delete_prognoses(
depopulate_prognoses(app.db, sensor_id)


@fm_delete_data.command("unchanged_beliefs")
@fm_delete_data.command("unchanged-beliefs")
@with_appcontext
@click.option(
"--sensor-id",
Expand Down Expand Up @@ -275,7 +275,7 @@ def delete_unchanged_beliefs(
print(f"Done! {num_beliefs_after} beliefs left")


@fm_delete_data.command("nan_beliefs")
@fm_delete_data.command("nan-beliefs")
@with_appcontext
@click.option(
"--sensor-id",
Expand Down

0 comments on commit 3c6d3a7

Please sign in to comment.