Issue 247 generic asset crud (#290)

* add a new method to specify an auth context (by callable) and discuss steps forward for use cases not covered yet by auth policy * add ACL for GenericAsset * added /dev/generic_assets API, CRUD uses it (passing manual testing) * fix asset icon in asset view * remove verbs from /api/dev/generic_assets API routes (HTTP method already defines what happens) * make CRUD UI tests (mocking the asset API) work * auth policy: change 'create' to 'create-children' permission (removing the need for issue#200); fix modelling problem if several distinct principal formulations exist for one permission * separate fresh tests from others; only generate copies of generic assets when we don't already have them (distinguish by name); re-add old asset tests; * add missing mock in another ui test * add unique contraints: on GenericAssetType.name & on GenericAssets for name+account_id * improve explanation of our authorization policy implementation * update changelog * improve changelog, also mention temporary API location * comments next to test cases Signed-off-by: Nicolas Höning <nicolas@seita.nl> * create_user function hashes the password -e Signed-off-by: Nicolas Höning <nicolas@seita.nl> * more informative validation errror Signed-off-by: Nicolas Höning <nicolas@seita.nl> * allow creation of public Assets in UI Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Fix display of owning account Signed-off-by: Nicolas Höning <nicolas@seita.nl> * better changelog docs Signed-off-by: Nicolas Höning <nicolas@seita.nl> * list sensors on asset page Signed-off-by: Nicolas Höning <nicolas@seita.nl> * do not fail getting latest state if capacity_in_mw is not in attributes Signed-off-by: Nicolas Höning <nicolas@seita.nl> * better default for capacity_in_mw Signed-off-by: Nicolas Höning <nicolas@seita.nl> * use id in the dict, for sensors lookup Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Suggested fixes to pr 290 (#299) Remove the entity addresses from the generic asset listing, and list the number of sensors instead. * grammar Signed-off-by: F.N. Claessen <felix@seita.nl> * Update generic asset listing, replacing entity addresses with number of sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * Avoid crashing on formatting None values Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix wrong variable name Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix incorrect type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * Partly revert cefe320 Signed-off-by: F.N. Claessen <felix@seita.nl> * small improvement on locating ea addresses Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Describe current state of transition Signed-off-by: Nicolas Höning <nicolas@seita.nl> * deprecation notice in old-style asset API documentation Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Fix type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * Update documentation Signed-off-by: F.N. Claessen <felix@seita.nl> * Update v2.0 connection entity addresses in documentation Signed-off-by: F.N. Claessen <felix@seita.nl> * Correct quickrefs in v2.0 documentation Signed-off-by: F.N. Claessen <felix@seita.nl> * Update other v2.0 sensor entity addresses in documentation Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix entity address dates corresponding to older previous domain name Signed-off-by: F.N. Claessen <felix@seita.nl> * Adapt more occurrences of older dates in entity addresses, and switch to fm1 scheme. Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Co-authored-by: F.N. Claessen <felix@seita.nl>
FlexMeasures · Jan 5, 2022 · 1bb2819 · 1bb2819
1 parent e0af47a
commit 1bb2819
Show file tree

Hide file tree

Showing 52 changed files with 1,150 additions and 588 deletions.
diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst
@@ -6,6 +6,17 @@ API change log
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
 
 
+v2.0-4 | 2022-01-04
+"""""""""""""""""""
+
+- Updated entity addresses in documentation, according to the fm1 scheme.
+- Changed the Introduction section:
+
+    - Rewrote the subsection on entity addresses to refer users to where they can find the entity addresses of their sensors.
+    - Rewrote the subsection on sensor identification (formerly known as asset identification) to place the fm1 scheme front and center.
+
+- Fixed the categorisation of the *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints from the User category to the Data category.
+
 v2.0-3 | 2021-06-07
 """""""""""""""""""
 
@@ -152,14 +163,14 @@ v1.2-1 | 2018-09-24
 
     {
         "type": "PostUdiEventRequest",
-        "event": "ea1.2018-06.io.flexmeasures.company:7:10:203:soc",
+        "event": "ea1.2021-01.io.flexmeasures.company:7:10:203:soc",
     }
 
     rather than the erroneously double-keyed:
 
     {
         "type": "PostUdiEventRequest",
-        "event": "ea1.2018-06.io.flexmeasures.company:7:10:203",
+        "event": "ea1.2021-01.io.flexmeasures.company:7:10:203",
         "type": "soc"
     }
 

diff --git a/documentation/api/introduction.rst b/documentation/api/introduction.rst
@@ -141,7 +141,7 @@ We distinguish the following roles with different access rights to the individua
 - admin
 - Aggregator
 - Supplier: an energy retailer (see :ref:`supplier`)
-- Prosumer: an asset owner (see :ref:`prosumer`)
+- Prosumer: owner of a grid connection (see :ref:`prosumer`)
 - ESCo: an energy service company (see :ref:`esco`)
 - MDC: a meter data company (see :ref:`mdc`)
 - DSO: a distribution system operator (see :ref:`dso`)
@@ -182,7 +182,7 @@ The API, however, does not distinguish between singular and plural key notation.
 Connections and entity addresses
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Connections are end points of the grid at which an asset is located. 
+A connection represents an end point of the grid, at which an electricity sensor (power meter) is located.
 Connections should be identified with an entity address following the EA1 addressing scheme prescribed by USEF[1],
 which is mostly taken from IETF RFC 3720 [2]:
 
@@ -199,17 +199,23 @@ Here is a full example for a FlexMeasures connection address:
 .. code-block:: json
 
     {
-        "connection": "ea1.2021-02.io.flexmeasures.company:fm0.30:73"
+        "connection": "ea1.2021-02.io.flexmeasures.company:fm1.73"
     }
 
-where FlexMeasures runs at `company.flexmeasures.io` (which the current domain owner started using in February 2021), and the locally unique string is of scheme `fm0` (see below) and the asset ID is 73. The asset's owner ID is 30, but this part is optional.
+where FlexMeasures runs at `company.flexmeasures.io` (which the current domain owner started using in February 2021), and the locally unique string uses the `fm1` scheme (see below) to identify sensor ID 73.
 
-Both the owner ID and the asset ID, as well as the full entity address can be obtained on the asset's listing:
+Assets are listed at:
 
 .. code-block:: html
 
     https://company.flexmeasures.io/assets
 
+The full entity addresses of all of the asset's sensors can be obtained on the asset's page, e.g. for asset 81:
+
+.. code-block:: html
+
+    https://company.flexmeasures.io/assets/81
+
 
 Entity address structure
 """"""""""""""""""""""""""
@@ -231,41 +237,31 @@ Some deeper explanations about an entity address:
 [3] https://tools.ietf.org/html/rfc3721
 
 
-Types of asset identifications used in FlexMeasures
+Types of sensor identification used in FlexMeasures
 """"""""""""""""""""""""""""""""""""""""""""""""""""
 
-FlexMeasures expects the locally unique string string to contain information in
-a certain structure. We distinguish type ``fm0`` and type ``fm1`` FlexMeasures entity addresses.
-
-The ``fm0`` scheme is the original scheme. It identifies connected assets, weather stations, markets and UDI events in different ways. 
+FlexMeasures expects the locally unique string string to contain information in a certain structure.
+We distinguish type ``fm0`` and type ``fm1`` FlexMeasures entity addresses.
 
-Examples for the fm0 scheme:
+The ``fm1`` scheme is the latest version.
+It uses the fact that all FlexMeasures sensors have unique IDs.
 
-- connection = ea1.2021-01.localhost:fm0.40:30
-- connection = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>
-- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.temperature:52:73.0
-- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.<sensor_type>:<latitude>:<longitude>
-- market = ea1.2021-01.io.flexmeasures:fm0.epex_da
-- market = ea1.2021-01.io.flexmeasures:fm0.<market_name>
-- event = ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
-- event = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>:<event_id>:<event_type>
+.. code-block::
 
-This scheme is explicit but also a little cumbersome to use, as one needs to look up the type or even owner (for assets), and weather sensors are identified by coordinates.
-For the fm0 scheme, the 'fm0.' part is optional, for backwards compatibility.
+    ea1.2021-01.io.flexmeasures:fm1.42
+    ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
 
+.. todo:: UDI events are not yet modelled in the fm1 scheme
 
-The ``fm1`` scheme is the latest version, currently under development. It works with the database structure 
-we are developing in the background, where all connected sensors have unique IDs. This makes it more straightforward (the scheme works the same way for all types of sensors), if less explicit.
+The ``fm0`` scheme is the original scheme.
+It identified different types of sensors (such as connections, weather sensors and markets) in different ways.
+The ``fm0`` scheme has been deprecated for the most part and is no longer supported officially.
+Only UDI events still need to be sent using the fm0 scheme.
 
-Examples for the fm1 scheme:
+.. code-block::
 
-- sensor = ea1.2021-01.io.flexmeasures:fm1.42
-- sensor = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- connection = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- market = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- weather_station = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-
-.. todo:: UDI events are not yet modelled in the fm1 scheme, but will probably be ea1.2021-01.io.flexmeasures:fm1.<actuator_id>
+    ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
+    ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<sensor_id>:<event_id>:<event_type>
 
 
 Groups
@@ -280,8 +276,8 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
         "groups": [
             {
                 "connections": [
-                    "ea1.2021-02.io.flexmeasures.company:fm0.30:71",
-                    "ea1.2021-02.io.flexmeasures.company:fm0.30:72"
+                    "ea1.2021-02.io.flexmeasures.company:fm1.71",
+                    "ea1.2021-02.io.flexmeasures.company:fm1.72"
                 ],
                 "values": [
                     306.66,
@@ -293,7 +289,7 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
                 ]
             },
             {
-                "connection": "ea1.2021-02.io.flexmeasures.company:fm0.30:73"
+                "connection": "ea1.2021-02.io.flexmeasures.company:fm1.73"
                 "values": [
                     306.66,
                     0,
@@ -315,8 +311,8 @@ In case of a single group of connections, the message may be flattened to:
 
     {
         "connections": [
-            "ea1.2021-02.io.flexmeasures.company:fm0.30:71",
-            "ea1.2021-02.io.flexmeasures.company:fm0.30:72"
+            "ea1.2021-02.io.flexmeasures.company:fm1.71",
+            "ea1.2021-02.io.flexmeasures.company:fm1.72"
         ],
         "values": [
             306.66,

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -6,7 +6,7 @@ v0.8.0 | November XX, 2021
 ===========================
 
 .. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
-.. warning:: Changes to asset attributes made via the UI or API are not reflected in the new data model until `issue #247 <https://github.com/FlexMeasures/flexmeasures/issues/247>`_ is resolved.
+.. note:: v0.8.0 is doing much of the work we need to do to move to the new data model (see :ref:`note_on_datamodel_transition`). We hope to keep the migration steps for users very limited. One thing you'll notice is that we are copying over existing data to the new model (which will be kept in sync) with the `db upgrade` command (see warning above), which can take a few minutes.
 
 New features
 -----------
@@ -25,6 +25,7 @@ Infrastructure / Support
 * Add sensor method to obtain just its latest state (excl. forecasts) [see `PR #235 <http://www.github.com/FlexMeasures/flexmeasures/pull/235>`_]
 * Migrate attributes of assets, markets and weather sensors to our new sensor model [see `PR #254 <http://www.github.com/FlexMeasures/flexmeasures/pull/254>`_ and `project 9 <http://www.github.com/FlexMeasures/flexmeasures/projects/9>`_]
 * Migrate all time series data to our new sensor data model based on the `timely beliefs <https://github.com/SeitaBV/timely-beliefs>`_ lib [see `PR #286 <http://www.github.com/FlexMeasures/flexmeasures/pull/286>`_ and `project 9 <http://www.github.com/FlexMeasures/flexmeasures/projects/9>`_]
+* Support the new asset model (which describes the organisational structure, rather than sensors and data) in UI and API. Until the transition to our new data model is completed, the new API for assets is at `/api/dev/generic_assets`. [see `PR #251 <http://www.github.com/FlexMeasures/flexmeasures/pull/251>`_ and `PR #290 <http://www.github.com/FlexMeasures/flexmeasures/pulls/290>`_]
 
 
 v0.7.1 | November 08, 2021

diff --git a/documentation/dev/note-on-datamodel-transition.rst b/documentation/dev/note-on-datamodel-transition.rst
@@ -58,3 +58,13 @@ Here is a brief list:
 - `Sensor relations and GeneralizedAssets with metadata <https://github.com/FlexMeasures/flexmeasures/projects/9>`_: We are generalizing our database structure for organising energy data, to support all sorts of sensors and relationships between them. We do this so we can better support the diverse set of use cases for energy flexibility.
 - `UI views for GeneralizedAssets <https://github.com/FlexMeasures/flexmeasures/projects/10>`_: We are updating our UI views (dashboard maps and analytics charts) according to our new database structure for organising energy data. We do this so users can customize what they want to see.
 - `Deprecate old database models <https://github.com/FlexMeasures/flexmeasures/projects/11>`_: We are deprecating the Power, Price and Weather tables in favour of the TimedBelief table, and deprecating the Asset, Market and WeatherSensor tables in favour of the Sensor and GeneralizedAsset tables. We are doing this so users can move their data to the new database model.
+
+
+The state of the transition (January 2022, v0.8.0)
+---------------------------------------------------
+
+Project 9 was implemented, which moved a lot of structure over, as well as actual data and some UI (dashboard, assets). We believe that was the hardest part.
+
+We are now close to being able to deprecate the old database models and route the API to the new model (see project 11). The API for assets is still in place, but the new one is already working (at /api/dev/generic_assets) and is powering what is shown in the UI.
+
+We take care to support people on the old data mode so the transition will be as smooth as possible, as we said above. One part of this is that the ``flexmeasures db upgrade`` command copies your data to the new model. Also, creating new data (e.g. old-style assets) creates new-style data (e.g. assets/sensors) automatically. However, some edge cases are not supported in this way. For instance, edited asset meta data might have to be re-entered later. Feel free to contact us to discuss the transition if needed.
diff --git a/documentation/tut/forecasting_scheduling.rst b/documentation/tut/forecasting_scheduling.rst
@@ -89,7 +89,7 @@ As an example, consider the same UDI event as we saw earlier (in :ref:`posting_f
 
     {
         "type": "PostUdiEventRequest",
-        "event": "ea1.2018-06.io.flexmeasures.company:7:10:204:soc-with-targets",
+        "event": "ea1.2021-01.io.flexmeasures.company:7:10:204:soc-with-targets",
         "value": 12.1,
         "datetime": "2015-06-02T10:00:00+00:00",
         "unit": "kWh",
@@ -132,7 +132,7 @@ This example requests a prognosis for 24 hours, with a rolling horizon of 6 hour
 
     {
         "type": "GetPrognosisRequest",
-        "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+        "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
         "start": "2015-01-01T00:00:00+00:00",
         "duration": "PT24H",
         "horizon": "PT6H",
@@ -159,7 +159,7 @@ This example of a request body shows that we want to look up a control signal fo
 
         {
             "type": "GetDeviceMessageRequest",
-            "event": "ea1.2018-06.io.flexmeasures.company:7:10:203:soc"
+            "event": "ea1.2021-01.io.flexmeasures.company:7:10:203:soc"
         }
 
 The following example response indicates that FlexMeasures planned ahead 45 minutes for this battery.
@@ -170,7 +170,7 @@ Each value represents the average power over a 15 minute time interval.
 
         {
             "type": "GetDeviceMessageResponse",
-            "event": "ea1.2018-06.io.flexmeasures.company:7:10:203",
+            "event": "ea1.2021-01.io.flexmeasures.company:7:10:203",
             "values": [
                 2.15,
                 3,

diff --git a/documentation/tut/posting_data.rst b/documentation/tut/posting_data.rst
@@ -35,22 +35,23 @@ Weather data (both observations and forecasts) can be posted to `POST  /api/v2_0
 
     https://company.flexmeasures.io/api/<version>/postWeatherData
 
-Weather data can be posted for the following three types of weather sensors:
+Weather data can be posted for different types of sensors, such as:
 
 - "radiation" (with kW/m² as unit)
 - "temperature" (with °C as unit)
-- "wind_speed" (with m/s as unit)
+- "wind speed" (with m/s as unit)
 
 The sensor type is part of the unique entity address for each sensor, together with the sensor's latitude and longitude.
 
-This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute intervals between 3.00pm and 4.30pm for a weather sensor located at latitude 33.4843866 and longitude 126.477859. This sensor is located in Korea's timezone ― we also reflect that in the datetimes.
+This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute intervals between 3.00pm and 4.30pm for a weather sensor with id 602.
+As this sensor is located in Korea's timezone ― we also reflect that in the datetimes.
 The forecasts were made at noon, as the ``prior`` field indicates.
 
 .. code-block:: json
 
         {
             "type": "PostWeatherDataRequest",
-            "sensor": "ea1.2018-06.io.flexmeasures.company:temperature:33.4843866:126.477859",
+            "sensor": "ea1.2021-01.io.flexmeasures.company:fm1.602",
             "values": [
                 20.04,
                 20.23,
@@ -106,14 +107,14 @@ Price data (both observations and forecasts) can be posted to `POST  /api/v2_0/p
     https://company.flexmeasures.io/api/<version>/postPriceData
 
 This example "PostPriceDataRequest" message posts prices for hourly intervals between midnight and midnight the next day
-for the Korean Power Exchange (KPX) day-ahead auction.
+for the Korean Power Exchange (KPX) day-ahead auction, registered under sensor 16.
 The ``prior`` indicates that the prices were published at 3pm on December 31st 2014 (i.e. the clearing time of the KPX day-ahead market, which is at 3 PM on the previous day ― see below for a deeper explanation).
 
 .. code-block:: json
 
     {
         "type": "PostPriceDataRequest",
-        "market": "ea1.2018-06.io.flexmeasures.company:kpx_da",
+        "market": "ea1.2021-01.io.flexmeasures.company:fm1.16",
         "values": [
             52.37,
             51.14,
@@ -187,7 +188,7 @@ A single average power value for a 15-minute time interval for a single connecti
 
     {
         "type": "PostMeterDataRequest",
-        "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+        "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
         "value": 220,
         "start": "2015-01-01T00:00:00+00:00",
         "duration": "PT0H15M",
@@ -204,7 +205,7 @@ Multiple values (indicating a univariate timeseries) for 15-minute time interval
 
     {
         "type": "PostMeterDataRequest",
-        "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+        "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
         "values": [
             220,
             210,
@@ -228,8 +229,8 @@ We recommend to use this notation for zero values only.
     {
         "type": "PostMeterDataRequest",
         "connections": [
-            "ea1.2018-06.io.flexmeasures.company:1:1",
-            "ea1.2018-06.io.flexmeasures.company:1:2"
+            "ea1.2021-01.io.flexmeasures.company:fm1.1",
+            "ea1.2021-01.io.flexmeasures.company:fm1.2"
         ],
         "value": 10,
         "start": "2015-01-01T00:00:00+00:00",
@@ -249,11 +250,11 @@ Single different values for a 15-minute time interval for two connections, poste
         "type": "PostMeterDataRequest",
         "groups": [
             {
-                "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+                "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
                 "value": 220
             },
             {
-                "connection": "ea1.2018-06.io.flexmeasures.company:1:2",
+                "connection": "ea1.2021-01.io.flexmeasures.company:fm1.2",
                 "value": 300
             }
         ],
@@ -274,15 +275,15 @@ Multiple values (indicating a univariate timeseries) for 15-minute time interval
         "type": "PostMeterDataRequest",
         "groups": [
             {
-                "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+                "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
                 "values": [
                     220,
                     210,
                     200
                 ]
             },
             {
-                "connection": "ea1.2018-06.io.flexmeasures.company:1:2",
+                "connection": "ea1.2021-01.io.flexmeasures.company:fm1.2",
                 "values": [
                     300,
                     303,
@@ -318,7 +319,7 @@ From this, FlexMeasures derives the energy flexibility this battery has in the n
 
         {
             "type": "PostUdiEventRequest",
-            "event": "ea1.2018-06.io.flexmeasures.company:7:10:203:soc",
+            "event": "ea1.2021-01.io.flexmeasures.company:7:10:203:soc",
             "value": 12.1,
             "datetime": "2015-06-02T10:00:00+00:00",
             "unit": "kWh"

diff --git a/flexmeasures/api/common/schemas/users.py b/flexmeasures/api/common/schemas/users.py
@@ -10,21 +10,23 @@ class AccountIdField(fields.Integer):
     Field that represents an account ID. It de-serializes from the account id to an account instance.
     """
 
-    def __init__(self, *args, **kwargs):
-        kwargs["load_default"] = (
-            lambda: current_user.account if not current_user.is_anonymous else None
-        )
-        super().__init__(*args, **kwargs)
-
-    def _deserialize(self, account_id: int, attr, obj, **kwargs) -> Account:
+    def _deserialize(self, account_id: str, attr, obj, **kwargs) -> Account:
         account: Account = Account.query.filter_by(id=int(account_id)).one_or_none()
         if account is None:
-            raise abort(404, f"Account {id} not found")
+            raise abort(404, f"Account {account_id} not found")
         return account
 
     def _serialize(self, account: Account, attr, data, **kwargs) -> int:
         return account.id
 
+    @classmethod
+    def load_current(cls):
+        """
+        Use this with the load_default arg to __init__ if you want the current user's account
+        by default.
+        """
+        return current_user.account if not current_user.is_anonymous else None
+
 
 class UserIdField(fields.Integer):
     """