Issue 247 generic asset crud

documentation/api/change_log.rst

@@ -6,6 +6,17 @@ API change log
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
 
 
+v2.0-4 | 2022-01-04
+"""""""""""""""""""
+
+- Updated entity addresses in documentation, according to the fm1 scheme.
+- Changed the Introduction section:
+
+    - Rewrote the subsection on entity addresses to refer users to where they can find the entity addresses of their sensors.
+    - Rewrote the subsection on sensor identification (formerly known as asset identification) to place the fm1 scheme front and center.
+
+- Fixed the categorisation of the *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints from the User category to the Data category.
+
 v2.0-3 | 2021-06-07
 """""""""""""""""""
 
@@ -152,14 +163,14 @@ v1.2-1 | 2018-09-24
 
     {
         "type": "PostUdiEventRequest",
-        "event": "ea1.2018-06.io.flexmeasures.company:7:10:203:soc",
+        "event": "ea1.2021-01.io.flexmeasures.company:7:10:203:soc",
     }
 
     rather than the erroneously double-keyed:
 
     {
         "type": "PostUdiEventRequest",
-        "event": "ea1.2018-06.io.flexmeasures.company:7:10:203",
+        "event": "ea1.2021-01.io.flexmeasures.company:7:10:203",
         "type": "soc"
     }
 

documentation/api/introduction.rst

@@ -141,7 +141,7 @@ We distinguish the following roles with different access rights to the individua
 - admin
 - Aggregator
 - Supplier: an energy retailer (see :ref:`supplier`)
-- Prosumer: an asset owner (see :ref:`prosumer`)
+- Prosumer: owner of a grid connection (see :ref:`prosumer`)
 - ESCo: an energy service company (see :ref:`esco`)
 - MDC: a meter data company (see :ref:`mdc`)
 - DSO: a distribution system operator (see :ref:`dso`)
@@ -182,7 +182,7 @@ The API, however, does not distinguish between singular and plural key notation.
 Connections and entity addresses
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Connections are end points of the grid at which an asset is located. 
+A connection represents an end point of the grid, at which an electricity sensor (power meter) is located.
 Connections should be identified with an entity address following the EA1 addressing scheme prescribed by USEF[1],
 which is mostly taken from IETF RFC 3720 [2]:
 
@@ -199,17 +199,23 @@ Here is a full example for a FlexMeasures connection address:
 .. code-block:: json
 
     {
-        "connection": "ea1.2021-02.io.flexmeasures.company:fm0.30:73"
+        "connection": "ea1.2021-02.io.flexmeasures.company:fm1.73"
     }
 
-where FlexMeasures runs at `company.flexmeasures.io` (which the current domain owner started using in February 2021), and the locally unique string is of scheme `fm0` (see below) and the asset ID is 73. The asset's owner ID is 30, but this part is optional.
+where FlexMeasures runs at `company.flexmeasures.io` (which the current domain owner started using in February 2021), and the locally unique string uses the `fm1` scheme (see below) to identify sensor ID 73.
 
-Both the owner ID and the asset ID, as well as the full entity address can be obtained on the asset's listing:
+Assets are listed at:
 
 .. code-block:: html
 
     https://company.flexmeasures.io/assets
 
+The full entity addresses of all of the asset's sensors can be obtained on the asset's page, e.g. for asset 81:
+
+.. code-block:: html
+
+    https://company.flexmeasures.io/assets/81
+
 
 Entity address structure
 """"""""""""""""""""""""""
@@ -231,41 +237,31 @@ Some deeper explanations about an entity address:
 [3] https://tools.ietf.org/html/rfc3721
 
 
-Types of asset identifications used in FlexMeasures
+Types of sensor identification used in FlexMeasures
 """"""""""""""""""""""""""""""""""""""""""""""""""""
 
-FlexMeasures expects the locally unique string string to contain information in
-a certain structure. We distinguish type ``fm0`` and type ``fm1`` FlexMeasures entity addresses.
-
-The ``fm0`` scheme is the original scheme. It identifies connected assets, weather stations, markets and UDI events in different ways. 
+FlexMeasures expects the locally unique string string to contain information in a certain structure.
+We distinguish type ``fm0`` and type ``fm1`` FlexMeasures entity addresses.
 
-Examples for the fm0 scheme:
+The ``fm1`` scheme is the latest version.
+It uses the fact that all FlexMeasures sensors have unique IDs.
 
-- connection = ea1.2021-01.localhost:fm0.40:30
-- connection = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>
-- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.temperature:52:73.0
-- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.<sensor_type>:<latitude>:<longitude>
-- market = ea1.2021-01.io.flexmeasures:fm0.epex_da
-- market = ea1.2021-01.io.flexmeasures:fm0.<market_name>
-- event = ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
-- event = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>:<event_id>:<event_type>
+.. code-block::
 
-This scheme is explicit but also a little cumbersome to use, as one needs to look up the type or even owner (for assets), and weather sensors are identified by coordinates.
-For the fm0 scheme, the 'fm0.' part is optional, for backwards compatibility.
+    ea1.2021-01.io.flexmeasures:fm1.42
+    ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
 
+.. todo:: UDI events are not yet modelled in the fm1 scheme
 
-The ``fm1`` scheme is the latest version, currently under development. It works with the database structure 
-we are developing in the background, where all connected sensors have unique IDs. This makes it more straightforward (the scheme works the same way for all types of sensors), if less explicit.
+The ``fm0`` scheme is the original scheme.
+It identified different types of sensors (such as connections, weather sensors and markets) in different ways.
+The ``fm0`` scheme has been deprecated for the most part and is no longer supported officially.
+Only UDI events still need to be sent using the fm0 scheme.
 
-Examples for the fm1 scheme:
+.. code-block::
 
-- sensor = ea1.2021-01.io.flexmeasures:fm1.42
-- sensor = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- connection = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- market = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- weather_station = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-
-.. todo:: UDI events are not yet modelled in the fm1 scheme, but will probably be ea1.2021-01.io.flexmeasures:fm1.<actuator_id>
+    ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
+    ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<sensor_id>:<event_id>:<event_type>
 
 
 Groups
@@ -280,8 +276,8 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
         "groups": [
             {
                 "connections": [
-                    "ea1.2021-02.io.flexmeasures.company:fm0.30:71",
-                    "ea1.2021-02.io.flexmeasures.company:fm0.30:72"
+                    "ea1.2021-02.io.flexmeasures.company:fm1.71",
+                    "ea1.2021-02.io.flexmeasures.company:fm1.72"
                 ],
                 "values": [
                     306.66,
@@ -293,7 +289,7 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
                 ]
             },
             {
-                "connection": "ea1.2021-02.io.flexmeasures.company:fm0.30:73"
+                "connection": "ea1.2021-02.io.flexmeasures.company:fm1.73"
                 "values": [
                     306.66,
                     0,
@@ -315,8 +311,8 @@ In case of a single group of connections, the message may be flattened to:
 
     {
         "connections": [
-            "ea1.2021-02.io.flexmeasures.company:fm0.30:71",
-            "ea1.2021-02.io.flexmeasures.company:fm0.30:72"
+            "ea1.2021-02.io.flexmeasures.company:fm1.71",
+            "ea1.2021-02.io.flexmeasures.company:fm1.72"
         ],
         "values": [
             306.66,

documentation/changelog.rst

@@ -6,7 +6,7 @@ v0.8.0 | November XX, 2021
 ===========================
 
 .. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
-.. warning:: Changes to asset attributes made via the UI or API are not reflected in the new data model until `issue #247 <https://github.com/FlexMeasures/flexmeasures/issues/247>`_ is resolved.
+.. note:: v0.8.0 is doing much of the work we need to do to move to the new data model (see :ref:`note_on_datamodel_transition`). We hope to keep the migration steps for users very limited. One thing you'll notice is that we are copying over existing data to the new model (which will be kept in sync) with the `db upgrade` command (see warning above), which can take a few minutes.
 
 New features
 -----------
@@ -25,6 +25,7 @@ Infrastructure / Support
 * Add sensor method to obtain just its latest state (excl. forecasts) [see `PR #235 <http://www.github.com/FlexMeasures/flexmeasures/pull/235>`_]
 * Migrate attributes of assets, markets and weather sensors to our new sensor model [see `PR #254 <http://www.github.com/FlexMeasures/flexmeasures/pull/254>`_ and `project 9 <http://www.github.com/FlexMeasures/flexmeasures/projects/9>`_]
 * Migrate all time series data to our new sensor data model based on the `timely beliefs <https://github.com/SeitaBV/timely-beliefs>`_ lib [see `PR #286 <http://www.github.com/FlexMeasures/flexmeasures/pull/286>`_ and `project 9 <http://www.github.com/FlexMeasures/flexmeasures/projects/9>`_]
+* Support the new asset model (which describes the organisational structure, rather than sensors and data) in UI and API. Until the transition to our new data model is completed, the new API for assets is at `/api/dev/generic_assets`. [see `PR #251 <http://www.github.com/FlexMeasures/flexmeasures/pull/251>`_ and `PR #290 <http://www.github.com/FlexMeasures/flexmeasures/pulls/290>`_]
 
 
 v0.7.1 | November 08, 2021

documentation/dev/note-on-datamodel-transition.rst

@@ -58,3 +58,13 @@ Here is a brief list:
 - `Sensor relations and GeneralizedAssets with metadata <https://github.com/FlexMeasures/flexmeasures/projects/9>`_: We are generalizing our database structure for organising energy data, to support all sorts of sensors and relationships between them. We do this so we can better support the diverse set of use cases for energy flexibility.
 - `UI views for GeneralizedAssets <https://github.com/FlexMeasures/flexmeasures/projects/10>`_: We are updating our UI views (dashboard maps and analytics charts) according to our new database structure for organising energy data. We do this so users can customize what they want to see.
 - `Deprecate old database models <https://github.com/FlexMeasures/flexmeasures/projects/11>`_: We are deprecating the Power, Price and Weather tables in favour of the TimedBelief table, and deprecating the Asset, Market and WeatherSensor tables in favour of the Sensor and GeneralizedAsset tables. We are doing this so users can move their data to the new database model.
+
+
+The state of the transition (January 2022, v0.8.0)
+---------------------------------------------------
+
+Project 9 was implemented, which moved a lot of structure over, as well as actual data and some UI (dashboard, assets). We believe that was the hardest part.
+
+We are now close to being able to deprecate the old database models and route the API to the new model (see project 11). The API for assets is still in place, but the new one is already working (at /api/dev/generic_assets) and is powering what is shown in the UI.
+
+We take care to support people on the old data mode so the transition will be as smooth as possible, as we said above. One part of this is that the ``flexmeasures db upgrade`` command copies your data to the new model. Also, creating new data (e.g. old-style assets) creates new-style data (e.g. assets/sensors) automatically. However, some edge cases are not supported in this way. For instance, edited asset meta data might have to be re-entered later. Feel free to contact us to discuss the transition if needed.

documentation/tut/forecasting_scheduling.rst

@@ -89,7 +89,7 @@ As an example, consider the same UDI event as we saw earlier (in :ref:`posting_f
 
     {
         "type": "PostUdiEventRequest",
-        "event": "ea1.2018-06.io.flexmeasures.company:7:10:204:soc-with-targets",
+        "event": "ea1.2021-01.io.flexmeasures.company:7:10:204:soc-with-targets",
         "value": 12.1,
         "datetime": "2015-06-02T10:00:00+00:00",
         "unit": "kWh",
@@ -132,7 +132,7 @@ This example requests a prognosis for 24 hours, with a rolling horizon of 6 hour
 
     {
         "type": "GetPrognosisRequest",
-        "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+        "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
         "start": "2015-01-01T00:00:00+00:00",
         "duration": "PT24H",
         "horizon": "PT6H",
@@ -159,7 +159,7 @@ This example of a request body shows that we want to look up a control signal fo
 
         {
             "type": "GetDeviceMessageRequest",
-            "event": "ea1.2018-06.io.flexmeasures.company:7:10:203:soc"
+            "event": "ea1.2021-01.io.flexmeasures.company:7:10:203:soc"
         }
 
 The following example response indicates that FlexMeasures planned ahead 45 minutes for this battery.
@@ -170,7 +170,7 @@ Each value represents the average power over a 15 minute time interval.
 
         {
             "type": "GetDeviceMessageResponse",
-            "event": "ea1.2018-06.io.flexmeasures.company:7:10:203",
+            "event": "ea1.2021-01.io.flexmeasures.company:7:10:203",
             "values": [
                 2.15,
                 3,

documentation/tut/posting_data.rst

@@ -35,22 +35,23 @@ Weather data (both observations and forecasts) can be posted to `POST  /api/v2_0
 
     https://company.flexmeasures.io/api/<version>/postWeatherData
 
-Weather data can be posted for the following three types of weather sensors:
+Weather data can be posted for different types of sensors, such as:
 
 - "radiation" (with kW/m² as unit)
 - "temperature" (with °C as unit)
-- "wind_speed" (with m/s as unit)
+- "wind speed" (with m/s as unit)
 
 The sensor type is part of the unique entity address for each sensor, together with the sensor's latitude and longitude.
 
-This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute intervals between 3.00pm and 4.30pm for a weather sensor located at latitude 33.4843866 and longitude 126.477859. This sensor is located in Korea's timezone ― we also reflect that in the datetimes.
+This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute intervals between 3.00pm and 4.30pm for a weather sensor with id 602.
+As this sensor is located in Korea's timezone ― we also reflect that in the datetimes.
 The forecasts were made at noon, as the ``prior`` field indicates.
 
 .. code-block:: json
 
         {
             "type": "PostWeatherDataRequest",
-            "sensor": "ea1.2018-06.io.flexmeasures.company:temperature:33.4843866:126.477859",
+            "sensor": "ea1.2021-01.io.flexmeasures.company:fm1.602",
             "values": [
                 20.04,
                 20.23,
@@ -106,14 +107,14 @@ Price data (both observations and forecasts) can be posted to `POST  /api/v2_0/p
     https://company.flexmeasures.io/api/<version>/postPriceData
 
 This example "PostPriceDataRequest" message posts prices for hourly intervals between midnight and midnight the next day
-for the Korean Power Exchange (KPX) day-ahead auction.
+for the Korean Power Exchange (KPX) day-ahead auction, registered under sensor 16.
 The ``prior`` indicates that the prices were published at 3pm on December 31st 2014 (i.e. the clearing time of the KPX day-ahead market, which is at 3 PM on the previous day ― see below for a deeper explanation).
 
 .. code-block:: json
 
     {
         "type": "PostPriceDataRequest",
-        "market": "ea1.2018-06.io.flexmeasures.company:kpx_da",
+        "market": "ea1.2021-01.io.flexmeasures.company:fm1.16",
         "values": [
             52.37,
             51.14,
@@ -187,7 +188,7 @@ A single average power value for a 15-minute time interval for a single connecti
 
     {
         "type": "PostMeterDataRequest",
-        "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+        "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
         "value": 220,
         "start": "2015-01-01T00:00:00+00:00",
         "duration": "PT0H15M",
@@ -204,7 +205,7 @@ Multiple values (indicating a univariate timeseries) for 15-minute time interval
 
     {
         "type": "PostMeterDataRequest",
-        "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+        "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
         "values": [
             220,
             210,
@@ -228,8 +229,8 @@ We recommend to use this notation for zero values only.
     {
         "type": "PostMeterDataRequest",
         "connections": [
-            "ea1.2018-06.io.flexmeasures.company:1:1",
-            "ea1.2018-06.io.flexmeasures.company:1:2"
+            "ea1.2021-01.io.flexmeasures.company:fm1.1",
+            "ea1.2021-01.io.flexmeasures.company:fm1.2"
         ],
         "value": 10,
         "start": "2015-01-01T00:00:00+00:00",
@@ -249,11 +250,11 @@ Single different values for a 15-minute time interval for two connections, poste
         "type": "PostMeterDataRequest",
         "groups": [
             {
-                "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+                "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
                 "value": 220
             },
             {
-                "connection": "ea1.2018-06.io.flexmeasures.company:1:2",
+                "connection": "ea1.2021-01.io.flexmeasures.company:fm1.2",
                 "value": 300
             }
         ],
@@ -274,15 +275,15 @@ Multiple values (indicating a univariate timeseries) for 15-minute time interval
         "type": "PostMeterDataRequest",
         "groups": [
             {
-                "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
+                "connection": "ea1.2021-01.io.flexmeasures.company:fm1.1",
                 "values": [
                     220,
                     210,
                     200
                 ]
             },
             {
-                "connection": "ea1.2018-06.io.flexmeasures.company:1:2",
+                "connection": "ea1.2021-01.io.flexmeasures.company:fm1.2",
                 "values": [
                     300,
                     303,
@@ -318,7 +319,7 @@ From this, FlexMeasures derives the energy flexibility this battery has in the n
 
         {
             "type": "PostUdiEventRequest",
-            "event": "ea1.2018-06.io.flexmeasures.company:7:10:203:soc",
+            "event": "ea1.2021-01.io.flexmeasures.company:7:10:203:soc",
             "value": 12.1,
             "datetime": "2015-06-02T10:00:00+00:00",
             "unit": "kWh"

flexmeasures/api/common/schemas/users.py

@@ -10,21 +10,23 @@ class AccountIdField(fields.Integer):
     Field that represents an account ID. It de-serializes from the account id to an account instance.
     """
 
-    def __init__(self, *args, **kwargs):
-        kwargs["load_default"] = (
-            lambda: current_user.account if not current_user.is_anonymous else None
-        )
-        super().__init__(*args, **kwargs)
-
-    def _deserialize(self, account_id: int, attr, obj, **kwargs) -> Account:
+    def _deserialize(self, account_id: str, attr, obj, **kwargs) -> Account:
         account: Account = Account.query.filter_by(id=int(account_id)).one_or_none()
         if account is None:
-            raise abort(404, f"Account {id} not found")
+            raise abort(404, f"Account {account_id} not found")
         return account
 
     def _serialize(self, account: Account, attr, data, **kwargs) -> int:
         return account.id
 
+    @classmethod
+    def load_current(cls):
+        """
+        Use this with the load_default arg to __init__ if you want the current user's account
+        by default.
+        """
+        return current_user.account if not current_user.is_anonymous else None
+
 
 class UserIdField(fields.Integer):
     """