Merge pull request #254 from FlexMeasures/project-9

Merge project 9: support Sensor and Asset diversity

Readme.md

@@ -29,6 +29,7 @@ FlexMeasures provides three core values:
 FlexMeasures is developed by [Seita BV](https://www.seita.nl) in The Netherlands.
 
 We made FlexMeasures freely available under the Apache2.0 licence. Please get in contact if you use FlexMeasures or are considering it.
+See also [Seita's Github profile](https://github.com/SeitaBV), e.g. for FlexMeasures plugin examples.
 
 Head over to our [documentation](https://flexmeasures.readthedocs.io), e.g. the [getting started guide](https://flexmeasures.readthedocs.io/en/latest/getting-started.html). Or find more information on [FlexMeasures.io](https://flexmeasures.io).
 

documentation/api/change_log.rst

@@ -6,6 +6,17 @@ API change log
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
 
 
+v2.0-4 | 2022-01-04
+"""""""""""""""""""
+
+- Updated entity addresses in documentation, according to the fm1 scheme.
+- Changed the Introduction section:
+
+    - Rewrote the subsection on entity addresses to refer users to where they can find the entity addresses of their sensors.
+    - Rewrote the subsection on sensor identification (formerly known as asset identification) to place the fm1 scheme front and center.
+
+- Fixed the categorisation of the *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints from the User category to the Data category.
+
 v2.0-3 | 2021-06-07
 """""""""""""""""""
 
@@ -44,10 +55,11 @@ v1.3-11 | 2022-01-05
 
 *Affects all versions since v1.3*.
 
-- Changed the *postUdiEvent* endpoint:
+- Changed and extended the *postUdiEvent* endpoint:
 
     - The recording time of new schedules triggered by calling the endpoint is now the time at which the endpoint was called, rather than the datetime of the sent state of charge (SOC).
     - Introduced the "prior" field for the purpose of communicating an alternative recording time, thereby keeping support for simulations.
+    - Introduced an optional "roundtrip_efficiency" field, for use in scheduling.
 
 v1.3-10 | 2021-11-08
 """"""""""""""""""""
@@ -154,14 +166,14 @@ v1.2-1 | 2018-09-24
 
     {
         "type": "PostUdiEventRequest",
-        "event": "ea1.2018-06.io.flexmeasures.company:7:10:203:soc",
+        "event": "ea1.2021-01.io.flexmeasures.company:7:10:203:soc",
     }
 
     rather than the erroneously double-keyed:
 
     {
         "type": "PostUdiEventRequest",
-        "event": "ea1.2018-06.io.flexmeasures.company:7:10:203",
+        "event": "ea1.2021-01.io.flexmeasures.company:7:10:203",
         "type": "soc"
     }
 

documentation/api/introduction.rst

@@ -141,7 +141,7 @@ We distinguish the following roles with different access rights to the individua
 - admin
 - Aggregator
 - Supplier: an energy retailer (see :ref:`supplier`)
-- Prosumer: an asset owner (see :ref:`prosumer`)
+- Prosumer: owner of a grid connection (see :ref:`prosumer`)
 - ESCo: an energy service company (see :ref:`esco`)
 - MDC: a meter data company (see :ref:`mdc`)
 - DSO: a distribution system operator (see :ref:`dso`)
@@ -182,7 +182,7 @@ The API, however, does not distinguish between singular and plural key notation.
 Connections and entity addresses
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Connections are end points of the grid at which an asset is located. 
+A connection represents an end point of the grid, at which an electricity sensor (power meter) is located.
 Connections should be identified with an entity address following the EA1 addressing scheme prescribed by USEF[1],
 which is mostly taken from IETF RFC 3720 [2]:
 
@@ -199,17 +199,23 @@ Here is a full example for a FlexMeasures connection address:
 .. code-block:: json
 
     {
-        "connection": "ea1.2021-02.io.flexmeasures.company:fm0.30:73"
+        "connection": "ea1.2021-02.io.flexmeasures.company:fm1.73"
     }
 
-where FlexMeasures runs at `company.flexmeasures.io` (which the current domain owner started using in February 2021), and the locally unique string is of scheme `fm0` (see below) and the asset ID is 73. The asset's owner ID is 30, but this part is optional.
+where FlexMeasures runs at `company.flexmeasures.io` (which the current domain owner started using in February 2021), and the locally unique string uses the `fm1` scheme (see below) to identify sensor ID 73.
 
-Both the owner ID and the asset ID, as well as the full entity address can be obtained on the asset's listing:
+Assets are listed at:
 
 .. code-block:: html
 
     https://company.flexmeasures.io/assets
 
+The full entity addresses of all of the asset's sensors can be obtained on the asset's page, e.g. for asset 81:
+
+.. code-block:: html
+
+    https://company.flexmeasures.io/assets/81
+
 
 Entity address structure
 """"""""""""""""""""""""""
@@ -231,41 +237,31 @@ Some deeper explanations about an entity address:
 [3] https://tools.ietf.org/html/rfc3721
 
 
-Types of asset identifications used in FlexMeasures
+Types of sensor identification used in FlexMeasures
 """"""""""""""""""""""""""""""""""""""""""""""""""""
 
-FlexMeasures expects the locally unique string string to contain information in
-a certain structure. We distinguish type ``fm0`` and type ``fm1`` FlexMeasures entity addresses.
-
-The ``fm0`` scheme is the original scheme. It identifies connected assets, weather stations, markets and UDI events in different ways. 
+FlexMeasures expects the locally unique string string to contain information in a certain structure.
+We distinguish type ``fm0`` and type ``fm1`` FlexMeasures entity addresses.
 
-Examples for the fm0 scheme:
+The ``fm1`` scheme is the latest version.
+It uses the fact that all FlexMeasures sensors have unique IDs.
 
-- connection = ea1.2021-01.localhost:fm0.40:30
-- connection = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>
-- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.temperature:52:73.0
-- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.<sensor_type>:<latitude>:<longitude>
-- market = ea1.2021-01.io.flexmeasures:fm0.epex_da
-- market = ea1.2021-01.io.flexmeasures:fm0.<market_name>
-- event = ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
-- event = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>:<event_id>:<event_type>
+.. code-block::
 
-This scheme is explicit but also a little cumbersome to use, as one needs to look up the type or even owner (for assets), and weather sensors are identified by coordinates.
-For the fm0 scheme, the 'fm0.' part is optional, for backwards compatibility.
+    ea1.2021-01.io.flexmeasures:fm1.42
+    ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
 
+.. todo:: UDI events are not yet modelled in the fm1 scheme
 
-The ``fm1`` scheme is the latest version, currently under development. It works with the database structure 
-we are developing in the background, where all connected sensors have unique IDs. This makes it more straightforward (the scheme works the same way for all types of sensors), if less explicit.
+The ``fm0`` scheme is the original scheme.
+It identified different types of sensors (such as connections, weather sensors and markets) in different ways.
+The ``fm0`` scheme has been deprecated for the most part and is no longer supported officially.
+Only UDI events still need to be sent using the fm0 scheme.
 
-Examples for the fm1 scheme:
+.. code-block::
 
-- sensor = ea1.2021-01.io.flexmeasures:fm1.42
-- sensor = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- connection = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- market = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-- weather_station = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
-
-.. todo:: UDI events are not yet modelled in the fm1 scheme, but will probably be ea1.2021-01.io.flexmeasures:fm1.<actuator_id>
+    ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
+    ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<sensor_id>:<event_id>:<event_type>
 
 
 Groups
@@ -280,8 +276,8 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
         "groups": [
             {
                 "connections": [
-                    "ea1.2021-02.io.flexmeasures.company:fm0.30:71",
-                    "ea1.2021-02.io.flexmeasures.company:fm0.30:72"
+                    "ea1.2021-02.io.flexmeasures.company:fm1.71",
+                    "ea1.2021-02.io.flexmeasures.company:fm1.72"
                 ],
                 "values": [
                     306.66,
@@ -293,7 +289,7 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
                 ]
             },
             {
-                "connection": "ea1.2021-02.io.flexmeasures.company:fm0.30:73"
+                "connection": "ea1.2021-02.io.flexmeasures.company:fm1.73"
                 "values": [
                     306.66,
                     0,
@@ -315,8 +311,8 @@ In case of a single group of connections, the message may be flattened to:
 
     {
         "connections": [
-            "ea1.2021-02.io.flexmeasures.company:fm0.30:71",
-            "ea1.2021-02.io.flexmeasures.company:fm0.30:72"
+            "ea1.2021-02.io.flexmeasures.company:fm1.71",
+            "ea1.2021-02.io.flexmeasures.company:fm1.72"
         ],
         "values": [
             306.66,

documentation/changelog.rst

@@ -5,21 +5,42 @@ FlexMeasures Changelog
 v0.8.0 | November XX, 2021
 ===========================
 
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
+.. note:: v0.8.0 is doing much of the work we need to do to move to the new data model (see :ref:`note_on_datamodel_transition`). We hope to keep the migration steps for users very limited. One thing you'll notice is that we are copying over existing data to the new model (which will be kept in sync) with the `db upgrade` command (see warning above), which can take a few minutes.
+
 New features
 -----------
+* Bar charts of sensor data for individual sensors, that can be navigated using a calendar [see `PR #99 <http://www.github.com/FlexMeasures/flexmeasures/pull/99>`_ and `PR #290 <http://www.github.com/FlexMeasures/flexmeasures/pull/290>`_]
 * Charts with sensor data can be requested in one of the supported  [`vega-lite themes <https://github.com/vega/vega-themes#included-themes>`_] (incl. a dark theme) [see `PR #221 <http://www.github.com/FlexMeasures/flexmeasures/pull/221>`_]
+* Mobile friendly (responsive) charts of sensor data, and such charts can be requested with a custom width and height [see `PR #313 <http://www.github.com/FlexMeasures/flexmeasures/pull/313>`_]
+* Schedulers take into account round-trip efficiency if set [see `PR #291 <http://www.github.com/FlexMeasures/flexmeasures/pull/291>`_]
+* Fallback policies for charging schedules of batteries and Charge Points, in cases where the solver is presented with an infeasible problem [see `PR #267 <http://www.github.com/FlexMeasures/flexmeasures/pull/267>`_ and `PR #270 <http://www.github.com/FlexMeasures/flexmeasures/pull/270>`_]
+
+Deprecations
+------------
+* The Portfolio and Analytics views are deprecated [see `PR #321 <http://www.github.com/FlexMeasures/flexmeasures/pull/321>`_]
 
 Bugfixes
 -----------
 * Fix recording time of schedules triggered by UDI events [see `PR #300 <http://www.github.com/FlexMeasures/flexmeasures/pull/300>`_]
+* Set bar width of bar charts based on sensor resolution [see `PR #310 <http://www.github.com/FlexMeasures/flexmeasures/pull/310>`_]
+* Fix bug in sensor data charts where data from multiple sources would be stacked, which incorrectly suggested that the data should be summed, whereas the data represents alternative beliefs [see `PR #228 <http://www.github.com/FlexMeasures/flexmeasures/pull/228>`_]
 
 Infrastructure / Support
 ----------------------
 * Account-based authorization, incl. new decorators for endpoints [see `PR #210 <http://www.github.com/FlexMeasures/flexmeasures/pull/210>`_]
 * Central authorization policy which lets database models codify who can do what (permission-based) and relieve API endpoints from this [see `PR #234 <http://www.github.com/FlexMeasures/flexmeasures/pull/234>`_]
 * Improve data specification for forecasting models using timely-beliefs data [see `PR #154 <http://www.github.com/FlexMeasures/flexmeasures/pull/154>`_]
+* Properly attribute Mapbox and OpenStreetMap [see `PR #292 <http://www.github.com/FlexMeasures/flexmeasures/pull/292>`_]
 * Allow plugins to register their custom config settings, so that FlexMeasures can check whether they are set up correctly [see `PR #230 <http://www.github.com/FlexMeasures/flexmeasures/pull/230>`_ and `PR #237 <http://www.github.com/FlexMeasures/flexmeasures/pull/237>`_]
-* Added sensor method to obtain just its latest state (excl. forecasts) [see `PR #235 <http://www.github.com/FlexMeasures/flexmeasures/pull/235>`_]
+* Add sensor method to obtain just its latest state (excl. forecasts) [see `PR #235 <http://www.github.com/FlexMeasures/flexmeasures/pull/235>`_]
+* Migrate attributes of assets, markets and weather sensors to our new sensor model [see `PR #254 <http://www.github.com/FlexMeasures/flexmeasures/pull/254>`_ and `project 9 <http://www.github.com/FlexMeasures/flexmeasures/projects/9>`_]
+* Migrate all time series data to our new sensor data model based on the `timely beliefs <https://github.com/SeitaBV/timely-beliefs>`_ lib [see `PR #286 <http://www.github.com/FlexMeasures/flexmeasures/pull/286>`_ and `project 9 <http://www.github.com/FlexMeasures/flexmeasures/projects/9>`_]
+* Support the new asset model (which describes the organisational structure, rather than sensors and data) in UI and API. Until the transition to our new data model is completed, the new API for assets is at `/api/dev/generic_assets`. [see `PR #251 <http://www.github.com/FlexMeasures/flexmeasures/pull/251>`_ and `PR #290 <http://www.github.com/FlexMeasures/flexmeasures/pulls/290>`_]
+* Internal search methods return most recent beliefs by default, also for charts, which can make them load a lot faster [see `PR #307 <http://www.github.com/FlexMeasures/flexmeasures/pull/307>`_ and `PR #312 <http://www.github.com/FlexMeasures/flexmeasures/pull/312>`_]
+* Support unit conversion for posting sensor data [see `PR #283 <http://www.github.com/FlexMeasures/flexmeasures/pull/283>`_ and `PR #293 <http://www.github.com/FlexMeasures/flexmeasures/pull/293>`_]
+* Improve the core device scheduler to support dealing with asymmetric efficiency losses of individual devices, and with asymmetric up and down prices for deviating from previous commitments (such as a different feed-in tariff) [see `PR #291 <http://www.github.com/FlexMeasures/flexmeasures/pull/291>`_]
+* Stop automatically triggering forecasting jobs when API calls save nothing new to the database, thereby saving redundant computation [see `PR #303 <http://www.github.com/FlexMeasures/flexmeasures/pull/303>`_]
 
 
 v0.7.1 | November 08, 2021
@@ -33,6 +54,7 @@ Bugfixes
 v0.7.0 | October 26, 2021
 ===========================
 
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
 .. warning:: The config setting ``FLEXMEASURES_PLUGIN_PATHS`` has been renamed to ``FLEXMEASURES_PLUGINS``. The old name still works but is deprecated.
 
 New features

documentation/configuration.rst

@@ -197,6 +197,14 @@ Interval in which viewing the queues dashboard refreshes itself, in milliseconds
 Default: ``3000`` (3 seconds) 
 
 
+FLEXMEASURES_ASSET_TYPE_GROUPS
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+How to group asset types together, e.g. in a dashboard.
+
+Default: ``{"renewables": ["solar", "wind"], "EVSE": ["one-way_evse", "two-way_evse"]}``
+
+
 Timing
 ------
 

documentation/dev/data.rst

@@ -118,7 +118,7 @@ Finally, test if you can log in as the flexmeasures user:
 Add Postgres Extensions to your database(s)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-To find the nearest sensors, FlexMeasures needs some extra POstgres support. 
+To find the nearest sensors, FlexMeasures needs some extra Postgres support.
 Add the following extensions while logged in as the postgres superuser:
 
 .. code-block:: bash