Issue 6 entity address scheme improvements 2

Makefile

@@ -15,13 +15,13 @@ test:
 # ---- Documentation ---
 
 update-docs:
-	pip3 install sphinx sphinx-rtd-theme sphinxcontrib.httpdomain
+	pip3 install sphinx==3.5.4 sphinxcontrib.httpdomain  # sphinx4 is not supported yet by sphinx-contrib/httpdomain, see https://github.com/sphinx-contrib/httpdomain/issues/46
 	cd documentation; make clean; make html; cd ..
 
 update-docs-pdf:
 	@echo "NOTE: PDF documentation requires packages (on Debian: latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended)"
 	@echo "NOTE: Currently, the docs require some pictures which are not in the git repo atm. Ask the devs."
-	pip3 install sphinx sphinxcontrib.httpdomain
+	pip3 install sphinx sphinx-rtd-theme sphinxcontrib.httpdomain
 	cd documentation; make clean; make latexpdf; make latexpdf; cd ..  # make latexpdf can require two passes
 
 # ---- Installation ---

documentation/api/introduction.rst

@@ -129,8 +129,8 @@ Throughout this document, keys are written in singular if a single value is list
 
 The API, however, does not distinguish between singular and plural key notation.
 
-Connections
-^^^^^^^^^^^
+Connections and entity addresses
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Connections are end points of the grid at which an asset is located. 
 Connections should be identified with an entity address following the EA1 addressing scheme prescribed by USEF[1],
@@ -160,36 +160,62 @@ The owner ID is optional. Both the owner ID and the asset ID, as well as the ful
     https://company.flexmeasures.io/assets
 
 
+Entity address structure
+""""""""""""""""""""""""""
 Some deeper explanations about an entity address:
 
 - "ea1" is a constant, indicating this is a type 1 USEF entity address
 - The date code "must be a date during which the naming authority owned the domain name used in this format, and should be the first month in which the domain name was owned by this naming authority at 00:01 GMT of the first day of the month.
 - The reversed domain name is taken from the naming authority (person or organization) creating this entity address
 - The locally unique string can be used for local purposes, and FlexMeasures uses it to identify the resource (more information in parse_entity_address).
+  Fields in the locally unique string are separated by colons, see for other examples
+  IETF RFC 3721, page 6 [3]. While [2] says it's possible to use dashes, dots or colons as separators, we might use dashes and dots in
+  latitude/longitude coordinates of sensors, so we settle on colons.
+
 
 [1] https://www.usef.energy/app/uploads/2020/01/USEF-Flex-Trading-Protocol-Specifications-1.01.pdf
 
 [2] https://tools.ietf.org/html/rfc3720
 
+[3] https://tools.ietf.org/html/rfc3721
 
-Notation for simulation
-"""""""""""""""""""""""
 
-For version 1 of the API, the following simplified addressing scheme may be used:
+Types of asset identifications used in FlexMeasures
+""""""""""""""""""""""""""""""""""""""""""
 
-.. code-block:: json
+FlexMeasures expects the locally unique string string to contain information in
+a certain structure. We distinguish type ``fm0`` and type ``fm1`` FlexMeasures entity addresses.
 
-    {
-        "connection": "<owner-id>:<asset-id>"
-    }
+The ``fm0`` scheme is the original scheme. It identifies connected assets, sensors and markets with a combined key of type and ID. 
 
-or even simpler:
+Examples for the fm0 scheme:
 
-.. code-block:: json
+- connection = ea1.2021-01.localhost:fm0.40:30
+- connection = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>
+- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.temperature:52:73.0
+- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.<sensor_type>:<latitude>:<longitude>
+- market = ea1.2021-01.io.flexmeasures:fm0.epex_da
+- market = ea1.2021-01.io.flexmeasures:fm0.<market_name>
+- event = ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
+- event = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>:<event_id>:<event_type>
+
+This scheme is explicit but also a little cumbersome to use, as one needs to look up the type or even owner (for assets), and weather sensors are identified by coordinates.
+For the fm0 scheme, the 'fm0.' part is optional, for backwards compatibility.
+
+
+The ``fm1`` scheme is the latest version, currently under development. It works with the database structure 
+we are developing in the background, where all connected sensors have unique IDs. This makes it more straightforward, if less explicit.
+
+Examples for the fm1 scheme:
+
+- sensor = ea1.2021-01.io.flexmeasures:fm1.42
+- sensor = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
+- connection = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
+- market = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
+- weather_station = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
+
+.. todo:: UDI events are not yet modelled in the fm1 scheme, but will probably be ea1.2021-01.io.flexmeasures:fm1.<actuator_id>
 
-    {
-        "connection": "<asset-id>"
-    }
 
 Groups
 ^^^^^^
@@ -203,8 +229,8 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
         "groups": [
             {
                 "connections": [
-                    "CS 1",
-                    "CS 2"
+                    "ea1.2021-02.io.flexmeasures.company:30:71",
+                    "ea1.2021-02.io.flexmeasures.company:30:72"
                 ],
                 "values": [
                     306.66,
@@ -216,7 +242,7 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
                 ]
             },
             {
-                "connection": "CS 3",
+                "connection": "ea1.2021-02.io.flexmeasures.company:30:73"
                 "values": [
                     306.66,
                     0,
@@ -238,8 +264,8 @@ In case of a single group of connections, the message may be flattened to:
 
     {
         "connections": [
-            "CS 1",
-            "CS 2"
+            "ea1.2021-02.io.flexmeasures.company:30:71",
+            "ea1.2021-02.io.flexmeasures.company:30:72"
         ],
         "values": [
             306.66,

flexmeasures/api/common/schemas/sensors.py

@@ -0,0 +1,135 @@
+from typing import Union
+
+from marshmallow import fields
+
+from flexmeasures.api import FMValidationError
+from flexmeasures.api.common.utils.api_utils import get_weather_sensor_by
+from flexmeasures.utils.entity_address_utils import (
+    parse_entity_address,
+    EntityAddressException,
+)
+from flexmeasures.data.models.assets import Asset
+from flexmeasures.data.models.markets import Market
+from flexmeasures.data.models.weather import WeatherSensor
+from flexmeasures.data.models.time_series import Sensor
+
+
+class EntityAddressValidationError(FMValidationError):
+    status = "INVALID_DOMAIN"  # USEF error status
+
+
+class SensorField(fields.Str):
+    """Field that deserializes to a Sensor, Asset, Market or WeatherSensor
+    and serializes back to an entity address (string)."""
+
+    # todo: when Actuators also get an entity address, refactor this class to EntityField,
+    #       where an Entity represents anything with an entity address: we currently foresee Sensors and Actuators
+
+    def __init__(
+        self,
+        entity_type: str,
+        fm_scheme: str,
+        *args,
+        **kwargs,
+    ):
+        """
+        :param entity_type: "sensor", "connection", "market" or "weather_sensor"
+        :param fm_scheme:   "fm0" or "fm1"
+        """
+        self.entity_type = entity_type
+        self.fm_scheme = fm_scheme
+        super().__init__(*args, **kwargs)
+
+    def _deserialize(  # noqa: C901 todo: the noqa can probably be removed after refactoring Asset/Market/WeatherSensor to Sensor
+        self, value, attr, obj, **kwargs
+    ) -> Union[Sensor, Asset, Market, WeatherSensor]:
+        """Deserialize to a Sensor, Asset, Market or WeatherSensor."""
+        # TODO: After refactoring, unify 3 generic_asset cases -> 1 sensor case
+        try:
+            ea = parse_entity_address(value, self.entity_type, self.fm_scheme)
+            if self.fm_scheme == "fm0":
+                if self.entity_type == "connection":
+                    asset = Asset.query.filter(Asset.id == ea["asset_id"]).one_or_none()
+                    if asset is not None:
+                        return asset
+                    else:
+                        raise EntityAddressValidationError(
+                            f"Asset with entity address {value} doesn't exist."
+                        )
+                elif self.entity_type == "market":
+                    market = Market.query.filter(
+                        Market.name == ea["market_name"]
+                    ).one_or_none()
+                    if market is not None:
+                        return market
+                    else:
+                        raise EntityAddressValidationError(
+                            f"Market with entity address {value} doesn't exist."
+                        )
+                elif self.entity_type == "weather_sensor":
+                    weather_sensor = get_weather_sensor_by(
+                        ea["weather_sensor_type_name"], ea["latitude"], ea["longitude"]
+                    )
+                    if weather_sensor is not None and isinstance(
+                        weather_sensor, WeatherSensor
+                    ):
+                        return weather_sensor
+                    else:
+                        raise EntityAddressValidationError(
+                            f"Weather sensor with entity address {value} doesn't exist."
+                        )
+            else:
+                if self.entity_type == "sensor":
+                    sensor = Sensor.query.filter(
+                        Sensor.id == ea["sensor_id"]
+                    ).one_or_none()
+                    if sensor is not None:
+                        return sensor
+                    else:
+                        raise EntityAddressValidationError(
+                            f"Sensor with entity address {value} doesn't exist."
+                        )
+                elif self.entity_type == "connection":
+                    asset = Asset.query.filter(
+                        Asset.id == ea["sensor_id"]
+                    ).one_or_none()
+                    if asset is not None:
+                        return asset
+                    else:
+                        raise EntityAddressValidationError(
+                            f"Asset with entity address {value} doesn't exist."
+                        )
+                elif self.entity_type == "market":
+                    market = Market.query.filter(
+                        Market.id == ea["sensor_id"]
+                    ).one_or_none()
+                    if market is not None:
+                        return market
+                    else:
+                        raise EntityAddressValidationError(
+                            f"Market with entity address {value} doesn't exist."
+                        )
+                elif self.entity_type == "weather_sensor":
+                    weather_sensor = WeatherSensor.query.filter(
+                        WeatherSensor.id == ea["sensor_id"]
+                    ).one_or_none()
+                    if weather_sensor is not None and isinstance(
+                        weather_sensor, WeatherSensor
+                    ):
+                        return weather_sensor
+                    else:
+                        raise EntityAddressValidationError(
+                            f"Weather sensor with entity address {value} doesn't exist."
+                        )
+        except EntityAddressException as eae:
+            raise EntityAddressValidationError(str(eae))
+        return NotImplemented
+
+    def _serialize(
+        self, value: Union[Sensor, Asset, Market, WeatherSensor], attr, data, **kwargs
+    ):
+        """Serialize to an entity address."""
+        if self.fm_scheme == "fm0":
+            return value.entity_address_fm0
+        else:
+            return value.entity_address

flexmeasures/api/common/schemas/tests/test_sensors.py

@@ -0,0 +1,97 @@
+import pytest
+
+from flexmeasures.api.common.schemas.sensors import (
+    SensorField,
+    EntityAddressValidationError,
+)
+from flexmeasures.utils.entity_address_utils import build_entity_address
+
+
+@pytest.mark.parametrize(
+    "entity_address, entity_type, fm_scheme, exp_deserialization_name",
+    [
+        (
+            build_entity_address(dict(sensor_id=9), "sensor"),
+            "sensor",
+            "fm1",
+            "my daughter's height",
+        ),
+        (
+            build_entity_address(
+                dict(market_name="epex_da"), "market", fm_scheme="fm0"
+            ),
+            "market",
+            "fm0",
+            "epex_da",
+        ),
+        (
+            build_entity_address(
+                dict(owner_id=1, asset_id=3), "connection", fm_scheme="fm0"
+            ),
+            "connection",
+            "fm0",
+            "Test battery with no known prices",
+        ),
+        (
+            build_entity_address(
+                dict(
+                    weather_sensor_type_name="temperature",
+                    latitude=33.4843866,
+                    longitude=126.0,
+                ),
+                "weather_sensor",
+                fm_scheme="fm0",
+            ),
+            "weather_sensor",
+            "fm0",
+            "temperature_sensor",
+        ),
+    ],
+)
+def test_sensor_field_straightforward(
+    entity_address, entity_type, fm_scheme, exp_deserialization_name
+):
+    """Testing straightforward cases"""
+    sf = SensorField(entity_type, fm_scheme)
+    deser = sf.deserialize(entity_address, None, None)
+    assert deser.name == exp_deserialization_name
+    assert sf.serialize(entity_type, {entity_type: deser}) == entity_address
+
+
+@pytest.mark.parametrize(
+    "entity_address, entity_type, fm_scheme, error_msg",
+    [
+        (
+            "ea1.2021-01.io.flexmeasures:some.weird:identifier%that^is*not)used",
+            "market",
+            "fm0",
+            "Could not parse",
+        ),
+        (
+            "ea1.2021-01.io.flexmeasures:fm1.some.weird:identifier%that^is*not)used",
+            "market",
+            "fm1",
+            "Could not parse",
+        ),
+        (
+            build_entity_address(
+                dict(market_name="non_existing_market"), "market", fm_scheme="fm0"
+            ),
+            "market",
+            "fm0",
+            "doesn't exist",
+        ),
+        (
+            build_entity_address(dict(sensor_id=-1), "sensor", fm_scheme="fm1"),
+            "market",
+            "fm1",
+            "Could not parse",
+        ),
+        ("ea1.2021-13.io.flexmeasures:fm1.9", "sensor", "fm1", "date specification"),
+    ],
+)
+def test_sensor_field_invalid(entity_address, entity_type, fm_scheme, error_msg):
+    sf = SensorField(entity_type, fm_scheme)
+    with pytest.raises(EntityAddressValidationError) as ve:
+        sf.deserialize(entity_address, None, None)
+    assert error_msg in str(ve)

flexmeasures/api/common/schemas/tests/test_times.py

@@ -8,7 +8,7 @@
 
 
 @pytest.mark.parametrize(
-    "duration_input,exp_deserialization",
+    "duration_input, exp_deserialization",
     [
         ("PT1H", timedelta(hours=1)),
         ("PT6M", timedelta(minutes=6)),
@@ -25,7 +25,7 @@ def test_duration_field_straightforward(duration_input, exp_deserialization):
 
 
 @pytest.mark.parametrize(
-    "duration_input,exp_deserialization,grounded_timedelta",
+    "duration_input, exp_deserialization, grounded_timedelta",
     [
         ("P1M", isodate.Duration(months=1), timedelta(days=29)),
         ("PT24H", isodate.Duration(hours=24), timedelta(hours=24)),
@@ -59,7 +59,7 @@ def test_duration_field_nominal_grounded(
 
 
 @pytest.mark.parametrize(
-    "duration_input,error_msg",
+    "duration_input, error_msg",
     [
         ("", "Unable to parse duration string"),
         ("1H", "Unable to parse duration string"),