Issue 6 entity address scheme improvements 2

Makefile

@@ -15,13 +15,13 @@ test:
 # ---- Documentation ---
 
 update-docs:
-	pip3 install sphinx sphinx-rtd-theme sphinxcontrib.httpdomain
+	pip3 install sphinx==3.5.4 sphinxcontrib.httpdomain  # sphinx4 is not supported yet by sphinx-contrib/httpdomain, see https://github.com/sphinx-contrib/httpdomain/issues/46
 	cd documentation; make clean; make html; cd ..
 
 update-docs-pdf:
 	@echo "NOTE: PDF documentation requires packages (on Debian: latexmk texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended)"
 	@echo "NOTE: Currently, the docs require some pictures which are not in the git repo atm. Ask the devs."
-	pip3 install sphinx sphinxcontrib.httpdomain
+	pip3 install sphinx sphinx-rtd-theme sphinxcontrib.httpdomain
 	cd documentation; make clean; make latexpdf; make latexpdf; cd ..  # make latexpdf can require two passes
 
 # ---- Installation ---

documentation/api/introduction.rst

@@ -129,8 +129,8 @@ Throughout this document, keys are written in singular if a single value is list
 
 The API, however, does not distinguish between singular and plural key notation.
 
-Connections
-^^^^^^^^^^^
+Connections and entity addresses
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Connections are end points of the grid at which an asset is located. 
 Connections should be identified with an entity address following the EA1 addressing scheme prescribed by USEF[1],
@@ -160,36 +160,62 @@ The owner ID is optional. Both the owner ID and the asset ID, as well as the ful
     https://company.flexmeasures.io/assets
 
 
+Entity address structure
+""""""""""""""""""""""""""
 Some deeper explanations about an entity address:
 
 - "ea1" is a constant, indicating this is a type 1 USEF entity address
 - The date code "must be a date during which the naming authority owned the domain name used in this format, and should be the first month in which the domain name was owned by this naming authority at 00:01 GMT of the first day of the month.
 - The reversed domain name is taken from the naming authority (person or organization) creating this entity address
 - The locally unique string can be used for local purposes, and FlexMeasures uses it to identify the resource (more information in parse_entity_address).
+  Fields in the locally unique string are separated by colons, see for other examples
+  IETF RFC 3721, page 6 [3]. While [2] says it's possible to use dashes, dots or colons as separators, we might use dashes and dots in
+  latitude/longitude coordinates of sensors, so we settle on colons.
+
 
 [1] https://www.usef.energy/app/uploads/2020/01/USEF-Flex-Trading-Protocol-Specifications-1.01.pdf
 
 [2] https://tools.ietf.org/html/rfc3720
 
+[3] https://tools.ietf.org/html/rfc3721
 
-Notation for simulation
-"""""""""""""""""""""""
 
-For version 1 of the API, the following simplified addressing scheme may be used:
+Types of asset identifications used in FlexMeasures
+""""""""""""""""""""""""""""""""""""""""""
 
-.. code-block:: json
+FlexMeasures expects the locally unique string string to contain information in
+a certain structure. We distinguish type ``fm0`` and type ``fm1`` FlexMeasures entity addresses.
 
-    {
-        "connection": "<owner-id>:<asset-id>"
-    }
+The ``fm0`` scheme is the original scheme. It identifies connected assets, sensors and markets with a combined key of type and ID. 
 
-or even simpler:
+Examples for the fm0 scheme:
 
-.. code-block:: json
+- connection = ea1.2021-01.localhost:fm0.40:30
+- connection = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>
+- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.temperature:52:73.0
+- weather_sensor = ea1.2021-01.io.flexmeasures:fm0.<sensor_type>:<latitude>:<longitude>
+- market = ea1.2021-01.io.flexmeasures:fm0.epex_da
+- market = ea1.2021-01.io.flexmeasures:fm0.<market_name>
+- event = ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
+- event = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>:<event_id>:<event_type>
+
+This scheme is explicit but also a little cumbersome to use, as one needs to look up the type or even owner (for assets), and weather sensors are identified by coordinates.
+For the fm0 scheme, the 'fm0.' part is optional, for backwards compatibility.
+
+
+The ``fm1`` scheme is the latest version, currently under development. It works with the database structure 
+we are developing in the background, where all connected sensors have unique IDs. This makes it more straightforward, if less explicit.
+
+Examples for the fm1 scheme:
+
+- sensor = ea1.2021-01.io.flexmeasures:fm1.42
+- sensor = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
+- connection = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
+- market = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
+- weather_station = ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
+
+.. todo:: UDI events are not yet modelled in the fm1 scheme, but will probably be ea1.2021-01.io.flexmeasures:fm1.<actuator_id>
 
-    {
-        "connection": "<asset-id>"
-    }
 
 Groups
 ^^^^^^
@@ -203,8 +229,8 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
         "groups": [
             {
                 "connections": [
-                    "CS 1",
-                    "CS 2"
+                    "ea1.2021-02.io.flexmeasures.company:30:71",
+                    "ea1.2021-02.io.flexmeasures.company:30:72"
                 ],
                 "values": [
                     306.66,
@@ -216,7 +242,7 @@ When the attributes "start", "duration" and "unit" are stated outside of "groups
                 ]
             },
             {
-                "connection": "CS 3",
+                "connection": "ea1.2021-02.io.flexmeasures.company:30:73"
                 "values": [
                     306.66,
                     0,
@@ -238,8 +264,8 @@ In case of a single group of connections, the message may be flattened to:
 
     {
         "connections": [
-            "CS 1",
-            "CS 2"
+            "ea1.2021-02.io.flexmeasures.company:30:71",
+            "ea1.2021-02.io.flexmeasures.company:30:72"
         ],
         "values": [
             306.66,

flexmeasures/utils/entity_address_utils.py

@@ -11,33 +11,10 @@
 
 
 """
-Functionality to support parsing and building USEF's EA1 addressing scheme [1],
-which is mostly taken from IETF RFC 3720 [2]:
-
-This is the complete structure of an EA1 address:
-
-ea1.{date code}.{reversed domain name}:{locally unique string}
-
-for example "ea1.2021-01.io.flexmeasures.company:sensor14"
-
-- "ea1" is a constant, indicating this is a type 1 USEF entity address
-- The date code "must be a date during which the naming authority owned
-    the domain name used in this format, and should be the first month in which the domain name was
-    owned by this naming authority at 00:01 GMT of the first day of the month.
-- The reversed domain name is taken from the naming authority
-    (person or organization) creating this entity address
-- The locally unique string can be used for local purposes, and FlexMeasures
-  uses it to identify the resource (more information in parse_entity_address).
-  Fields in the locally unique string are separated by colons, see for other examples
-  IETF RFC 3721, page 6 [3].
-  ([2] says it's possible to use dashes, dots or colons ― dashes and dots might come up in
-  latitude/longitude coordinates of sensors)
-
-TODO: This needs to be in the FlexMeasures documentation.
+Functionality to support parsing and building Entity Addresses as defined by USEF [1].
+See our documentation for more details.
 
 [1] https://www.usef.energy/app/uploads/2020/01/USEF-Flex-Trading-Protocol-Specifications-1.01.pdf
-[2] https://tools.ietf.org/html/rfc3720
-[3] https://tools.ietf.org/html/rfc3721
 """
 
 
@@ -130,10 +107,10 @@ def parse_entity_address(  # noqa: C901
     """
     Parses a generic asset name into an info dict.
 
-    The entity_address must be a valid type 1 USEF entity address.
-    That is, it must follow the EA1 addressing scheme recommended by USEF.
-    In addition, FlexMeasures expects the identifying string to contain information in
-    a certain structure. We distinguish type 0 and type 1 FlexMeasures entity addresses.
+    Returns a dictionary with scheme, naming_authority and various other fields,
+    depending on the entity type and FlexMeasures scheme (see examples above).
+    Returns None if entity type is unknown or entity_address is not parse-able.
+    We recommend to `return invalid_domain()` in that case.
 
     Examples for the fm1 scheme:
 
@@ -156,11 +133,6 @@ def parse_entity_address(  # noqa: C901
         event = ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<asset_id>:<event_id>:<event_type>
 
     For the fm0 scheme, the 'fm0.' part is optional, for backwards compatibility.
-
-    Returns a dictionary with scheme, naming_authority and various other fields,
-    depending on the entity type and FlexMeasures scheme (see examples above).
-    Returns None if entity type is unknown or entity_address is not parseable.
-    We recommend to `return invalid_domain()` in that case.
     """
 
     # Check the scheme and naming authority date
@@ -207,17 +179,16 @@ def validate_ea_for_fm_scheme(ea: dict, fm_scheme: str):
             r"$",
             entity_address,
         )
-        if match:
-            value_types = {
-                "scheme": str,
-                "naming_authority": str,
-                "fm_scheme": str,
-                "sensor_id": int,
-            }
-        else:
+        if match is None:
             raise EntityAddressException(
                 f"Could not parse {entity_type} {entity_address}."
             )
+        value_types = {
+            "scheme": str,
+            "naming_authority": str,
+            "fm_scheme": str,
+            "sensor_id": int,
+        }
     elif fm_scheme != FM0_ADDR_SCHEME:
         raise EntityAddressException(
             f"Unrecognized FlexMeasures scheme for entity addresses: {fm_scheme}"
@@ -234,17 +205,16 @@ def validate_ea_for_fm_scheme(ea: dict, fm_scheme: str):
             r"$",
             entity_address,
         )
-        if match:
-            value_types = {
-                "scheme": str,
-                "naming_authority": str,
-                "owner_id": int,
-                "asset_id": int,
-            }
-        else:
+        if match is None:
             raise EntityAddressException(
                 f"Could not parse {entity_type} {entity_address}."
             )
+        value_types = {
+            "scheme": str,
+            "naming_authority": str,
+            "owner_id": int,
+            "asset_id": int,
+        }
     elif entity_type == "weather_sensor":
         match = re.search(
             r"^"
@@ -261,18 +231,17 @@ def validate_ea_for_fm_scheme(ea: dict, fm_scheme: str):
             r"$",
             entity_address,
         )
-        if match:
-            value_types = {
-                "scheme": str,
-                "naming_authority": str,
-                "weather_sensor_type_name": str,
-                "latitude": float,
-                "longitude": float,
-            }
-        else:
+        if match is None:
             raise EntityAddressException(
                 f"Could not parse {entity_type} {entity_address}."
             )
+        value_types = {
+            "scheme": str,
+            "naming_authority": str,
+            "weather_sensor_type_name": str,
+            "latitude": float,
+            "longitude": float,
+        }
     elif entity_type == "market":
         match = re.search(
             r"^"
@@ -285,12 +254,11 @@ def validate_ea_for_fm_scheme(ea: dict, fm_scheme: str):
             r"$",
             entity_address,
         )
-        if match:
-            value_types = {"scheme": str, "naming_authority": str, "market_name": str}
-        else:
+        if match is None:
             raise EntityAddressException(
                 f"Could not parse {entity_type} {entity_address}."
             )
+        value_types = {"scheme": str, "naming_authority": str, "market_name": str}
     elif entity_type == "event":
         match = re.search(
             r"^"
@@ -308,19 +276,18 @@ def validate_ea_for_fm_scheme(ea: dict, fm_scheme: str):
             r"$",
             entity_address,
         )
-        if match:
-            value_types = {
-                "scheme": str,
-                "naming_authority": str,
-                "owner_id": int,
-                "asset_id": int,
-                "event_id": int,
-                "event_type": str,
-            }
-        else:
+        if match is None:
             raise EntityAddressException(
                 f"Could not parse {entity_type} {entity_address}."
             )
+        value_types = {
+            "scheme": str,
+            "naming_authority": str,
+            "owner_id": int,
+            "asset_id": int,
+            "event_id": int,
+            "event_type": str,
+        }
     else:
         # Finally, we simply raise without precise information what went wrong
         raise EntityAddressException(f"Could not parse {entity_address}.")