API package gets and sets metadata on GenericAssets and Sensors (#243)

Make JSON attributes mutable and have the API package get and set metadata on GenericAssets and Sensors. Applies to all version of the API, but excludes CRUD functionality on assets in API v2.0.


* Create draft PR for #239

* Db migration that copies over attributes from old data models

* - In Asset.__init__, copy over attributes to GenericAsset.
- Start having our model_spec_factory get attributes it needs from GenericAsset.
- Rename variables, specifically, variables that were annotated as a union of our old sensor models were named generic_asset, which was too easily confused with instances of our GenericAsset class.

* model_spec_factory now gets its attributes from GenericAsset instead of old sensor model types

* More renaming to avoid confusion

* Have db migration copy over sensor attributes: unit, event_resolution and knowledge horizons

* In Asset.__init__, copy over sensor attributes: unit, event_resolution and knowledge horizons

* model_spec_factory now gets event_resolution and name from Sensor

* Fix tests

* Factor out use of corresponding_generic_asset attribute

* Factor out use of corresponding_generic_asset attribute

* More renaming

* Pass time series class to model configurator explicitly

* Finally, model_spec_factory doesn't need the old sensor model anymore

* Allow setting the collect function name for TBSeriesSpecs to something custom

* In Asset.__init__, copy over additional asset attributes to GenericAsset

* Planning subpackage uses sensors instead of assets

* Move some simple attributes in the UI package

* Refactor to stop explicitly passing the market to the scheduler, and instead have the scheduler check for an applicable market

* Revert "Move some simple attributes in the UI package", because this needs to be done jointly with moving over asset crud (which we test for)

This reverts commit 56ff279.

* Create draft PR for #242

* Allow config setting specs as module variables, too.


Support reading config setting specs from module (#237)

* Support reading config setting specs from module

* Add additional documentation (review suggestion)

* Amend changelog entry

* Create draft PR for #242

* Make JSON attributes mutable

* Set Asset owner at initialization, so it is copied to GenericAsset

* API v1_2 gets and sets asset attributes in the new data model

* Deprecate use of Asset class in api v1_2

* Deprecate use of Asset class in api v1_3

* Revert "Allow config setting specs as module variables, too."

This reverts commit 327b8b6.

* Work around black issue by updating pre-commit-config

* Deprecate use of Asset class in api v1_0

* Deprecate use of Asset class in part of api v2_0

* Fix docstring

* Revert upgrade of black in pre-commit-config

* Still battling black

* Correct docstring

* Deprecate use of Asset class in SensorField deserialization

* Set Asset owner at initialization, so it is copied to GenericAsset

* Simplify conftest

* Add notes about how each attribute is to be copied from an old class to a new class

* Rename variables

* Refactor attribute copying to util function

* In Market.__init__, copy over attributes from old models to new models

* In Weather.__init__, copy over attributes from old models to new models

* Deprecate use of Market class in SensorField deserialization

* Deprecate use of WeatherSensor class in SensorField deserialization for fm1

* Simplify fm1 entity type schema for deserialization

* Deprecate use of WeatherSensor class in SensorField deserialization for fm0

* Refactor query

* Rename variable

* Refactor query to arrive at a single combined query

* Update todos

* Intend to copy display_name to both GenericAsset and Sensor

* Introduce Sensor attributes and copy most old model attributes there instead of to GenericAsset attributes

* Adjust attribute copying in Asset.__init__

* Implement Sensor method to get an attribute

* Give old sensor classes a generic_asset property

* Give old sensor classes a get_attribute property

* Derive Sensor class lat/lng location from GenericAsset

* Get attributes from Sensor rather than from GenericAsset

* Resolve merge conflict

* Refactor after resolving merge conflict

* Adjust attribute copying in Market.__init__

* Adjust attribute copying in WeatherSensor.__init__

* Post-merge cherry-pick: Set default attributes on generic assets, too

* Post-merge cherry-pick: Add clarity to method docstring

* Introduce has_attribute and set_attribute on the Sensor class, too

* Remove redundant import

* Get attributes from Sensor

* Simplify (as requested in PR review)

* Add docstring to migration util functions, explaining their parameters

* Add module docstring

* Add todos

* Make Sensor attributes mutable, too

* Avoid assumptions on db type (specifically, postgres)

* Update upgrade migration docstring

* Deprecate use of Market class in api v1_1

* Separate setup of market types and markets for tests, otherwise we run into flush issues

* Remove redundant copy, now that we initialize super() first

* Fix bugs: work on kwargs before copying from it and move up initialization of super()

* Increase the chance of identifying a unique sensor by just its name, if you also know the name of its generic asset type

* Simplify API tests by removing the owner id from event-type entity addresses, as the server ignores this optional field anyways

* Simplify API tests by removing the owner id from event-type entity addresses, as the server ignores this optional field anyways

* Deprecate use of Asset class in v1_3 tests

* Deprecate use of Asset class in v1_2 tests

* Deprecate use of Asset class in v1 tests

* Deprecate use of Market class in v1_1 tests

* Fix merge errors

* Update string status code

* Rename legacy migration module

Co-authored-by: Flix6x <Flix6x@users.noreply.github.com>
Co-authored-by: F.N. Claessen <felix@seita.nl>
Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>

flexmeasures/api/common/responses.py

@@ -30,6 +30,11 @@ def my_logic(*args, **kwargs):
         return my_logic
 
 
+@BaseMessage("The requested API version is deprecated for this feature.")
+def deprecated_api_version(message: str) -> ResponseTuple:
+    return dict(result="Rejected", status="INVALID_API_VERSION", message=message), 400
+
+
 @BaseMessage("Some of the data has already been received and successfully processed.")
 def already_received_and_successfully_processed(message: str) -> ResponseTuple:
     return (

flexmeasures/api/common/schemas/sensors.py

@@ -3,7 +3,9 @@
 from marshmallow import fields
 
 from flexmeasures.api import FMValidationError
-from flexmeasures.api.common.utils.api_utils import get_weather_sensor_by
+from flexmeasures.api.common.utils.api_utils import (
+    get_sensor_by_generic_asset_type_and_location,
+)
 from flexmeasures.utils.entity_address_utils import (
     parse_entity_address,
     EntityAddressException,
@@ -19,8 +21,8 @@ class EntityAddressValidationError(FMValidationError):
 
 
 class SensorField(fields.Str):
-    """Field that de-serializes to a Sensor, Asset, Market or WeatherSensor
-    and serializes back to an entity address (string)."""
+    """Field that de-serializes to a Sensor,
+    and serializes a Sensor, Asset, Market or WeatherSensor into an entity address (string)."""
 
     # todo: when Actuators also get an entity address, refactor this class to EntityField,
     #       where an Entity represents anything with an entity address: we currently foresee Sensors and Actuators
@@ -42,85 +44,50 @@ def __init__(
 
     def _deserialize(  # noqa: C901 todo: the noqa can probably be removed after refactoring Asset/Market/WeatherSensor to Sensor
         self, value, attr, obj, **kwargs
-    ) -> Union[Sensor, Asset, Market, WeatherSensor]:
-        """De-serialize to a Sensor, Asset, Market or WeatherSensor."""
+    ) -> Sensor:
+        """De-serialize to a Sensor."""
         # TODO: After refactoring, unify 3 generic_asset cases -> 1 sensor case
         try:
             ea = parse_entity_address(value, self.entity_type, self.fm_scheme)
             if self.fm_scheme == "fm0":
                 if self.entity_type == "connection":
-                    asset = Asset.query.filter(Asset.id == ea["asset_id"]).one_or_none()
-                    if asset is not None:
-                        return asset
-                    else:
-                        raise EntityAddressValidationError(
-                            f"Asset with entity address {value} doesn't exist."
-                        )
-                elif self.entity_type == "market":
-                    market = Market.query.filter(
-                        Market.name == ea["market_name"]
-                    ).one_or_none()
-                    if market is not None:
-                        return market
-                    else:
-                        raise EntityAddressValidationError(
-                            f"Market with entity address {value} doesn't exist."
-                        )
-                elif self.entity_type == "weather_sensor":
-                    weather_sensor = get_weather_sensor_by(
-                        ea["weather_sensor_type_name"], ea["latitude"], ea["longitude"]
-                    )
-                    if weather_sensor is not None and isinstance(
-                        weather_sensor, WeatherSensor
-                    ):
-                        return weather_sensor
-                    else:
-                        raise EntityAddressValidationError(
-                            f"Weather sensor with entity address {value} doesn't exist."
-                        )
-            else:
-                if self.entity_type == "sensor":
                     sensor = Sensor.query.filter(
-                        Sensor.id == ea["sensor_id"]
+                        Sensor.id == ea["asset_id"]
                     ).one_or_none()
                     if sensor is not None:
                         return sensor
-                    else:
-                        raise EntityAddressValidationError(
-                            f"Sensor with entity address {value} doesn't exist."
-                        )
-                elif self.entity_type == "connection":
-                    asset = Asset.query.filter(
-                        Asset.id == ea["sensor_id"]
-                    ).one_or_none()
-                    if asset is not None:
-                        return asset
                     else:
                         raise EntityAddressValidationError(
                             f"Asset with entity address {value} doesn't exist."
                         )
                 elif self.entity_type == "market":
-                    market = Market.query.filter(
-                        Market.id == ea["sensor_id"]
+                    sensor = Sensor.query.filter(
+                        Sensor.name == ea["market_name"]
                     ).one_or_none()
-                    if market is not None:
-                        return market
+                    if sensor is not None:
+                        return sensor
                     else:
                         raise EntityAddressValidationError(
                             f"Market with entity address {value} doesn't exist."
                         )
                 elif self.entity_type == "weather_sensor":
-                    weather_sensor = WeatherSensor.query.filter(
-                        WeatherSensor.id == ea["sensor_id"]
-                    ).one_or_none()
-                    if weather_sensor is not None and isinstance(
-                        weather_sensor, WeatherSensor
-                    ):
-                        return weather_sensor
+                    sensor = get_sensor_by_generic_asset_type_and_location(
+                        ea["weather_sensor_type_name"], ea["latitude"], ea["longitude"]
+                    )
+                    if sensor is not None:
+                        return sensor
                     else:
                         raise EntityAddressValidationError(
                             f"Weather sensor with entity address {value} doesn't exist."
                         )
+            else:
+                sensor = Sensor.query.filter(Sensor.id == ea["sensor_id"]).one_or_none()
+                if sensor is not None:
+                    return sensor
+                else:
+                    raise EntityAddressValidationError(
+                        f"{self.entity_type} with entity address {value} doesn't exist."
+                    )
         except EntityAddressException as eae:
             raise EntityAddressValidationError(str(eae))
         return NotImplemented

flexmeasures/api/common/schemas/tests/test_sensors.py

@@ -62,6 +62,9 @@ def test_sensor_field_straightforward(
     sf = SensorField(entity_type, fm_scheme)
     deser = sf.deserialize(entity_address, None, None)
     assert deser.name == exp_deserialization_name
+    if fm_scheme == "fm0" and entity_type in ("connection", "market", "weather_sensor"):
+        # These entity types are deserialized to Sensors, which have no entity address under the fm0 scheme
+        return
     assert sf.serialize(entity_type, {entity_type: deser}) == entity_address
 
 

flexmeasures/api/common/utils/api_utils.py

@@ -14,7 +14,9 @@
 
 from flexmeasures.data import db
 from flexmeasures.data.models.assets import Asset, Power
+from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
 from flexmeasures.data.models.markets import Price
+from flexmeasures.data.models.time_series import Sensor
 from flexmeasures.data.models.weather import WeatherSensor, Weather
 from flexmeasures.data.services.time_series import drop_unchanged_beliefs
 from flexmeasures.data.utils import save_to_session
@@ -284,24 +286,26 @@ def asset_replace_name_with_id(connections_as_name: List[str]) -> List[str]:
     return connections_as_ea
 
 
-def get_weather_sensor_by(
-    weather_sensor_type_name: str, latitude: float = 0, longitude: float = 0
-) -> Union[WeatherSensor, ResponseTuple]:
+def get_sensor_by_generic_asset_type_and_location(
+    generic_asset_type_name: str, latitude: float = 0, longitude: float = 0
+) -> Union[Sensor, ResponseTuple]:
     """
-    Search a weather sensor by type and location.
-    Can create a weather sensor if needed (depends on API mode)
+    Search a sensor by generic asset type and location.
+    Can create a sensor if needed (depends on API mode)
     and then inform the requesting user which one to use.
     """
-    # Look for the WeatherSensor object
-    weather_sensor = (
-        WeatherSensor.query.filter(
-            WeatherSensor.weather_sensor_type_name == weather_sensor_type_name
-        )
-        .filter(WeatherSensor.latitude == latitude)
-        .filter(WeatherSensor.longitude == longitude)
+    # Look for the Sensor object
+    sensor = (
+        Sensor.query.join(GenericAsset)
+        .join(GenericAssetType)
+        .filter(GenericAssetType.name == generic_asset_type_name)
+        .filter(GenericAsset.generic_asset_type_id == GenericAssetType.id)
+        .filter(GenericAsset.latitude == latitude)
+        .filter(GenericAsset.longitude == longitude)
+        .filter(Sensor.generic_asset_id == GenericAsset.id)
         .one_or_none()
     )
-    if weather_sensor is None:
+    if sensor is None:
         create_sensor_if_unknown = False
         if current_app.config.get("FLEXMEASURES_MODE", "") == "play":
             create_sensor_if_unknown = True
@@ -311,13 +315,14 @@ def get_weather_sensor_by(
             current_app.logger.info("CREATING NEW WEATHER SENSOR...")
             weather_sensor = WeatherSensor(
                 name="Weather sensor for %s at latitude %s and longitude %s"
-                % (weather_sensor_type_name, latitude, longitude),
-                weather_sensor_type_name=weather_sensor_type_name,
+                % (generic_asset_type_name, latitude, longitude),
+                weather_sensor_type_name=generic_asset_type_name,
                 latitude=latitude,
                 longitude=longitude,
             )
             db.session.add(weather_sensor)
             db.session.flush()  # flush so that we can reference the new object in the current db session
+            sensor = weather_sensor.corresponding_sensor
 
         # or query and return the nearest sensor and let the requesting user post to that one
         else:
@@ -333,7 +338,7 @@ def get_weather_sensor_by(
                 )
             else:
                 return unrecognized_sensor()
-    return weather_sensor
+    return sensor
 
 
 def save_to_db(

flexmeasures/api/common/utils/migration_utils.py

@@ -0,0 +1,42 @@
+"""
+This module is part of our data model migration (see https://github.com/SeitaBV/flexmeasures/projects/9).
+It will become obsolete when we deprecate the fm0 scheme for entity addresses.
+"""
+
+from typing import List, Optional, Union
+
+from flexmeasures.api.common.responses import (
+    deprecated_api_version,
+    unrecognized_market,
+    ResponseTuple,
+)
+from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
+from flexmeasures.data.models.time_series import Sensor
+
+
+def get_sensor_by_unique_name(
+    sensor_name: str, generic_asset_type_names: Optional[List[str]] = None
+) -> Union[Sensor, ResponseTuple]:
+    """Search a sensor by unique name, returning a ResponseTuple if it is not found.
+
+    Optionally specify a list of generic asset type names to filter on.
+    This function should be used only for sensors that correspond to the old Market class.
+    """
+    # Look for the Sensor object
+    query = Sensor.query.filter(Sensor.name == sensor_name)
+    if generic_asset_type_names is not None:
+        query = (
+            query.join(GenericAsset)
+            .join(GenericAssetType)
+            .filter(GenericAssetType.name.in_(generic_asset_type_names))
+            .filter(GenericAsset.generic_asset_type_id == GenericAssetType.id)
+            .filter(Sensor.generic_asset_id == GenericAsset.id)
+        )
+    sensor = query.all()
+    if len(sensor) == 0:
+        return unrecognized_market(sensor_name)
+    elif len(sensor) > 1:
+        return deprecated_api_version(
+            f"Multiple sensors were found named {sensor_name}."
+        )
+    return sensor[0]

flexmeasures/api/common/utils/validators.py

@@ -750,9 +750,9 @@ def post_data_checked_for_required_resolution(
     entity_type: str, fm_scheme: str
 ):  # noqa: C901
     """Decorator which checks that a POST request receives time series data with the event resolutions
-    required by the sensor (asset). It sets the "resolution" keyword argument.
-    If the resolution in the data is a multiple of the asset resolution, values are upsampled to the asset resolution.
-    Finally, this decorator also checks if all assets have the same event_resolution and complains otherwise.
+    required by the sensor. It sets the "resolution" keyword argument.
+    If the resolution in the data is a multiple of the sensor resolution, values are upsampled to the sensor resolution.
+    Finally, this decorator also checks if all sensors have the same event_resolution and complains otherwise.
 
     The resolution of the data is inferred from the duration and the number of values.
     Therefore, the decorator should follow after the values_required, period_required and assets_required decorators.
@@ -800,30 +800,30 @@ def decorated_service(*args, **kwargs):
                 (kwargs["start"] + kwargs["duration"]) - kwargs["start"]
             ) / len(kwargs["value_groups"][0])
 
-            # Finding the required resolution for assets affected in this request
+            # Finding the required resolution for sensors affected in this request
             required_resolution = None
-            last_asset = None
+            last_sensor = None
             for asset_group in kwargs["generic_asset_name_groups"]:
                 for asset_descriptor in asset_group:
-                    # Getting the asset
-                    generic_asset = SensorField(entity_type, fm_scheme).deserialize(
+                    # Getting the sensor
+                    sensor = SensorField(entity_type, fm_scheme).deserialize(
                         asset_descriptor
                     )
-                    if generic_asset is None:
+                    if sensor is None:
                         return unrecognized_asset(
                             f"Failed to look up asset by {asset_descriptor}"
                         )
-                    # Complain if assets don't all require the same resolution
+                    # Complain if sensors don't all require the same resolution
                     if (
                         required_resolution is not None
-                        and generic_asset.event_resolution != required_resolution
+                        and sensor.event_resolution != required_resolution
                     ):
                         return conflicting_resolutions(
-                            f"Cannot send data for both {generic_asset} and {last_asset}."
+                            f"Cannot send data for both {sensor} and {last_sensor}."
                         )
-                    # Setting the resolution & remembering last looked-at asset
-                    required_resolution = generic_asset.event_resolution
-                    last_asset = generic_asset
+                    # Setting the resolution & remembering last looked-at sensor
+                    required_resolution = sensor.event_resolution
+                    last_sensor = sensor
 
             # if inferred resolution is a multiple from required_solution, we can upsample_values
             # todo: next line fails on sensors with 0 resolution
@@ -855,12 +855,12 @@ def decorated_service(*args, **kwargs):
 def get_data_downsampling_allowed(entity_type: str, fm_scheme: str):
     """Decorator which allows downsampling of data which a GET request returns.
     It checks for a form parameter "resolution".
-    If that is given and is a multiple of the asset's event_resolution,
+    If that is given and is a multiple of the sensor's event_resolution,
     downsampling is performed on the data. This is done by setting the "resolution"
     keyword parameter, which is obeyed by collect_time_series_data and used
     in resampling.
 
-    The original resolution of the data is the event_resolution of the asset.
+    The original resolution of the data is the event_resolution of the sensor.
     Therefore, the decorator should follow after the assets_required decorator.
 
     Example:
@@ -891,19 +891,19 @@ def decorated_service(*args, **kwargs):
                 ds_resolution = parse_duration(form["resolution"])
                 if ds_resolution is None:
                     return invalid_resolution_str(form["resolution"])
-                # Check if the resolution can be applied to all assets (if it is a multiple
+                # Check if the resolution can be applied to all sensors (if it is a multiple
                 # of the event_resolution(s) and thus downsampling is possible)
                 for asset_group in kwargs["generic_asset_name_groups"]:
                     for asset_descriptor in asset_group:
-                        generic_asset = SensorField(entity_type, fm_scheme).deserialize(
+                        sensor = SensorField(entity_type, fm_scheme).deserialize(
                             asset_descriptor
                         )
-                        if generic_asset is None:
+                        if sensor is None:
                             return unrecognized_asset()
-                        asset_resolution = generic_asset.event_resolution
-                        if ds_resolution % asset_resolution != timedelta(minutes=0):
+                        sensor_resolution = sensor.event_resolution
+                        if ds_resolution % sensor_resolution != timedelta(minutes=0):
                             return unapplicable_resolution(
-                                f"{isodate.duration_isoformat(asset_resolution)} or a multiple hereof."
+                                f"{isodate.duration_isoformat(sensor_resolution)} or a multiple hereof."
                             )
                 kwargs["resolution"] = to_offset(
                     isodate.parse_duration(form["resolution"])