Support showing instantaneous sensor data with a custom event frequency

documentation/api/notation.rst

@@ -92,23 +92,16 @@ It uses the fact that all FlexMeasures sensors have unique IDs.
     ea1.2021-01.io.flexmeasures:fm1.42
     ea1.2021-01.io.flexmeasures:fm1.<sensor_id>
 
-.. todo:: UDI events are not yet modelled in the fm1 scheme
-
 The ``fm0`` scheme is the original scheme.
 It identified different types of sensors (such as grid connections, weather sensors and markets) in different ways.
-The ``fm0`` scheme has been deprecated for the most part and is no longer supported officially.
-Only UDI events still need to be sent using the fm0 scheme.
-
-.. code-block::
-
-    ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc
-    ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<sensor_id>:<event_id>:<event_type>
+The ``fm0`` scheme has been deprecated and is no longer supported officially.
 
 
 Timeseries
 ^^^^^^^^^^
 
-Timestamps and durations are consistent with the ISO 8601 standard. The resolution of the data is implicit (from duration and number of values), see :ref:`resolutions`.
+Timestamps and durations are consistent with the ISO 8601 standard.
+The frequency of the data is implicit (from duration and number of values), while the resolution of the data is explicit, see :ref:`frequency_and_resolution`.
 
 All timestamps in requests to the API must be timezone-aware. For instance, in the below example, the timezone indication "Z" indicates a zero offset from UTC.
 
@@ -267,19 +260,36 @@ For example, the following message implies that all prognosed values were made 1
 Note that, for a horizon indicating a belief 10 minutes after the *start* of each 15-minute interval, the "horizon" would have been "PT5M".
 This denotes that the prognosed interval has 5 minutes left to be concluded.
 
-.. _resolutions:
+.. _frequency_and_resolution:
+
+Frequency and resolution
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+FlexMeasures handles two types of time series, which can be distinguished by defining the following timing properties for events recorded by sensors:
+
+- Frequency: how far apart events occur (a constant duration between event starts)
+- Resolution: how long an event lasts (a constant duration between the start and end of an event)
+
+.. note:: FlexMeasures runs on Pandas, and follows Pandas terminology accordingly.
+          The term frequency as used by Pandas is the reciprocal of the `SI quantity for frequency <https://en.wikipedia.org/wiki/SI_derived_unit>`_.
+
+1. The first type of time series describes non-instantaneous events such as average hourly wind speed.
+   For this case, it is commonly assumed that ``frequency == resolution``.
+   That is, events follow each other sequentially and without delay.
 
-Resolutions
-^^^^^^^^^^^
+2. The second type of time series describes instantaneous events (zero resolution) such as temperature at a given time.
+   For this case, we have ``frequency != resolution``.
 
-Specifying a resolution is redundant for POST requests that contain both "values" and a "duration" ― FlexMeasures computes the resolution by dividing the duration by the number of values.
+Specifying a frequency and resolution is redundant for POST requests that contain both "values" and a "duration" ― FlexMeasures computes the frequency by dividing the duration by the number of values, and, for sensors that record non-instantaneous events, assumes the resolution of the data is equal to the frequency.
 
-When POSTing data, FlexMeasures checks this computed resolution against the required resolution of the sensors which are posted to. If these can't be matched (through upsampling), an error will occur.
+When POSTing data, FlexMeasures checks this inferred resolution against the required resolution of the sensors that are posted to.
+If these can't be matched (through upsampling), an error will occur.
 
-GET requests (such as *getMeterData*) return data in the resolution which the sensor is configured for.
-A "resolution" may be specified explicitly to obtain the data in downsampled form, 
-which can be very beneficial for download speed. The specified resolution needs to be a multiple
-of the sensor's resolution, e.g. hourly or daily values if the sensor's resolution is 15 minutes.
+GET requests (such as */sensors/data*) return data with a frequency either equal to the resolution that the sensor is configured for (for non-instantaneous sensors), or a default frequency befitting (in our opinion) the requested time interval.
+A "resolution" may be specified explicitly to obtain the data in downsampled form, which can be very beneficial for download speed.
+For non-instantaneous sensors, the specified resolution needs to be a multiple of the sensor's resolution, e.g. hourly or daily values if the sensor's resolution is 15 minutes.
+For instantaneous sensors, the specified resolution is interpreted as a request for data in a specific frequency.
+The resolution of the underlying data will remain zero (and the returned message will say so).
 
 
 .. _sources:

documentation/changelog.rst

@@ -17,6 +17,7 @@ New features
 * The CLI command ``flexmeasures show beliefs`` supports showing beliefs data in a custom resolution and/or timezone, and also saving the shown beliefs data to a CSV file [see `PR #519 <http://www.github.com/FlexMeasures/flexmeasures/pull/519>`_]
 * Improved import of time series data from CSV file: 1) drop duplicate records with warning, 2) allow configuring which column contains explicit recording times for each data point (use case: import forecasts) [see `PR #501 <http://www.github.com/FlexMeasures/flexmeasures/pull/501>`_], 3) localize timezone naive data, 4) support reading in datetime and timedelta values, 5) remove rows with NaN values, and 6) filter by values in specific columns [see `PR #521 <http://www.github.com/FlexMeasures/flexmeasures/pull/521>`_]
 * Filter data by source in the API endpoint `/sensors/data` (GET) [see `PR #543 <http://www.github.com/FlexMeasures/flexmeasures/pull/543>`_]
+* New resampling functionality for instantaneous sensor data: 1) ``flexmeasures show beliefs`` can now handle showing (and saving) instantaneous sensor data and non-instantaneous sensor data together, and 2) the API endpoint `/sensors/data` (GET) now allows fetching instantaneous sensor data in a custom frequency, by using the "resolution" field [see `PR #542 <http://www.github.com/FlexMeasures/flexmeasures/pull/542>`_]
 
 Bugfixes
 -----------

documentation/tut/posting_data.rst

@@ -21,7 +21,7 @@ Prerequisites
 - FlexMeasures needs some structural meta data for data to be understood. For example, for adding weather data we need to define a weather sensor, and what kind of weather sensors there are. You also need a user account. If you host FlexMeasures yourself, you need to add this info first. Head over to :ref:`getting_started`, where these steps are covered, study our :ref:`cli` or look into plugins which do this like `flexmeasures-entsoe <https://github.com/SeitaBV/flexmeasures-entsoe>`_ or `flexmeasures-openweathermap <https://github.com/SeitaBV/flexmeasures-openweathermap>`_.
 - You should be familiar with where to find your API endpoints (see :ref:`api_versions`) and how to authenticate against the API (see :ref:`api_auth`).
 
-.. note:: For deeper explanations of the data and the meta fields we'll send here, You can always read the :ref:`api_introduction`, to the FlexMeasures API, e.g. :ref:`signs`, :ref:`resolutions`, :ref:`prognoses` and :ref:`units`.
+.. note:: For deeper explanations of the data and the meta fields we'll send here, You can always read the :ref:`api_introduction`, to the FlexMeasures API, e.g. :ref:`signs`, :ref:`frequency_and_resolution`, :ref:`prognoses` and :ref:`units`.
 
 .. note:: To address assets and sensors, these tutorials assume entity addresses valid in the namespace ``fm1``. See :ref:`api_introduction` for more explanations. 
 

flexmeasures/api/common/schemas/sensor_data.py

@@ -17,7 +17,11 @@
 from flexmeasures.data.models.planning.utils import initialize_index
 from flexmeasures.data.schemas import AwareDateTimeField, DurationField, SourceIdField
 from flexmeasures.data.services.time_series import simplify_index
-from flexmeasures.utils.time_utils import duration_isoformat, server_now
+from flexmeasures.utils.time_utils import (
+    decide_resolution,
+    duration_isoformat,
+    server_now,
+)
 from flexmeasures.utils.unit_utils import (
     convert_units,
     units_are_convertible,
@@ -155,6 +159,14 @@ def dump_bdf(self, sensor_data_description: dict, **kwargs) -> dict:
         resolution = sensor_data_description.get("resolution")
         source = sensor_data_description.get("source")
 
+        # Post-load configuration of event frequency
+        if resolution is None:
+            if sensor.event_resolution != timedelta(hours=0):
+                resolution = sensor.event_resolution
+            else:
+                # For instantaneous sensors, choose a default resolution given the requested time window
+                resolution = decide_resolution(start, end)
+
         # Post-load configuration of belief timing against message type
         horizons_at_least = sensor_data_description.get("horizon", None)
         horizons_at_most = None
@@ -183,7 +195,7 @@ def dump_bdf(self, sensor_data_description: dict, **kwargs) -> dict:
         )
 
         # Convert to desired time range
-        index = initialize_index(start=start, end=end, resolution=df.event_resolution)
+        index = initialize_index(start=start, end=end, resolution=resolution)
         df = df.reindex(index)
 
         # Convert to desired unit
@@ -202,6 +214,7 @@ def dump_bdf(self, sensor_data_description: dict, **kwargs) -> dict:
             start=datetime_isoformat(start),
             duration=duration_isoformat(duration),
             unit=unit,
+            resolution=duration_isoformat(df.event_resolution),
         )
 
         return response
@@ -256,12 +269,21 @@ def check_schema_unit_against_type(self, data, **kwargs):
             )
 
     @validates_schema
-    def check_resolution_compatibility_of_values(self, data, **kwargs):
-        inferred_resolution = data["duration"] / len(data["values"])
+    def check_resolution_compatibility_of_sensor_data(self, data, **kwargs):
+        """Ensure event frequency is compatible with the sensor's event resolution.
+
+        For a sensor recording instantaneous values, any event frequency is compatible.
+        For a sensor recording non-instantaneous values, the event frequency must fit the sensor's event resolution.
+        Currently, only upsampling is supported (e.g. converting hourly events to 15-minute events).
+        """
         required_resolution = data["sensor"].event_resolution
-        # TODO: we don't yet have a good policy w.r.t. zero-resolution (direct measurement)
         if required_resolution == timedelta(hours=0):
+            # For instantaneous sensors, any event frequency is compatible
             return
+
+        # The event frequency is inferred by assuming sequential, equidistant values within a time interval.
+        # The event resolution is assumed to be equal to the event frequency.
+        inferred_resolution = data["duration"] / len(data["values"])
         if inferred_resolution % required_resolution != timedelta(hours=0):
             raise ValidationError(
                 f"Resolution of {inferred_resolution} is incompatible with the sensor's required resolution of {required_resolution}."
@@ -305,13 +327,15 @@ def possibly_upsample_values(data):
         Upsample the data if needed, to fit to the sensor's resolution.
         Marshmallow runs this after validation.
         """
-        inferred_resolution = data["duration"] / len(data["values"])
         required_resolution = data["sensor"].event_resolution
-
-        # TODO: we don't yet have a good policy w.r.t. zero-resolution (direct measurement)
         if required_resolution == timedelta(hours=0):
+            # For instantaneous sensors, no need to upsample
             return data
 
+        # The event frequency is inferred by assuming sequential, equidistant values within a time interval.
+        # The event resolution is assumed to be equal to the event frequency.
+        inferred_resolution = data["duration"] / len(data["values"])
+
         # we already know resolutions are compatible (see validation)
         if inferred_resolution != required_resolution:
             data["values"] = upsample_values(

flexmeasures/api/dev/tests/conftest.py

@@ -1,6 +1,6 @@
 import pytest
 
-from flexmeasures.api.v3_0.tests.conftest import add_gas_sensor
+from flexmeasures.api.v3_0.tests.conftest import add_incineration_line
 from flexmeasures.data.models.time_series import Sensor
 
 
@@ -10,7 +10,7 @@ def setup_api_test_data(db, setup_roles_users, setup_generic_assets):
     Set up data for API dev tests.
     """
     print("Setting up data for API dev tests on %s" % db.engine)
-    add_gas_sensor(db, setup_roles_users["Test Supplier User"])
+    add_incineration_line(db, setup_roles_users["Test Supplier User"])
 
 
 @pytest.fixture(scope="function")
@@ -23,4 +23,4 @@ def setup_api_fresh_test_data(
     print("Setting up fresh data for API dev tests on %s" % fresh_db.engine)
     for sensor in Sensor.query.all():
         fresh_db.delete(sensor)
-    add_gas_sensor(fresh_db, setup_roles_users_fresh_db["Test Supplier User"])
+    add_incineration_line(fresh_db, setup_roles_users_fresh_db["Test Supplier User"])

flexmeasures/api/v1/routes.py

@@ -38,7 +38,7 @@ def get_meter_data():
 
     **Optional fields**
 
-    - "resolution" (see :ref:`resolutions`)
+    - "resolution" (see :ref:`frequency_and_resolution`)
     - "horizon" (see :ref:`beliefs`)
     - "prior" (see :ref:`beliefs`)
     - "source" (see :ref:`sources`)

flexmeasures/api/v1_1/routes.py

@@ -275,7 +275,7 @@ def get_prognosis():
 
     **Optional fields**
 
-    - "resolution" (see :ref:`resolutions`)
+    - "resolution" (see :ref:`frequency_and_resolution`)
     - "horizon" (see :ref:`beliefs`)
     - "prior" (see :ref:`beliefs`)
     - "source" (see :ref:`sources`)

flexmeasures/api/v3_0/sensors.py

@@ -183,7 +183,7 @@ def get_data(self, response: dict):
 
         **Optional fields**
 
-        - "resolution" (see :ref:`resolutions`)
+        - "resolution" (see :ref:`frequency_and_resolution`)
         - "horizon" (see :ref:`beliefs`)
         - "prior" (see :ref:`beliefs`)
         - "source" (see :ref:`sources`)

flexmeasures/api/v3_0/tests/conftest.py

@@ -17,8 +17,8 @@ def setup_api_test_data(
     Set up data for API v3.0 tests.
     """
     print("Setting up data for API v3.0 tests on %s" % db.engine)
-    gas_sensor = add_gas_sensor(db, setup_roles_users["Test Supplier User"])
-    return {gas_sensor.name: gas_sensor}
+    sensors = add_incineration_line(db, setup_roles_users["Test Supplier User"])
+    return sensors
 
 
 @pytest.fixture(scope="function")
@@ -31,22 +31,10 @@ def setup_api_fresh_test_data(
     print("Setting up fresh data for API 3.0 tests on %s" % fresh_db.engine)
     for sensor in Sensor.query.all():
         fresh_db.delete(sensor)
-    gas_sensor = add_gas_sensor(
+    sensors = add_incineration_line(
         fresh_db, setup_roles_users_fresh_db["Test Supplier User"]
     )
-    return {gas_sensor.name: gas_sensor}
-
-
-@pytest.fixture(scope="function")
-def setup_api_fresh_gas_measurements(
-    fresh_db, setup_api_fresh_test_data, setup_roles_users_fresh_db
-):
-    """Set up some measurements for the gas sensor."""
-    add_gas_measurements(
-        fresh_db,
-        setup_roles_users_fresh_db["Test Supplier User"].data_source[0],
-        setup_api_fresh_test_data["some gas sensor"],
-    )
+    return sensors
 
 
 @pytest.fixture(scope="module")
@@ -66,15 +54,15 @@ def setup_inactive_user(db, setup_accounts, setup_roles_users):
     )
 
 
-def add_gas_sensor(db, test_supplier_user) -> Sensor:
+def add_incineration_line(db, test_supplier_user) -> dict[str, Sensor]:
     incineration_type = GenericAssetType(
         name="waste incinerator",
     )
     db.session.add(incineration_type)
     incineration_asset = GenericAsset(
         name="incineration line",
         generic_asset_type=incineration_type,
-        account_id=test_supplier_user.account_id,
+        owner=test_supplier_user.account,
     )
     db.session.add(incineration_asset)
     gas_sensor = Sensor(
@@ -84,18 +72,50 @@ def add_gas_sensor(db, test_supplier_user) -> Sensor:
         generic_asset=incineration_asset,
     )
     db.session.add(gas_sensor)
-    return gas_sensor
+    add_gas_measurements(db, test_supplier_user.data_source[0], gas_sensor)
+    temperature_sensor = Sensor(
+        name="some temperature sensor",
+        unit="°C",
+        event_resolution=timedelta(0),
+        generic_asset=incineration_asset,
+    )
+    db.session.add(temperature_sensor)
+    add_temperature_measurements(
+        db, test_supplier_user.data_source[0], temperature_sensor
+    )
+
+    db.session.flush()  # assign sensor ids
+    return {gas_sensor.name: gas_sensor, temperature_sensor.name: temperature_sensor}
 
 
-def add_gas_measurements(db, source: Source, gas_sensor: Sensor):
+def add_gas_measurements(db, source: Source, sensor: Sensor):
     event_starts = [
-        pd.Timestamp("2021-08-02T00:00:00+02:00") + timedelta(minutes=minutes)
+        pd.Timestamp("2021-05-02T00:00:00+02:00") + timedelta(minutes=minutes)
         for minutes in range(0, 30, 10)
     ]
     event_values = [91.3, 91.7, 92.1]
     beliefs = [
         TimedBelief(
-            sensor=gas_sensor,
+            sensor=sensor,
+            source=source,
+            event_start=event_start,
+            belief_horizon=timedelta(0),
+            event_value=event_value,
+        )
+        for event_start, event_value in zip(event_starts, event_values)
+    ]
+    db.session.add_all(beliefs)
+
+
+def add_temperature_measurements(db, source: Source, sensor: Sensor):
+    event_starts = [
+        pd.Timestamp("2021-05-02T00:00:00+02:00") + timedelta(minutes=minutes)
+        for minutes in range(0, 30, 10)
+    ]
+    event_values = [815, 817, 818]
+    beliefs = [
+        TimedBelief(
+            sensor=sensor,
             source=source,
             event_start=event_start,
             belief_horizon=timedelta(0),