diff --git a/documentation/api/notation.rst b/documentation/api/notation.rst index e96519d86..e240a1c54 100644 --- a/documentation/api/notation.rst +++ b/documentation/api/notation.rst @@ -92,23 +92,16 @@ It uses the fact that all FlexMeasures sensors have unique IDs. ea1.2021-01.io.flexmeasures:fm1.42 ea1.2021-01.io.flexmeasures:fm1. -.. todo:: UDI events are not yet modelled in the fm1 scheme - The ``fm0`` scheme is the original scheme. It identified different types of sensors (such as grid connections, weather sensors and markets) in different ways. -The ``fm0`` scheme has been deprecated for the most part and is no longer supported officially. -Only UDI events still need to be sent using the fm0 scheme. - -.. code-block:: - - ea1.2021-01.io.flexmeasures:fm0.40:30:302:soc - ea1.2021-01.io.flexmeasures:fm0.::: +The ``fm0`` scheme has been deprecated and is no longer supported officially. Timeseries ^^^^^^^^^^ -Timestamps and durations are consistent with the ISO 8601 standard. The resolution of the data is implicit (from duration and number of values), see :ref:`resolutions`. +Timestamps and durations are consistent with the ISO 8601 standard. +The frequency of the data is implicit (from duration and number of values), while the resolution of the data is explicit, see :ref:`frequency_and_resolution`. All timestamps in requests to the API must be timezone-aware. For instance, in the below example, the timezone indication "Z" indicates a zero offset from UTC. @@ -267,19 +260,36 @@ For example, the following message implies that all prognosed values were made 1 Note that, for a horizon indicating a belief 10 minutes after the *start* of each 15-minute interval, the "horizon" would have been "PT5M". This denotes that the prognosed interval has 5 minutes left to be concluded. -.. _resolutions: +.. _frequency_and_resolution: + +Frequency and resolution +^^^^^^^^^^^^^^^^^^^^^^^^ + +FlexMeasures handles two types of time series, which can be distinguished by defining the following timing properties for events recorded by sensors: + +- Frequency: how far apart events occur (a constant duration between event starts) +- Resolution: how long an event lasts (a constant duration between the start and end of an event) + +.. note:: FlexMeasures runs on Pandas, and follows Pandas terminology accordingly. + The term frequency as used by Pandas is the reciprocal of the `SI quantity for frequency `_. + +1. The first type of time series describes non-instantaneous events such as average hourly wind speed. + For this case, it is commonly assumed that ``frequency == resolution``. + That is, events follow each other sequentially and without delay. -Resolutions -^^^^^^^^^^^ +2. The second type of time series describes instantaneous events (zero resolution) such as temperature at a given time. + For this case, we have ``frequency != resolution``. -Specifying a resolution is redundant for POST requests that contain both "values" and a "duration" ― FlexMeasures computes the resolution by dividing the duration by the number of values. +Specifying a frequency and resolution is redundant for POST requests that contain both "values" and a "duration" ― FlexMeasures computes the frequency by dividing the duration by the number of values, and, for sensors that record non-instantaneous events, assumes the resolution of the data is equal to the frequency. -When POSTing data, FlexMeasures checks this computed resolution against the required resolution of the sensors which are posted to. If these can't be matched (through upsampling), an error will occur. +When POSTing data, FlexMeasures checks this inferred resolution against the required resolution of the sensors that are posted to. +If these can't be matched (through upsampling), an error will occur. -GET requests (such as *getMeterData*) return data in the resolution which the sensor is configured for. -A "resolution" may be specified explicitly to obtain the data in downsampled form, -which can be very beneficial for download speed. The specified resolution needs to be a multiple -of the sensor's resolution, e.g. hourly or daily values if the sensor's resolution is 15 minutes. +GET requests (such as */sensors/data*) return data with a frequency either equal to the resolution that the sensor is configured for (for non-instantaneous sensors), or a default frequency befitting (in our opinion) the requested time interval. +A "resolution" may be specified explicitly to obtain the data in downsampled form, which can be very beneficial for download speed. +For non-instantaneous sensors, the specified resolution needs to be a multiple of the sensor's resolution, e.g. hourly or daily values if the sensor's resolution is 15 minutes. +For instantaneous sensors, the specified resolution is interpreted as a request for data in a specific frequency. +The resolution of the underlying data will remain zero (and the returned message will say so). .. _sources: diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 682436185..9a9b0a8b8 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -7,7 +7,6 @@ v0.12.0 | October XX, 2022 .. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``). - New features ------------- @@ -20,6 +19,7 @@ New features * Improved import of time series data from CSV file: 1) drop duplicate records with warning, 2) allow configuring which column contains explicit recording times for each data point (use case: import forecasts) [see `PR #501 `_], 3) localize timezone naive data, 4) support reading in datetime and timedelta values, 5) remove rows with NaN values, and 6) filter by values in specific columns [see `PR #521 `_] * Filter data by source in the API endpoint `/sensors/data` (GET) [see `PR #543 `_] * Allow posting ``null`` values to `/sensors/data` (POST) to correctly space time series that include missing values (the missing values are not stored) [see `PR #549 `_] +* New resampling functionality for instantaneous sensor data: 1) ``flexmeasures show beliefs`` can now handle showing (and saving) instantaneous sensor data and non-instantaneous sensor data together, and 2) the API endpoint `/sensors/data` (GET) now allows fetching instantaneous sensor data in a custom frequency, by using the "resolution" field [see `PR #542 `_] Bugfixes ----------- @@ -42,7 +42,6 @@ Infrastructure / Support * Scheduling test for maximizing self-consumption, and improved time series db queries for fixed tariffs (and other long-term constants) [see `PR #532 `_] * Clean up table formatting for ``flexmeasures show`` CLI commands [see `PR #540 `_] - .. warning:: The CLI command ``flexmeasures monitor tasks`` has been renamed to ``flexmeasures monitor last-run``. The old name will stop working in version 0.13. @@ -90,7 +89,6 @@ New features .. note:: Read more on these features on `the FlexMeasures blog `__. - Bugfixes ----------- * Do not fail asset page if entity addresses cannot be built [see `PR #457 `_] @@ -199,7 +197,6 @@ New features .. note:: Read more on these features on `the FlexMeasures blog `__. - Bugfixes ----------- @@ -445,8 +442,6 @@ Infrastructure / Support * Ensured unique sensor ids for all sensors [see `PR #70 `_ and (fix) `PR #77 `_] - - v0.2.3 | February 27, 2021 =========================== @@ -463,7 +458,6 @@ Bugfixes * Fix maps on new asset page (update MapBox lib) [see `PR #27 `_] * Some asset links were broken [see `PR #20 `_] * Password reset link on account page was broken [see `PR #23 `_] - Infrastructure / Support ---------------------- diff --git a/documentation/tut/posting_data.rst b/documentation/tut/posting_data.rst index 809f0d6d6..36dbedf77 100644 --- a/documentation/tut/posting_data.rst +++ b/documentation/tut/posting_data.rst @@ -21,7 +21,7 @@ Prerequisites - FlexMeasures needs some structural meta data for data to be understood. For example, for adding weather data we need to define a weather sensor, and what kind of weather sensors there are. You also need a user account. If you host FlexMeasures yourself, you need to add this info first. Head over to :ref:`getting_started`, where these steps are covered, study our :ref:`cli` or look into plugins which do this like `flexmeasures-entsoe `_ or `flexmeasures-openweathermap `_. - You should be familiar with where to find your API endpoints (see :ref:`api_versions`) and how to authenticate against the API (see :ref:`api_auth`). -.. note:: For deeper explanations of the data and the meta fields we'll send here, You can always read the :ref:`api_introduction`, to the FlexMeasures API, e.g. :ref:`signs`, :ref:`resolutions`, :ref:`prognoses` and :ref:`units`. +.. note:: For deeper explanations of the data and the meta fields we'll send here, You can always read the :ref:`api_introduction`, to the FlexMeasures API, e.g. :ref:`signs`, :ref:`frequency_and_resolution`, :ref:`prognoses` and :ref:`units`. .. note:: To address assets and sensors, these tutorials assume entity addresses valid in the namespace ``fm1``. See :ref:`api_introduction` for more explanations. diff --git a/flexmeasures/api/common/schemas/sensor_data.py b/flexmeasures/api/common/schemas/sensor_data.py index 82d1d3e0d..1502d8286 100644 --- a/flexmeasures/api/common/schemas/sensor_data.py +++ b/flexmeasures/api/common/schemas/sensor_data.py @@ -17,7 +17,11 @@ from flexmeasures.data.models.planning.utils import initialize_index from flexmeasures.data.schemas import AwareDateTimeField, DurationField, SourceIdField from flexmeasures.data.services.time_series import simplify_index -from flexmeasures.utils.time_utils import duration_isoformat, server_now +from flexmeasures.utils.time_utils import ( + decide_resolution, + duration_isoformat, + server_now, +) from flexmeasures.utils.unit_utils import ( convert_units, units_are_convertible, @@ -155,6 +159,14 @@ def dump_bdf(self, sensor_data_description: dict, **kwargs) -> dict: resolution = sensor_data_description.get("resolution") source = sensor_data_description.get("source") + # Post-load configuration of event frequency + if resolution is None: + if sensor.event_resolution != timedelta(hours=0): + resolution = sensor.event_resolution + else: + # For instantaneous sensors, choose a default resolution given the requested time window + resolution = decide_resolution(start, end) + # Post-load configuration of belief timing against message type horizons_at_least = sensor_data_description.get("horizon", None) horizons_at_most = None @@ -183,7 +195,7 @@ def dump_bdf(self, sensor_data_description: dict, **kwargs) -> dict: ) # Convert to desired time range - index = initialize_index(start=start, end=end, resolution=df.event_resolution) + index = initialize_index(start=start, end=end, resolution=resolution) df = df.reindex(index) # Convert to desired unit @@ -202,6 +214,7 @@ def dump_bdf(self, sensor_data_description: dict, **kwargs) -> dict: start=datetime_isoformat(start), duration=duration_isoformat(duration), unit=unit, + resolution=duration_isoformat(df.event_resolution), ) return response @@ -256,12 +269,21 @@ def check_schema_unit_against_type(self, data, **kwargs): ) @validates_schema - def check_resolution_compatibility_of_values(self, data, **kwargs): - inferred_resolution = data["duration"] / len(data["values"]) + def check_resolution_compatibility_of_sensor_data(self, data, **kwargs): + """Ensure event frequency is compatible with the sensor's event resolution. + + For a sensor recording instantaneous values, any event frequency is compatible. + For a sensor recording non-instantaneous values, the event frequency must fit the sensor's event resolution. + Currently, only upsampling is supported (e.g. converting hourly events to 15-minute events). + """ required_resolution = data["sensor"].event_resolution - # TODO: we don't yet have a good policy w.r.t. zero-resolution (direct measurement) if required_resolution == timedelta(hours=0): + # For instantaneous sensors, any event frequency is compatible return + + # The event frequency is inferred by assuming sequential, equidistant values within a time interval. + # The event resolution is assumed to be equal to the event frequency. + inferred_resolution = data["duration"] / len(data["values"]) if inferred_resolution % required_resolution != timedelta(hours=0): raise ValidationError( f"Resolution of {inferred_resolution} is incompatible with the sensor's required resolution of {required_resolution}." @@ -305,13 +327,15 @@ def possibly_upsample_values(data): Upsample the data if needed, to fit to the sensor's resolution. Marshmallow runs this after validation. """ - inferred_resolution = data["duration"] / len(data["values"]) required_resolution = data["sensor"].event_resolution - - # TODO: we don't yet have a good policy w.r.t. zero-resolution (direct measurement) if required_resolution == timedelta(hours=0): + # For instantaneous sensors, no need to upsample return data + # The event frequency is inferred by assuming sequential, equidistant values within a time interval. + # The event resolution is assumed to be equal to the event frequency. + inferred_resolution = data["duration"] / len(data["values"]) + # we already know resolutions are compatible (see validation) if inferred_resolution != required_resolution: data["values"] = upsample_values( diff --git a/flexmeasures/api/dev/tests/conftest.py b/flexmeasures/api/dev/tests/conftest.py index 6887661e2..9b98647af 100644 --- a/flexmeasures/api/dev/tests/conftest.py +++ b/flexmeasures/api/dev/tests/conftest.py @@ -1,6 +1,6 @@ import pytest -from flexmeasures.api.v3_0.tests.conftest import add_gas_sensor +from flexmeasures.api.v3_0.tests.conftest import add_incineration_line from flexmeasures.data.models.time_series import Sensor @@ -10,7 +10,7 @@ def setup_api_test_data(db, setup_roles_users, setup_generic_assets): Set up data for API dev tests. """ print("Setting up data for API dev tests on %s" % db.engine) - add_gas_sensor(db, setup_roles_users["Test Supplier User"]) + add_incineration_line(db, setup_roles_users["Test Supplier User"]) @pytest.fixture(scope="function") @@ -23,4 +23,4 @@ def setup_api_fresh_test_data( print("Setting up fresh data for API dev tests on %s" % fresh_db.engine) for sensor in Sensor.query.all(): fresh_db.delete(sensor) - add_gas_sensor(fresh_db, setup_roles_users_fresh_db["Test Supplier User"]) + add_incineration_line(fresh_db, setup_roles_users_fresh_db["Test Supplier User"]) diff --git a/flexmeasures/api/v1/routes.py b/flexmeasures/api/v1/routes.py index 5fe5c3cd5..6727ccf42 100644 --- a/flexmeasures/api/v1/routes.py +++ b/flexmeasures/api/v1/routes.py @@ -38,7 +38,7 @@ def get_meter_data(): **Optional fields** - - "resolution" (see :ref:`resolutions`) + - "resolution" (see :ref:`frequency_and_resolution`) - "horizon" (see :ref:`beliefs`) - "prior" (see :ref:`beliefs`) - "source" (see :ref:`sources`) diff --git a/flexmeasures/api/v1_1/routes.py b/flexmeasures/api/v1_1/routes.py index b79fb6701..0c5b27dac 100644 --- a/flexmeasures/api/v1_1/routes.py +++ b/flexmeasures/api/v1_1/routes.py @@ -275,7 +275,7 @@ def get_prognosis(): **Optional fields** - - "resolution" (see :ref:`resolutions`) + - "resolution" (see :ref:`frequency_and_resolution`) - "horizon" (see :ref:`beliefs`) - "prior" (see :ref:`beliefs`) - "source" (see :ref:`sources`) diff --git a/flexmeasures/api/v3_0/sensors.py b/flexmeasures/api/v3_0/sensors.py index f8ad94c02..9a6191c3b 100644 --- a/flexmeasures/api/v3_0/sensors.py +++ b/flexmeasures/api/v3_0/sensors.py @@ -184,7 +184,7 @@ def get_data(self, response: dict): **Optional fields** - - "resolution" (see :ref:`resolutions`) + - "resolution" (see :ref:`frequency_and_resolution`) - "horizon" (see :ref:`beliefs`) - "prior" (see :ref:`beliefs`) - "source" (see :ref:`sources`) diff --git a/flexmeasures/api/v3_0/tests/conftest.py b/flexmeasures/api/v3_0/tests/conftest.py index f73f54b94..be9ddd120 100644 --- a/flexmeasures/api/v3_0/tests/conftest.py +++ b/flexmeasures/api/v3_0/tests/conftest.py @@ -17,8 +17,8 @@ def setup_api_test_data( Set up data for API v3.0 tests. """ print("Setting up data for API v3.0 tests on %s" % db.engine) - gas_sensor = add_gas_sensor(db, setup_roles_users["Test Supplier User"]) - return {gas_sensor.name: gas_sensor} + sensors = add_incineration_line(db, setup_roles_users["Test Supplier User"]) + return sensors @pytest.fixture(scope="function") @@ -31,22 +31,10 @@ def setup_api_fresh_test_data( print("Setting up fresh data for API 3.0 tests on %s" % fresh_db.engine) for sensor in Sensor.query.all(): fresh_db.delete(sensor) - gas_sensor = add_gas_sensor( + sensors = add_incineration_line( fresh_db, setup_roles_users_fresh_db["Test Supplier User"] ) - return {gas_sensor.name: gas_sensor} - - -@pytest.fixture(scope="function") -def setup_api_fresh_gas_measurements( - fresh_db, setup_api_fresh_test_data, setup_roles_users_fresh_db -): - """Set up some measurements for the gas sensor.""" - add_gas_measurements( - fresh_db, - setup_roles_users_fresh_db["Test Supplier User"].data_source[0], - setup_api_fresh_test_data["some gas sensor"], - ) + return sensors @pytest.fixture(scope="module") @@ -66,7 +54,7 @@ def setup_inactive_user(db, setup_accounts, setup_roles_users): ) -def add_gas_sensor(db, test_supplier_user) -> Sensor: +def add_incineration_line(db, test_supplier_user) -> dict[str, Sensor]: incineration_type = GenericAssetType( name="waste incinerator", ) @@ -74,7 +62,7 @@ def add_gas_sensor(db, test_supplier_user) -> Sensor: incineration_asset = GenericAsset( name="incineration line", generic_asset_type=incineration_type, - account_id=test_supplier_user.account_id, + owner=test_supplier_user.account, ) db.session.add(incineration_asset) gas_sensor = Sensor( @@ -84,18 +72,50 @@ def add_gas_sensor(db, test_supplier_user) -> Sensor: generic_asset=incineration_asset, ) db.session.add(gas_sensor) - return gas_sensor + add_gas_measurements(db, test_supplier_user.data_source[0], gas_sensor) + temperature_sensor = Sensor( + name="some temperature sensor", + unit="°C", + event_resolution=timedelta(0), + generic_asset=incineration_asset, + ) + db.session.add(temperature_sensor) + add_temperature_measurements( + db, test_supplier_user.data_source[0], temperature_sensor + ) + + db.session.flush() # assign sensor ids + return {gas_sensor.name: gas_sensor, temperature_sensor.name: temperature_sensor} -def add_gas_measurements(db, source: Source, gas_sensor: Sensor): +def add_gas_measurements(db, source: Source, sensor: Sensor): event_starts = [ - pd.Timestamp("2021-08-02T00:00:00+02:00") + timedelta(minutes=minutes) + pd.Timestamp("2021-05-02T00:00:00+02:00") + timedelta(minutes=minutes) for minutes in range(0, 30, 10) ] event_values = [91.3, 91.7, 92.1] beliefs = [ TimedBelief( - sensor=gas_sensor, + sensor=sensor, + source=source, + event_start=event_start, + belief_horizon=timedelta(0), + event_value=event_value, + ) + for event_start, event_value in zip(event_starts, event_values) + ] + db.session.add_all(beliefs) + + +def add_temperature_measurements(db, source: Source, sensor: Sensor): + event_starts = [ + pd.Timestamp("2021-05-02T00:00:00+02:00") + timedelta(minutes=minutes) + for minutes in range(0, 30, 10) + ] + event_values = [815, 817, 818] + beliefs = [ + TimedBelief( + sensor=sensor, source=source, event_start=event_start, belief_horizon=timedelta(0), diff --git a/flexmeasures/api/v3_0/tests/test_sensor_data_fresh_db.py b/flexmeasures/api/v3_0/tests/test_sensor_data_fresh_db.py index b2f803735..fe924f776 100644 --- a/flexmeasures/api/v3_0/tests/test_sensor_data_fresh_db.py +++ b/flexmeasures/api/v3_0/tests/test_sensor_data_fresh_db.py @@ -42,7 +42,11 @@ def test_post_sensor_data( num_values=num_values, unit=unit, include_a_null=include_a_null ) sensor = Sensor.query.filter(Sensor.name == "some gas sensor").one_or_none() - beliefs_before = TimedBelief.query.filter(TimedBelief.sensor_id == sensor.id).all() + filters = ( + TimedBelief.sensor_id == sensor.id, + TimedBelief.event_start >= post_data["start"], + ) + beliefs_before = TimedBelief.query.filter(*filters).all() print(f"BELIEFS BEFORE: {beliefs_before}") assert len(beliefs_before) == 0 @@ -54,7 +58,7 @@ def test_post_sensor_data( ) print(response.json) assert response.status_code == 200 - beliefs = TimedBelief.query.filter(TimedBelief.sensor_id == sensor.id).all() + beliefs = TimedBelief.query.filter(*filters).all() print(f"BELIEFS AFTER: {beliefs}") assert len(beliefs) == expected_num_values # check that values are scaled to the sensor unit correctly @@ -65,19 +69,16 @@ def test_post_sensor_data( def test_get_sensor_data( client, - db, setup_api_fresh_test_data: dict[str, Sensor], - setup_api_fresh_gas_measurements, setup_roles_users_fresh_db: dict[str, User], ): """Check the /sensors/data endpoint for fetching 1 hour of data of a 10-minute resolution sensor.""" sensor = setup_api_fresh_test_data["some gas sensor"] source: Source = setup_roles_users_fresh_db["Test Supplier User"].data_source[0] assert sensor.event_resolution == timedelta(minutes=10) - db.session.flush() # assign sensor id message = { "sensor": f"ea1.2021-01.io.flexmeasures:fm1.{sensor.id}", - "start": "2021-08-02T00:00:00+02:00", + "start": "2021-05-02T00:00:00+02:00", "duration": "PT1H20M", "horizon": "PT0H", "unit": "m³/h", @@ -94,5 +95,37 @@ def test_get_sensor_data( assert response.status_code == 200 values = response.json["values"] # We expect two data points (from conftest) followed by 2 null values (which are converted to None by .json) - # The first data point averages 91.3 and 91.7, and the second data point averages 92.1 and None. + # The first data point averages [91.3, 91.7], and the second data point averages [92.1, None]. assert all(a == b for a, b in zip(values, [91.5, 92.1, None, None])) + + +def test_get_instantaneous_sensor_data( + client, + setup_api_fresh_test_data: dict[str, Sensor], + setup_roles_users_fresh_db: dict[str, User], +): + """Check the /sensors/data endpoint for fetching 1 hour of data of an instantaneous sensor.""" + sensor = setup_api_fresh_test_data["some temperature sensor"] + source: Source = setup_roles_users_fresh_db["Test Supplier User"].data_source[0] + assert sensor.event_resolution == timedelta(minutes=0) + message = { + "sensor": f"ea1.2021-01.io.flexmeasures:fm1.{sensor.id}", + "start": "2021-05-02T00:00:00+02:00", + "duration": "PT1H20M", + "horizon": "PT0H", + "unit": "°C", + "source": source.id, + "resolution": "PT20M", + } + auth_token = get_auth_token(client, "test_supplier_user_4@seita.nl", "testtest") + response = client.get( + url_for("SensorAPI:get_data"), + query_string=message, + headers={"content-type": "application/json", "Authorization": auth_token}, + ) + print("Server responded with:\n%s" % response.json) + assert response.status_code == 200 + values = response.json["values"] + # We expect two data point (from conftest) followed by 2 null values (which are converted to None by .json) + # The first data point is the first of [815, 817], and the second data point is the first of [818, None]. + assert all(a == b for a, b in zip(values, [815, 818, None, None])) diff --git a/flexmeasures/data/models/time_series.py b/flexmeasures/data/models/time_series.py index d1acf04f6..17c8f7fe6 100644 --- a/flexmeasures/data/models/time_series.py +++ b/flexmeasures/data/models/time_series.py @@ -611,7 +611,10 @@ def search( * If user_source_ids is specified, the "user" source type is automatically included (and not excluded). Somewhat redundant, though still allowed, is to set both source_types and exclude_source_types. - ** Note that timely-beliefs converts string resolutions to datetime.timedelta objects (see https://github.com/SeitaBV/timely-beliefs/issues/13). + ** Note that: + - timely-beliefs converts string resolutions to datetime.timedelta objects (see https://github.com/SeitaBV/timely-beliefs/issues/13). + - for sensors recording non-instantaneous data: updates both the event frequency and the event resolution + - for sensors recording instantaneous data: updates only the event frequency (and event resolution remains 0) """ # todo: deprecate the 'sensor' argument in favor of 'sensors' (announced v0.8.0) sensors = tb_utils.replace_deprecated_argument(