Remove obsolete data classes (#695)

Removed obsolete code dealing with deprecated data models (e.g. assets, markets and weather sensors), and sunset the fm0 scheme for entity addresses.


* Fix doctests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete view utils

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete api utils

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Sunset old save_to_db, already announced in v0.8.0

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete portfolio queries

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete analytics queries

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete resource utils

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete resource class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete AssetSchema

Signed-off-by: F.N. Claessen <felix@seita.nl>

* refactor: Rename module

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update test: user deletion does not remove their organisation's data

Signed-off-by: F.N. Claessen <felix@seita.nl>

* temporary fix for printing out old Asset instances

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clean up UI conftest

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete Power class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete Price class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete Weather class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update conftest to stop using Market class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove obsolete Market and MarketType classes

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: Update notation.rst by removing mentions of sunset API versions

* fix merge conflict residue

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* fix: broken import after merge

Signed-off-by: F.N. Claessen <felix@seita.nl>

* sunset: fm0 scheme

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: sunset fm0 scheme

Signed-off-by: F.N. Claessen <felix@seita.nl>

* erratum: fm0 was not compatible with API v3

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: obsolete API utils

Signed-off-by: F.N. Claessen <felix@seita.nl>

* update docstring and set default to fm1 scheme

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: remove obsolete WeatherSensor class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: remove obsolete WeatherSensorType class and weather module

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix: find closest weather sensor

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: remove obsolete Asset and AssetType class from conftests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: remove obsolete Asset and AssetType class and corresponding module

Signed-off-by: F.N. Claessen <felix@seita.nl>

* style: black and flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: remove obsolete deprecation utils for fm0 scheme

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: remove obsolete TimedValue class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: remove obsolete API utils

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix: update test

Signed-off-by: F.N. Claessen <felix@seita.nl>

* style: black and flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: daterangepicker.js and moment.js dependencies

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update documentation notes on data model transition

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: obsolete coding utils

Signed-off-by: F.N. Claessen <felix@seita.nl>

* chore: update visualize_data_model.py

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* refactor: move towards a single function to set session variables from request variables

Signed-off-by: F.N. Claessen <felix@seita.nl>

* delete: remove obsolete session variables and corresponding view utils

Signed-off-by: F.N. Claessen <felix@seita.nl>

* feat: set 'sensor' as default entity type

Signed-off-by: F.N. Claessen <felix@seita.nl>

* refactor: retire the 'modern' label on save_to_db

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix: inline comment

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix fixture

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: remove mentions of portfolio, analytics and control pages

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui: remove mentions of portfolio, analytics and control pages from base template

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: remove documentation pages for portfolio, analytics and control pages

Signed-off-by: F.N. Claessen <felix@seita.nl>

* style: black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* style: flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: changelog entry

Signed-off-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Co-authored-by: Victor Garcia Reolid <victor@seita.nl>

documentation/api/notation.rst

@@ -94,7 +94,7 @@ It uses the fact that all FlexMeasures sensors have unique IDs.
 
 The ``fm0`` scheme is the original scheme.
 It identified different types of sensors (such as grid connections, weather sensors and markets) in different ways.
-The ``fm0`` scheme has been deprecated and is no longer supported officially.
+The ``fm0`` scheme has been sunset since API version 3.
 
 
 Timeseries
@@ -393,22 +393,17 @@ For example, to obtain data originating from data source 42, include the followi
 
 Data source IDs can be found by hovering over data in charts.
 
-.. note:: Older API version (< 3) accepted user IDs (integers), account roles (strings) and lists thereof, instead of data source IDs (integers).
-
-
 .. _units:
 
 Units
 ^^^^^
 
-From API version 3 onwards, we are much more flexible with sent units.
+The FlexMeasures API is quite flexible with sent units.
 A valid unit for timeseries data is any unit that is convertible to the configured sensor unit registered in FlexMeasures.
 So, for example, you can send timeseries data with "W" unit to a "kW" sensor.
 And if you wish to do so, you can even send a timeseries with "kWh" unit to a "kW" sensor.
 In this case, FlexMeasures will convert the data using the resolution of the timeseries.
 
-For API versions 1 and 2, the unit sent needs to be an exact match with the sensor unit, and only "MW" is allowed for power sensors.
-
 .. _signs:
 
 Signs of power values

documentation/changelog.rst

@@ -35,6 +35,7 @@ Infrastructure / Support
 * Document the `device_scheduler` linear program [see `PR #764 <https://www.github.com/FlexMeasures/flexmeasures/pull/764>`_].
 * Add support for `HiGHS <https://highs.dev/>`_ solver [see `PR #766 <https://www.github.com/FlexMeasures/flexmeasures/pull/766>`_].
 * Add support for installing FlexMeasures under Python 3.11 [see `PR #771 <https://www.github.com/FlexMeasures/flexmeasures/pull/771>`_].
+* Removed obsolete code dealing with deprecated data models (e.g. assets, markets and weather sensors), and sunset the fm0 scheme for entity addresses [see `PR #695 <https://www.github.com/FlexMeasures/flexmeasures/pull/695>`_ and `project 11 <https://www.github.com/FlexMeasures/flexmeasures/projects/11>`_]
 
 v0.14.2 | July 25, 2023
 ============================

documentation/configuration.rst

@@ -179,7 +179,7 @@ For more fine-grained control, the entries can also be tuples of view names and
 
 .. note:: This fine-grained control requires FlexMeasures version 0.6.0
 
-Default: ``["dashboard", "analytics", "portfolio", "assets", "users"]``
+Default: ``["dashboard"]``
 
 
 FLEXMEASURES_MENU_LISTED_VIEW_ICONS

documentation/dev/note-on-datamodel-transition.rst

@@ -11,41 +11,42 @@
 A note on the ongoing data model transition
 ============================================
 
-FlexMeasures is already ~3 years in the making. It's a normal process for well-maintained software to update architectural principles during such a time.
+FlexMeasures is already ~5 years in the making. It's a normal process for well-maintained software to update architectural principles during such a time.
 
-We are in the middle of a refactoring which affects the data model, and if you are using FlexMeasures on your own server, we want you to know the following:
+We are finishing up a refactoring which affects the data model, and if you are using FlexMeasures on your own server, we want you to know the following:
 
 
 We have your back
 ------------------
 
-If you work with the current model, there will be support to transition data to the new model once it's active. Actually, we are already working with the new model in some projects, so talk to us if you're interested.
+By upgrading FlexMeasures one minor version at a time, you get the most out of our transition tools, including database upgrades (moving data over from the old to the new model automatically), plugin compatibility warnings, deprecation warnings for upcoming sunsets, and blackout tests (:ref:`more info here<api_deprecation>`).
+If you still work with the old model and are having trouble to transition data to the current model, let us know.
 
 
 This transition is in your interest, as well
 ----------------------------------------------
 
-We do this transition so we can make FlexMeasures even more useful. For instance: support for more kinds of assets (energy plus related sensors). Or better forecasting and scheduling support.
+We did this transition so we could make FlexMeasures even more useful. For instance: support for more kinds of assets (energy plus related sensors), and better support for forecasting, scheduling and reporting.
 
 
 What are the big changes?
 -----------------------------
 
-There are two important transitions happening in this transition:
+There are two important transitions that happened in this transition:
 
-1. First, we'll be deprecating the specific data types ``Asset``, ``Market`` and ``WeatherSensor``. We learned that to manage energy flexibility, you need all sort of sensors, and thus a more generalisable data model. When we model assets and sensors, we'll also better be able to differentiate the business from the data world.
-2. Second, we'll fully integrate the `timely-beliefs framework <https://github.com/SeitaBV/timely-beliefs>`_ as the model for our time series data, which brings some major benefits for programmers as it lets us handle uncertain, multi-source time series data in a special Pandas data frame.
+1. First, we deprecated the specific data types ``Asset``, ``Market`` and ``WeatherSensor``. We learned that to manage energy flexibility, you need all sort of sensors, and thus a more generalisable data model. When we modelled assets and sensors, we were also better able to differentiate the business from the data world.
+2. Second, we fully integrated the `timely-beliefs framework <https://github.com/SeitaBV/timely-beliefs>`_ as the model for our time series data, which brings some major benefits for programmers as it lets us handle uncertain, multi-source time series data in a special Pandas data frame.
 
-For the curious, here are visualisations of where we're now and where we're going (click image for large versions).
+For the curious, here are visualisations of where we were before and where we're going (click image for large versions).
 
-The current model:
+The old model:
 
 .. image:: https://raw.githubusercontent.com/FlexMeasures/screenshots/main/architecture/FlexMeasures-CurrentDataModel.png
     :target: https://raw.githubusercontent.com/FlexMeasures/screenshots/main/architecture/FlexMeasures-CurrentDataModel.png
     :align: center
 ..    :scale: 40%
 
-The new model (work in progress): 
+The future model (work in progress):
 
 .. image:: https://raw.githubusercontent.com/FlexMeasures/screenshots/main/architecture/FlexMeasures-NewDataModel.png
     :target: https://raw.githubusercontent.com/FlexMeasures/screenshots/main/architecture/FlexMeasures-NewDataModel.png
@@ -64,19 +65,22 @@ Here is a brief list:
 - |check_| `Support Sensor and Asset diversity <https://github.com/FlexMeasures/flexmeasures/projects/9>`_: We are generalizing our database structure for organising energy data, to support all sorts of sensors and assets, and are letting users move their data to the new database model. We do this so we can better support the diverse set of use cases for energy flexibility.
 - |check_| `Update API endpoints for time series communication <https://github.com/FlexMeasures/flexmeasures/projects/13>`_: We are updating our API with new endpoints for communicating time series data, thereby consolidating a few older endpoints into a better standard. We do this so we can both simplify our API and documentation, and support a diversity of sensors.
 - |check_| `Update CLI commands for setting up Sensors and Assets <https://github.com/FlexMeasures/flexmeasures/projects/14>`_: We are updating our CLI commands to reflect the new database structure. We do this to facilitate setting up structure for new users.
-- |uncheck_| `Update UI views for Sensors and Assets <https://github.com/FlexMeasures/flexmeasures/projects/10>`_: We are updating our UI views (dashboard maps and analytics charts) according to our new database structure for organising energy data. We do this so users can customize what they want to see.
+- |check_| `Update UI views for Sensors and Assets <https://github.com/FlexMeasures/flexmeasures/projects/10>`_: We are updating our UI views (dashboard maps and analytics charts) according to our new database structure for organising energy data. We do this so users can customize what they want to see.
+- |check_| `Deprecate old database models <https://github.com/FlexMeasures/flexmeasures/projects/11>`_: We are deprecating the Power, Price and Weather tables in favour of the TimedBelief table, and deprecating the Asset, Market and WeatherSensor tables in favour of the Sensor and GenericAsset tables. We are doing this to clean up the code and database structure.
+- |uncheck_| `Infrastructure for reporting on sensors <https://github.com/FlexMeasures/flexmeasures/projects/19>`_: We are working on a backend infrastructure for sensors that record reports based on other sensors, like daily costs and aggregate power flow.
 - |uncheck_| `Scheduling of sensors <https://github.com/FlexMeasures/flexmeasures/projects/6>`_: We are extending our database structure for Sensors with actuator functionality, and are moving to a model store where scheduling models can be registered. We do this so we can provide better plugin support for scheduling a diverse set of devices.
 - |uncheck_| `Forecasting of sensors <https://github.com/FlexMeasures/flexmeasures/projects/8>`_: We are revising our forecasting tooling to support fixed-viewpoint forecasts. We do this so we can better support decision moments with the most recent expectations about relevant sensors.
-- |uncheck_| `Deprecate old database models <https://github.com/FlexMeasures/flexmeasures/projects/11>`_: We are deprecating the Power, Price and Weather tables in favour of the TimedBelief table, and deprecating the Asset, Market and WeatherSensor tables in favour of the Sensor and GeneralizedAsset tables. We are doing this to clean up the code and database structure.
 
 
-The state of the transition (March 2022, v0.9.0)
+The state of the transition (July 2023, v0.15.0)
 ---------------------------------------------------
 
 Project 9 was implemented with the release of v0.8.0. This work moved a lot of structure over, as well as actual data and some UI (dashboard, assets). We believe that was the hardest part.
 
-We are now working on deprecating the old database models (see project 11). As part of that move, we decided to begin the work on a new API version (v3) which supports only the new data model (and is more REST-like). That work was done in project 13. The new APIs for assets and sensor data had already been working before (at /api/dev) and had been powering what is shown in the UI since v0.8.0.
+In project 13, we began work on a new API version (v3) that supports only the new data model (and is more REST-like). The new APIs for assets and sensor data had already been working before (at /api/dev) and had been powering what is shown in the UI since v0.8.0.
 
 We also implemented many CLI commands which support the new model (project 14).
 
+We have deprecated and sunset all API versions before v3, while offering the ability for FlexMeasures hosts to organise blackout tests, and have removed the old database models (see project 11).
+
 We take care to support people on the old data model so the transition will be as smooth as possible, as we said above. One part of this is that the ``flexmeasures db upgrade`` command copies your data to the new model. Also, creating new data (e.g. old-style assets) creates new-style data (e.g. assets/sensors) automatically. However, some edge cases are not supported in this way. For instance, edited asset meta data might have to be re-entered later. Feel free to contact us to discuss the transition if needed.

documentation/host/modes.rst

@@ -16,8 +16,6 @@ In this mode, the server is assumed to be used as a demonstration tool. Most of
 - [UI] Logged-in users can view queues on the demo server (usually only admins can do that)
 - [UI] Demo servers often display login credentials, so visitors can try out functionality. Use the :ref:`demo-credentials-config` config setting to do this.
 - [UI] The dashboard shows all non-empty asset groups, instead of only the ones for the current user.
-- [UI] The analytics page mocks confidence intervals around power, price and weather data, so that the demo data doesn't need to have them. 
-- [UI] The portfolio page mocks flexibility numbers and a mocked control action.
 
 Play
 ------

documentation/index.rst

@@ -169,7 +169,6 @@ The platform operator of FlexMeasures can be an Aggregator.
     :maxdepth: 1
 
     concepts/benefits
-    concepts/benefits_of_flex
     concepts/inbuilt-smart-functionality
     concepts/algorithms
     concepts/security_auth

documentation/tut/forecasting_scheduling.rst

@@ -40,7 +40,7 @@ You can also clear the job queues:
 
 When the main FlexMeasures process runs (e.g. by ``flexmeasures run``\ ), the queues of forecasting and scheduling jobs can be visited at ``http://localhost:5000/tasks/forecasting`` and ``http://localhost:5000/tasks/schedules``\ , respectively (by admins).
 
-When forecasts and schedules have been generated, they should be visible at ``http://localhost:5000/analytics``.
+When forecasts and schedules have been generated, they should be visible at ``http://localhost:5000/assets/<id>``.
 
 
 .. note:: You can run workers who process jobs on different computers than the main server process. This can be a great architectural choice. Just keep in mind to use the same databases (postgres/redis) and to stick to the same FlexMeasures version on both.

documentation/views/dashboard.rst

@@ -24,8 +24,7 @@ Interactive map of assets
 =========================
 
 The map shows all of the user's assets with icons for each asset type.
-Clicking on an asset allows the user to see its current state (e.g. latest measurement of wind power production) and to navigate to the :ref:`analytics` page
-to see more details, for instance forecasts.
+Hovering over an asset allows users to see its name and ownership, and clicking on an asset allows the user to navigate to its page to see more details, for instance forecasts.
 
 
 .. _dashboard_summary:
@@ -34,7 +33,7 @@ Summary of asset types
 ======================
 
 The summary below the map lists all asset types that the user has hooked up to the platform and how many of each there are.
-Clicking on the asset type name leads to the :ref:`analytics` page, where data is shown aggregated for that asset type. 
+Clicking on the asset type name leads to the asset's page, where its data is shown.
 
 
 Grouping by accounts