Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 389 publish sensor data api documentation #390

Merged
merged 99 commits into from Mar 22, 2022
Merged
Show file tree
Hide file tree
Changes from 91 commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
f6f1b91
Move sensor data API from dev to v2 and use flask-classful
Flix6x Mar 7, 2022
f92d182
Publish dev API
Flix6x Mar 8, 2022
84f4169
Add quickrefs
Flix6x Mar 8, 2022
9eb7299
Add type annotations
Flix6x Mar 8, 2022
ea7a77d
Refactor use of auth_required in SensorAPI
Flix6x Mar 8, 2022
9d2f329
Add type annotations
Flix6x Mar 8, 2022
25ba2de
Add more quickrefs
Flix6x Mar 8, 2022
0126169
Indentation
Flix6x Mar 8, 2022
7ce3f14
valid JSON, rather than a python dict
Flix6x Mar 8, 2022
cb205ba
Move import to its preferred position
Flix6x Mar 8, 2022
841f1d3
Rename and add type annotation
Flix6x Mar 8, 2022
f126561
Allow AssetIdField and SensorIdField to be used for API requests, too…
Flix6x Mar 8, 2022
2e3bf11
Update tutorial for posting data
Flix6x Mar 8, 2022
e6db62f
Implement [GET] sensorData
Flix6x Mar 8, 2022
de46436
Add example GET request
Flix6x Mar 8, 2022
2ab771b
Update docstring for unit conversion
Flix6x Mar 8, 2022
1a7414e
Respect the passed horizon and prior fields
Flix6x Mar 8, 2022
70e3793
Convert NaN to null (otherwise invalid JSON)
Flix6x Mar 8, 2022
2ef234f
Pass event resolution to enable additional unit conversions
Flix6x Mar 8, 2022
bc5ffbe
Smarter unit conversion for BeliefsSeries
Flix6x Mar 8, 2022
3e85a5e
Revert to auth_token_required instead of login_required
Flix6x Mar 8, 2022
5738a29
Deserialize to BeliefsDataFrame using post_load decorator, and run th…
Flix6x Mar 8, 2022
9cd8f9b
Move serialization logic to schema
Flix6x Mar 8, 2022
c76766e
Fix tutorial for posting data
Flix6x Mar 8, 2022
271c435
Introduce util function to check for a valid price unit
Flix6x Mar 8, 2022
56a5f25
Comment out tutorial sections on posting data for multiple connection…
Flix6x Mar 8, 2022
8a14c64
Update tutorial for posting data: change connection to sensor and sto…
Flix6x Mar 8, 2022
e878f5c
Add validation based on POST message type
Flix6x Mar 8, 2022
dfa64b8
Publish documentation for SensorDataAPI
Flix6x Mar 8, 2022
7de8b6f
Add validation based on GET message type
Flix6x Mar 8, 2022
c0f0bee
mypy
Flix6x Mar 8, 2022
29d3381
Update getting-started.rst
Flix6x Mar 8, 2022
852bee8
Undoc old endpoints
Flix6x Mar 8, 2022
eef5d42
Rename with_appcontext_if_needed
Flix6x Mar 10, 2022
83b774e
refactor auth logic - separate checking access from the permission_re…
nhoening Mar 10, 2022
52b3ba9
add modern auth for sensor data get & post, remove checking account r…
nhoening Mar 12, 2022
9c0695e
more freedom to create inside accounts: allow all account members to …
nhoening Mar 12, 2022
bc9a100
use a user from the same account as the sensor for sensor data tests
nhoening Mar 12, 2022
2f31a17
adapt asset tests w.r.t. to a change in dev api conftest
nhoening Mar 12, 2022
352ae60
Remove unneeded import
Flix6x Mar 11, 2022
210ecbc
GET endpoints work for logged-in users when tried in the browser with…
Flix6x Mar 11, 2022
b76ebdb
Add docstring
Flix6x Mar 11, 2022
9b3f281
Make type a required field
Flix6x Mar 11, 2022
7b67a1f
Move @route to top
Flix6x Mar 14, 2022
2037394
SensorDataDescriptionSchema doesn't need the `type` field
Flix6x Mar 14, 2022
b17a9b8
Revert "Undoc old endpoints"
Flix6x Mar 14, 2022
4e804ba
Move sensor data API documentation to v3
Flix6x Mar 14, 2022
24ffc1a
Move sensor data API to v3
Flix6x Mar 14, 2022
77ffdaf
Move user API to class for v3
Flix6x Mar 14, 2022
daa43c6
Move user UI to v3
Flix6x Mar 14, 2022
3abb649
Update user API tests
Flix6x Mar 14, 2022
c4ea13b
Move user API tests to v3 module
Flix6x Mar 14, 2022
41371c7
Separate view for user index
Flix6x Mar 14, 2022
d2a2e5f
Fix test
Flix6x Mar 14, 2022
0c2d199
Update documentation tutorials to v3
Flix6x Mar 14, 2022
482cde9
No trailing slash
Flix6x Mar 14, 2022
820521e
Make `type` field in v3
Flix6x Mar 14, 2022
0fc14b6
Add v3 to API index
Flix6x Mar 14, 2022
70a9406
Update introduction
Flix6x Mar 14, 2022
281af3b
Fix comment about where to request auth token
Flix6x Mar 14, 2022
d14d85e
flake8
Flix6x Mar 14, 2022
89e8dff
black
Flix6x Mar 14, 2022
2f2cee0
Revert some documentation changes that are ahead of code changes
Flix6x Mar 14, 2022
abfc41a
Use one common base route
Flix6x Mar 14, 2022
3fec582
Prefer plural base route
Flix6x Mar 14, 2022
80ed32d
different shade of black
Flix6x Mar 14, 2022
a26fad1
typos
Flix6x Mar 15, 2022
6f34010
Transition SensorDataAPI:post to SensorAPI:post_data and SensorDataAP…
Flix6x Mar 15, 2022
0f2a8f0
Rename module
Flix6x Mar 15, 2022
bf277b0
Update documentation for moving from SensorDataAPI:post to SensorAPI:…
Flix6x Mar 15, 2022
dcb46f3
Update introduction
Flix6x Mar 15, 2022
d392423
API changelog entry
Flix6x Mar 15, 2022
64e1100
Implement /sensors [GET] to replace getConnection
Flix6x Mar 15, 2022
c06d768
Complete docstring
Flix6x Mar 15, 2022
257b5b5
Add extra tips to API changelog on how to upgrade from v2 to v3
Flix6x Mar 15, 2022
b889992
Fix check for belief timing against message type
Flix6x Mar 15, 2022
6037d60
No need to use implementations folder anymore
Flix6x Mar 16, 2022
16130a1
Contain endpoint logic within SensorAPI class
Flix6x Mar 16, 2022
1b6394f
Contain endpoint logic within UserAPI class
Flix6x Mar 16, 2022
c16a3bf
Fix mock email and type annotations
Flix6x Mar 17, 2022
d4e99db
Initialize schemas within SensorAPI and UserAPI class
Flix6x Mar 17, 2022
cdb7704
Typos
Flix6x Mar 17, 2022
4c982bb
Fix 404 error message
Flix6x Mar 17, 2022
ea1e1b6
Remove unused variable declaration
Flix6x Mar 17, 2022
38d3bc1
Clarify use of headers in test
Flix6x Mar 17, 2022
9d98f71
Concerning patching user ids and user account ids, set dump_only for …
Flix6x Mar 17, 2022
ca3154f
Add test cases for unexpected fields
Flix6x Mar 18, 2022
71add34
Also move patch endpoint logic into UserAPI class; this one was a bit…
Flix6x Mar 18, 2022
1f07045
Move test utils from dev to v3_0
Flix6x Mar 18, 2022
d8b2790
Finish moving sensor data tests from dev to v3_0
Flix6x Mar 18, 2022
0589009
Fix asset tests in dev API by moving back some conftest logic
Flix6x Mar 18, 2022
3743032
Instantiate schemas outside of endpoint logic to minimize response time
Flix6x Mar 22, 2022
bf10bf3
Add mentions of http status to docstrings
Flix6x Mar 22, 2022
f159094
Undocument getService from the API introduction
Flix6x Mar 22, 2022
8572b8f
Undocument USEF roles in introduction
Flix6x Mar 22, 2022
1c013e9
Use "sensor" instead of "connection" in introduction
Flix6x Mar 22, 2022
dd7c59e
Undocument group notation in introduction
Flix6x Mar 22, 2022
c808fe3
Update API changelog for rewrite of introduction
Flix6x Mar 22, 2022
0358dfa
Update API changelog entry date
Flix6x Mar 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
26 changes: 22 additions & 4 deletions documentation/api/change_log.rst
Expand Up @@ -6,6 +6,24 @@ API change log
.. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.


v3.0-0 | 2022-03-15
"""""""""""""""""""

- Added REST endpoint for listing sensors: `/sensors` (GET).
- Added REST endpoint for managing sensor data: `/sensors/data` (GET, POST).
- [**Breaking change**] Switched to plural resource names for REST endpoints: `/users/<id>` (GET, PATCH) and `/users/<id>/password-reset` (PATCH).
- [**Breaking change**] Deprecated the following endpoints:

- *getConnection* -> use `/sensors` (GET) instead
- *getMeterData* -> use `/sensors/data` (GET) instead, replacing the "connection" field with "sensor"
- *getPrognosis* -> use `/sensors/data` (GET) instead, replacing the "connection" field with "sensor"
- *getService*
- *postMeterData* -> use `/sensors/data` (POST) instead, replacing the "connection" field with "sensor"
- *postPriceData* -> use `/sensors/data` (POST) instead, replacing the "market" field with "sensor"
- *postPrognosis* -> use `/sensors/data` (POST) instead, replacing the "connection" field with "sensor"
- *postWeatherData* -> use `/sensors/data` (POST) instead
- *restoreData*

v2.0-4 | 2022-01-04
"""""""""""""""""""

Expand Down Expand Up @@ -42,12 +60,12 @@ v2.0-2 | 2021-04-02
v2.0-1 | 2021-02-19
"""""""""""""""""""

- REST endpoints for managing users: `/users/` (GET), `/user/<id>` (GET, PATCH) and `/user/<id>/password-reset` (PATCH).
- Added REST endpoints for managing users: `/users/` (GET), `/user/<id>` (GET, PATCH) and `/user/<id>/password-reset` (PATCH).

v2.0-0 | 2020-11-14
"""""""""""""""""""

- REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).
- Added REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).


v1.3-11 | 2022-01-05
Expand Down Expand Up @@ -80,7 +98,7 @@ v1.3-8 | 2020-04-02

*Affects all versions since v1.0*.

- [**Breaking change**, partially reverted in v1.3-9] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
- [**Breaking change**, partially reverted in v1.3-9] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API versions below v2.0.

v1.3-7 | 2020-12-16
"""""""""""""""""""
Expand All @@ -107,7 +125,7 @@ v1.3-5 | 2020-10-29
- Endpoints to POST meter data will now check incoming data to see if the required asset's resolution is being used ― upsampling is done if possible.
These endpoints can now return the REQUIRED_INFO_MISSING status 400 response.
- Endpoints to GET meter data will return data in the asset's resolution ― downsampling to the "resolution" field is done if possible.
- As they need to determine the asset, all of the mentioned POST and GET endpoints can now return the UNRECOGNIZED_ASSET status 4000 response.
- As they need to determine the asset, all of the mentioned POST and GET endpoints can now return the UNRECOGNIZED_ASSET status 400 response.

v1.3-4 | 2020-06-18
"""""""""""""""""""
Expand Down
22 changes: 22 additions & 0 deletions documentation/api/dev.rst
@@ -0,0 +1,22 @@
.. _dev:

Developer API
=============

These endpoints are still under development and are subject to change in new releases.

Summary
-------

.. qrefflask:: flexmeasures.app:create(env="documentation")
:modules: flexmeasures.api.dev.assets, flexmeasures.api.dev.sensors
:order: path
:include-empty-docstring:

API Details
-----------

.. autoflask:: flexmeasures.app:create(env="documentation")
:modules: flexmeasures.api.dev.assets, flexmeasures.api.dev.sensors
:order: path
:include-empty-docstring:
32 changes: 19 additions & 13 deletions documentation/api/introduction.rst
Expand Up @@ -23,13 +23,13 @@ So if you are running FlexMeasures on your computer, it would be:

https://localhost:5000/api

At Seita, we run servers for our clients at:
Let's assume we are running a server for a client at:

.. code-block:: html

https://company.flexmeasures.io/api

where `company` is a hosting customer of ours. All their accounts' data lives on that server.
where `company` is a client of ours. All their accounts' data lives on that server.

We assume in this document that the FlexMeasures instance you want to connect to is hosted at https://company.flexmeasures.io.

Expand All @@ -40,17 +40,17 @@ Let's see what the ``/api`` endpoint returns:
>>> import requests
>>> res = requests.get("https://company.flexmeasures.io/api")
>>> res.json()
{'flexmeasures_version': '0.4.0',
'message': 'For these API versions a public endpoint is available, listing its service. For example: /api/v1/getService and /api/v1_1/getService. An authentication token can be requested at: /api/requestAuthToken',
{'flexmeasures_version': '0.9.0',
'message': 'For these API versions a public endpoint is available, listing its service. For example: /api/v2_0/getService and /api/v3_0/getService. An authentication token can be requested at: /api/requestAuthToken',
'status': 200,
'versions': ['v1', 'v1_1', 'v1_2', 'v1_3', 'v2_0']
'versions': ['v1', 'v1_1', 'v1_2', 'v1_3', 'v2_0', 'v3_0']
}

So this tells us which API versions exist. For instance, we know that the latest API version is available at
So this tells us which API versions exist. For instance, we know that the latest API version is available at:

.. code-block:: html

https://company.flexmeasures.io/api/v2_0
https://company.flexmeasures.io/api/v3_0


Also, we can see that a list of endpoints which are available at (a version of) the FlexMeasures web service can be obtained by sending a ``getService`` request. An optional field "access" can be used to specify a user role for which to obtain only the relevant services.
Flix6x marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -374,10 +374,10 @@ Technically, this is equal to:

This intuitive convention allows us to reduce communication by sending univariate timeseries as arrays.

Notation for v1
"""""""""""""""
Notation for v1, v2 and v3
""""""""""""""""""""""""""

For version 1 and 2 of the API, only equidistant timeseries data is expected to be communicated. Therefore:
For version 1, 2 and 3 of the API, only equidistant timeseries data is expected to be communicated. Therefore:

- only the array notation should be used (first notation from above),
- "start" should be a timestamp on the hour or a multiple of the sensor resolution thereafter (e.g. "16:10" works if the resolution is 5 minutes), and
Expand Down Expand Up @@ -496,19 +496,25 @@ Resolutions

Specifying a resolution is redundant for POST requests that contain both "values" and a "duration" ― FlexMeasures computes the resolution by dividing the duration by the number of values.

When POSTing data, FlexMeasures checks this computed resolution against the required resolution of the assets which are posted to. If these can't be matched (through upsampling), an error will occur.
When POSTing data, FlexMeasures checks this computed resolution against the required resolution of the sensors which are posted to. If these can't be matched (through upsampling), an error will occur.

GET requests (such as *getMeterData*) return data in the resolution which the sensor is configured for.
A "resolution" may be specified explicitly to obtain the data in downsampled form,
which can be very beneficial for download speed. The specified resolution needs to be a multiple
of the asset's resolution, e.g. hourly or daily values if the asset's resolution is 15 minutes.
of the sensor's resolution, e.g. hourly or daily values if the sensor's resolution is 15 minutes.

.. _units:

Units
^^^^^

Valid units for timeseries data in version 1 of the API are "MW" only.
From API version 3 onwards, we are much more flexible with sent units.
A valid unit for timeseries data is any unit that is convertible to the configured sensor unit registered in FlexMeasures.
So, for example, you can send timeseries data with "W" unit to a "kW" sensor.
And if you wish to do so, you can even send a timeseries with "kWh" unit to a "kW" sensor.
In this case, FlexMeasures will convert the data using the resolution of the timeseries.

For API versions 1 and 2, the unit sent needs to be an exact match with the sensor unit, and only "MW" is allowed for power sensors.

.. _signs:

Expand Down
20 changes: 20 additions & 0 deletions documentation/api/v3_0.rst
@@ -0,0 +1,20 @@
.. _v3_0:

Version 3.0
===========

Summary
-------

.. qrefflask:: flexmeasures.app:create(env="documentation")
:modules: flexmeasures.api.v3_0.implementations.sensors, flexmeasures.api.v3_0.implementations.users
:order: path
:include-empty-docstring:

API Details
-----------

.. autoflask:: flexmeasures.app:create(env="documentation")
:modules: flexmeasures.api.v3_0.implementations.sensors, flexmeasures.api.v3_0.implementations.users
:order: path
:include-empty-docstring:
2 changes: 1 addition & 1 deletion documentation/getting-started.rst
Expand Up @@ -195,7 +195,7 @@ First, you can load in data from a file (CSV or Excel) via the ``flexmeasures``

This assumes you have a file `my-data.csv` with measurements, which was exported from some legacy database, and that the data is about our sensor with ID 1. This command has many options, so do use its ``--help`` function.

Second, you can use the `POST /api/v2_0/postMeterData <api/v2_0.html#post--api-v2_0-postMeterData>`_ endpoint in the FlexMeasures API to send meter data.
Second, you can use the `POST /api/v3_0/sensors/data <api/v3_0.html#post--api-v3_0-sensors-data>`_ endpoint in the FlexMeasures API to send meter data.

Finally, you can tell FlexMeasures to create forecasts for your meter data with the ``flexmeasures add forecasts`` command, here is an example:

Expand Down
2 changes: 2 additions & 0 deletions documentation/index.rst
Expand Up @@ -128,11 +128,13 @@ The platform operator of FlexMeasures can be an Aggregator.
:maxdepth: 1

api/introduction
api/v3_0
api/v2_0
api/v1_3
api/v1_2
api/v1_1
api/v1
api/dev
api/change_log

.. toctree::
Expand Down
2 changes: 1 addition & 1 deletion documentation/tut/building_uis.rst
Expand Up @@ -20,7 +20,7 @@ This tutorial will show how the FlexMeasures API can be used from JavaScript to
Get an authentication token
-----------------------

FlexMeasures provides the `POST /api/v2_0/requestAuthToken <../api/v2_0.html#post--api-v2_0-requestAuthToken>`_ endpoint, as discussed in :ref:`api_auth`.
FlexMeasures provides the `POST /api/requestAuthToken <../api/v2_0.html#post--api-v2_0-requestAuthToken>`_ endpoint, as discussed in :ref:`api_auth`.
Here is a JavaScript function to call it:

.. code-block:: JavaScript
Expand Down
2 changes: 1 addition & 1 deletion documentation/tut/forecasting_scheduling.rst
Expand Up @@ -116,7 +116,7 @@ Getting power forecasts (prognoses)

Prognoses (the USEF term used for power forecasts) are used by FlexMeasures to determine the best control signals to valorise on balancing opportunities.

You can access forecasts via the FlexMeasures API at `GET /api/v2_0/getPrognosis <../api/v2_0.html#get--api-v2_0-getPrognosis>`_.
You can access forecasts via the FlexMeasures API at `GET /api/v2_0/getPrognosis <../api/v2_0.html#get--api-v2_0-getPrognosis>`_.
Getting them might be useful if you want to use prognoses in your own system, or to check their accuracy against meter data, i.e. the realised power measurements.
The FlexMeasures UI also lists forecast accuracy, and visualises prognoses and meter data next to each other.

Expand Down