Merge branch 'Issue-389_Publish_SensorDataAPI_documentation' into ass…

…et-api-to-v3

documentation/api/change_log.rst

@@ -6,7 +6,7 @@ API change log
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
 
 
-v3.0-0 | 2022-03-19
+v3.0-0 | 2022-03-22
 """""""""""""""""""
 
 - Added REST endpoint for listing sensors: `/sensors` (GET).
@@ -25,6 +25,13 @@ v3.0-0 | 2022-03-19
     - *postWeatherData* -> use `/sensors/data` (POST) instead
     - *restoreData*
 
+- Changed the Introduction section:
+
+    - Rewrote the section on service listing for API versions to refer to the public documentation.
+    - Rewrote the section on entity addresses to refer to *sensors* instead of *connections*.
+    - Rewrote the sections on roles and sources into a combined section that refers to account roles rather than USEF roles.
+    - Deprecated the section on group notation.
+
 v2.0-4 | 2022-01-04
 """""""""""""""""""
 

documentation/api/introduction.rst

@@ -41,7 +41,7 @@ Let's see what the ``/api`` endpoint returns:
     >>> res = requests.get("https://company.flexmeasures.io/api")
     >>> res.json()
     {'flexmeasures_version': '0.9.0',
-     'message': 'For these API versions a public endpoint is available, listing its service. For example: /api/v2_0/getService and /api/v3_0/getService. An authentication token can be requested at: /api/requestAuthToken',
+     'message': 'For these API versions endpoints are available. An authentication token can be requested at: /api/requestAuthToken. For a list of services, see https://flexmeasures.readthedocs.io',
      'status': 200,
      'versions': ['v1', 'v1_1', 'v1_2', 'v1_3', 'v2_0', 'v3_0']
     }
@@ -53,37 +53,7 @@ So this tells us which API versions exist. For instance, we know that the latest
     https://company.flexmeasures.io/api/v3_0
 
 
-Also, we can see that a list of endpoints which are available at (a version of) the FlexMeasures web service can be obtained by sending a ``getService`` request. An optional field "access" can be used to specify a user role for which to obtain only the relevant services.
-
-**Example request**
-
-Let's ask which endpoints are available for meter data companies (MDC):
-
-.. code-block:: html
-
-    https://company.flexmeasures.io/api/v2_0/getService?access=MDC
-
-
-**Example response**
-
-.. code-block:: json
-
-    {
-        "type": "GetServiceResponse",
-        "version": "1.0",
-        "services": [
-            {
-                "name": "getMeterData",
-                "access": ["Aggregator", "Supplier", "MDC", "DSO", "Prosumer", "ESCo"],
-                "description": "Request meter reading"
-            },
-            {
-                "name": "postMeterData",
-                "access": ["MDC"],
-                "description": "Send meter reading"
-            }
-        ]
-    }
+Also, we can see that a list of endpoints is available on https://flexmeasures.readthedocs.io for each of these versions.
 
 .. _api_auth:
 
@@ -131,28 +101,13 @@ which gives a response like this if the credentials are correct:
 .. note:: Each access token has a limited lifetime, see :ref:`auth`.
 
 
-Roles
------
-
-We distinguish the following roles with different access rights to the individual services. Capitalised roles are defined by USEF:
-
-- public
-- user
-- admin
-- Aggregator
-- Supplier: an energy retailer (see :ref:`supplier`)
-- Prosumer: owner of a grid connection (see :ref:`prosumer`)
-- ESCo: an energy service company (see :ref:`esco`)
-- MDC: a meter data company (see :ref:`mdc`)
-- DSO: a distribution system operator (see :ref:`dso`)
-
 .. _sources:
 
 Sources
 -------
 
 Requests for data may limit the data selection by specifying a source, for example, a specific user.
-USEF roles are also valid source selectors.
+Account roles are also valid source selectors.
 For example, to obtain data originating from either a meter data company or user 42, include the following:
 
 .. code-block:: json
@@ -161,6 +116,8 @@ For example, to obtain data originating from either a meter data company or user
         "sources": ["MDC", "42"],
     }
 
+Here, "MDC" is the name of the account role for meter data companies.
+
 Notation
 --------
 All requests and responses to and from the web service should be valid JSON messages.
@@ -179,27 +136,26 @@ Throughout this document, keys are written in singular if a single value is list
 
 The API, however, does not distinguish between singular and plural key notation.
 
-Connections and entity addresses
+Sensors and entity addresses
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-A connection represents an end point of the grid, at which an electricity sensor (power meter) is located.
-Connections should be identified with an entity address following the EA1 addressing scheme prescribed by USEF[1],
-which is mostly taken from IETF RFC 3720 [2]:
+All sensors are identified with an entity address following the EA1 addressing scheme prescribed by USEF[1],
+which is mostly taken from IETF RFC 3720 [2].
 
 This is the complete structure of an EA1 address:
 
 .. code-block:: json
 
     {
-        "connection": "ea1.{date code}.{reversed domain name}:{locally unique string}"
+        "sensor": "ea1.{date code}.{reversed domain name}:{locally unique string}"
     }
 
-Here is a full example for a FlexMeasures connection address: 
+Here is a full example for an entity address of a sensor in FlexMeasures:
 
 .. code-block:: json
 
     {
-        "connection": "ea1.2021-02.io.flexmeasures.company:fm1.73"
+        "sensor": "ea1.2021-02.io.flexmeasures.company:fm1.73"
     }
 
 where FlexMeasures runs at `company.flexmeasures.io` (which the current domain owner started using in February 2021), and the locally unique string uses the `fm1` scheme (see below) to identify sensor ID 73.
@@ -254,7 +210,7 @@ It uses the fact that all FlexMeasures sensors have unique IDs.
 .. todo:: UDI events are not yet modelled in the fm1 scheme
 
 The ``fm0`` scheme is the original scheme.
-It identified different types of sensors (such as connections, weather sensors and markets) in different ways.
+It identified different types of sensors (such as grid connections, weather sensors and markets) in different ways.
 The ``fm0`` scheme has been deprecated for the most part and is no longer supported officially.
 Only UDI events still need to be sent using the fm0 scheme.
 
@@ -264,69 +220,6 @@ Only UDI events still need to be sent using the fm0 scheme.
     ea1.2021-01.io.flexmeasures:fm0.<owner_id>:<sensor_id>:<event_id>:<event_type>
 
 
-Groups
-^^^^^^
-
-Data such as measurements, load prognoses and tariffs are usually stated per group of connections.
-When the attributes "start", "duration" and "unit" are stated outside of "groups" they are inherited by each of the individual groups. For example:
-
-.. code-block:: json
-
-    {
-        "groups": [
-            {
-                "connections": [
-                    "ea1.2021-02.io.flexmeasures.company:fm1.71",
-                    "ea1.2021-02.io.flexmeasures.company:fm1.72"
-                ],
-                "values": [
-                    306.66,
-                    306.66,
-                    0,
-                    0,
-                    306.66,
-                    306.66
-                ]
-            },
-            {
-                "connection": "ea1.2021-02.io.flexmeasures.company:fm1.73"
-                "values": [
-                    306.66,
-                    0,
-                    0,
-                    0,
-                    306.66,
-                    306.66
-                ]
-            }
-        ],
-        "start": "2016-05-01T12:45:00Z",
-        "duration": "PT1H30M",
-        "unit": "MW"
-    }
-
-In case of a single group of connections, the message may be flattened to:
-
-.. code-block:: json
-
-    {
-        "connections": [
-            "ea1.2021-02.io.flexmeasures.company:fm1.71",
-            "ea1.2021-02.io.flexmeasures.company:fm1.72"
-        ],
-        "values": [
-            306.66,
-            306.66,
-            0,
-            0,
-            306.66,
-            306.66
-        ],
-        "start": "2016-05-01T12:45:00Z",
-        "duration": "PT1H30M",
-        "unit": "MW"
-    }
-
 Timeseries
 ^^^^^^^^^^
 

flexmeasures/api/dev/tests/conftest.py

@@ -1,12 +1,16 @@
 import pytest
 
+from flexmeasures.api.v3_0.tests.conftest import add_gas_sensor
+from flexmeasures.data.models.time_series import Sensor
+
 
 @pytest.fixture(scope="module")
 def setup_api_test_data(db, setup_roles_users, setup_generic_assets):
     """
     Set up data for API dev tests.
     """
     print("Setting up data for API dev tests on %s" % db.engine)
+    add_gas_sensor(db, setup_roles_users["Test Supplier User"])
 
 
 @pytest.fixture(scope="function")
@@ -17,3 +21,6 @@ def setup_api_fresh_test_data(
     Set up fresh data for API dev tests.
     """
     print("Setting up fresh data for API dev tests on %s" % fresh_db.engine)
+    for sensor in Sensor.query.all():
+        fresh_db.delete(sensor)
+    add_gas_sensor(fresh_db, setup_roles_users_fresh_db["Test Supplier User"])

flexmeasures/api/v3_0/sensors.py

@@ -17,6 +17,11 @@
 from flexmeasures.data.schemas.sensors import SensorSchema
 from flexmeasures.data.services.sensors import get_sensors
 
+# Instantiate schemas outside of endpoint logic to minimize response time
+get_sensor_schema = GetSensorDataSchema()
+post_sensor_schema = PostSensorDataSchema()
+sensors_schema = SensorSchema(many=True)
+
 
 class SensorAPI(FlaskView):
 
@@ -68,13 +73,14 @@ def index(self, account: Account):
         :status 400: INVALID_REQUEST
         :status 401: UNAUTHORIZED
         :status 403: INVALID_SENDER
+        :status 422: UNPROCESSABLE_ENTITY
         """
         sensors = get_sensors(account_name=account.name)
-        return SensorSchema(many=True).dump(sensors), 200
+        return sensors_schema.dump(sensors), 200
 
     @route("/data", methods=["POST"])
     @use_args(
-        PostSensorDataSchema(),
+        post_sensor_schema,
         location="json",
     )
     def post_data(self, bdf: BeliefsDataFrame):
@@ -102,13 +108,22 @@ def post_data(self, bdf: BeliefsDataFrame):
         The unit has to be convertible to the sensor's unit.
         The resolution of the data has to match the sensor's required resolution, but
         FlexMeasures will attempt to upsample lower resolutions.
+
+        :reqheader Authorization: The authentication token
+        :reqheader Content-Type: application/json
+        :resheader Content-Type: application/json
+        :status 200: PROCESSED
+        :status 400: INVALID_REQUEST
+        :status 401: UNAUTHORIZED
+        :status 403: INVALID_SENDER
+        :status 422: UNPROCESSABLE_ENTITY
         """
         response, code = save_and_enqueue(bdf)
         return response, code
 
     @route("/data", methods=["GET"])
     @use_args(
-        GetSensorDataSchema(),
+        get_sensor_schema,
         location="query",
     )
     def get_data(self, response: dict):
@@ -128,5 +143,14 @@ def get_data(self, response: dict):
             }
 
         The unit has to be convertible from the sensor's unit.
+
+        :reqheader Authorization: The authentication token
+        :reqheader Content-Type: application/json
+        :resheader Content-Type: application/json
+        :status 200: PROCESSED
+        :status 400: INVALID_REQUEST
+        :status 401: UNAUTHORIZED
+        :status 403: INVALID_SENDER
+        :status 422: UNPROCESSABLE_ENTITY
         """
         return json.dumps(response)

flexmeasures/api/v3_0/users.py

@@ -24,6 +24,11 @@
 Both POST (to create) and DELETE are not accessible via the API, but as CLI functions.
 """
 
+# Instantiate schemas outside of endpoint logic to minimize response time
+user_schema = UserSchema()
+users_schema = UserSchema(many=True)
+partial_user_schema = UserSchema(partial=True)
+
 
 class UserAPI(FlaskView):
     route_base = "/users"
@@ -78,9 +83,10 @@ def index(self, account: Account, include_inactive: bool = False):
         :status 400: INVALID_REQUEST
         :status 401: UNAUTHORIZED
         :status 403: INVALID_SENDER
+        :status 422: UNPROCESSABLE_ENTITY
         """
         users = get_users(account_name=account.name, only_active=not include_inactive)
-        return UserSchema(many=True).dump(users), 200
+        return users_schema.dump(users), 200
 
     @route("/<id>")
     @use_kwargs({"user": UserIdField(data_key="id")}, location="path")
@@ -115,11 +121,12 @@ def get(self, id: int, user: UserModel):
         :status 400: INVALID_REQUEST, REQUIRED_INFO_MISSING, UNEXPECTED_PARAMS
         :status 401: UNAUTHORIZED
         :status 403: INVALID_SENDER
+        :status 422: UNPROCESSABLE_ENTITY
         """
-        return UserSchema().dump(user), 200
+        return user_schema.dump(user), 200
 
     @route("/<id>", methods=["PATCH"])
-    @use_kwargs(UserSchema(partial=True))
+    @use_kwargs(partial_user_schema)
     @use_kwargs({"user": UserIdField(data_key="id")}, location="path")
     @permission_required_for_context("update", arg_name="user")
     @as_json
@@ -191,7 +198,7 @@ def patch(self, id: int, user: UserModel, **user_data):
                 dict(message="Duplicate user already exists", detail=ie._message()),
                 400,
             )
-        return UserSchema().dump(user), 200
+        return user_schema.dump(user), 200
 
     @route("/<id>/password-reset", methods=["PATCH"])
     @use_kwargs({"user": UserIdField(data_key="id")}, location="path")
@@ -216,6 +223,7 @@ def reset_user_password(self, id: int, user: UserModel):
         :status 400: INVALID_REQUEST, REQUIRED_INFO_MISSING, UNEXPECTED_PARAMS
         :status 401: UNAUTHORIZED
         :status 403: INVALID_SENDER
+        :status 422: UNPROCESSABLE_ENTITY
         """
         set_random_password(user)
         remove_cookie_and_token_access(user)