ACL-based auth policy and the @permission_required_for_context decora…

…tor. For now used with /users API (#234)

* first implementation of ACL-based auth policy and the @permission_required_for_context decorator. Used for one /user endpoint (GET /user/<id>)

* make GET /users and PATCH /user/<id> work. Inn the process simplifying /users to work for one account only, and moving API context factories to a common location

* resolve circular import

* always log the authorization failure for debugging purposes

* adapt UI crud so it lists users from all accounts again after API simplification

* check for valid permissions and allow the ADMIN_READER role to read

* separate checking admin access and principals within policy, add tests for principals matching

* changelog entry

* fix test

* Add test for multiple roles and multiple account roles

* smaller haul from review

* replace deprecated marshmallow parameters

* explicitly require marshmallow>3 as we use recent parameters (app.txt already pins it >3)

* replace load_account with Marshmallow field; enable the permission-checking decorator to get the resource from keyword args

* move load_user factory to marshmallow field, as well; refactor permission decorator a bit

* update documentation

* permissions: use update over write (-> CRUD)

* use @auth_required in place of @login_required in non-user-facing endpoints

* doc improvements from review

* nicer way to state conditions for account roles decorators

* Typo

* Auth decorators raise 403/401, simplify invalid_sender

* Add type annotation

* nicer error messages in auth decorators

Co-authored-by: F.N. Claessen <felix@seita.nl>

documentation/changelog.rst

@@ -15,6 +15,7 @@ Bugfixes
 Infrastructure / Support
 ----------------------
 * Account-based authorization, incl. new decorators for endpoints [see `PR #210 <http://www.github.com/SeitaBV/flexmeasures/pull/210>`_]
+* Central authorization policy which lets database models codify who can do what (permission-based) and relieve API endpoints from this [see `PR #234 <http://www.github.com/SeitaBV/flexmeasures/pull/234>`_]
 * Improve data specification for forecasting models using timely-beliefs data [see `PR #154 <http://www.github.com/SeitaBV/flexmeasures/pull/154>`_]
 * Allow plugins to register their custom config settings, so that FlexMeasures can check whether they are set up correctly [see `PR #230 <http://www.github.com/SeitaBV/flexmeasures/pull/230>`_ and `PR #237 <http://www.github.com/SeitaBV/flexmeasures/pull/237>`_]
 * Added sensor method to obtain just its latest state (excl. forecasts) [see `PR #235 <http://www.github.com/SeitaBV/flexmeasures/pull/235>`_]

documentation/concepts/security_auth.rst

@@ -15,7 +15,7 @@ There are two types of data on FlexMeasures servers - files (e.g. source code, i
 * Finally, The application communicates all data with HTTPS, the Hypertext Transfer Protocol encrypted by Transport Layer Security. This is used even if the application is accessed via ``http://``.
 
 
-.. _auth:
+.. _authentication:
 
 Authentication 
 ----------------
@@ -30,19 +30,24 @@ This involves a username/password combination ("credentials") or an access token
 .. note:: Authentication (and authorization, see below) affects the FlexMeasures API and UI. The CLI (command line interface) can only be used if the user is already on the server and can execute ``flexmeasures`` commands, thus we can safely assume they are admins.
 
 
+.. _authorization:
+
 Authorization
 --------------
 
-*Authorization* is the system by which the FlexMeasures platform decides whether an authenticated user can access a feature. For instance, many features are reserved for administrators, others for users belonging to certain accounts. An example for the latter is that a user might need to belong to an account with the "Prosumer" account role (usually the owner of assets).
+*Authorization* is the system by which the FlexMeasures platform decides whether an authenticated user can access data. Data about users and assets. Or metering data, forecasts and schedules.
+
+For instance, a user is authorized to update his or her personal data, like the surname. Other users should not be authorized to do that. We can also authorize users to do something because they belong to a certain account. An example for this is to read the meter data of the account's assets. Any regular user should *only* be able to read data that their account should be able to see.
 
 .. note:: Each user belongs to exactly one account.
 
-.. todo:: Data which belongs to a specific account should only be viewable by users within that account (and platform admins). We want to anchor this crucial security measure on a deep level, `see this ticket <https://github.com/SeitaBV/flexmeasures/issues/201>`_.
+In a nutshell, the way FlexMeasures implements authorization works as follows: The data models codify under which conditions a user can have certain permissions to work with their data. Permissions allow distinct ways of access like reading, writing or deleting. The API endpoints are where we know what needs to happen to what data, so there we make sure that the user has the necessary permissions.
 
-All other authorization is achieved via *roles*. 
+We already discussed certain conditions under which a user has access to data ― being a certain user or belonging to a specific account. Furthermore, authorization conditions can also be implemented via *roles*: 
 
-* Account roles are most commonly used for deciding who can access a resource (usually guarded by an API endpoint). We support several roles which are mentioned in the USEF framework but more roles are possible (e.g. defined by customer services, see below). 
-* User roles give a user personal authorizations. For instance, we have a few `admin`\ s who can perform all actions, and `admin-reader`\ s who can read everything. Other roles have only an effect within the user's account.
+* ``Account roles`` are often used for authorization. We support several roles which are mentioned in the USEF framework but more roles are possible (e.g. defined by custom-made services, see below). For example, a user might be authorized to write sensor data if they belong to an account with the "MDC" account role ("MDC" being short for meter data company).
+* ``User roles`` give a user personal authorizations. For instance, we have a few `admin`\ s who can perform all actions, and `admin-reader`\ s who can read everything. Other roles have only an effect within the user's account, e.g. there could be an "HR" role which allows to edit user data like surnames within the account.
 * Roles cannot be edited via the UI at the moment. They are decided when a user or account is created in the CLI (for adding roles later, we use the database for now). Editing roles in UI and CLI is future work.
 
-.. note:: Custom energy flexibility services developed on top of FlexMeasures can use account roles to achieve their custom authorization. E.g. if several services run on one FlexMeasures server, each service could define a "MyService-subscriber" account role, to make sure that only users of such accounts can use the endpoints. More on this in :ref:`auth-dev`.
+
+.. note:: Custom energy flexibility services developed on top of FlexMeasures also need to implement authorization. More on this in :ref:`auth-dev`. Here is an example for a custom authorization concept: services can use account roles to achieve their custom authorization. E.g. if several services run on one FlexMeasures server, each service could define a "MyService-subscriber" account role, to make sure that only users of such accounts can use the endpoints.

documentation/dev/auth.rst

@@ -3,9 +3,47 @@
 Custom authorization
 ======================
 
-Our :ref:`auth` section describes general authentication and authorization handling in FlexMeasures. However, custom energy flexibility services developed on top of FlexMeasures probably also need their custom authorization. 
+Our :ref:`authorization` section describes general authorization handling in FlexMeasures.
 
-One means for this is to define custom account roles. E.g. if several services run on one FlexMeasures server, each service could define a "MyService-subscriber" account role. To make sure that only users of such accounts can use the endpoints:
+If you are creating your own API endpoints for a custom energy flexibility service (on top of FlexMeasures), you should also get your authorization right. 
+It's recommended to get familiar with the decorators we provide. Here are some pointers, but feel free to read more in the ``flexmeasures.auth`` package. 
+
+In short, we recommend to use the ``@permission_required_for_context`` decorator (more explanation below).
+
+FlexMeasures also supports role-based decorators, e.g. ``@account_roles_required``. These authorization decorators are more straightforward to use than the  ``@permission_required_for_context`` decorator. However, they are a bit crude as they do not distinguish on what the context is, nor do they qualify on the required permission(e.g. read versus write). [#f1]_
+
+Finally, all decorators available through `Flask-Security-Too <https://flask-security-too.readthedocs.io/en/stable/patterns.html#authentication-and-authorization>`_ can be used, e.g. ``@auth_required`` (that's technically only checking authentication) or ``@permissions_required``.
+
+
+Permission-based authorization
+--------------------------------
+
+Via permissions, it's possible to define authorization access to data, distinguishing between create, read, update and delete access. It's a finer model than simply allowing per role.
+
+The data models codify under which conditions a user can have certain permissions to work with their data.
+You, as the endpoint author, need to make sure this is checked. Here is an example (taken from the decorator docstring):
+
+.. code-block:: python
+
+    @app.route("/resource/<resource_id>", methods=["GET"])
+    @use_kwargs(
+        {"the_resource": ResourceIdField(data_key="resource_id")},
+        location="path",
+    )
+    @permission_required_for_context("read", arg_name="the_resource")
+    @as_json
+    def view(resource_id: int, resource: Resource):
+        return dict(name=resource.name)
+
+As you see, there is some sorcery with ``@use_kwargs`` going on before we check the permissions. `That decorator <https://webargs.readthedocs.io>`_ is relaying to a `Marshmallow <https://marshmallow.readthedocs.io/>`_ field definition. Here, ``ResourceIdField`` is a definition which de-serializes an ID (passed in as a request parameter) into a ``Resource`` instance. This instance can then be asked if the current user may read it. That last part is what ``@permission_required_for_context`` is doing. You can find these Marshmallow fields in ``flexmeasures.api.common.schemas``. 
+
+
+Account roles
+---------------
+
+Another way to implement custom authorization is to define custom account roles. E.g. if several services run on one FlexMeasures server, each service could define a "MyService-subscriber" account role. 
+
+To make sure that only users of such accounts can use the endpoints:
 
 .. code-block:: python
 
@@ -14,9 +52,13 @@ One means for this is to define custom account roles. E.g. if several services r
     def bananas_view:
         pass
 
-.. note:: This endpoint decorator lists required roles, so the authenticated user's account needs to have each role. You can also use the ``account_roles_accepted`` decorator. Then the user's account only needs to have at least one of the roles.
+.. note:: This endpoint decorator lists required roles, so the authenticated user's account needs to have each role. You can also use the ``@account_roles_accepted`` decorator. Then the user's account only needs to have at least one of the roles.
 
-There are also decorators to check user roles:
+
+User roles
+---------------
+
+There are also decorators to check user roles. Here is an example:
 
 .. code-block:: python 
 
@@ -25,5 +67,9 @@ There are also decorators to check user roles:
     def bananas_view:
         pass
 
-.. note:: You can also use the ``roles_accepted`` decorator.
+.. note:: You can also use the ``@roles_accepted`` decorator.
+
+
+.. rubric:: Footnotes
 
+.. [#f1] Some authorization features are not possible for endpoints decorated in this way. For instance, we have an ``admin-reader`` role who should be able to read but not write everything ― with only role-based decorators we can not allow this user to read (as we don't know what permission the endpoint requires).

flexmeasures/api/common/responses.py

@@ -1,4 +1,4 @@
-from typing import Callable, List, Optional, Tuple, Union, Sequence
+from typing import List, Optional, Tuple, Union, Sequence
 import inflect
 from functools import wraps
 
@@ -142,20 +142,15 @@ def invalid_role(requested_access_role: str) -> ResponseTuple:
 
 
 def invalid_sender(
-    calling_function: Optional[Callable] = None,
-    allowed_role_names: Optional[List[str]] = None,
+    required_permissions: Optional[List[str]] = None,
 ) -> ResponseTuple:
     """
-    Signify that a sender is invalid.
-    We use this as a stand-in for flask-security auth handlers,
-    thus the arguments fit it.
-    - calling_function is usually an auth decorator like roles_required.
-    - allowed_role_names can be used if the security check involved roles.
+    Signify that the sender is invalid to perform the request. Fits well with 403 errors.
+    Optionally tell the user which permissions they should have.
     """
     message = FORBIDDEN_MSG
-    if allowed_role_names:
-        allowed_role_names = [pluralize(role_name) for role_name in allowed_role_names]
-        message += f" It is reserved for {p.join(allowed_role_names)}."
+    if required_permissions:
+        message += f" It requires {p.join(required_permissions)} permission(s)."
     return (
         dict(result="Rejected", status="INVALID_SENDER", message=message),
         FORBIDDEN_STATUS_CODE,

flexmeasures/api/common/schemas/sensor_data.py

@@ -66,7 +66,7 @@ class SensorDataDescriptionSchema(ma.Schema):
     start = AwareDateTimeField(required=True, format="iso")
     duration = DurationField(required=True)
     horizon = DurationField(
-        required=False, missing=timedelta(hours=0), default=timedelta(hours=0)
+        required=False, load_default=timedelta(hours=0), dump_default=timedelta(hours=0)
     )
     unit = fields.Str(required=True)
 

flexmeasures/api/common/schemas/users.py

@@ -0,0 +1,47 @@
+from flask import abort
+from flask_security import current_user
+from marshmallow import fields
+
+from flexmeasures.data.models.user import User, Account
+
+
+class AccountIdField(fields.Integer):
+    """
+    Field that represents an account ID. It de-serializes from the account id to an account instance.
+    """
+
+    def __init__(self, *args, **kwargs):
+        kwargs["load_default"] = (
+            lambda: current_user.account if not current_user.is_anonymous else None
+        )
+        super().__init__(*args, **kwargs)
+
+    def _deserialize(self, account_id: int, attr, obj, **kwargs) -> Account:
+        account: Account = Account.query.filter_by(id=int(account_id)).one_or_none()
+        if account is None:
+            raise abort(404, f"Account {id} not found")
+        return account
+
+    def _serialize(self, account: Account, attr, data, **kwargs) -> int:
+        return account.id
+
+
+class UserIdField(fields.Integer):
+    """
+    Field that represents a user ID. It de-serializes from the user id to a user instance.
+    """
+
+    def __init__(self, *args, **kwargs):
+        kwargs["load_default"] = (
+            lambda: current_user if not current_user.is_anonymous else None
+        )
+        super().__init__(*args, **kwargs)
+
+    def _deserialize(self, user_id: int, attr, obj, **kwargs) -> User:
+        user: User = User.query.filter_by(id=int(user_id)).one_or_none()
+        if user is None:
+            raise abort(404, f"User {id} not found")
+        return user
+
+    def _serialize(self, user: User, attr, data, **kwargs) -> int:
+        return user.id

flexmeasures/api/dev/sensors.py

@@ -1,8 +1,7 @@
 import json
 
 from flask_classful import FlaskView, route
-from flask_login import login_required
-from flask_security import current_user
+from flask_security import current_user, auth_required
 from marshmallow import fields
 from webargs.flaskparser import use_kwargs
 from werkzeug.exceptions import abort
@@ -20,7 +19,7 @@ class SensorAPI(FlaskView):
 
     route_base = "/sensor"
 
-    @login_required
+    @auth_required()
     @route("/<id>/chart/")
     @use_kwargs(
         {
@@ -38,7 +37,7 @@ def get_chart(self, id, **kwargs):
         sensor = get_sensor_or_abort(id)
         return json.dumps(sensor.chart(**kwargs))
 
-    @login_required
+    @auth_required()
     @route("/<id>/chart_data/")
     @use_kwargs(
         {
@@ -57,7 +56,7 @@ def get_chart_data(self, id, **kwargs):
         sensor = get_sensor_or_abort(id)
         return sensor.search_beliefs(as_json=True, **kwargs)
 
-    @login_required
+    @auth_required()
     def get(self, id: int):
         """GET from /sensor/<id>"""
         sensor = get_sensor_or_abort(id)

flexmeasures/api/v1/tests/test_api_v1.py

@@ -149,8 +149,7 @@ def test_invalid_sender_and_logout(client, user_email, get_message):
     )
     print("Server responded with:\n%s" % get_meter_data_response.json)
     assert get_meter_data_response.status_code == 403
-    assert get_meter_data_response.json["type"] == "GetMeterDataResponse"
-    assert get_meter_data_response.json["status"] == invalid_sender("MDC")[0]["status"]
+    assert get_meter_data_response.json["status"] == invalid_sender()[0]["status"]
 
     # log out
     logout_response = client.get(