544 Deprecation notice for API versions 1.0 to 2.0 (#554)

- Deprecation notices in response headers - Sunset notices in response headers - Documentation regarding monitoring of deprecation and sunsetting - Log warnings in case users call deprecated endpoints * Add and test deprecation warnings to API users Signed-off-by: F.N. Claessen <felix@seita.nl> * Provide deprecation information via headers instead of via JSON message, which is the approach recommended by the Internet Engineering Task Force (IETF) Signed-off-by: F.N. Claessen <felix@seita.nl> * API changelog entries Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * Log warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Clean up Signed-off-by: F.N. Claessen <felix@seita.nl> * Add release notice for FlexMeasures hosts Signed-off-by: F.N. Claessen <felix@seita.nl> * Changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix header links Signed-off-by: F.N. Claessen <felix@seita.nl> * Also test asset API v2 Signed-off-by: F.N. Claessen <felix@seita.nl> * Add missing deprecation checks Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl>
FlexMeasures · Dec 16, 2022 · c93f60d · c93f60d
1 parent 6532cc1
commit c93f60d
Show file tree

Hide file tree

Showing 28 changed files with 299 additions and 27 deletions.
diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst
@@ -69,6 +69,11 @@ v3.0-0 | 2022-03-25
     - Rewrote the sections on roles and sources into a combined section that refers to account roles rather than USEF roles.
     - Deprecated the section on group notation.
 
+v2.0-5 | 2022-02-13
+""""""""""""""""""""
+
+*API v2.0 is deprecated.*
+
 v2.0-4 | 2022-01-04
 """""""""""""""""""
 
@@ -110,6 +115,11 @@ v2.0-0 | 2020-11-14
 
 - Added REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).
 
+v1.3-12 | 2022-02-13
+""""""""""""""""""""
+
+*API v1.3 is deprecated.*
+
 v1.3-11 | 2022-01-05
 """"""""""""""""""""
 
@@ -196,6 +206,11 @@ v1.3-0 | 2020-01-28
 - The *postUdiEvent* endpoint now triggers scheduling jobs to be set up (rather than scheduling directly triggered by the *getDeviceMessage* endpoint)
 - The *getDeviceMessage* now queries the job queue and database for an available schedule
 
+v1.2-4 | 2022-02-13
+""""""""""""""""""""
+
+*API v1.2 is deprecated.*
+
 v1.2-3 | 2020-01-28
 """""""""""""""""""
 
@@ -242,6 +257,11 @@ v1.2-0 | 2018-09-08
 - Added a description of the *postUdiEvent* endpoint in the Prosumer and Simulation sections
 - Added a description of the *getDeviceMessage* endpoint in the Prosumer and Simulation sections
 
+v1.1-6 | 2022-02-13
+""""""""""""""""""""
+
+*API v1.1 is deprecated.*
+
 v1.1-5 | 2020-06-18
 """""""""""""""""""
 
@@ -295,6 +315,11 @@ v1.1-0 | 2018-07-15
 
 - Added a description of the *getPrognosis* endpoint in the Supplier section
 
+v1.0-2 | 2022-02-13
+""""""""""""""""""""
+
+*API v1.0 is deprecated.*
+
 v1.0-1 | 2018-07-10
 """""""""""""""""""
 

diff --git a/documentation/api/introduction.rst b/documentation/api/introduction.rst
@@ -101,3 +101,21 @@ which gives a response like this if the credentials are correct:
     }
 
 .. note:: Each access token has a limited lifetime, see :ref:`auth`.
+
+.. _api_deprecation:
+
+Deprecation and sunset
+----------------------
+
+Professional API users should monitor API responses for the ``"Deprecation"`` and ``"Sunset"`` response headers [see `draft-ietf-httpapi-deprecation-header-02 <https://datatracker.ietf.org/doc/draft-ietf-httpapi-deprecation-header/>`_ and `RFC 8594 <https://www.rfc-editor.org/rfc/rfc8594>`_, respectively], so system administrators can be warned when using API endpoints that are flagged for deprecation and/or are likely to become unresponsive in the future.
+
+The deprecation header field shows an `IMF-fixdate <https://www.rfc-editor.org/rfc/rfc7231#section-7.1.1.1>`_ indicating when the API endpoint was deprecated.
+The sunset header field shows an `IMF-fixdate <https://www.rfc-editor.org/rfc/rfc7231#section-7.1.1.1>`_ indicating when the API endpoint is likely to become unresponsive.
+
+More information about a deprecation, sunset, and possibly recommended replacements, can be found under the ``"Link"`` response header. Relevant relations are:
+
+- ``"deprecation"``
+- ``"successor-version"``
+- ``"latest-version"``
+- ``"alternate"``
+- ``"sunset"``
diff --git a/documentation/api/v1.rst b/documentation/api/v1.rst
@@ -3,6 +3,8 @@
 Version 1.0
 ===========
 
+.. warning:: This API version is deprecated since December 14, 2022, and will likely be sunset in February 2023. Please update to :ref:`v3_0`. For more information about how FlexMeasures handles deprecation and sunsetting, see :ref:`api_deprecation`.
+
 Summary
 -------
 

diff --git a/documentation/api/v1_1.rst b/documentation/api/v1_1.rst
@@ -3,6 +3,8 @@
 Version 1.1
 ===========
 
+.. warning:: This API version is deprecated since December 14, 2022, and will likely be sunset in February 2023. Please update to :ref:`v3_0`. For more information about how FlexMeasures handles deprecation and sunsetting, see :ref:`api_deprecation`.
+
 Summary
 -------
 

diff --git a/documentation/api/v1_2.rst b/documentation/api/v1_2.rst
@@ -3,6 +3,8 @@
 Version 1.2
 ===========
 
+.. warning:: This API version is deprecated since December 14, 2022, and will likely be sunset in February 2023. Please update to :ref:`v3_0`. For more information about how FlexMeasures handles deprecation and sunsetting, see :ref:`api_deprecation`.
+
 Summary
 -------
 

diff --git a/documentation/api/v1_3.rst b/documentation/api/v1_3.rst
@@ -3,6 +3,8 @@
 Version 1.3
 ===========
 
+.. warning:: This API version is deprecated since December 14, 2022, and will likely be sunset in February 2023. Please update to :ref:`v3_0`. For more information about how FlexMeasures handles deprecation and sunsetting, see :ref:`api_deprecation`.
+
 Summary
 -------
 

diff --git a/documentation/api/v2_0.rst b/documentation/api/v2_0.rst
@@ -3,6 +3,8 @@
 Version 2.0
 ===========
 
+.. warning:: This API version is deprecated since December 14, 2022, and will likely be sunset in February 2023. Please update to :ref:`v3_0`. For more information about how FlexMeasures handles deprecation and sunsetting, see :ref:`api_deprecation`.
+
 Summary
 -------
 

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -5,6 +5,9 @@ FlexMeasures Changelog
 v0.12.0 | October XX, 2022
 ============================
 
+.. warning:: After upgrading to ``flexmeasures==0.12``, users of API versions 1.0, 1.1, 1.2, 1.3 and 2.0 will receive ``"Deprecation"`` and ``"Sunset"`` response headers, and warnings are logged for FlexMeasures hosts whenever users call API endpoints in these deprecated API versions.
+             The relevant endpoints are planned to become unresponsive in ``flexmeasures==0.13``.
+
 .. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
 
 New features
@@ -41,6 +44,7 @@ Infrastructure / Support
 * Revised strategy for removing unchanged beliefs when saving data: retain the oldest measurement (ex-post belief), too [see `PR #518 <http://www.github.com/FlexMeasures/flexmeasures/pull/518>`_]
 * Scheduling test for maximizing self-consumption, and improved time series db queries for fixed tariffs (and other long-term constants) [see `PR #532 <http://www.github.com/FlexMeasures/flexmeasures/pull/532>`_]
 * Clean up table formatting for ``flexmeasures show`` CLI commands [see `PR #540 <http://www.github.com/FlexMeasures/flexmeasures/pull/540>`_]
+* Add  ``"Deprecation"`` and ``"Sunset"`` response headers for API users of deprecated API versions, and log warnings for FlexMeasures hosts when users still use them [see `PR #554 <http://www.github.com/FlexMeasures/flexmeasures/pull/554>`_]
 
 .. warning:: The CLI command ``flexmeasures monitor tasks`` has been renamed to ``flexmeasures monitor last-run``. The old name will stop working in version 0.13.
 

diff --git a/flexmeasures/api/common/utils/decorators.py b/flexmeasures/api/common/utils/decorators.py
@@ -1,6 +1,8 @@
+from __future__ import annotations
+
 from functools import wraps
 
-from flask import current_app, request
+from flask import current_app, request, Response
 from flask_json import as_json
 from werkzeug.datastructures import Headers
 
@@ -45,11 +47,7 @@ def decorated_service(*args, **kwargs):
                     "Response is not a Flask response object. I did not assign a response type."
                 )
                 return response
-            data = response.json
-            headers = dict(
-                zip(Headers.keys(response.headers), Headers.values(response.headers))
-            )
-            status_code = response.status_code
+            data, status_code, headers = split_response(response)
             if "type" in data:
                 current_app.logger.warning(
                     "Response already contains 'type' key. I did not assign a new response type."
@@ -63,3 +61,13 @@ def decorated_service(*args, **kwargs):
         return decorated_service
 
     return wrapper
+
+
+def split_response(response: Response) -> tuple[dict, int, dict]:
+    """Split Flask Response object into json data, status code and headers."""
+    data = response.json
+    headers = dict(
+        zip(Headers.keys(response.headers), Headers.values(response.headers))
+    )
+    status_code = response.status_code
+    return data, status_code, headers
diff --git a/flexmeasures/api/common/utils/deprecation_utils.py b/flexmeasures/api/common/utils/deprecation_utils.py
@@ -0,0 +1,89 @@
+from __future__ import annotations
+
+from flask import current_app, request, Blueprint, Response
+from flask_security.core import current_user
+import pandas as pd
+
+from flexmeasures.utils.time_utils import to_http_time
+
+
+def deprecate_blueprint(
+    blueprint: Blueprint,
+    deprecation_date: pd.Timestamp | str | None = None,
+    deprecation_link: str | None = None,
+    sunset_date: pd.Timestamp | str | None = None,
+    sunset_link: str | None = None,
+):
+    """Deprecates every route on a blueprint by adding the "Deprecation" header with a deprecation date.
+
+    >>> from flask import Flask, Blueprint
+    >>> app = Flask('some_app')
+    >>> deprecated_bp = Blueprint('API version 1', 'v1_bp')
+    >>> app.register_blueprint(deprecated_bp, url_prefix='/v1')
+    >>> deprecate_blueprint(
+            deprecated_bp,
+            deprecation_date="2022-12-14",
+            deprecation_link="https://flexmeasures.readthedocs.org/some-deprecation-notice",
+            sunset_date="2023-02-01",
+            sunset_link="https://flexmeasures.readthedocs.org/some-sunset-notice",
+        )
+
+    :param blueprint:        The blueprint to be deprecated
+    :param deprecation_date: date indicating when the API endpoint was deprecated, used for the "Deprecation" header
+                             if no date is given, defaults to "true"
+                             see https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-deprecation-header#section-2-1
+    :param deprecation_link: url providing more information about the deprecation
+    :param sunset_date:      date indicating when the API endpoint is likely to become unresponsive
+    :param sunset_link:      url providing more information about the sunset
+
+    References
+    ----------
+    - Deprecation field: https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-deprecation-header
+    - Sunset field: https://www.rfc-editor.org/rfc/rfc8594
+    """
+    if deprecation_date:
+        deprecation = to_http_time(pd.Timestamp(deprecation_date) - pd.Timedelta("1s"))
+    else:
+        deprecation = "true"
+    if sunset_date:
+        sunset = to_http_time(pd.Timestamp(sunset_date) - pd.Timedelta("1s"))
+
+    def _after_request_handler(response: Response) -> Response:
+        return _add_headers(
+            response,
+            deprecation,
+            deprecation_link,
+            sunset,
+            sunset_link,
+        )
+
+    blueprint.after_request(_after_request_handler)
+
+
+def _add_headers(
+    response: Response,
+    deprecation: str,
+    deprecation_link: str | None,
+    sunset: str | None,
+    sunset_link: str | None,
+) -> Response:
+    response.headers["Deprecation"] = deprecation
+    if sunset:
+        response.headers["Sunset"] = sunset
+    if deprecation_link:
+        response = _add_link(response, deprecation_link, "deprecation")
+    if sunset_link:
+        response = _add_link(response, sunset_link, "sunset")
+    current_app.logger.warning(
+        f"Deprecated endpoint {request.endpoint} called by {current_user}"
+    )
+    return response
+
+
+def _add_link(response: Response, link: str, rel: str) -> Response:
+    link_text = f'<{link}>; rel="{rel}"; type="text/html"'
+    if response.headers.get("Link"):
+        response.headers["Link"] += f", {link_text}"
+    else:
+        response.headers["Link"] = link_text
+    return response
diff --git a/flexmeasures/api/tests/utils.py b/flexmeasures/api/tests/utils.py
@@ -1,6 +1,6 @@
 import json
 
-from flask import url_for, current_app
+from flask import url_for, current_app, Response
 
 from flexmeasures.data import db
 from flexmeasures.data.services.users import find_user_by_email
@@ -102,3 +102,12 @@ def post_task_run(client, task_name: str, status: bool = True):
             "Authorization": get_auth_token(client, "task_runner@seita.nl", "testtest")
         },
     )
+
+
+def check_deprecation(response: Response):
+    print(response.headers)
+    assert "Tue, 13 Dec 2022 23:59:59 GMT" in response.headers["Deprecation"]
+    assert "Tue, 31 Jan 2023 23:59:59 GMT" in response.headers["Sunset"]
+    # Make sure we link to some url for both deprecation and sunset
+    assert 'rel="deprecation"' in response.headers["Link"]
+    assert 'rel="sunset"' in response.headers["Link"]
diff --git a/flexmeasures/api/v1/__init__.py b/flexmeasures/api/v1/__init__.py
@@ -1,8 +1,17 @@
 from flask import Flask, Blueprint
 
+from flexmeasures.api.common.utils.deprecation_utils import deprecate_blueprint
+
 
 # The api blueprint. It is registered with the Flask app (see register_at)
 flexmeasures_api = Blueprint("flexmeasures_api_v1", __name__)
+deprecate_blueprint(
+    flexmeasures_api,
+    deprecation_date="2022-12-14",
+    deprecation_link="https://flexmeasures.readthedocs.io/en/latest/api/introduction.html#deprecation-and-sunset",
+    sunset_date="2023-02-01",
+    sunset_link="https://flexmeasures.readthedocs.io/en/latest/api/v1.html",
+)
 
 
 def register_at(app: Flask):

diff --git a/flexmeasures/api/v1/tests/test_api_v1.py b/flexmeasures/api/v1/tests/test_api_v1.py
@@ -12,7 +12,7 @@
     request_processed,
     unrecognized_connection_group,
 )
-from flexmeasures.api.tests.utils import get_auth_token
+from flexmeasures.api.tests.utils import check_deprecation, get_auth_token
 from flexmeasures.api.common.utils.api_utils import message_replace_name_with_ea
 from flexmeasures.api.common.utils.validators import validate_user_sources
 from flexmeasures.api.v1.tests.utils import (
@@ -32,6 +32,7 @@ def test_get_service(client, query):
         headers={"content-type": "application/json"},
     )
     print("Server responded with:\n%s" % get_service_response.json)
+    check_deprecation(get_service_response)
     assert get_service_response.status_code == 200
     assert get_service_response.json["type"] == "GetServiceResponse"
     assert get_service_response.json["status"] == request_processed()[0]["status"]
@@ -47,6 +48,7 @@ def test_unauthorized_request(client):
         headers={"content-type": "application/json"},
     )
     print("Server responded with:\n%s" % get_meter_data_response.json)
+    check_deprecation(get_meter_data_response)
     assert get_meter_data_response.status_code == 401
     assert get_meter_data_response.json["type"] == "GetMeterDataResponse"
     assert get_meter_data_response.json["status"] == UNAUTH_ERROR_STATUS
@@ -63,6 +65,7 @@ def test_no_connection_in_get_request(client):
         },
     )
     print("Server responded with:\n%s" % get_meter_data_response.json)
+    check_deprecation(get_meter_data_response)
     assert get_meter_data_response.status_code == 400
     assert get_meter_data_response.json["type"] == "GetMeterDataResponse"
     assert (
@@ -82,6 +85,7 @@ def test_invalid_connection_in_get_request(client):
         },
     )
     print("Server responded with:\n%s" % get_meter_data_response.json)
+    check_deprecation(get_meter_data_response)
     assert get_meter_data_response.status_code == 400
     assert get_meter_data_response.json["type"] == "GetMeterDataResponse"
     assert get_meter_data_response.json["status"] == invalid_domain()[0]["status"]
@@ -116,6 +120,7 @@ def test_invalid_or_no_unit(client, method, message):
                 )
             },
         )
+        check_deprecation(get_meter_data_response)
     else:
         get_meter_data_response = []
     assert get_meter_data_response.status_code == 400
@@ -148,6 +153,7 @@ def test_invalid_sender_and_logout(client, user_email, get_message):
         headers={"Authorization": auth_token},
     )
     print("Server responded with:\n%s" % get_meter_data_response.json)
+    check_deprecation(get_meter_data_response)
     assert get_meter_data_response.status_code == 403
     assert get_meter_data_response.json["status"] == invalid_sender()[0]["status"]
 
@@ -169,6 +175,7 @@ def test_invalid_resolution_str(client):
         headers={"Authorization": auth_token},
     )
     print("Server responded with:\n%s" % get_meter_data_response.json)
+    check_deprecation(get_meter_data_response)
     assert get_meter_data_response.status_code == 400
     assert get_meter_data_response.json["type"] == "GetMeterDataResponse"
     assert get_meter_data_response.json["status"] == "INVALID_RESOLUTION"
@@ -240,6 +247,7 @@ def test_get_meter_data(db, app, client, message):
         headers={"content-type": "application/json", "Authorization": auth_token},
     )
     print("Server responded with:\n%s" % get_meter_data_response.json)
+    check_deprecation(get_meter_data_response)
     assert get_meter_data_response.status_code == 200
     assert get_meter_data_response.json["values"] == [(100.0 + i) for i in range(6)]
 
@@ -257,6 +265,7 @@ def test_post_meter_data_to_different_resolutions(app, client):
         headers={"Authorization": auth_token},
     )
     print("Server responded with:\n%s" % post_meter_data_response.json)
+    check_deprecation(post_meter_data_response)
     assert post_meter_data_response.json["type"] == "PostMeterDataResponse"
     assert post_meter_data_response.status_code == 400
     assert (