Also load plugins which are full Python packages

documentation/changelog.rst

@@ -5,6 +5,8 @@ FlexMeasures Changelog
 v0.7.0 | October XX, 2021
 ===========================
 
+.. warning:: The config setting ``FLEXMEASURES_PLUGIN_PATHS`` has been renamed to ``FLEXMEASURES_PLUGINS``. The old name still works but is deprecated.
+
 New features
 -----------
 
@@ -13,6 +15,7 @@ Bugfixes
 
 Infrastructure / Support
 ----------------------
+* FlexMeasures plugins can be Python packages now. We provide `a cookie-cutter template <https://github.com/SeitaBV/flexmeasures-plugin-template>`_ for this approach. [see `PR #182 <http://www.github.com/SeitaBV/flexmeasures/pull/182>`_]
 
 
 v0.6.1 | September XX, 2021

documentation/configuration.rst

@@ -58,12 +58,18 @@ Default: ``{"flexmeasures.io": "2021-01"}``
 
 .. _plugin-config:
 
-FLEXMEASURES_PLUGIN_PATHS
+FLEXMEASURES_PLUGINS
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-A list of absolute paths to Blueprint-based plugins for FlexMeasures (e.g. for custom views or CLI functions).
-Each plugin path points to a folder, which should contain an ``__init__.py`` file where the Blueprint is defined. 
-See :ref:`plugins` on what is expected for content.
+A list of plugins you want FlexMeasures to load (e.g. for custom views or CLI functions). 
+
+Two types of entries are possible here:
+
+* File paths (absolute or relative) to plugins. Each such path needs to point to a folder, which should contain an ``__init__.py`` file where the Blueprint is defined. 
+* Names of installed Python modules. 
+
+Added functionality in plugins needs to be based on Flask Blueprints. See :ref:`plugins` for more information and examples.
+
 
 Default: ``[]``
 

documentation/dev/plugins.rst

@@ -3,24 +3,35 @@
 Writing Plugins
 ====================
 
-You can extend FlexMeasures with functionality like UI pages or CLI functions.
+You can extend FlexMeasures with functionality like UI pages, API endpoints, or CLI functions.
+This is eventually how energy flexibility services are built on top of FlexMeasures!
 
-A FlexMeasures plugin works as a `Flask Blueprint <https://flask.palletsprojects.com/en/1.1.x/tutorial/views/>`_.
+In an nutshell, a FlexMeasures plugin adds functionality via one or more `Flask Blueprints <https://flask.palletsprojects.com/en/1.1.x/tutorial/views/>`_.
 
 .. todo:: We'll use this to allow for custom forecasting and scheduling algorithms, as well.
 
 
-How it works
-^^^^^^^^^^^^^^
+How to make FlexMeasures load your plugin
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Use the config setting :ref:`plugin-config` to point to your plugin(s).
+Use the config setting :ref:`plugin-config` to list your plugin(s).
 
-Here are the assumptions FlexMeasures makes to be able to import your Blueprint:
+A setting in this list can:
 
-- The plugin folder contains an ``__init__.py`` file.
-- In that file, you define a Blueprint object (or several).
-
-We'll refer to the plugin with the name of your plugin folder.
+1. point to a plugin folder containing an __init__.py file
+2. be the name of an installed module (i.e. in a Python console `import <module_name>` would work)
+
+Each plugin defines at least one Blueprint object. These will be registered with the Flask app,
+so their functionality (e.g. routes) becomes available.
+
+We'll discuss an example below.
+
+In that example, we use the first option from above to tell FlexMeasures about the plugin. It is the simplest way to start playing around.
+
+The second option (the plugin being an importable Python package) allows for more professional software development. For instance, it is more straightforward in that case to add code hygiene, version management and dependencies (your plugin can depend on a specific FlexMeasures version and other plugins can depend on yours).
+
+To hit the ground running with that approach, we provide a `CookieCutter template <https://github.com/SeitaBV/flexmeasures-plugin-template>`_.
+It also includes a few Blueprint examples and best practices.
 
 
 Showcase
@@ -29,7 +40,7 @@ Showcase
 Here is a showcase file which constitutes a FlexMeasures plugin called ``our_client``.
 
 * We demonstrate adding a view, which can be rendered using the FlexMeasures base templates.
-* We also showcase a CLI function which has access to the FlexMeasures `app` object. It can be called via ``flexmeasures our_client test``. 
+* We also showcase a CLI function which has access to the FlexMeasures `app` object. It can be called via ``flexmeasures our-client test``. 
 
 We first create the file ``<some_folder>/our_client/__init__.py``. This means that ``our_client`` is the plugin folder and becomes the plugin name.
 
@@ -45,15 +56,15 @@ With the ``__init__.py`` below, plus the custom Jinja2 template, ``our_client``
     from flexmeasures.ui.utils.view_utils import render_flexmeasures_template
 
 
-    our_client_bp = Blueprint('our_client', __name__,
+    our_client_bp = Blueprint('our-client', __name__,
                               template_folder='templates')
 
     # Showcase: Adding a view
 
     @our_client_bp.route('/')
     @our_client_bp.route('/my-page')
     @login_required
-    def metrics():
+    def my_page():
         msg = "I am a FlexMeasures plugin !"
         # Note that we render via the in-built FlexMeasures way
         return render_flexmeasures_template(
@@ -73,7 +84,7 @@ With the ``__init__.py`` below, plus the custom Jinja2 template, ``our_client``
 
     @our_client_bp.cli.command("test")
     @with_appcontext
-    def oc_test():
+    def our_client_test():
         print(f"I am a CLI command, part of FlexMeasures: {current_app}")
 
 
@@ -121,11 +132,14 @@ Starting the template with ``{% extends "base.html" %}`` integrates your page co
 We'd name this file ``our_client_base.html``. Then, we'd extend our page template from ``our_client_base.html``, instead of ``base.html``.
 
 
-Using other code files in your plugin
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Using other code files in your non-package plugin
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Say you want to include other Python files in your plugin, importing them in your ``__init__.py`` file.
-This can be done if you put the plugin path on the import path. Do it like this in your ``__init__.py``:
+With this file-only version of loading the plugin (if your plugin isn't imported as a package),
+this is a bit tricky.
+
+But it can be achieved if you put the plugin path on the import path. Do it like this in your ``__init__.py``:
 
 .. code-block:: python
 

flexmeasures/utils/app_utils.py

@@ -21,9 +21,9 @@
 def flexmeasures_cli():
     """
     Management scripts for the FlexMeasures platform.
-    We use @app_context here so things from the app setup are initialised
-    only once. This is crucial for Sentry, for example.
     """
+    # We use @app_context above, so things from the app setup are initialised
+    # only once! This is crucial for Sentry, for example.
     pass
 
 
@@ -179,41 +179,72 @@ def parse_config_entry_by_account_roles(
 def register_plugins(app: Flask):
     """
     Register FlexMeasures plugins as Blueprints.
-    This is configured by the config setting FLEXMEASURES_PLUGIN_PATHS.
+    This is configured by the config setting FLEXMEASURES_PLUGINS.
 
     Assumptions:
-    - Your plugin folders contains an __init__.py file.
-    - In that file, you define a Blueprint object (or several).
+    - a setting EITHER points to a plugin folder containing an __init__.py file
+      OR it is the name of an installed module, which can be imported.
+    - each plugin defines at least one Blueprint object. These will be registered with the Flask app,
+      so their functionality (e.g. routes) becomes available.
 
-    We'll refer to the plugins with the name of your plugin folders (last part of the path).
+    If you load a plugin via a file path, we'll refer to the plugin with the name of your plugin folder
+    (last part of the path).
     """
-    plugin_paths = app.config.get("FLEXMEASURES_PLUGIN_PATHS", "")
-    if not isinstance(plugin_paths, list):
-        app.logger.warning(
-            f"The value of FLEXMEASURES_PLUGIN_PATHS is not a list: {plugin_paths}. Cannot install plugins ..."
+    plugins = app.config.get("FLEXMEASURES_PLUGINS", [])
+    if not plugins:
+        # this is deprecated behaviour which we should remove in version 1.0
+        app.logger.debug(
+            "No plugins configured. Attempting deprecated setting FLEXMEASURES_PLUGIN_PATHS ..."
+        )
+        plugins = app.config.get("FLEXMEASURES_PLUGIN_PATHS", [])
+    if not isinstance(plugins, list):
+        app.logger.error(
+            f"The value of FLEXMEASURES_PLUGINS is not a list: {plugins}. Cannot install plugins ..."
         )
         return
     app.config["LOADED_PLUGINS"] = {}
-    for plugin_path in plugin_paths:
-        plugin_name = plugin_path.split("/")[-1]
-        if not os.path.exists(os.path.join(plugin_path, "__init__.py")):
-            app.logger.warning(
-                f"Plugin {plugin_name} does not contain an '__init__.py' file. Cannot load plugin {plugin_name}."
+    for plugin in plugins:
+        plugin_name = plugin.split("/")[-1]
+        app.logger.info(f"Importing plugin {plugin_name} ...")
+        module = None
+        if not os.path.exists(plugin):  # assume plugin is a package
+            pkg_name = os.path.split(plugin)[
+                -1
+            ]  # rule out attempts for relative package imports
+            app.logger.debug(
+                f"Attempting to import {pkg_name} as an installed package ..."
             )
-            continue
-        app.logger.debug(f"Importing plugin {plugin_name} ...")
-        spec = importlib.util.spec_from_file_location(
-            plugin_name, os.path.join(plugin_path, "__init__.py")
-        )
-        if spec is None:
-            app.logger.warning(
-                f"Could not load specs for plugin {plugin_name} at {plugin_path}."
+            try:
+                module = importlib.import_module(pkg_name)
+            except ModuleNotFoundError:
+                app.logger.error(
+                    f"Attempted to import module {pkg_name} (as it is not a valid file path), but it is not installed."
+                )
+                continue
+        else:  # assume plugin is a file path
+            if not os.path.exists(os.path.join(plugin, "__init__.py")):
+                app.logger.error(
+                    f"Plugin {plugin_name} is a valid file path, but does not contain an '__init__.py' file. Cannot load plugin {plugin_name}."
+                )
+                continue
+            spec = importlib.util.spec_from_file_location(
+                plugin_name, os.path.join(plugin, "__init__.py")
             )
+            if spec is None:
+                app.logger.error(
+                    f"Could not load specs for plugin {plugin_name} at {plugin}."
+                )
+                continue
+            module = importlib.util.module_from_spec(spec)
+            sys.modules[plugin_name] = module
+            assert isinstance(spec.loader, Loader)
+            spec.loader.exec_module(module)
+
+        if module is None:
+            app.logger.error(f"Plugin {plugin} could not be loaded.")
             continue
-        module = importlib.util.module_from_spec(spec)
-        sys.modules[plugin_name] = module
-        assert isinstance(spec.loader, Loader)
-        spec.loader.exec_module(module)
+
+        plugin_version = getattr(module, "__version__", "0.1")
 
         # Look for blueprints in the plugin's main __init__ module and register them
         plugin_blueprints = [
@@ -223,13 +254,13 @@ def register_plugins(app: Flask):
         ]
         if not plugin_blueprints:
             app.logger.warning(
-                f"No blueprints found for plugin {plugin_name} at {plugin_path}."
+                f"No blueprints found for plugin {plugin_name} at {plugin}."
             )
             continue
         for plugin_blueprint in plugin_blueprints:
+            app.logger.debug(f"Registering {plugin_blueprint} ...")
             app.register_blueprint(plugin_blueprint)
 
-        plugin_version = getattr(module, "__version__", "0.1")
         app.config["LOADED_PLUGINS"][plugin_name] = plugin_version
     app.logger.info(f"Loaded plugins: {app.config['LOADED_PLUGINS']}")
     sentry_sdk.set_context("plugins", app.config.get("LOADED_PLUGINS", {}))

flexmeasures/utils/config_defaults.py

@@ -94,7 +94,7 @@ class Config(object):
     # This setting contains the domain on which FlexMeasures runs
     # and the first month when the domain was under the current owner's administration
     FLEXMEASURES_HOSTS_AND_AUTH_START: dict = {"flexmeasures.io": "2021-01"}
-    FLEXMEASURES_PLUGIN_PATHS: List[str] = []
+    FLEXMEASURES_PLUGINS: List[str] = []
     FLEXMEASURES_PROFILE_REQUESTS: bool = False
     FLEXMEASURES_DB_BACKUP_PATH: str = "migrations/dumps"
     FLEXMEASURES_ROOT_VIEW: Union[str, List[Union[str, Tuple[str, List[str]]]]] = []