Resolving indirect and direct (ciruclar) dependencies between extensions

[eganma-umich]

summary

I'm looking for feedback on a design that has resulted in circular dependencies between flask extensions. It currently works as intended, but I'm not really satisfied with where it ended up. While I'm looking for options to remove the circular dependency, I'd also be interested in hearing more about how extensions are supposed to deal with dependencies on other extensions

background

I've got an API where want to be able to configure some functionality used in application blueprints for different deployments. For example, registration for production API access uses info from the org's official IdP to validate a user is eligible for access whereas a test deployment will just check that the person trying to register is one of a few specific users.

Since there were a few pieces of functionality like this, I rolled a pseudo-exetension plugin registry class that could be subclassed to implement the different concrete versions of the functionality I needed.

This was fine until some of those implementations started to require other extensions to work. In this case, it was needing Flask-SQLAlchemy, where I needed to use the db.session and model classes to query for info in the database.

I initially imported the plugin implementations module and passed it directly to the plugin registry on construction, which eventually revealed the issue. Deferring loading the module, by passing the name as a string for the plugin registry to load on its own or setting up a flask app config default to specify the module in the app factory, got around the error so long as the plugin was created after the SQLAlchemy extension instance.

example

folder strucutre
circimport/
	__init__.py
	__main__.py
	models/
		__init__.py
		core.py
	plugins/
		__init__.py
		plugins.py
		development.py
		production.py
		
`pythom -m circimport` runs create_app

# cicimport/__init__.py
# the rest of the __init__.py files are empty

from circimport.plugins.plugins import Type1PluginRegistry

# plugin implementation module import causes circular import
# import circimport.plugins.production as implementations


class FlaskFake:
    def __init__(self) -> None:
        self.config = {}


def create_db():
    return lambda: 'SQLALchemy'


def create_plugin():
    return Type1PluginRegistry(
        plugin_module='circimport.plugins.production'
    )


def create_app():
    app = FlaskFake()
    app.config['PLUGIN_MODULE'] = 'circimport.plugins.development'

    # db.init_app(app)
    plugin.init_app(app)

    plugin.set_plugin('Impl1')
    plugin.exec()

    plugin.set_plugin('Impl2')
    plugin.exec()

    plugin.set_plugin('Impl3')
    plugin.exec()


# initializing before db causes circular import
# plugin = create_plugin()
db = create_db()
plugin = create_plugin()

# circimport/__main__.py

from circimport import create_app

create_app()

# circimport/models/core.py

from circimport import db


def model():
    return f'model using {db()}'

# circimport/plugins/plugins.py

from abc import ABC, abstractmethod
from inspect import ismodule, isclass, getmembers
from importlib import import_module


class AbstractPluginBase(ABC):
    def __init__(self, app=None):
        if app is not None:
            self.init_app(app)

    def init_app(self, app):
        pass

    @abstractmethod
    def exec(*args, **kwargs):
        pass


class PluginRegistry:
    MODULE_ENVVAR = 'PLUGIN_MODULE'
    PLUGIN_ENVVAR = 'PLUGIN'

    def __init__(self, base_cls, plugin_module=None, app=None) -> None:
        self._registry = {}
        self._selected = None

        self._base_cls = base_cls

        # load modules at registry instantiation
        if plugin_module is not None:
            self.load_plugins(plugin_module)

        if app is not None:
            self.init_app(app)

    def load_plugins(self, module):
        if not ismodule(module):
            module = import_module(module)

        predicate = lambda x: isclass(x) and issubclass(x, self._base_cls) and x is not self._base_cls

        self._registry.update({name: cls() for name, cls in getmembers(module, predicate)})

    def set_plugin(self, plugin):
        self._selected = plugin

    def init_app(self, app):
        # load additional modules at app instantiation
        if self.MODULE_ENVVAR in app.config:
            self.load_plugins(app.config[self.MODULE_ENVVAR])

        if self.PLUGIN_ENVVAR in app.config:
            self.set_plugin(app.config[self.PLUGIN_ENVVAR])

        for impl in self._registry.values():
            impl.init_app(app)

    def exec(self, *args, **kwargs):
        print(self._selected, end=' ')
        return self._registry[self._selected].exec(*args, **kwargs)


class Type1Plugin(AbstractPluginBase):
    pass


class Type1PluginRegistry(PluginRegistry):
    def __init__(self, plugin_module=None, app=None) -> None:
        super().__init__(Type1Plugin, plugin_module, app)

# circimport/plugins/development.py

from circimport.plugins.plugins import Type1Plugin
from circimport.models.core import model


class Impl3(Type1Plugin):
    def exec(*args, **kwargs):
        print(f'loaded on app create importing {model()}')

# circimport/plugins/production.py

from circimport import db
from circimport.plugins.plugins import Type1Plugin


class Impl1(Type1Plugin):
    def exec(*args, **kwargs):
        print('doesn\'t touch db')


class Impl2(Type1Plugin):
    def exec(*args, **kwargs):
        print(f'touches {db()}')

[ThiefMaster]

TBH, my first impression there is "over-engineered"... ;)

My approach would be to keep core stuff in your core app, and then create plugins for your application (e.g. using setup.cfg entry points, and there's also flask-pluginengine (disclaimer: I'm the author)). Those plugins then import stuff directly from the core app, e.g. from yourapp.flaskstuff import db to get the SQLAlchemy object; same for the rest.

When a plugin depends on another one it declares that dependency so the other one gets loaded first; and so far I never had to deal w/ circular dependencies but I'd simply do the imports locally within a function if it otherwise creates problems - but again, try to avoid having circular dependencies altogether.

[eganma-umich]

TBH, my first impression there is "over-engineered"... ;)

I admittedly approached this as a prototype for another part of the program. Similar since there's going to be a set of different implementations. Different since there're going to be many implementations, and which implementation is used depends on the data being processed rather than being set at app init. Tried to come up with something that made managing these sets mostly automatic.

My approach would be to keep core stuff in your core app, and then create plugins for your application

I cribbed the init_app pattern since it seemed like a common way to configure program behavior in the app factory, but other projects I looked were true third-party extensions that expected to be provided their dependencies up front. This code is just part of this project, and I think treating it as a pseudo-extension is where I started to muddle things.

I now realize that this is actually the same problem that requires you to import the blueprints inside the app factory if you're initializing Flask-SQLAlchemy in the same module. With that in mind, I'm more comfortable with the circular dependency so long as it reads like other parts of the program that have to account for it too.

from flask import Flask
from flask_sqlalchemy import SQLAlchemy


def create_db():
    return SQLAlchemy()


def create_app():
    app = Flask(__name__)
    app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///database.sqlite'
    app.config['REGISTRATION_VALIDATOR'] = 'IdPValidator'

    db.init_app(app)

    # registration_validators would be an instance of PluginRegsitry in previous example
    from refactor.validators.registration import registration_validators
    registration_validators.load_from_app_config(app)

    from refactor.views.api import bp
    app.register_blueprint(bp)

    return app


db = create_db()

Looking at some other projects, there's a mix of locating the extensions with the app factory and just dealing with the circularity or in other modules to avoid it. Are there any flask-specific reasons to favor one or the other?

[ThiefMaster]

Creating the extension should be done globally. They need to be available at import time.

Initializing the extensions should be one in the app factory, since you need the Flask app instance to do so, and you generally want to have your config loaded etc at that point.

[eganma-umich]

So those are the two operational requirements, and it's up to you to structure things to meet them.