Downloading a model from within Python without interrupting your script

[BramVanroy]

I just posted this on Twitter and I figured I'd share it here as well.

I've often ran into the question where I use spaCy under the hood and I want to make my code base accessible and easy to use, but was hindered by the need of users to pre-install models (spacy -m download ...). For some users, this is a technical, frightening or simply "extra" step to take. They just want to follow example scripts and run commands with whichever model/language they want. This may seem silly to many of you, but I've had people knocking on my door with this a couple of times.

So recently, I've started to add this little snippet in my codebases, which automatically downloads and loads (technically imports) a model when it is not installed. It therefore mimics behavior of other libraries like stanza or transformers where models are downloaded on-the-fly. I am aware that spaCy models are a different beast altogether, with their own dependencies, configs, and all kinds of magic. But for the end-user, the behavior is the same.

I hope it helps some of you! And if there are any issues with this snippet (security, portability, usability wise), I'd be very happy to hear it, and keep track of them in this first post!

def download_and_init_nlp(model_name: str, **kwargs) -> Language:
    """Load a spaCy model, download it if it has not been installed yet.
    :param model_name: the model name, e.g., en_core_web_sm
    :param kwargs: options passed to the spaCy loader, such as component exclusion
    :return: an initialized spaCy Language
    """
    try:
        model_module = import_module(model_name)
    except ModuleNotFoundError:
        download(model_name)
        model_module = import_module(model_name)

    return model_module.load(**kwargs)

EDIT: I packaged it for easy use so you can just pip-install it. If it turns out that there are too many/intricate underlying issues with this, I am happy to hear it and re-adjust.

Potential issues

1. Because spaCy's download invokes pip to install the module/models, it is technically possible that the models depend on different versions of installed packages, leading to the on-the-fly uninstallation/reinstallation of packages, potentially even of spaCy itself. That is not wanted! But as long as official spaCy packages are installed, such problems should not occur I think. However, because of this I would not recommend to also do this for any spaCy model on the Hugging Face hub, for which you have no control over its dependencies.

[adrianeboyd]

I totally understand the frustration and confusion around this, but I don't think it's going to work in general unless you do install separately and restart python afterwards. In practice, however, it will probably currently work for all or nearly all of the current sm/md/lg models.

The main problem is that this doesn't work for packages that have dependencies that add entry points. So if you have a base spacy install and load a trf package with this method (aside from the fact that you're probably not going to get the right version of torch), it doesn't work because the spacy-transformers entry points aren't available from the package that was installed in the running session.

There are potentially workarounds for reloading the entry points on the fly, but they are probably going to be brittle and different across different versions of python. We've looked into it in the past and decided it wasn't worth pursuing, but it's also possible this could change as we drop older versions of python.

At any rate, there are a ton of advantages in packaging pipelines as full-fledged pip packages rather than just treating them as data.

[BramVanroy]

I did not intend to discredit how spaCy handles package/model installations. I definitely appreciate the upsides that it brings. But, as you mention, one drawback is the requirement to always manually install any model. This is a manual step that you could argue is simply one additional step and not too bad. However, I also find myself wanting to make all models available if requested by a user, on-the-fly. As an example, this simple demo where I do not want to put all of the possible spaCy models wheels in my pip file, but where they can be downloaded when needed.

I do understand that spaCy does not have this built-in. From what I have tested, this seems to work for the sm,md,lg models and as well as the spacy-transformers models if that package has been installed manually. But, as you say, this may change at any given time when a new model has a dependency that does something that cannot be caught by simply importing as a module. For instance reloading entry-points. I also wrote a snippet to load any spaCy model from the Hugging Face hub but quickly found that dependencies are the major issue there, because oftentimes models are trained on/require a specific spaCy version, and you do not want to uninstall spaCy while your script is running. I'll see if with --no-deps and catching errors I get something working.

Note: I work in research, not in production environments. So for me it isn't as much of a problem if spacy_download would stop working. But I understand your concern that it should not be something to rely on, esp. in production environments. I'll emphasize that in the readme!

[adrianeboyd]

We're obviously not going to tell you you can't do this (if it works for your demo that's fine!), but we're going to strongly recommend against it in the general case.

(Also, did you mean to name the demo "spacey"?)

[BramVanroy]

Hah, yes. The library itself is called spacy-conll, but the Hugging Face Space is called space-y conll. A bit silly, I admit.