What is a good basic deployment workflow on Linux?

[schuderer]

I've been asked for a step-for-step guide on deploying a model in this example situation:

• Not using "fancy" technologies, done by hand or easily scriptable
• The model has been developed on Windows
• The model has Python dependencies which may themselves have Python dependencies
• The model is to be deployed on a Linux machine without connection to the internet
• We want to use gunicorn and nginx
• The developer should not have to maintain more than one or two config files for the solution
• HTTPS
• We cannot use Docker or similar :-P

To make the question a little more interesting:

• What if we want to train the model in the development environment (or any environment that is different from the production environment)?
• What if we need to use a Python version different from the default?
• What if we use python packages of which some may rely on C?
• What if we rely on non-python applications or libraries (OCR, image manipulation, Tensorflow, ...)?

The idea of this question to give some more guidance beyond what can be found in the deployment section of the docs, and the general recommendation to just "let your (dev)ops experts do their thing." 😉

[schuderer]

Summary:

• Have a look at the project template at https://github.com/schuderer/mllaunchpad-template, whose build.py has been created to perform the points below, using a central deploy: section in the model's config file.
• Freeze your (clean, non-dev) requirements into a requirements file: pip freeze >requirements_frozen.txt (I like to maintain an unfrozen requirements.txt with my direct requirements and a requirements_frozen.txt with all pinned dependencies; you can also call these files requirements.in vs. requirements.txt)
• Download the requirements for the target platform pip download --platform=manylinux2010_x86_64 --no-deps -d wheels -r requirements_frozen.txt (if manylinux2010_x86_64 results in source downloads where you expect wheels, try older platform definitions like manylinux1_x86_64 and linux_x86_64 in this order). Sometimes, the version you pinned on your development platform does not have a wheel on your target platform, which is a special kind of cross-platform dependency hell and you'd have to dig around in the target platform's dependency tree.
• Put everything in an archive/zip file (code, config-to-deploy, downloaded wheels) and transfer and unarchive it into a suitable directory on your server (we assume a suitable user and path exists).
• Create a virtualenv/venv on the server and install the wheels (pip install --no-index --find-links wheels/ -r requirements_frozen.txt)
• Run e.g. on gunicorn or uwsgi, optionally behind nginx.

Disclaimer: First of all, the setting sketched above is far from ideal -- which is part of what makes this question interesting. Honestly, however, particular the second block of bullets just screams containering to me personally, and, generally, if the restrictions become extremely "interesting" (i.e. difficult), one should take a moment and talk about changing the restrictions instead of still trying to work around them.
Disclaimer 2: Whenever I write"production" below, you can substitute this with "test", "acceptance" or similar, depending on your deployment target.
Disclaimer 3: Q: Why not setup.py? A: No cross-platform wheel building supported.

I will first try to answer the first block of bullets (the more modest requirements), and talk about the what-if points one after another below that.

We assume that you have ssh/scp access to the Linux server, user(s) with suitable privileges to deploy and also a user to run it. We also assume that you have tested your model and API on your local development machine, and that your dev/build environment has a python version that is at least compatible with the python version in production. When creating environments, you can tell your environment which Python version to use. For example, a way to create such a development/build environment on conda could be conda create --name mydevenv python=3.6 pip ipython, followed by installing the dependencies from requirements_dev.txt.

Note that we don't actually recommend using conda because dependency management relies on pip, and mixing pip with conda is discouraged. We would rather recommend to install Python from python.org and use venv for managing environments, e.g. using python -m venv --clear --copies .mydevenv (this is for Windows -- leave out the --copies option on Linux).

Note: If you get strange pip-related errors, try python -m pip install --user --upgrade pip.

If you have not done so before and this is your first release, prepare your first freeze your dependencies. That way, the complete dependency tree is available, with versions, and you will produce a defined, reproducible artifact. You fist need to create a pristine environment with a production-compatible Python version, then install the requirements, and then freeze them. Use these commands (in your project directory):

> cd myprojectdir  &:: Change into project directory (if necessary)
> .mydevenv\Scripts\activate  &:: Activate dev environment (if necessary; if using conda, use `conda activate mydevenv`)
> python --version &:: Validate that the dev/build environment's Python is compatible with that of production
Python 3.6.10
> python -m venv .venv_temp_deploy
> deactivate &:: (if you activated your dev environment above)
> .venv_temp_deploy\Scripts\activate
> pip install -r requirements.txt
> pip freeze --local > requirements_frozen.txt
> deactivate
> rmdir /s /q .venv_temp_deploy

Important: Freezing requirements is usually only done once. From now on, you shall have to maintain your requirements_frozen.txt (including the version of each package listed inside) in your version control system. That said, for major releases that break compatibility anyway, you would probably re-freeze.

Steps to prepare the artifact to deploy:

1. Activate your development/build environment
2. Download the dependencies to the wheels subdirectory using the command pip download --platform=manylinux1_x86_64 --no-deps -d wheels -r requirements_frozen.txt

Transfer your code, config, wheels as well as any other local resources your API needs to the server.

Steps to install on the server

1. Create an empty venv, e.g. cd my_app_dir && python -m venv .venv
2. pip install --no-index --find-links wheels/ -r requirements_frozen.txt

How to run with waitress/gunicorn/uwsgi/nginx can be found in https://flask.palletsprojects.com/en/1.1.x/deploying/ and those specific tools' docs, including the HTTPS bullet point from the issue description.

This should at least answer the first block of bullet points above.

PS: Here's a nice rundown of more advanced Python deployment approaches: https://www.nylas.com/blog/packaging-deploying-python/

[schuderer]

At least some short answers on the second block of bulleted questions:

• What if we want to train the model in the development environment (or any environment that is different from the production environment)?

• You need production train- (and optionally test)-tagged datasources in your deployed config file.
• You need to create some sort of wrapper/trigger mechanism which calls mllaunchpad.train_model() when you want to retrain, e.g. periodically.
• Alternatively, you can monitor the model's performance by periodically running mllaunchpad.retest() and, when the test metrics indicate it, scheduling a run of mllaunchpad.train_model().
• The process (training and testing) will need to write to the config-specified model_store location, so make sure it can.
• You need to nudge the deployed Flask app to reload to make use of the new model.

• What if we need to use a Python version different from the default?

Just install it, e.g. somewhere in /opt, and use the python interpreter there to create your app's venvs.

• What if we use python packages of which some may rely on C?

If they are wheels, and could be pip-downloaded for your target platform as described in my earlier answer, you're fine. Anyway, it's a good idea to install python-dev on your server so that some (basic) compiling can take place. So for maybe 95% of the cases, this should work. Some C-backed source distributions can then be compiled already, but others require additional dependencies specifically for compiling which the download/deploy mechanism above does not cover (shakes fist at pandas). For those, you could either build a wheel to deyloy yourself, or download the sources and build-install on the server, or, if this happens often, think about not cross-deploying, i.e. building your deployment artifact on a proper build environment/machine whose platform is compatible with the target platform.

• What if we rely on non-python applications or libraries (OCR, image manipulation, Tensorflow, ...)?

You need to install them (not tensorflow BTW) using your Linux distro's package management system or similar. Or use Docker. 😬