airflow-pex-example

Deploying Apache Airflow typically involves installing Airflow and your custom libraries in a virtualenv on the production host, along with your and DAGs & other files (e.g.: SQL). To simplify deployment, here we explore using pex.

Build an Airflow pex by running:

./pants binary src/python/example/airflow

This produces airflow.pex, a single file that is analogous to a statically-lined binary. It's a self-contained, runnable Airflow you can scp to another machine and run.

You can then run Airflow commands as usual, using dist/airflow.pex.

$ ./dist/airflow.pex list_dags
[2018-04-13 17:34:39,885] {__init__.py:45} INFO - Using executor SequentialExecutor
[2018-04-13 17:34:39,924] {models.py:189} INFO - Filling up the DagBag from /home/travis/airflow/dags


-------------------------------------------------------------------
DAGS
-------------------------------------------------------------------
example_bash_operator
example_branch_dop_operator_v3
example_branch_operator
example_http_operator
example_passing_params_via_test_command
example_python_operator
example_short_circuit_operator
example_skip_dag
example_subdag_operator
example_subdag_operator.section-1
example_subdag_operator.section-2
example_trigger_controller_dag
example_trigger_target_dag
example_xcom
latest_only
latest_only_with_trigger
test_utils
tutorial

You can then scp or otherwise distribute this file to a production host.

python_app example

When using Pants 1.7.0rc0 or later which contains python_app support we can build a self-contained binary along with DAGs in a deployable artifact.

Note how src/dags:analytics contains a directory of DAGs, which may be useful if multiple teams require separate DAGs, or you can use just one DAG dir.

$ ./pants bundle src/dags:analytics --bundle-py-archive=tgz
$ $ cd dist/src.dags.analytics-bundle/
$ $ find .
.
./main.pex
./analytics
./analytics/analytics_daily.pyc
./analytics/analytics_daily.py
$ AIRFLOW_HOME=$(pwd) AIRFLOW__CORE__DAGS_FOLDER=$(pwd)/analytics ./main.pex list_tasks analytics_daily
[2018-06-01 17:34:47,127] {__init__.py:45} INFO - Using executor SequentialExecutor
[2018-06-01 17:34:47,174] {models.py:189} INFO - Filling up the DagBag from /Users/travis/src/airflow-pex-example/dist/src.dags.analytics-bundle/analytics
print_date

Working environment

We show a working example of Airflow integrated into pants for development. direnv is required to load bin/airflow and bin/gunicorn directly into the environment. This is explained in detail in bin/README.md. We use pyenv in the example, which is recommended but not required.

Install dependencies

brew install direnv pyenv openssl

Copy the environment file

cp .envrc.example .envrc && direnv allow

Build

make

View the help

airflow --help

List the available DAGs

airflow list_dags

Initialize the database

airflow initdb 

Run the example workflow

airflow backfill analytics_daily -s 2018-01-01 -e 2018-01-01

List projects

change_project

Change project

change_project eng