airflow-branching-demo

This repo contains the results of some exploratory research on Airflow features focused on learning how to implement dynamic task flows. The question I wanted to repy is basically, how we can define task flows where the execution can be controlled by the results from previous tasks.

Covered Airflow features

• Trigger rules
• Variables
• Branch Operators
• XComs
• Context
• Session

DAGs

As a result of the study, a list of operative Airflow DAGs are available:

• 00_branches_dag.py
  • Examples with the BranchPythonOperator, ShortCircuitOperator and BranchDayOfWeekOperator
  • Conditions are set on pipeline start time
• 00_branches_extended_dag.py
  • The last of this example should always fail
• 01_trigger_rules_dag.py
  • Shows how different trigger rules apply to successful and failed tasks in a pipeline
• 02_xcom_push_dag.py
  • The most basic example about writing and reading XComs ever
• 03_xcom_return_dag.py
  • A different approach about how a task can write and read XComs
• 04_xcom_advanced_dag.py
  • An example of tasks returning more than one XCOM
  • An example of task reading XCOMs from a different DAG
• 04_xcom_with_context_dag.py
  • An alternative way to read and write XComs accessing the context
  • Suitable for those operators that are not based on a callable
• 05_xcom_with_session__dag.py
  • An alternative way to read and write XComs directly accessing the Airflow DB
  • Suitable for those operators that are not based on a callable
• 07_clean_xcom_dag.py
  • A one-task pipeline to remove all XComs generated by previous pipelines
  • This task can be included in a pipeline to clean the pipeline state before running it
• 08_wrapup_dag.py
  • A kind of more or less realistic application of the concepts we learned

Setup

How to install Airflow locally with Docker is detailed in this Quick Start. Anyway, all you require is initializing the environment for the first time by running this from the root folder:

mkdir -p ./logs ./plugins
echo -e "AIRFLOW_UID=$(id -u)" > .env
docker-compose up airflow-init

Run environment

After that, just run

docker-compose up -d

Now, you are ready to test your DAGS on a local Airflow instance (user:airflow / pwd:airflow)

Instance configuration

Some examples require using a few preconfigured Airflow variables. To set them up just go to Admin > Variables and import the file airflow_config/variables.json.

Stop environment

To stop the full environment just run:

docker-compose down --volumes --remove-orphans