This repo contains the results of some exploratory research on Airflow features focused on learning how to implement dynamic task flows. The question I wanted to repy is basically, how we can define task flows where the execution can be controlled by the results from previous tasks.
- Trigger rules
- Variables
- Branch Operators
- XComs
- Context
- Session
As a result of the study, a list of operative Airflow DAGs are available:
00_branches_dag.py
- Examples with the
BranchPythonOperator
,ShortCircuitOperator
andBranchDayOfWeekOperator
- Conditions are set on pipeline start time
- Examples with the
00_branches_extended_dag.py
- The last of this example should always fail
01_trigger_rules_dag.py
- Shows how different trigger rules apply to successful and failed tasks in a pipeline
02_xcom_push_dag.py
- The most basic example about writing and reading XComs ever
03_xcom_return_dag.py
- A different approach about how a task can write and read XComs
04_xcom_advanced_dag.py
- An example of tasks returning more than one XCOM
- An example of task reading XCOMs from a different DAG
04_xcom_with_context_dag.py
- An alternative way to read and write XComs accessing the context
- Suitable for those operators that are not based on a callable
05_xcom_with_session__dag.py
- An alternative way to read and write XComs directly accessing the Airflow DB
- Suitable for those operators that are not based on a callable
07_clean_xcom_dag.py
- A one-task pipeline to remove all XComs generated by previous pipelines
- This task can be included in a pipeline to clean the pipeline state before running it
08_wrapup_dag.py
- A kind of more or less realistic application of the concepts we learned
How to install Airflow locally with Docker is detailed in this Quick Start. Anyway, all you require is initializing the environment for the first time by running this from the root folder:
mkdir -p ./logs ./plugins
echo -e "AIRFLOW_UID=$(id -u)" > .env
docker-compose up airflow-init
After that, just run
docker-compose up -d
Now, you are ready to test your DAGS on a local Airflow instance (user:airflow / pwd:airflow)
Some examples require using a few preconfigured Airflow variables.
To set them up just go to Admin > Variables
and import the file airflow_config/variables.json
.
To stop the full environment just run:
docker-compose down --volumes --remove-orphans