This module is an alternative Airflow XCom backend.
Airflow XCom is a way Tasks exchange values in a DAG. By default the values are stored in the default Airflow database. This makes it unsuitable for sharing large volumes of data for two reasons:
- PostgreSQL has a limit of 1GB per XCom value
- Using the same database as other Airflow data may degrade performance
For these reasons Airflow allows for implementing a custom XCom backend. This module is one such implementation for use with Redis.
Currently the module uses the Python Redis module which is already provided by Airflow.
The only requirements is for Airflow to have a Redis Connection defined called xcom_cache.
Possible future improvements:
- Get Connection ID from configuration or environment variables
- Splitting of values into different keys to avoid size limits