feat!: Adds support for Airflow 2 Cloud Composer environment and operators

[adlersantos]

Description

This PR is the last to be merged in a series of breaking pull requests that upgrades the architecture to Airflow 2:

• First, merge feat: Upgrade usa_names pipeline to use Airflow 2 operators #136
• Then, merge feat!: Upgrade dependencies to Airflow 2.1.1 #135
• Then, merge feat!: Pipeline YAML template using Airflow 2 operators #138
• And finally this PR, which targets the main branch

Changes

First, DAGs are now generated using Airflow 2.0+ operators by specifying an airflow_version: 2 in the dag block of the YAML config

# pipeline.yaml

dag:
  airflow_version: 2

  # .. more DAG config

For backwards compatibility, pipelines default to using Airflow 1.10 operators if airflow_version is unspecified (tested it with the usa_names dataset in #136.

Next, we can now import Airflow 2.x equivalents of commonly used Airflow 1.10.15 operators.

Next, because Airflow 1.10 DAGs can run in an Airflow 2.x environment, but not the other way around, I added some version compatibility checks when deploying a DAG.

Finally, I revised deploy_dag to use the new Airflow 2 CLI:

# old cli
$ airflow variables --import 

# new cli
$ airflow variables import

Checklist

Note: Delete items below that aren't applicable to your pull request.

• Please merge this PR for me once it is approved.
• If this PR adds or edits a feature, I have updated the README accordingly.
• This PR is appropriately labeled.