Airflow 2.8.1: awscli installation hindered by version conflicts in requirements.txt

[gameofdatas]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.8.1

What happened?

After upgrading to Airflow 2.8.1, DAGs are failing with a 'No such file or directory' error related to awscli. It seems awscli needs to be installed as part of the environment setup. However, when adding awscli to 'requirements.txt', the installation fails due to version conflicts with dependent packages like botocore, docutils, colorama, and rsa. These conflicts arise from the constraint list pinned for Airflow 2.8.1.

To match the botocore version from the constraint list i.e 1.33.13, which is a dependent package of awscli, I installed specific version of awscli i.e 1.31.13
installation logs:

The conflict is caused by:
    awscli 1.31.13 depends on docutils<0.17 and >=0.10
    The user requested (constraint) botocore==1.33.13
    The user requested (constraint) docutils==0.20.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Although after matching botocore version, docutils and other dependent versions were having version conflict.

What you think should happen instead?

awscli should be installed via requirements.txt without any version conflict

How to reproduce

Steps to be followed for reproducing the error:

> git clone https://github.com/aws/aws-mwaa-local-runner.git

> cd aws-mwaa-local-runner && ./mwaa-local-env build-image

> paste the below content to `requirements/requirement.txt`
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt"
awscli==1.31.13

> ./mwaa-local-env test-requirements

Operating System

docker

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.16.0
apache-airflow-providers-celery==3.5.1
apache-airflow-providers-cncf-kubernetes==7.13.0
apache-airflow-providers-common-io==1.2.0
apache-airflow-providers-common-sql==1.10.0
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-http==4.8.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-mysql==5.10.0
apache-airflow-providers-sqlite==37.0

Deployment

Amazon (AWS) MWAA

Deployment details

No response

Anything else?

awscli related packages:

botocore
docutils
colorama
rsa

reducing the package version of docutils, colorama and rsa did the trick for me.

botocore==1.33.13
docutils==0.16          // docutils version supported by awscli 1.31.13 was docutils<0.17,>=0.10
colorama==0.4.4     // colorama version supported by awscli 1.31.13 was  colorama<0.4.5 and >=0.2.5
rsa==4.7                  // colorama version supported by awscli 1.31.13 was rsa<4.8 and >=3.1.2

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Taragolis]

awscli should be installed via requirements.txt without any version conflict

That is upstream dependency issue and not related to the Apache Airflow itself

aws cli v1 known as "I would turn your life into the hell until you might resolve dependencies". Personal recommendation: swhich to the aws cli v2: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

You also might use https://github.com/apache/airflow/blob/main/scripts/in_container/bin/install_aws.sh as a reference