adding apache airflow docker using "Add Stack"

[eshirvana]

Bug description
I'm trying to install airflow on my docker using docker-compose , when I follow instruction explained on Airflow documenation, I'm able to install and config airflow using cli but I'm not able to deploy the stack using portainer,and I get error as shown below:

Expected behaviour
A clear and concise description of what you expected to happen.

Portainer Logs

level=info msg="2021/11/14 00:44:34 http error: failed to deploy a stack: The Compose file './docker-compose.yml' is invalid because:"
level=info msg="Unsupported config option for services.airflow-cli: 'profiles'"
level=info msg="services.airflow-scheduler.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-triggerer.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-webserver.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-worker.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.flower.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg=": exit status 1 (err=failed to deploy a stack: The Compose file './docker-compose.yml' is invalid because:"
level=info msg="Unsupported config option for services.airflow-cli: 'profiles'"
level=info msg="services.airflow-scheduler.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-triggerer.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-webserver.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-worker.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.flower.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg=": exit status 1) (code=500)"

Steps to reproduce the issue:

1. Go to 'Stacks' click on 'Add Stack', click on 'Upload'
2. upload file downloaded from : 'https://airflow.apache.org/docs/apache-airflow/2.2.1/docker-compose.yaml'
3. Click on "Add an Environment variable" : Name : "AIRFLOW_UID" , Value = "1000"
4. Click on "Deploy the stack"
   Technical details:

• Portainer version: 2.9.2
• Docker version (managed by Portainer): 20.10.10 (API: 1.41)
• Docker-compose version: 1.29.2, build 5becea4c
• Kubernetes version (managed by Portainer):
• Platform (windows/linux): linux x86_64 Ubuntu 20.04.3 LTS
• Browser: Firefox 94
• Use Case (delete as appropriate): Using Portainer at Home
• Have you reviewed our technical documentation and knowledge base? Yes

[jamescarppe]

Portainer doesn't currently support docker-compose profiles, so I'd suggest either removing the profiles section from the service definition and trying again, or removing the airflow-cli service entirely (as the profile implies you should only be running it when debugging).

We have a feature request - #5796 - open for adding profile support and you can add your support for the feature there.

[eshirvana]

Thank you @jamescarppe , I removed the airflow-cli section totally , but still getting error:

level=info msg=": exit status 1 (err=failed to deploy a stack: The Compose file './docker-compose.yml' is invalid because:"
level=info msg="services.airflow-scheduler.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-triggerer.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-webserver.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.airflow-worker.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg="services.flower.depends_on.airflow-init.condition contains \"service_completed_successfully\", which is an invalid type, it should be a service_started, or a service_healthy"
level=info msg=": exit status 1) (code=500)"

[jamescarppe]

As the error indicates, service_completed_successfully isn't a valid option according to the version of compose-cli that Portainer uses. It's likely this will be resolved with an update to compose-cli which we are looking into.

In the meantime, you could try replacing service_completed_successfully with service_healthy.

[huib-portainer]

Compose v2 mentions service_started and service_healthy:
https://docs.docker.com/compose/compose-file/compose-file-v2/#depends_on

But v3 doesn't even have those any more:
https://docs.docker.com/compose/compose-file/compose-file-v3/#depends_on
docker/compose#4305

And looks like Docker Compose never implemented service_completed_successfully from the Compose Spec anyway.

[eshirvana]

@huib-portainer I'm no expert , but when I run my stack over command line , It runs successfully with no error messages,

> docker-compose up airflow-init

@jamescarppe and even if I replace it with service_healthy , I see the added the stack to the list and cretaed couple of the containers and it fails in middle of the process with these errors:

level=info msg="2021/11/16 16:35:48 http error: failed to deploy a stack: Creating network \"apache-airflow_default\" with the default driver"
level=info msg="Creating apache-airflow_redis_1 ... "
level=info msg="Creating apache-airflow_postgres_1 ... "
level=info msg="\x1b[2A\x1b[2K\rCreating apache-airflow_redis_1    ... \x1b[32mdone\x1b[0m\r\x1b[2B\x1b[1A\x1b[2K\rCreating apache-airflow_postgres_1 ... \x1b[32mdone\x1b[0m\r\x1b[1BCreating apache-airflow_airflow-init_1 ... "
level=info msg="Creating apache-airflow_airflow-cli_1  ... "
level=info msg="\x1b[2A\x1b[2K\rCreating apache-airflow_airflow-init_1 ... \x1b[32mdone\x1b[0m\r\x1b[2B\x1b[1A\x1b[2K\rCreating apache-airflow_airflow-cli_1  ... \x1b[32mdone\x1b[0m\r\x1b[1B"
level=info msg="ERROR: for airflow-worker  Service \"airflow-init\" is missing a healthcheck configuration"
level=info
level=info msg="ERROR: for airflow-scheduler  Service \"airflow-init\" is missing a healthcheck configuration"
level=info
level=info msg="ERROR: for airflow-triggerer  Service \"airflow-init\" is missing a healthcheck configuration"
level=info
level=info msg="ERROR: for flower  Service \"airflow-init\" is missing a healthcheck configuration"
level=info
level=info msg="ERROR: for airflow-webserver  Service \"airflow-init\" is missing a healthcheck configuration"
level=info msg="Encountered errors while bringing up the project."
level=info msg=": exit status 1 (err=failed to deploy a stack: Creating network \"apache-airflow_default\" with the default driver"
level=info msg="Creating apache-airflow_redis_1 ... "
level=info msg="Creating apache-airflow_postgres_1 ... "
level=info msg="\x1b[2A\x1b[2K\rCreating apache-airflow_redis_1    ... \x1b[32mdone\x1b[0m\r\x1b[2B\x1b[1A\x1b[2K\rCreating apache-airflow_postgres_1 ... \x1b[32mdone\x1b[0m\r\x1b[1BCreating apache-airflow_airflow-init_1 ... "
level=info msg="Creating apache-airflow_airflow-cli_1  ... "
level=info msg="\x1b[2A\x1b[2K\rCreating apache-airflow_airflow-init_1 ... \x1b[32mdone\x1b[0m\r\x1b[2B\x1b[1A\x1b[2K\rCreating apache-airflow_airflow-cli_1  ... \x1b[32mdone\x1b[0m\r\x1b[1B"
level=info msg="ERROR: for airflow-worker  Service \"airflow-init\" is missing a healthcheck configuration"
level=info
level=info msg="ERROR: for airflow-scheduler  Service \"airflow-init\" is missing a healthcheck configuration"
level=info
level=info msg="ERROR: for airflow-triggerer  Service \"airflow-init\" is missing a healthcheck configuration"
level=info
level=info msg="ERROR: for flower  Service \"airflow-init\" is missing a healthcheck configuration"
level=info
level=info msg="ERROR: for airflow-webserver  Service \"airflow-init\" is missing a healthcheck configuration"
level=info msg="Encountered errors while bringing up the project."
level=info msg=": exit status 1) (code=500)"

[jamescarppe]

Ah yep, good catch - service_healthy assumes a healthcheck. Sorry about that. Perhaps try service_started instead.

@huib-portainer can provide more specifics on this, but to provide for some of the cross-platform functionality that Portainer includes, we use our own copy of compose-cli within Portainer, which can mean we are out of sync with functionality available in the latest version of docker-compose on your local system.

[huib-portainer]

Portainer is currently using compose-cli v1.0.17
And we are considering upgrading to v1.0.18 (#5799)
But I don't know if that will make any difference.

[eshirvana]

@jamescarppe , @huib-portainer
eventually by removing profile section and replacing service_completed_successfully with service_started .
but that would be nice if you could update the builtin compose version

[Mehdi-YC]

hello , im trying to build my own templates file and i fon't know if portainer supports apache smaprflow compose file right now
if someone managed to make it please tell me how , or if you jave your own templates file containing it mmlonk it please

[baflo]

I added a new issue on the Docker clients problem: #6720

[shawnngtq]

Using v2.13.1 to install airflow, same steps as above. Would also love to see compose v3 works 😁

[eshirvana]

Compose v2 mentions service_started and service_healthy: https://docs.docker.com/compose/compose-file/compose-file-v2/#depends_on

But v3 doesn't even have those any more: https://docs.docker.com/compose/compose-file/compose-file-v3/#depends_on docker/compose#4305

And looks like Docker Compose never implemented service_completed_successfully from the Compose Spec anyway.

@huib-portainer I see service_completed_successfully has been mentioned in compose file reference documentation under depends_on -> long syntax section:

https://docs.docker.com/compose/compose-file/#depends_on

and I see docker Compose file1.x is deprecated, so any plan to upgrade the built-in compose to at least v 2.x or 3.x??