No LocalWorkers started when scheduler is launched in daemon mode. #39385
Replies: 4 comments 1 reply
-
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
-
The documented pattern for running Airflow in Docker can be found in the Airflow guide Running Airflow in Docker. The docker compose file is a bit more extensive than the one you've shared. Can you try following that guide? You can run everything in detached mode with |
Beta Was this translation helpful? Give feedback.
-
Thanks Ryan. I've worked around this issue by starting the scheduler last. I'm actually using AWS ECS and only made the compose file to debug the issue. I thought I would submit a bug as it did waste a couple hours of my time... and I thought it was time I try and get more involved with an open source project. Again: thanks for your help. |
Beta Was this translation helpful? Give feedback.
-
Run Airflow services in daemon it is useful only on case if there is not possible for some reason to use external daemons (e.g. systemd) or containers. In case of containers it's best practice to separate areas of concern by using one service per container.
Even if you use cloud based container service the principal pretty the same, 1 service = 1 task. |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
2.9.0
If "Other Airflow 2 version" selected, which one?
No response
What happened?
LocalExecutor workers aren't started when launching the airflow scheduler in daemon mode. I get this when running
apache/airflow:latest
docker image. Additionally, the Web UI says the scheduler isn't running and Jobs aren't started.What you think should happen instead?
I would expect the same scheduler behavior whether launched as a deamon or not.
How to reproduce
compose.yml.txt
apt-get update && apt-get install -y procps
ps -ef
and note that none of the LocalExecutor workers are running.I can workaround this issue by starting the webserver first and the scheduler last -- not in daemon mode.
Operating System
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
Versions of Apache Airflow Providers
N/A
Deployment
Other Docker-based deployment
Deployment details
Docker Compose version v2.26.1-desktop.1
Also observed run AWS Elastic Container Service.
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions