Airflow task initdb fails

[ClaraBuettner]

I installed eGon-data as you desctibed it yesterday and tried to run the DAG. But it fails with this log:

[2020-10-22 17:04:36,647] {taskinstance.py:901} INFO - Executing <Task(PythonOperator): initdb> on 2020-10-22T14:52:52.202105+00:00
[2020-10-22 17:04:36,649] {standard_task_runner.py:54} INFO - Started process 26382 to run task
[2020-10-22 17:04:36,665] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'egon-data-processing-pipeline', 'initdb', '2020-10-22T14:52:52.202105+00:00', '--job_id', '3', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/pipeline.py', '--cfg_path', '/tmp/tmpsibdxiuj']
[2020-10-22 17:04:36,666] {standard_task_runner.py:78} INFO - Job 3: Subtask initdb
[2020-10-22 17:04:36,704] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: egon-data-processing-pipeline.initdb 2020-10-22T14:52:52.202105+00:00 [running]> clara-LIFEBOOK-U749
[2020-10-22 17:04:36,717] {taskinstance.py:1150} ERROR - [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'
Traceback (most recent call last):
File "/home/clara/egon_data_env/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/clara/egon_data_env/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
File "/home/clara/egon_data_env/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/clara/GitHub/eGon-data/src/egon/data/airflow/tasks.py", line 9, in initdb
cwd=os.path.dirname(file),
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 729, in init
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'docker-compose': 'docker-compose'
[2020-10-22 17:04:36,719] {taskinstance.py:1194} INFO - Marking task as FAILED. dag_id=egon-data-processing-pipeline, task_id=initdb, execution_date=20201022T145252, start_date=20201022T150436, end_date=20201022T150436
[2020-10-22 17:04:41,627] {local_task_job.py:102} INFO - Task exited with return code 1

I just installed egon-data with pip. Do I need to install something else manually (maybe docker)?

[gnn]

That is correct. You need to have Docker installed and the Docker daemon, dockerd, must be running. On my system, i.e. Ubuntu 20.04, dockerd is provided by the docker.io package in the standard package sources, but the Docker maintainers also provide custom packages and installation instructions for various systems. This is definitely something we should put into the documentation.
If you'd rather not use Docker and use your own persistent local database, don't hesitate to open a feature request, so others can +1 it and we can gauge interest in this feature. :)

[ClaraBuettner]

Thank you. Now I installed docker.io and run dockerd parallel to the egon-data serve process. But now the DAG is running since 8 o'clock in the morning and doesn't stop. So I think I'm still doing something wrong... Do you have an idea?

[gplssm]

Could you please try one thing? Go into src/egon/data/airflow and execute

docker-compose up -d --build

Maybe also try it without --build.

This does the same as the function initdb() which is the first task called in the pipeline. It should set up a Postgres DB in a docker container. Verify with docker ps that the container is running.

[ClaraBuettner]

I get this error:

ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

When I run sudo docker-compose up -d --build it works fine and a db is set up. So I did something wrong when installing docker, and can use it only as sudo user. Sorry, it has noting to do with the repo but with my bad installation. I try to fix this.

[gplssm]

Seems to be a problem with permissions.

The docker documentation gives instructions on how to add your user to the docker group for allowing to use docker-compose without sudo.
https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

If this doesn't help, here's more discussion along the error message:
docker/compose#4181

And please have no worries, asking these questions is totally fine. It might help others a we should preserved the essence in the docs.

[ClaraBuettner]

Seems to be a problem with permissions.

The docker documentation gives instructions on how to add your user to the docker group for allowing to use docker-compose without sudo.
https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

Yes, I already did this and it still doesn't work.

If this doesn't help, here's more discussion along the error message:
docker/compose#4181

And please have no worries, asking these questions is totally fine. It might help others a we should preserved the essence in the docs.

Thank you. I will try to find a solution with the help of the issue and hopefully post my solution here soon.

[ClaraBuettner]

Ok, so I re-installed docker (and added my user to the docker group) and created a new egon-data environement. Now I can run docker-compose up -d --build from the terminal.
But when I trigger the DAG via the airflow interface, it's still not working. The dag doesn't stop running and in my terminal I get repeating prints like this:

[2020-10-30 08:56:58 +0100] [6151] [INFO] Handling signal: ttou
[2020-10-30 08:56:58 +0100] [6265] [INFO] Worker exiting (pid: 6265)
[2020-10-30 08:57:27 +0100] [6151] [INFO] Handling signal: ttin
[2020-10-30 08:57:27 +0100] [6437] [INFO] Booting worker with pid: 6437

[gnn]

How did you open the Airflow interface? Did you start up Airflow manually or via egon-data serve?

[ClaraBuettner]

I started it via egon-data serve in my egon-data-environment

[ClaraBuettner]

I re-installed egon-data in a new environment and now I can run the dag via the airflow interface.
So I don't really know what has caused the error, but a new installation solved it.

[gplssm]

@ClaraBuettner if you're happy with the result and the issue is resolved for, please close it. Re-opening is possible for all of us if the error appears again.

[IlkaCu]

Ok, so I re-installed docker (and added my user to the docker group) and created a new egon-data environement. Now I can run docker-compose up -d --build from the terminal.
But when I trigger the DAG via the airflow interface, it's still not working. The dag doesn't stop running and in my terminal I get repeating prints like this:

[2020-10-30 08:56:58 +0100] [6151] [INFO] Handling signal: ttou
[2020-10-30 08:56:58 +0100] [6265] [INFO] Worker exiting (pid: 6265)
[2020-10-30 08:57:27 +0100] [6151] [INFO] Handling signal: ttin
[2020-10-30 08:57:27 +0100] [6437] [INFO] Booting worker with pid: 6437

I am experiencing exactly the same. Re-installing egon-data in a fresh environment didn't help.

[gnn]

There's an, admittedly not to prominent, On/Off button in the second column of the DAGs table in the Airflow webinterface. I've attached a picture with a red arrow pointing to it. While the button is in the "Off" state, no tasks in a DAG will run, regardless of how often it is triggered. Maybe that fixes the problem?

[gplssm]

There's an, admittedly not to prominent, On/Off button in the second column of the DAGs table in the Airflow webinterface. I've attached a picture with a red arrow pointing to it. While the button is in the "Off" state, no tasks in a DAG will run, regardless of how often it is triggered. Maybe that fixes the problem?

How embarrassing! 🤦‍♂️

Many thanks for finding that!

[IlkaCu]

How embarrassing! man_facepalming

You said it!

Maybe that fixes the problem?

And yes, it fixes the problem. @gnn: Thanks a lot.