Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local runner fails to bind port of dask dashboard #870

Closed
mrchtr opened this issue Feb 21, 2024 · 4 comments · Fixed by #872
Closed

Local runner fails to bind port of dask dashboard #870

mrchtr opened this issue Feb 21, 2024 · 4 comments · Fixed by #872
Labels
bug Something isn't working

Comments

@mrchtr
Copy link
Contributor

mrchtr commented Feb 21, 2024

Encountered a bug with:

  • fondant 0.11dev1
  • Docker version 25.0.1, build 29cf629
  • Docker Compose version v2.24.2

The Docker container cannot be started accordingly because during the start of a component, the port of the Dask dashboard is still allocated from the previous completed component.

E.g.

Error response from daemon: driver failed programming external connectivity on endpoint controlnet-pipeline-write_to_file-1 (cc30d244e30a23e8d565000b34c48bbc0e9444b8f706a3b4ecf22db073e7abfe): Bind for 0.0.0.0:8787 failed: port is already allocated

Temporary work around could be to disable the dask dashboard port binding, since it isn't used really at the moment.
@mrchtr mrchtr added the bug Something isn't working label Feb 21, 2024
@mrchtr mrchtr mentioned this issue Feb 22, 2024
Merged
@RobbeSneyders
Copy link
Member

Did you run into this issue yourself? Can you check that your environment is clean and you don't have any detached services still active?

Also, we might not have to move the port completely, but just define it differently: docker/compose#4950

@RobbeSneyders
Copy link
Member

FYI, I can't reproduce this myself.

@mrchtr
Copy link
Contributor Author

mrchtr commented Feb 22, 2024

Did you run into this issue yourself? Can you check that your environment is clean and you don't have any detached services still active?

Yes. This was my first thought too. I run into it on my local machine. Later I run into the same issue using a vertexai notebook. Also it seems to be occurring randomly. Maybe related to speed of the execution.

@RobbeSneyders
Copy link
Member

If it happens and you re-run the pipeline, do you then already get the error from the first step again?

RobbeSneyders added a commit that referenced this issue Mar 6, 2024
@mrchtr @RobbeSneyders
Temporarily disabled exposing the Dask diagnostic dashboard (#872)
ec4f730 
Temporarily disabled exposing the Dask diagnostic dashboard port to the
docker setup.
Executing docker containers after each other using docker compose, leads
sometimes to the case that the port is already in use.

We should come up with a solution if we want to use the dask dashboard
again.

Fix: #870

---------

Co-authored-by: Robbe Sneyders <robbe.sneyders@ml6.eu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Fondant development
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Temporarily disabled exposing the Dask diagnostic dashboard ml6team/fondant
2 participants
@mrchtr @RobbeSneyders