Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shuffle Worker searching for docker in microk8s Environment #1370

Open
Petix992 opened this issue Apr 18, 2024 · 1 comment
Open

Shuffle Worker searching for docker in microk8s Environment #1370

Petix992 opened this issue Apr 18, 2024 · 1 comment

Comments

@Petix992
Copy link

Describe the bug
Deploying shuffle packeges on a microk8s environment I ran in to a problem where the shuffle-worker pod try to contact Docker, but since Docker is not the present the pod goes in ERROR

To Reproduce
Steps to reproduce the behavior:

  1. Run installation on microk8s environment
    ...

Expected behavior
should not search docker

17:19:27 user@ubuntu1:~$ kubectl -n shuffle logs -f worker-8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d
2024/04/17 17:18:53 [DEBUG] Inside Files Init with org bucket name "shuffle_org_files"
2024/04/17 17:18:53 [DEBUG] Starting with memcached address '' (SHUFFLE_MEMCACHED). If this is empty, fallback to default (appengine / local). Name: 'worker'
2024/04/17 17:18:53 [DEBUG] Ran init for worker to set up cache system. Docker version: 1.40
2024/04/17 17:18:53 [INFO] Setting up worker environment
2024/04/17 17:18:53 [INFO] Running with timezone Europe/Amsterdam and swarm config ""
2024/04/17 17:18:53 [INFO] Running normal execution with auth 9a33b6c6-8585-498c-918c-375c4b60ee32 and ID 8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d
2024/04/17 17:18:53 [DEBUG] Looking for subflow in []string{"f8a44502-e350-4180-933c-f7c3d7e8460b", "240b5c73-72eb-4ff0-b177-1dbf5a3cb854"} to check execution pattern as child of c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4
2024/04/17 17:18:53 [DEBUG] Environments: [Shuffle]. Source: default. 1 env = webserver, 0 or >1 = default. Subflow exists: false
2024/04/17 17:18:53 [DEBUG] Running NON-OPTIMIZED execution for type default with 1 environment(s). This only happens when ran manually OR when running with subflows. Status: EXECUTING
2024/04/17 17:18:53 [INFO][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] shuffle.Actions: 3 + Special shuffle.Triggers: 0
2024/04/17 17:18:53 [INFO][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] shuffle.Actions: 3 + Special shuffle.Triggers: 0
2024/04/17 17:18:53 [DEBUG] DEFAULT EXECUTION Startaction: c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4
2024/04/17 17:18:53 [INFO][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Inside Decide execution with 0 / 3 results (extra: 0). Status: EXECUTING
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Running Change Me (c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4) with 0 parents. Names: []string{}
2024/04/17 17:18:53 [INFO][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Should execute Shuffle Tools:1.2.0 (c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4) with label Change Me
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Action: Send, Label: 'Change Me', Action: 'Shuffle Tools', Run status: EXECUTING, Extra=
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Running app set_cache_value in docker NORMALLY as there is no delay set with identifier Shuffle-Tools_1-2-0_c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4_8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d
2024/04/17 17:18:53 [ERROR] Container CREATE error (1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024/04/17 17:18:53 [DEBUG] Trying to download image frikky/shuffle:shuffle-tools_1.2.0 from backend http://shuffle-backend:5001 as it doesn't exist. All images: []string(nil)
2024/04/17 17:18:53 [ERROR] Docker download for image frikky/shuffle:shuffle-tools_1.2.0 (backend) StatusCode (1): 401
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Running app set_cache_value in docker NORMALLY as there is no delay set with identifier Shuffle-Tools_1-2-0_c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4_8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d
2024/04/17 17:18:53 [ERROR] Container CREATE error (1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024/04/17 17:18:53 [WARNING] Failed to download image frikky/shuffle:shuffle-tools_1.2.0 (CLEANUP): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024/04/17 17:18:53 [ERROR] Failed getting frikky/shuffle:shuffle-tools_1.2.0. Couldn't be find locally, AND is missing.
2024/04/17 17:18:53 [DEBUG] Shutting down (4)
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Shutdown (EXECUTING) started with reason "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?". Result amount: 0. ResultsSent: 0, Send result: true, Parent: ""
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] NEWRESP (from backend): {"success": true, "reason": "success"}
2024/04/17 17:18:53 [WARNING][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Sent shutdown update with 0 results and result value Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024/04/17 17:18:54 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Abort URL: http://shuffle-backend:5001/api/v1/workflows/01b2a418-d8b9-438b-a7a0-99e403f7f60e/executions/8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d/abort?reason=Cannot+connect+to+the+Docker+daemon+at+unix%3A%2F%2F%2Fvar%2Frun%2Fdocker.sock.+Is+the+docker+daemon+running%3F&node=c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4&env=Shuffle
2024/04/17 17:18:54 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Finished shutdown (after 1 seconds).
@frikky
Copy link
Member

frikky commented Apr 21, 2024

Describe the bug Deploying shuffle packeges on a microk8s environment I ran in to a problem where the shuffle-worker pod try to contact Docker, but since Docker is not the present the pod goes in ERROR

To Reproduce Steps to reproduce the behavior:

  1. Run installation on microk8s environment
    ...

Expected behavior should not search docker

17:19:27 user@ubuntu1:~$ kubectl -n shuffle logs -f worker-8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d
2024/04/17 17:18:53 [DEBUG] Inside Files Init with org bucket name "shuffle_org_files"
2024/04/17 17:18:53 [DEBUG] Starting with memcached address '' (SHUFFLE_MEMCACHED). If this is empty, fallback to default (appengine / local). Name: 'worker'
2024/04/17 17:18:53 [DEBUG] Ran init for worker to set up cache system. Docker version: 1.40
2024/04/17 17:18:53 [INFO] Setting up worker environment
2024/04/17 17:18:53 [INFO] Running with timezone Europe/Amsterdam and swarm config ""
2024/04/17 17:18:53 [INFO] Running normal execution with auth 9a33b6c6-8585-498c-918c-375c4b60ee32 and ID 8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d
2024/04/17 17:18:53 [DEBUG] Looking for subflow in []string{"f8a44502-e350-4180-933c-f7c3d7e8460b", "240b5c73-72eb-4ff0-b177-1dbf5a3cb854"} to check execution pattern as child of c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4
2024/04/17 17:18:53 [DEBUG] Environments: [Shuffle]. Source: default. 1 env = webserver, 0 or >1 = default. Subflow exists: false
2024/04/17 17:18:53 [DEBUG] Running NON-OPTIMIZED execution for type default with 1 environment(s). This only happens when ran manually OR when running with subflows. Status: EXECUTING
2024/04/17 17:18:53 [INFO][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] shuffle.Actions: 3 + Special shuffle.Triggers: 0
2024/04/17 17:18:53 [INFO][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] shuffle.Actions: 3 + Special shuffle.Triggers: 0
2024/04/17 17:18:53 [DEBUG] DEFAULT EXECUTION Startaction: c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4
2024/04/17 17:18:53 [INFO][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Inside Decide execution with 0 / 3 results (extra: 0). Status: EXECUTING
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Running Change Me (c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4) with 0 parents. Names: []string{}
2024/04/17 17:18:53 [INFO][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Should execute Shuffle Tools:1.2.0 (c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4) with label Change Me
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Action: Send, Label: 'Change Me', Action: 'Shuffle Tools', Run status: EXECUTING, Extra=
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Running app set_cache_value in docker NORMALLY as there is no delay set with identifier Shuffle-Tools_1-2-0_c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4_8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d
2024/04/17 17:18:53 [ERROR] Container CREATE error (1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024/04/17 17:18:53 [DEBUG] Trying to download image frikky/shuffle:shuffle-tools_1.2.0 from backend http://shuffle-backend:5001 as it doesn't exist. All images: []string(nil)
2024/04/17 17:18:53 [ERROR] Docker download for image frikky/shuffle:shuffle-tools_1.2.0 (backend) StatusCode (1): 401
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Running app set_cache_value in docker NORMALLY as there is no delay set with identifier Shuffle-Tools_1-2-0_c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4_8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d
2024/04/17 17:18:53 [ERROR] Container CREATE error (1): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024/04/17 17:18:53 [WARNING] Failed to download image frikky/shuffle:shuffle-tools_1.2.0 (CLEANUP): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024/04/17 17:18:53 [ERROR] Failed getting frikky/shuffle:shuffle-tools_1.2.0. Couldn't be find locally, AND is missing.
2024/04/17 17:18:53 [DEBUG] Shutting down (4)
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Shutdown (EXECUTING) started with reason "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?". Result amount: 0. ResultsSent: 0, Send result: true, Parent: ""
2024/04/17 17:18:53 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] NEWRESP (from backend): {"success": true, "reason": "success"}
2024/04/17 17:18:53 [WARNING][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Sent shutdown update with 0 results and result value Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024/04/17 17:18:54 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Abort URL: http://shuffle-backend:5001/api/v1/workflows/01b2a418-d8b9-438b-a7a0-99e403f7f60e/executions/8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d/abort?reason=Cannot+connect+to+the+Docker+daemon+at+unix%3A%2F%2F%2Fvar%2Frun%2Fdocker.sock.+Is+the+docker+daemon+running%3F&node=c93c2ce0-e42a-4d30-8a2e-e9adb7ee7cc4&env=Shuffle
2024/04/17 17:18:54 [DEBUG][8ff8b3d3-5b3c-4cf7-af6d-dc53c530140d] Finished shutdown (after 1 seconds).

Hey!

There is an environment variable that is required to run on K8s, as it's for now not auto discovered: IS_KUBERNETES=true

`

Shuffle/.env

Line 67 in 06ac921

IS_KUBERNETES=false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Shuffle Roadmap
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@frikky @Petix992