ksync init cannot connect to docker daemon #517

tetchel · 2021-03-29T15:03:42Z

hi, I am trying to run ksync init to set up ksync on a CodeReady Containers cluster running locally on my Fedora machine

ksync init failed with this message:

[ /redhat-actions/openshift-actions-connector/containerize ] 47 (main) $ ksync init
==== Preflight checks ====

==== Cluster Environment ====

==== Postflight checks ====
↳	rpc error: code = Unknown desc = Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version": dial unix /var/run/docker.sock: connect: permission denied
FATA[0001]

Adding sudo results in the same error message.

dockerd is running and i can run all docker commands as my current user since I am part of the docker group.

Thanks for any help

[ /redhat-actions/openshift-actions-connector/containerize ] 06 (main) $ ksync version
ksync:
	Version:    Release
	Go Version: go1.16.2
	Git Commit: 14ec9e2
	Git Tag:    0.4.7-hotfix
	Built:      Wed Mar 24 22:04:50 +0000 2021
	OS/Arch:    linux/amd64
service:
	Version:    Release
	Go Version: go1.16.2
	Git Commit: 14ec9e2
	Git Tag:    0.4.7-hotfix
	Built:      Wed Mar 24 22:08:03 +0000 2021
[ /redhat-actions/openshift-actions-connector/containerize ] 06 (main) $ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 14
  Running: 0
  Paused: 0
  Stopped: 14
 Images: 616
 Server Version: 20.10.5
 Storage Driver: btrfs
  Build Version: Btrfs v5.10 
  Library Version: 102
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.23-200.fc33.x86_64
 Operating System: Fedora 33 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.11GiB
 Name: tims-fedora
 ID: 2TP6:X5QS:VJ36:ZABX:GMIV:J6R3:Q5DW:44JZ:EGSR:MR3G:R3IZ:G42Z
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Support for cgroup v2 is experimental

The text was updated successfully, but these errors were encountered:

tetchel · 2021-03-29T15:04:33Z

I see #289 but it seems closed without resolution

timfallmk · 2021-03-30T17:54:16Z

@tetchel Looks like you're using BTRFs as a storage driver. Unfortunately ksync doesn't support that right now.

tetchel · 2021-04-07T21:13:38Z

thanks for the response. I switched to overlay2 storage driver and am still getting the same result.

[ /src/redhat-actions/openshift-actions-connector ] 08 (main) $ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 59
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.23-200.fc33.x86_64
 Operating System: Fedora 33 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.11GiB
 Name: tims-fedora
 ID: 2TP6:X5QS:VJ36:ZABX:GMIV:J6R3:Q5DW:44JZ:EGSR:MR3G:R3IZ:G42Z
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: tetchel
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Support for cgroup v2 is experimental

[ /src/redhat-actions/openshift-actions-connector ] 12 (main) $ sudo ksync init
[sudo] password for tim: 
==== Preflight checks ====

==== Cluster Environment ====

==== Postflight checks ====
↳	rpc error: code = Unknown desc = Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version": dial unix /var/run/docker.sock: connect: permission denied
FATA[0001]

timfallmk · 2021-04-08T18:36:06Z

That would indicate the local docker daemon isn't available. Is there some special configuration for redhat when running the daemon? Maybe check your local docker client setup to see how it connects.

tetchel · 2021-04-08T18:41:38Z

I am just on a regular Fedora distribution. Docker is definitely running as I can build images, run docker ps, etc.

/run/docker.sock is there as expected, I'm not sure what else I would check.

timfallmk · 2021-04-08T19:19:15Z

Well that would do it 😄 . The trace above shows it’s looking for the socket at /var/run/docker.sock, which is the default for most platforms. I vaguely remember RedHat doing something different.

There’s a ksync config option to give a path to the Docker socket for this reason. I can’t look it up at the moment, as I’m not at my machine, but it should™️ be in the documentation.

tetchel · 2021-04-08T19:22:17Z

sorry, my mistake, it's in both places

[ /src/redhat-actions/openshift-actions-connector ] 21 (main) $ ls -l /var/run/docker.sock
srw-rw----. 1 root docker 0 Apr  2 16:49 /var/run/docker.sock
[ /src/redhat-actions/openshift-actions-connector ] 21 (main) $ sudo ls -l /run/docker.sock
[sudo] password for tim: 
srw-rw----. 1 root docker 0 Apr  2 16:49 /run/docker.sock
[ /src/redhat-actions/openshift-actions-connector ] 22 (main) $

timfallmk · 2021-04-08T19:26:28Z

Hmm. There’s still an itch in the back of my brain about RH/Fedora releases doing something different here. Btw, I’m not sure it’s actually documented (oops), but here’s the relevant flag

ksync/cmd/ksync/ksync.go

Line 148 in 87c8aa0

flags.String(

SELinux issue @grampelberg ?

timfallmk · 2021-04-08T19:28:07Z

Two things to try:

If you haven’t, run the doctor command.
Try both that and init as root and see if there’s a difference

tetchel · 2021-04-15T16:55:09Z

It has the same problem

[ /tim/downloads/crc-linux-1.25.0-amd64 ] 53 $ ksync doctor
↳	rpc error: code = Unknown desc = Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version": dial unix /var/run/docker.sock: connect: permission denied
[ /tim/downloads/crc-linux-1.25.0-amd64 ] 53 $ sudo ksync doctor
↳	rpc error: code = Unknown desc = Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version": dial unix /var/run/docker.sock: connect: permission denied

I tried some other stuff like sudo -i and chmod 666 /var/run/docker.sock and still somehow get 'permission denied'

w/ debug output:

[ /tim/downloads/crc-linux-1.25.0-amd64 ] 00 $ sudo ksync doctor --log-level=debug
Extra Binaries                              ✓
DEBU[0000] initializing kubernetes client                context=
DEBU[0000] kubernetes client created                     host="https://api.crc.testing:6443"
Cluster Config                              ✓
Cluster Connection                          ✓
Cluster Version                             ✓
Cluster Permissions                         ✓
Cluster Service                             ✓
DEBU[0000] radar nodes                                   count=1
DEBU[0000] checking to see if radar is ready             nodeName=crc-xl2km-master-0
DEBU[0000] found pod name                                Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=crc-xl2km-master-0 podName=ksync-nfpmr
DEBU[0000] found pod                                     nodeName=crc-xl2km-master-0 podName=ksync-nfpmr status=Running
Service Health                              ✓
DEBU[0000] radar nodes                                   count=1
DEBU[0000] checking to see if radar is ready             nodeName=crc-xl2km-master-0
DEBU[0000] found pod name                                Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=crc-xl2km-master-0 podName=ksync-nfpmr
DEBU[0000] found pod                                     nodeName=crc-xl2km-master-0 podName=ksync-nfpmr status=Running
DEBU[0000] checking to see if radar is ready             nodeName=crc-xl2km-master-0
DEBU[0000] found pod name                                Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=crc-xl2km-master-0 podName=ksync-nfpmr
DEBU[0000] found pod                                     nodeName=crc-xl2km-master-0 podName=ksync-nfpmr status=Running
DEBU[0000] starting tunnel                               LocalPort=46315 Namespace=kube-system Out= PodName=ksync-nfpmr RemotePort=40321 url="https://api.crc.testing:6443/api/v1/namespaces/kube-system/pods/ksync-nfpmr/portforward"
DEBU[0000] tunnel running                                LocalPort=46315 Namespace=kube-system Out="Forwarding from 127.0.0.1:46315 -> 40321\nForwarding from [::1]:46315 -> 40321\n" PodName=ksync-nfpmr RemotePort=40321
Service Version                             ✓
DEBU[0000] radar nodes                                   count=1
DEBU[0000] checking to see if radar is ready             nodeName=crc-xl2km-master-0
DEBU[0000] found pod name                                Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=crc-xl2km-master-0 podName=ksync-nfpmr
DEBU[0000] found pod                                     nodeName=crc-xl2km-master-0 podName=ksync-nfpmr status=Running
DEBU[0000] starting tunnel                               LocalPort=45559 Namespace=kube-system Out= PodName=ksync-nfpmr RemotePort=40321 url="https://api.crc.testing:6443/api/v1/namespaces/kube-system/pods/ksync-nfpmr/portforward"
DEBU[0000] tunnel running                                LocalPort=45559 Namespace=kube-system Out="Forwarding from 127.0.0.1:45559 -> 40321\nForwarding from [::1]:45559 -> 40321\n" PodName=ksync-nfpmr RemotePort=40321
Docker Version                              ✘
↳	rpc error: code = Unknown desc = Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/version": dial unix /var/run/docker.sock: connect: permission denied

timfallmk · 2021-04-15T19:02:52Z

I'm just shooting in the dark now, but does your docker startup config (wherever that is on Fedora) have anything in it that would be a non-standard setting? It sounds like there's something Fedora does that's non-standard but I don't know what.

timfallmk · 2021-04-15T19:13:32Z

Something SELinux related? https://stackoverflow.com/questions/63215054/portainer-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-so

Edit: a grammar

zawias-pro · 2021-05-26T16:47:05Z

I have the same issue on my Linux mint. (Ubuntu derivative)

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.7.0)

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 99
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-73-generic
 Operating System: Linux Mint 20.1
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.13GiB
 Name: ***
 ID: ***
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: ***
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

mrtimp · 2021-07-16T07:08:48Z

I'm not sure if this helps or not but I ran into a similar issue - not 100% the same as the one described here (maybe closer to #289). I'm currently using microk8s on macOS Big Sur (microk8s is installed/configured/managed via multipass and is a VirtualBox VM). After a bit of debugging it turned out that the rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? error that I was getting was because the ksync pod couldn't access the Docker socket - which makes total sense because in my situation Docker is running on macOS not inside the VirtualBox VM.

What I did to work around this until I can come up with a better solution was run two socat instances, the first on macOS exposing /var/run/docker.sock as a TCP service:

socat -d TCP-LISTEN:17443,reuseaddr,fork UNIX-CLIENT:/var/run/docker.sock

then another inside the VM connecting to the TCP service on macOS and recreating the socket in /var/run/docker.sock:

sudo socat -d UNIX-LISTEN:/var/run/docker.sock,reuseaddr,fork TCP:[macOS host IP]:17443

This is a total hack for now but I just needed to determine where the issue was.

I first attempted to share /var/run/docker.sock into the VM but due to it being a unix socket it looses access to the kernel inside the VM and becomes a file - socat to the rescue ;-)

timfallmk · 2021-07-30T23:07:18Z

Very creative!

…

On Fri, Jul 16, 2021 at 12:08 AM Tim Philips ***@***.***> wrote: I'm not sure if this helps or not but I ran into a similar issue - not 100% the same as the one described here. I'm currently using microk8s on macOS Big Sur (microk8s is installed/configured/managed via multipass and is a VirtualBox VM). After a bit of debugging it turned out that the rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? error that I was getting was because the ksync pod couldn't access the Docker socket - which makes total sense because in my situation Docker is running on macOS not inside the VirtualBox VM. What I did to work around this until I can come up with a better solution was run two socat instances, the first on macOS exposing /var/run/docker.sock as a TCP service: socat -d TCP-LISTEN:17443,reuseaddr,fork UNIX-CLIENT:/var/run/docker.sock then another inside the VM connecting to the TCP service on macOS and recreating the socket in /var/run/docker.sock: sudo socat -d UNIX-LISTEN:/var/run/docker.sock,reuseaddr,fork TCP:[macOS host IP]:17443 This is a total hack for now but I just needed to determine where the issue was. I first attempted to share /var/run/docker.sock into the VM but due to it being a unix socket it looses access to the kernel inside the VM and becomes a file - socat to the rescue ;-) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#517 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIR4FOSTD3MDLHZTUCOUQ3TX7LIXANCNFSM4Z7YNQOQ> .

kaiivoschneider · 2022-06-29T09:51:55Z

Alright it's been a year. Where is the support.

timfallmk · 2022-07-01T21:48:06Z

@kaiivoschneider Was there something in particular you were looking for. In general this project is largely defunct and both @grampelberg and myself have moved on to other things. I do my best to answer what I can, but there won't be any significant work on the project.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ksync init cannot connect to docker daemon #517

ksync init cannot connect to docker daemon #517

tetchel commented Mar 29, 2021 •

edited

tetchel commented Mar 29, 2021 •

edited

timfallmk commented Mar 30, 2021

tetchel commented Apr 7, 2021 •

edited

timfallmk commented Apr 8, 2021

tetchel commented Apr 8, 2021

timfallmk commented Apr 8, 2021

tetchel commented Apr 8, 2021

timfallmk commented Apr 8, 2021

timfallmk commented Apr 8, 2021 •

edited

tetchel commented Apr 15, 2021 •

edited

timfallmk commented Apr 15, 2021

timfallmk commented Apr 15, 2021 •

edited

zawias-pro commented May 26, 2021 •

edited

mrtimp commented Jul 16, 2021 •

edited

timfallmk commented Jul 30, 2021 via email

kaiivoschneider commented Jun 29, 2022

timfallmk commented Jul 1, 2022

ksync init cannot connect to docker daemon #517

ksync init cannot connect to docker daemon #517

Comments

tetchel commented Mar 29, 2021 • edited

tetchel commented Mar 29, 2021 • edited

timfallmk commented Mar 30, 2021

tetchel commented Apr 7, 2021 • edited

timfallmk commented Apr 8, 2021

tetchel commented Apr 8, 2021

timfallmk commented Apr 8, 2021

tetchel commented Apr 8, 2021

timfallmk commented Apr 8, 2021

timfallmk commented Apr 8, 2021 • edited

tetchel commented Apr 15, 2021 • edited

timfallmk commented Apr 15, 2021

timfallmk commented Apr 15, 2021 • edited

zawias-pro commented May 26, 2021 • edited

mrtimp commented Jul 16, 2021 • edited

timfallmk commented Jul 30, 2021 via email

kaiivoschneider commented Jun 29, 2022

timfallmk commented Jul 1, 2022

tetchel commented Mar 29, 2021 •

edited

tetchel commented Mar 29, 2021 •

edited

tetchel commented Apr 7, 2021 •

edited

timfallmk commented Apr 8, 2021 •

edited

tetchel commented Apr 15, 2021 •

edited

timfallmk commented Apr 15, 2021 •

edited

zawias-pro commented May 26, 2021 •

edited

mrtimp commented Jul 16, 2021 •

edited