Could not connect to Ryuk - Linux, Docker 20.10.0, 1.15.0-rc2

[ullrichp]

As this is my first issue i would like to say "Thank you" to all maintainers and contributors.

Apparently, out of nothing my tests started to throw:

Caused by: java.lang.IllegalStateException: Could not connect to Ryuk at localhost:49160
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:171)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:201)
	at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
	at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
	at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
	at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
	at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:65)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:26)
	at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
	at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1280)
	... 266 more

I am on Linux with:
openjdk 11.0.8 2020-07-14
testcontainers 1.15.0-rc2
docker 20.10.0

The container seems to start:

$ docker ps -a
CONTAINER ID   IMAGE                       COMMAND   CREATED        STATUS                  PORTS                     NAMES
85335add1def   testcontainers/ryuk:0.3.0   "/app"    1 second ago   Up Less than a second   0.0.0.0:49160->8080/tcp   testcontainers-ryuk-2b08ab75-8559-48fc-8f73-8d233b45c430

and the logs do not seem too wrong:

$ docker logs 85335add1def
2020/12/09 09:47:37 Pinging Docker...
2020/12/09 09:47:37 Docker daemon is available!
2020/12/09 09:47:37 Starting on port 8080...
2020/12/09 09:47:37 Started!

Still, the above exception happens after the default timeout of 30 seconds.

Let me know if you need more information, stack traces or logs.

[bsideup]

Hi @ullrichp,

Thanks for your kind words! :)

Have you tried the latest release (1.15.0)? Also, does it work if you attempt to connect to Ryuk from CLI?

[arhohuttunen]

Experiencing the same out of the blue when executing in EC2, this was still working this morning. The stack trace has this:

Caused by:
    org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:11.8-alpine, imagePullPolicy=DefaultPullPolicy())
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1278)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:612)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:317)
    ... 114 more
        Caused by:
        com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"}

But looking at the Docker Hub that image does exists there.

[bsideup]

@arhohuttunen you issue seems to be different and caused by the registry returning 404 for an image that exists.

[arhohuttunen]

My mistake, was too hasty to draw conclusions between the two issues.

[swierszczu4j]

I have exactly the same issue on latest (1.15.0). What is more seems to be not-deterministic one :/ Anyway seems like they are some compatibility issues with the newest docker.

[amelnikov-mylivn]

Experiencing the same out of the blue when executing in EC2, this was still working this morning. The stack trace has this:

Caused by:
    org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:11.8-alpine, imagePullPolicy=DefaultPullPolicy())
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1278)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:612)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:317)
    ... 114 more
        Caused by:
        com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"}

But looking at the Docker Hub that image does exists there.

We had the same issue today. Temporary fixed by pinning docker version to 19.03.

[jdelucaa]

@amelnikov-mylivn what does it have to do with the docker version? have they changed anything in the latest version?

[KyleAure]

I am also seeing the same issue with the stack trace:

[12/09/2020 11:09:31:957 CST] 040 ResourceReaper                 lambda$null$1                  W Can not connect to Ryuk at localhost:32769
java.net.SocketException: Protocol wrong type for socket (Write failed)
	at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
	at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:138)
	at org.testcontainers.utility.ResourceReaper$FilterRegistry.register(ResourceReaper.java:406)
	at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:135)
	at org.testcontainers.utility.ResourceReaper$$Lambda$131/000000002833D020.run(Unknown Source)
	at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
	at org.testcontainers.utility.ResourceReaper$$Lambda$130/00000000279F0A20.run(Unknown Source)
	at java.base/java.lang.Thread.run(Thread.java:834)

[croudet]

Also got com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"

Looks like related to docker version:
Working version:

13:02:19.695 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
13:02:19.718 [main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
13:02:19.725 [main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor)
13:02:19.735 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
13:02:19.764 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /usr/lib/jvm/java-1.8-openjdk/jre/bin, /usr/lib/jvm/java-1.8-openjdk/bin])
13:02:19.764 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - $XDG_RUNTIME_DIR is not set.
13:02:19.765 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/root/.docker/run' does not exist.
13:02:19.807 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/run/user/0' does not exist.
13:02:20.330 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
13:02:20.352 [ducttape-0] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.658 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
13:02:20.659 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Transport type: 'okhttp', Docker host: 'unix:///var/run/docker.sock'
13:02:20.659 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Checking Docker OS type for local Unix socket (unix:///var/run/docker.sock)
13:02:20.668 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: bridge
13:02:20.684 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
13:02:20.686 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.700 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.718 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 19.03.14
  API Version: 1.40
  Operating System: Ubuntu 16.04.4 LTS
  Total Memory: 31907 MB
13:02:20.718 [main] DEBUG org.testcontainers.DockerClientFactory - Ryuk is enabled
13:02:20.721 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for testcontainers/ryuk:0.3.0 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
13:02:20.724 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=testcontainers/ryuk:0.3.0,showAll=false,filters=org.testcontainers.shaded.com.github.dockerjava.core.util.FiltersBuilder@0]
13:02:20.803 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.0 at registry: index.docker.io
13:02:20.804 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (does not exist) and commandPathPrefix: 
13:02:20.804 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.0, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
13:02:20.805 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
13:02:20.805 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [null]
13:02:21.019 [docker-java-stream--1668592714] INFO org.testcontainers.DockerClientFactory - Starting to pull image
13:02:21.041 [docker-java-stream--1668592714] DEBUG com.github.dockerjava.api.command.PullImageResultCallback - ResponseItem(stream=null, status=Pulling from testcontainers/ryuk, progressDetail=null, progress=null, id=0.3.0, from=null, time=null, errorDetail=null, error=null, aux=null)

Non working version:

19:03:19.418 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
19:03:19.441 [main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
19:03:19.448 [main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor)
19:03:19.459 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
19:03:19.489 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /usr/lib/jvm/java-1.8-openjdk/jre/bin, /usr/lib/jvm/java-1.8-openjdk/bin])
19:03:19.490 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - $XDG_RUNTIME_DIR is not set.
19:03:19.490 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/root/.docker/run' does not exist.
19:03:19.533 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/run/user/0' does not exist.
19:03:20.117 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
19:03:20.140 [ducttape-0] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.458 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
19:03:20.458 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Transport type: 'okhttp', Docker host: 'unix:///var/run/docker.sock'
19:03:20.458 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Checking Docker OS type for local Unix socket (unix:///var/run/docker.sock)
19:03:20.468 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: bridge
19:03:20.486 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
19:03:20.488 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.505 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.528 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.0
  API Version: 1.41
  Operating System: Ubuntu 16.04.4 LTS
  Total Memory: 31907 MB
19:03:20.528 [main] DEBUG org.testcontainers.DockerClientFactory - Ryuk is enabled
19:03:20.531 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for testcontainers/ryuk:0.3.0 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
19:03:20.534 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=testcontainers/ryuk:0.3.0,showAll=false,filters=org.testcontainers.shaded.com.github.dockerjava.core.util.FiltersBuilder@0]
19:03:20.585 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.0 at registry: index.docker.io
19:03:20.585 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (does not exist) and commandPathPrefix: 
19:03:20.586 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.0, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
19:03:20.586 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
19:03:20.587 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [null]
19:03:20.593 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl@4ff8d125[name=testcontainers-ryuk-478be443-d6a0-46f7-93af-512bc9f73bfa,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,healthcheck=<null>,argsEscaped=<null>,entrypoint=<null>,image=testcontainers/ryuk:0.3.0,volumes=com.github.dockerjava.api.model.Volumes@19c65cdc,workingDir=<null>,macAddress=<null>,onBuild=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@74bf1791,stopSignal=<null>,stopTimeout=<null>,hostConfig=HostConfig(binds=[/var/run/docker.sock:/var/run/docker.sock:rw], blkioWeight=null, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, memorySwappiness=null, nanoCPUs=null, capAdd=null, capDrop=null, containerIDFile=null, cpuPeriod=null, cpuRealtimePeriod=null, cpuRealtimeRuntime=null, cpuShares=null, cpuQuota=null, cpusetCpus=null, cpusetMems=null, devices=null, deviceCgroupRules=null, deviceRequests=null, diskQuota=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=null, groupAdd=null, ipcMode=null, cgroup=null, links=[], logConfig=LogConfig(type=null, config=null), lxcConf=null, memory=null, memorySwap=null, memoryReservation=null, kernelMemory=null, networkMode=null, oomKillDisable=null, init=null, autoRemove=true, oomScoreAdj=null, portBindings=null, privileged=false, publishAllPorts=true, readonlyRootfs=null, restartPolicy=null, ulimits=null, cpuCount=null, cpuPercent=null, ioMaximumIOps=null, ioMaximumBandwidth=null, volumesFrom=null, mounts=null, pidMode=null, isolation=null, securityOpts=null, storageOpt=null, cgroupParent=null, volumeDriver=null, shmSize=null, pidsLimit=null, runtime=null, tmpFs=null, utSMode=null, usernsMode=null, sysctls=null, consoleSize=null),labels={org.testcontainers=true},shell=<null>,networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>]
19:03:20.695 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))

[jdelucaa]

I am also seeing this com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: quay.io/testcontainers/ryuk:0.3.0"}

[amelnikov-mylivn]

@amelnikov-mylivn what does it have to do with the docker version? have they changed anything in the latest version?

Honestly idk, I did not dig enough into. Our CI runs on spot EC2 instances which are created for each task. Today builds started to fail with 404 No such image. We wasted most of the day, nothing helped, but when found that docker 20.10 appeared in repos yesterday, we pinned its version to an old one, and it worked.

P.S. Sorry for the off-topic.

[gesellix]

A manual docker pull testcontainers/ryuk:0.3.0 helps. Without having debugged the issue, yet, it seems like docker-java doesn't transparently pull the missing image on status code 404. This should be the default behaviour.
I'm also running into this issue using the recent Docker for Mac 2.5.4/Docker Engine 20.10.0-rc2 and Testcontainers 1.15.0.

[bsideup]

Without having debugged the issue, yet, it seems like docker-java doesn't transparently pull the missing image on status code 404.

It doesn't, yes.

This should be the default behaviour.

Why? 404 is "not found" and 4xx type of an http code, non-retriable. I don't think registries should return 404 for images that exist.

Although the original issue report is talking about some other, TCP related issue, so please consider reporting 404 separately, this is some other issue.

[gesellix]

Filed an issue for "No such image: testcontainers/ryuk:0.3.0" (Status 404) at #3574.
Sorry for the spam.

[rnorth]

I'm trying to reproduce this TCP/connection to Ryuk issue, but haven't managed to so far 😬

Using a fresh VM with Ubuntu 20.04 (LTS) x64 and Docker version 20.10.0, build 7287ab3

I'm testing with 1.15-0rc2, 1.15.0 and master/HEAD and so far not able to reproduce. Will continue to try.