Can't connect to Ryuk container when using remote Docker host

[alexcase52]

Version 1.6.0
Not sure if this came from multiple threads but never seen it with single thread.
Many tests use several types of containers.

java.lang.ExceptionInInitializerError
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
	at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:158)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:116)
	at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:126)
	at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:35)
	at org.testcontainers.containers.PostgreSQLContainer.<init>(PostgreSQLContainer.java:28)
	at com.tests.container.DbContainer.<init>()
	at com.tests.MyTest.<clinit>()
	... 27 more

[bsideup]

Hi @alexcase52,

Could you please try with 1.5.1 + multiple threads? I want to make sure that this is a regression in 1.6.0 :)

[llfbandit]

Hi @alexcase52,

I get also the error only on 1.6.0, 1.5.1 is fine.
Testing on Windows 10 with Java9.

Is deamon exposition on tcp://xy:2375 without TLS mandatory ?

[kiview]

@llfbandit

Is deamon exposition on tcp://xy:2375 without TLS mandatory ?

This is currently still mandatory (until we will be able to support npipes or Docker for Windows will support Unix sockets).

[llfbandit]

https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

Well... kind of tiny scope for now...

[alvarosanchez]

I'm experiencing a similar issue when running on Jenkins, with a DOCKER_HOST pointing to some other host:

  2018-04-03 17:59:19.719  WARN --- [containers-ryuk[] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at 10.17.48.167:32800
  java.net.ConnectException: Connection refused
  	at java.net.PlainSocketImpl.socketConnect(Native Method)
  	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
  	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  	at java.net.Socket.connect(Socket.java:589)
  	at java.net.Socket.connect(Socket.java:538)
  	at java.net.Socket.<init>(Socket.java:434)
  	at java.net.Socket.<init>(Socket.java:211)
  	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:111)
  	at org.testcontainers.utility.ResourceReaper$$Lambda$41/1921545475.run(Unknown Source)
  	at java.lang.Thread.run(Thread.java:745)

Downgrading to 1.5.1 fixes the issue.

[kiview]

@alvarosanchez
Thanks for the info, is 10.17.48.167 the ip of your DOCKER_HOST? Should be. Could this be a problem with a firewall?
I assume unrelated, but I've experienced this problem when inside a network with a captive portal, so this can be some DNS hickup.

[alvarosanchez]

is 10.17.48.167 the ip of your DOCKER_HOST?

Yes, it is.

Could this be a problem with a firewall?

No, because then version 1.5.1 wouldn't work either. Note that the client machine (a Jenkins instance) is able to run Docker commands on that DOCKER_HOST via docker CLI and also the Gradle plugin.

[kiview]

@alvarosanchez
With version 1.6.0 we've introduced a new implementation for the ResourceReaper component, which will now run outside of the JVM in its own Docker container (the container is called Ryuk and the error appears when trying to connect to this container).
Since we don't have an automated CI build for this scenario (remote DOCKER_HOST), I'm not sure if this implementation broke the compatibility with some remote scenarios.

I've changed the title of the issue to better reflect the current state of knowledge we have, I hope it's okay for everyone involved?

@alexcase52 Were you able to find out if this issue is related to multiple Gradle threads?

[kiview]

@alvarosanchez
Have you tried using 1.7.0? Maybe the container startup ran into a timeout as mentioned in #621.

[bsideup]

@kiview 1.7.2 :)

[lon-pilot-mpf]

Hi, Just encountered this on 1.7.3 running a test on our Jenkins server:

9:26:52.538 [main] INFO  org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
09:26:52.555 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://NAME_OF_MY_DOCKER_HOST:2375
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='jenkins'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://NAME_OF_MY_DOCKER_HOST:2375,registryUsername=jenkins,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/var/lib/jenkins/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

09:26:52.557 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is NAME_OF_MY_DOCKER_HOST
09:26:52.921 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 17.09.0-ce
  API Version: 1.32
  Operating System: CentOS Linux 7 (Core)
  Total Memory: 7823 MB
09:26:54.359 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at NAME_OF_MY_DOCKER_HOST:32774
java.net.NoRouteToHostException: No route to host (Host unreachable)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
	at java.lang.Thread.run(Thread.java:748)

[deshchenko]

Hi, I have similar error on 1.7.3 inside Jenkins image.
In docker processes I see that Ruak image has been started, but it impossible to connect to it for some Reason...

10:00:00.109 [main] INFO  org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
10:00:00.129 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

10:00:00.989 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
10:00:01.134 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 17.03.2-ce
  API Version: 1.27
  Operating System: Ubuntu 14.04.5 LTS
  Total Memory: 16047 MB
10:00:02.686 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:39972
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
	at java.lang.Thread.run(Thread.java:748)
10:00:03.692 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:39972
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
	at `java.lang.Thread.run(Thread.java:748)