tests stopped working on Docker desktop #399

gavinking · 2020-10-04T13:41:53Z

I've no idea what's going on here, but it seems to have coincided with me updating Docker desktop on Mac to the latest version:

Gradle Test Executor 2 > org.hibernate.reactive.FilterTest > testFilterCollectionFetch FAILED
    org.testcontainers.containers.ContainerLaunchException: Container startup failed
        at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:330)
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
        at org.hibernate.reactive.containers.PostgreSQLDatabase.getJdbcUrl(PostgreSQLDatabase.java:39)
        at org.hibernate.reactive.containers.DatabaseConfiguration.getJdbcUrl(DatabaseConfiguration.java:78)
        at org.hibernate.reactive.BaseReactiveTest.constructConfiguration(BaseReactiveTest.java:88)
        at org.hibernate.reactive.FilterTest.constructConfiguration(FilterTest.java:26)

        Caused by:
        org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:13.0, imagePullPolicy=DefaultPullPolicy())
            at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1279)
            at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:613)
            at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:320)
            ... 5 more

            Caused by:
            java.lang.IllegalStateException: Can not connect to Ryuk at localhost:32768
                at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:176)
                at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:168)
                at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
                at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
                at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
                at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
                at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
                at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:59)
                at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:26)
                at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
                at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:27)
                at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1277)
                ... 7 more

    java.lang.NullPointerException
        at org.hibernate.reactive.BaseReactiveTest.after(BaseReactiveTest.java:156)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at io.vertx.ext.unit.junit.VertxUnitRunner.invokeTestMethod(VertxUnitRunner.java:95)
        at io.vertx.ext.unit.junit.VertxUnitRunner.lambda$invokeExplosively$0(VertxUnitRunner.java:114)
        at io.vertx.ext.unit.impl.TestContextImpl.run(TestContextImpl.java:90)
        at io.vertx.ext.unit.junit.VertxUnitRunner.invokeExplosively(VertxUnitRunner.java:130)
        at io.vertx.ext.unit.junit.VertxUnitRunner.access$000(VertxUnitRunner.java:39)
        at io.vertx.ext.unit.junit.VertxUnitRunner$3.evaluate(VertxUnitRunner.java:217)
        at io.vertx.ext.unit.junit.Timeout$1.evaluate(Timeout.java:45)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
        at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
        at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
        at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
        at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
        at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
        at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
        at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
        at java.lang.Thread.run(Thread.java:748)

Any ideas @Sanne, @DavideD ?

gavinking · 2020-10-04T13:43:09Z

What is a Ryuk and why should it be running at localhost:32768?

gavinking · 2020-10-04T13:44:10Z

Oh, wait, I just found this: testcontainers/testcontainers-java#3241

Sanne · 2020-10-04T13:44:20Z

PosgreSQL recently released version 13, so I reconfigured our test images to use image postgres:13.0 . Sounds like you're unable to download it?

Weird, it's supposedly no different from fetching the previous versions.

gavinking · 2020-10-04T13:48:39Z

PosgreSQL recently released version 13, so I reconfigured our test images to use image postgres:13.0 . Sounds like you're unable to download it?

I think—but I'm not sure—that I already had it working on Postgres 13 up until I updated Docker.

Sanne · 2020-10-04T13:51:45Z

First time I hear about Ryuk .. from the very few doc references I found, it seems like an optional local server one is supposed to start?
It's responsible for fetching images on behalf of Testcontainers - so not having a new container (such as postgresql 13) would trigger this.

Seems a bug in Testcontainers.

You might be able to workaround it by downloading the container once manually?

try

docker run --rm -it --name HibernateTestingPGSQL \
    -e POSTGRES_USER=hreact -e POSTGRES_PASSWORD=hreact -e POSTGRES_DB=hreact \
    -p 5432:5432 postgres:13.0

gavinking · 2020-10-04T13:52:23Z

Changing to testcontainers 1.15.0-rc2 seems to have fixed it for me.

Sanne · 2020-10-04T13:53:37Z

Cool.
Alternatively: https://www.testcontainers.org/features/configuration/#disabling-ryuk

gavinking · 2020-10-04T13:55:17Z

Alternatively: https://www.testcontainers.org/features/configuration/#disabling-ryuk

Yep, that worked.

Is there a right way to set an environment variable from the gradle script?

gavinking · 2020-10-04T14:00:03Z

Apparently:

test {
    environment "TESTCONTAINERS_RYUK_DISABLED", "true"

Sanne · 2020-10-04T14:00:30Z

right, that should do it.

BTW please hold on pushing for some minutes :) release is running

Sanne · 2020-10-04T14:18:04Z

Release done.. I'll do the Quarkus updates now.

gavinking · 2020-10-04T14:29:52Z

Release done.. I'll do the Quarkus updates now.

Excellent!

DavideD · 2020-10-13T16:48:38Z

Is this still an issue?

gavinking · 2020-10-13T16:51:19Z

I believe the bug is still there, but the only people who are going to experience it are people who don't already have the thing fetched to their local machine.

gavinking · 2020-10-26T10:08:23Z

So I wonder if this was caused by the need to upgrade the test database when migrating from Postgres 12 to 13?

See, I have testcontainers.reuse.enable=true set, which perhaps reuses the database between test runs?

Sanne · 2020-10-26T10:17:09Z

I don't believe we strictly need that upgrade to PostgreSQL 13 ?

AFAIR when I upgraded it I didn't need to make any related changes, so I'd expect everything to work fine on 12 as well - unless a regression was included in the meantime but that seems less likely than having some problems with docker/testcontainers.

Sanne · 2020-10-26T10:17:32Z

Maybe try to revert the configuration to 12 and see?

gavinking · 2020-11-06T16:35:19Z

I'm experiencing this again :-(

See, I have testcontainers.reuse.enable=true set, which perhaps reuses the database between test runs?

I set it to false, and it didn't fix the problem.

gavinking · 2020-11-06T16:37:29Z

environment "TESTCONTAINERS_RYUK_DISABLED", "true"

This fixed it again.

@Sanne Do you think I should push this change to the gradle script?

Sanne · 2020-11-06T21:12:09Z

yes, sure why not. It seems to be a new feature, and we've been happy so far without it.

bsideup · 2020-11-07T19:54:43Z

It's responsible for fetching images on behalf of Testcontainers - so not having a new container (such as postgresql 13) would trigger this.

It is actually responsible for terminating the containers after running the tests. If you kill the JVM, the shutdown hooks won't get executed and the containers will continue running, creating dangling containers.
The environment variable is there for CI systems where Ryuk cannot be used (e.g. BitBucket Pipelines) but they guarantee the cleanup themselves.

tl;dr: do not disable Ryuk ;)

gavinking · 2020-11-07T21:53:26Z

If you kill the JVM, the shutdown hooks won't get executed and the containers will continue running, creating dangling containers.

Well, TBH, I run with testcontainers.reuse.enable=true always, so I guess that's what happens anyway.

And, on balance, dangling containers is less worse than my tests just not running. :-)

But sure, if this is fixed in 1.15 (which just came out two days ago, I think) then that's great, and we'll just upgrade instead.

Sanne · 2020-11-07T21:58:49Z

Thanks for explaning that @bsideup !

gavinking added the bug Something isn't working label Oct 4, 2020

gavinking added the testing Improvements to test suite label Nov 6, 2020

gavinking added a commit to gavinking/hibernate-reactive that referenced this issue Nov 6, 2020

disable ryuk for hibernate#399

855085b

gavinking linked a pull request Nov 6, 2020 that will close this issue

disable ryuk for #399 #431

Closed

gavinking linked a pull request Nov 7, 2020 that will close this issue

update testcontainers to 1.15 #435

Merged

DavideD closed this as completed in #435 Nov 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests stopped working on Docker desktop #399

tests stopped working on Docker desktop #399

gavinking commented Oct 4, 2020

gavinking commented Oct 4, 2020

gavinking commented Oct 4, 2020

Sanne commented Oct 4, 2020

gavinking commented Oct 4, 2020

Sanne commented Oct 4, 2020 •

edited

gavinking commented Oct 4, 2020

Sanne commented Oct 4, 2020

gavinking commented Oct 4, 2020

gavinking commented Oct 4, 2020

Sanne commented Oct 4, 2020

Sanne commented Oct 4, 2020

gavinking commented Oct 4, 2020

DavideD commented Oct 13, 2020

gavinking commented Oct 13, 2020

gavinking commented Oct 26, 2020

Sanne commented Oct 26, 2020

Sanne commented Oct 26, 2020

gavinking commented Nov 6, 2020

gavinking commented Nov 6, 2020

Sanne commented Nov 6, 2020

bsideup commented Nov 7, 2020

gavinking commented Nov 7, 2020

Sanne commented Nov 7, 2020

tests stopped working on Docker desktop #399

tests stopped working on Docker desktop #399

Comments

gavinking commented Oct 4, 2020

gavinking commented Oct 4, 2020

gavinking commented Oct 4, 2020

Sanne commented Oct 4, 2020

gavinking commented Oct 4, 2020

Sanne commented Oct 4, 2020 • edited

gavinking commented Oct 4, 2020

Sanne commented Oct 4, 2020

gavinking commented Oct 4, 2020

gavinking commented Oct 4, 2020

Sanne commented Oct 4, 2020

Sanne commented Oct 4, 2020

gavinking commented Oct 4, 2020

DavideD commented Oct 13, 2020

gavinking commented Oct 13, 2020

gavinking commented Oct 26, 2020

Sanne commented Oct 26, 2020

Sanne commented Oct 26, 2020

gavinking commented Nov 6, 2020

gavinking commented Nov 6, 2020

Sanne commented Nov 6, 2020

bsideup commented Nov 7, 2020

gavinking commented Nov 7, 2020

Sanne commented Nov 7, 2020

Sanne commented Oct 4, 2020 •

edited