Intermittent couchbase failures in CI

[rnorth]

For a long time we've been getting intermittent failures in CI, such as this:

shouldInsertDocument - org.testcontainers.couchbase.Couchbase4_6Test
com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:686)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:667)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:626)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:878)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:844)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:836)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:44)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1286)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:176)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:162)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:821)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:776)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Unfortunately this is a nuisance for us and for contributors. I've tried suggestions such as this but it hasn't improved reliability.

Can anyone with more Couchbase expertise help investigate?

To reproduce, run the test suites under modules/couchbase repeatedly.

[rnorth]

Example build failures (not sure if these will be overwritten when we re-run tests):
https://circleci.com/gh/testcontainers/testcontainers-java/12930
https://circleci.com/gh/testcontainers/testcontainers-java/12796
https://circleci.com/gh/testcontainers/testcontainers-java/12791

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[rnorth]

This continues to be a problem; our tests fail even if retried. This gives me some doubt that the module is actually stable enough for real use.

I think we're going to have to have to reach out to Couchbase/Couchbase community for help. In the worst case scenario, if we can't fix it, I'm inclined to remove this module.

[SudharakaP]

@rnorth : Just for the record, we've observed problems with couchbase testcontainers as well; jhipster/generator-jhipster#10594

[rnorth]

@SudharakaP thanks for confirming, and I’m sorry to hear that.

It sounds like a couple of Couchbase experts may be willing to lend a hand - hopefully there’s a straightforward solution so that we can make this module more reliable. Fingers crossed!

_{Sent with GitHawk}

[deniswsrosa]

Couchbase employee here, we will investigate this issue and come back to you as soon as possible.

[SudharakaP]

@rnorth : You are welcome. 😄

@deniswsrosa : Just for the record, I was wondering if this issue is similar to #715 where the startup timeout in Couchbase is 120 being inadequate. I was trying to create a custom wait strategy to test it out, but couldn't get it working yet (due to my lack of knowledge about Couchbase probably) 😄

[deniswsrosa]

@rnorth : You are welcome. 😄

@deniswsrosa : Just for the record, I was wondering if this issue is similar to #715 where the startup timeout in Couchbase is 120 being inadequate. I was trying to create a custom wait strategy to test it out, but couldn't get it working yet (due to my lack of knowledge about Couchbase probably) 😄

thanks for the heads up, I will ping this back to the team.

[pascalgrimaud]

Some additional information, if it can help.
In JHipster project, we encounter the similar issue as mentionned by @SudharakaP.

In our CI, it works perfectly well with OpenJDK11 :

      echo '*** Using OpenJDK 11'
      sudo add-apt-repository ppa:openjdk-r/ppa
      sudo apt-get update
      sudo apt-get install -y openjdk-11-jdk
      sudo update-java-alternatives -s java-1.11.0-openjdk-amd64
      java -version

But, it failed randomly when using OpenJDK8:

      echo '*** Using OpenJDK 8'
      sudo add-apt-repository ppa:openjdk-r/ppa
      sudo apt-get update
      sudo apt-get install -y openjdk-8-jdk
      sudo update-java-alternatives -s java-1.8.0-openjdk-amd64
      java -version

[enbrent]

Are there any alternatives while we wait for a solution?

[daschl]

Trying to triage this a bit. Am I correct that the initial exception reported in this ticket is different from the failing runs linked? In the linked run I see:

Caused by:
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://localhost:32934/ui/index.html should return HTTP 200)

while this ticket reports

shouldInsertDocument - org.testcontainers.couchbase.Couchbase4_6Test
com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:686)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:667)
	at

The first one looks like the server is not coming up in time until it stops waiting, the second one is completely different: the SDK sent a request into the cluster but while the request was in-flight on the socket the socket got closed- and as a result the client has no other option as to cancel the request.