"Initialization has been cancelled" starting container

[paulomorgado]

I'm running into this:

Error Message:
   DotNet.Testcontainers.Containers.ResourceReaperException : Initialization has been cancelled.
Stack Trace:
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartDefaultAsync(IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, Boolean isWindowsEngineEnabled, CancellationToken ct)
   at DotNet.Testcontainers.Clients.TestcontainersClient.RunAsync(IContainerConfiguration configuration, CancellationToken ct)
   at DotNet.Testcontainers.Containers.DockerContainer.UnsafeCreateAsync(CancellationToken ct)
   at DotNet.Testcontainers.Containers.DockerContainer.StartAsync(CancellationToken ct)
...

How do I increase the start timeout?

There doesn't seem to be a way to do that when invoking ResourceReaper.GetAndStartDefaultAsync or to use a resource reaper from ResourceReaper.GetAndStartNewAsync.

[HofmeisterAn]

This usually indicates a misconfiguration. Which container runtime are you using? Are you running DinD?

[paulomorgado]

Can RYUK_CONNECTION_TIMEOUT be used?

[HofmeisterAn]

If, for some reason, we need more time

Could you please elaborate on why this should be the case? Please use the following command to manually start a Ryuk container instance and test if the container starts successfully:

docker run -v /var/run/docker.sock:/var/run/docker.sock -e RYUK_PORT=8080 -p 8080:8080 testcontainers/ryuk:0.5.1

As mentioned earlier, the container will start in less than a second. I do not see any reason why it should take more than 60 seconds.

I believe Testcontainers cannot establish a connection within 60 seconds to the created and started instance. This indicates a misconfiguration, and other containers will not run properly either. I am pretty sure that increasing the timeout will not fix the root cause. Again, which container runtime and configuration are you using? Can you please share the log messages?

[paulomorgado]

I'm getting this log in our build pipeline:

[testcontainers.org 00:00:00.14] Connected to Docker:
  Host: tcp://linux.runner.git.intra.vision-box.com:2376/
  Server Version: 20.10.5
  Kernel Version: 5.4.0-67-generic
  API Version: 1.41
  Operating System: Ubuntu 20.04.2 LTS
  Total Memory: 7.78 GB
[testcontainers.org 00:00:00.26] Docker config "C:\Users\gitlab-runner\.docker\config.json" not found
[testcontainers.org 00:00:00.26] Searching Docker registry credential in Auths
[testcontainers.org 00:00:00.26] Docker registry credential https://index.docker.io/v1/ not found
[testcontainers.org 00:00:04.24] Docker image testcontainers/ryuk:0.5.1 created
[testcontainers.org 00:00:04.38] Docker container 02e287af9fad created
[testcontainers.org 00:00:04.48] Start Docker container 02e287af9fad
[testcontainers.org 00:00:05.02] Wait for Docker container 02e287af9fad to complete readiness checks
[testcontainers.org 00:00:05.05] Docker container 02e287af9fad ready

It looks like the container was started in less than a second.

I'm explicitly starting the reaper before starting the containers:

_ = await ResourceReaper.GetAndStartDefaultAsync(TestcontainersSettings.OS.DockerEndpointAuthConfig);

But I'm getting:

  Error Message:
   DotNet.Testcontainers.Containers.ResourceReaperException : Initialization has been cancelled.
  Stack Trace:
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartNewAsync(Guid sessionId, IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, IImage resourceReaperImage, IMount dockerSocket, Boolean requiresPrivilegedMode, TimeSpan initTimeout, CancellationToken ct)
   at DotNet.Testcontainers.Containers.ResourceReaper.GetAndStartDefaultAsync(IDockerEndpointAuthenticationConfiguration dockerEndpointAuthConfig, Boolean isWindowsEngineEnabled, CancellationToken ct)

docker run -v /var/run/docker.sock:/var/run/docker.sock -e RYUK_PORT=8080 -p 8080:8080 testcontainers/ryuk:0.5.1

Is this for test purposes only, or is ti supposed to write something to disk?
From the stack trace, I speculate that the task completion source is completing with:

ryukInitializedTaskSource.SetException(new ResourceReaperException("Initialization has been cancelled."));

[HofmeisterAn]

I'm explicitly starting the reaper before starting the containers:

This will not resolve your issue. The container starts, but Testcontainers cannot establish or maintain a connection to it.

Is this for test purposes only, or is ti supposed to write something to disk?

This was solely for testing purposes.

I'm getting this log in our build pipeline:

As I suspected, you are not using a "simple" container runtime configuration on your test host. Testcontainers tries to establish a connection to the Ryuk container and fails. This is likely due to the remote host configuration. Does your host (linux.runner.git.intra.vision-box.com) allow incoming connections? Starting Ryuk exposes the container port 8080 to a random host port. Testcontainers attempts to connect to this random host port. I guess this connection is blocked.

You can simply start a container on linux.runner.git.intra.vision-box.com, publish a port, and try to connect from your test host. I guess this will also fail.

It seems you are using or running on GitLab. Even though it appears that you are using your own infrastructure, configuring a container runtime like Docker for GitLab requires specific configurations. Here is a quick explanation about DinD configuration. You will find further information here (these are for Java, but it is pretty similar for .NET too). Maybe they help too.

[paulomorgado]

It was, indeed, an "environmental" issue.

Thank you so much.