Run Containerlab in a GitLab-CI Docker-in-Docker environment

[neilschark]

Hi,

I got containerlab running in a docker-in-docker setup without problems with a VSCode dev container setup as it is mentioned in the documentation of containerlab.

The next step for us is to have containerlab running in GitLab-CI. We currently use a shell runner for it, but this setup is not very good for obvious reasons. So we want to migrate it so that each test uses its own containerized instance of containerlab and a topology.

I configured a docker-in-docker runner and based my image on the default docker-in-docker alpine image.
I then added all dependencies listed in containerlabs own dockerfile: https://github.com/srl-labs/containerlab/blob/29bf4912f127d5ccd28e71a59d6938046ac0755b/goreleaser.dockerfile#L6C1-L15C17

With this image, docker runs fine:

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
719385e32844: Pulling fs layer
719385e32844: Verifying Checksum
719385e32844: Download complete
719385e32844: Pull complete
Digest: sha256:88ec0acaa3ec199d3b7eaf73588f4518c25f9d34f58ce9a0df68429c5af48e8d
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
[...]

Docker can also create networks:

$ docker network create -d bridge my-bridge-network
3734c8d908be1011ea1addc0d00df57747d899499c38b7b3e60f9c43003fbbe1

But unfortunately containerlab isn't able to create the docker network:

$ containerlab deploy --log-level trace -t dev_env_data/clab/basic_two_aristas.yaml
[...]
time="2023-11-01T09:47:05Z" level=debug msg="Error opening host DNS config run/systemd/resolve/resolv.conf: open run/systemd/resolve/resolv.conf: no such file or directory"
time="2023-11-01T09:47:05Z" level=debug msg="lab Conf: &{Name:gosdn_two_aristas Prefix:0xc00057c1a0 Mgmt:0xc000220090 Settings:<nil> Topology:0xc0006a4180 Debug:false}"
time="2023-11-01T09:47:05Z" level=debug msg="Env: CLAB_VERSION_CHECK="
time="2023-11-01T09:47:05Z" level=debug msg="Checking if docker network \"gosdn-net\" exists"
time="2023-11-01T09:47:05Z" level=debug msg="Network \"gosdn-net\" does not exist"
time="2023-11-01T09:47:05Z" level=info msg="Creating docker network: Name=\"gosdn-net\", IPv4Subnet=\"172.100.0.0/16\", IPv6Subnet=\"2001:db8::/64\", MTU='ל'"
time="2023-11-01T09:47:06Z" level=warning msg="failed gleaning v4 and/or v6 addresses from bridge via netlink, falling back to docker network inspect data"
Error: failed to lookup link "br-c785b71806d6": Link not found

I then tried using the containerlab docker image directly as my image for this CI job. But this doesn't work, as GitLab-CI doesn't mount the docker socket into the container, but rather exposes the API to the container as a network service: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-in-docker

Which leads to the following obvious error:

$ containerlab deploy --log-level trace -t dev_env_data/clab/basic_two_aristas.yaml
[...]
Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

How is it possible to run containerlab in a GitLab-CI docker-in-docker environment?

[hellt]

Hi @neilschark
can you try using the d-in-d image we use in clabernetes to run docker-in-docker with containerlab? The dockerfile is here https://github.com/srl-labs/clabernetes/blob/main/build/launcher.Dockerfile

/cc @carlmontanari

[hellt]

yes, in contianerlab's docker client code we respect the env vars known to docker.
Not sure why it refuses to connect to the daemon over TCP
I bet if you use tcpdump you will requests coming from containerlab towards docker:2375

[neilschark]

I will look into running tcpdump on it next week and report back.

[neilschark]

I tried using tcpdump to see if containerlab tries to connect to the docker daemon, but apparently because of docker iptables magic, you apparently can't simply use tcpdump in the same container. You have to connect a separate container into the same docker network. This is a relatively complicated task simply to see if containerlab tries to connect to the docker daemon, as the main problem would likely remain. As is shown by my self-build docker image, even if containerlab can connect, it still can't create the network.

Do any of you still have an idea how I can proceed and get containerlab running in GitLab-CI with docker-in-docker?

[hellt]

I haven't tried dind in gitlab ci.
Do you use self-hosted runners with gilab ci?

[neilschark]

Yes, I use a self-hosted runner. It is a root server where I have full root access. The server is running all our pipelines for the project.

But I have access to servers on demand, so I could also create a different GitLab-Runner just for testing, if some configurations should be tested which could potentially break the runner, so that my coworkers are not disturbed in their work.