Unreliable DNS during Container Builds

[tpanum]

I have a GitHub Actions pipeline that roughly process like so:

1. Connect to a private Tailscale network using this Github Action.
2. Start a docker build of a multi-stage Dockerfile where an internal python package index[1] is accessed to pull dependencies.

[1]: This package index is available within the Tailscale network and is discovered using DNS of an internal DNS server configured in Tailscale.

Step 2 occasionally fail due to DNS lookups for the package index failing, thus DNS resolving not working reliably.

[mloberg]

Are you using buildx? I was having this issue, but it seems to be resolved by setting network=host in driver-opts

[tpanum]

I have been using buildx, yeah. I have tried using the network=host in the past, but did not make it more reliable from my experiences. I ended up doing dig example.com > $IP and then use --add-host.

[henworth]

I was dealing with basically the same issue--container builds that rely on internal resources failing to resolve dns--and the thing that finally fixed it in my case was to configure buildx with the internal dns hosts:

- name: Setup buildx with internal DNS
  uses: docker/setup-buildx-action@v3
  with:
    config-inline: |
      [dns]
        nameservers="<comma separated list>"

[kdpuvvadi]

Even that is also incosistant, it works sometimes. Mostly doesn't.

For some reason, it's using 168.63.129.16 for dns resolution.

[tpanum]

Following @henworth's example, I had to change it slightly to get it working:

with:
  buildkitd-config-inline: |
    [dns]
      nameservers=["..."]

[kdpuvvadi]

Still same though @tpanum.

mines looks like this

- name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3.2.0
        with:
          buildkitd-config-inline: |
            [dns]
              nameservers=["100.78.xx.xx","100.78.xx.xx"]

And errors out

#45 ERROR: failed to push git.local.puvvadi.net/***/blog:e234247: failed to do request: Head "https://git.local.puvvadi.net/v2/***/blog/blobs/sha256:bbba97e7b63ba8e2a28aa20a0a10b6ba491f29a395e8f7cc5bdf6ff4fe783000": dial tcp: lookup git.local.puvvadi.net on 168.63.129.16:53: no such host

[marcelofernandez]

We've also seen this same issue on docker run github actions steps, so it's not a docker build issue only. For example, a job containing this bash step:

runner@fv-az1797-395:~/work/repo/repo$ docker run -it --rm \
                --env ENV_VAR \
                <aws_account_id>.dkr.ecr.us-east-2.amazonaws.com/repo:latest /bin/bash
root@35f3013f9caf:/opt/code# cat /etc/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 168.63.129.16
nameserver 100.100.100.100
search vnw05d5vvvpeplv1mpaxmbipab.bx.internal.cloudapp.net tail1abc1.ts.net
root@35f3013f9caf:/opt/code#

This way we had many issues resolving Tailscale hosts inside this container. Fortunately, we managed to fix it using the --dns docker run option:

runner@fv-az1797-395:~/work/repo/repo$ docker run --dns=100.100.100.100 -it --rm \
                --env ENV_VAR \
                <aws_account_id>.dkr.ecr.us-east-2.amazonaws.com/repo:latest /bin/bash
root@35f3013f9caf:/opt/code# cat /etc/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 100.100.100.100
search vnw05d5vvvpeplv1mpaxmbipab.bx.internal.cloudapp.net tail1abc1.ts.net
root@35f3013f9caf:/opt/code#

Regards