Bug: 403 Forbidden when pulling pause containers

[cloudziu]

This issue is related to: kubernetes/registry.k8s.io#261 and #783

In a cluster deployed on Hetzner, after a cluster is deployed one of the compute nodes stays in NotReady state. From the node logs we can see:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "registry.k8s.io/pause:3.9": failed to pull image "registry.k8s.io/pause:3.9": failed to pull and unpack image "registry.k8s.io/pause:3.9": failed to resolve reference "registry.k8s.io/pause:3.9": unexpected status from HEAD request to https://europe-west3-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9: 403 Forbidden

I've tried to manually pull the images from the VM with and the result is the same 403 Forbidden:

# ctr --debug image pull --http-dump --http-trace -k registry.k8s.io/pause:3.9
DEBU[0000] fetching                                      image="registry.k8s.io/pause:3.9"
DEBU[0000] resolving                                     host=registry.k8s.io
DEBU[0000] do request                                    host=registry.k8s.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.28 request.method=HEAD url="https://registry.k8s.io/v2/pause/manifests/3.9"
INFO[0000] HEAD /v2/pause/manifests/3.9 HTTP/1.1
INFO[0000] Host: registry.k8s.io
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*
INFO[0000] User-Agent: containerd/1.6.28
INFO[0000]
DEBU[0000] DNS lookup                                    host=registry.k8s.io
DEBU[0000] DNS lookup complete                           coalesced=false result=34.96.108.209
DEBU[0000] Connection successful                         remote_addr="34.96.108.209:443" reused=false
INFO[0000] HTTP/1.1 307 Temporary Redirect
INFO[0000] Transfer-Encoding: chunked
INFO[0000] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
INFO[0000] Content-Type: text/html; charset=utf-8
INFO[0000] Date: Wed, 13 Mar 2024 13:02:46 GMT
INFO[0000] Location: https://europe-west3-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9
INFO[0000] Server: Google Frontend
INFO[0000] Via: 1.1 google, 1.1 google
INFO[0000] X-Cloud-Trace-Context: 7c0c3ef8fc6400fae1f36fdb1d0e223e
INFO[0000]
INFO[0000]
INFO[0000] HEAD /v2/k8s-artifacts-prod/images/pause/manifests/3.9 HTTP/0.0
INFO[0000] Host: europe-west3-docker.pkg.dev
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*
INFO[0000] Referer: https://registry.k8s.io/v2/pause/manifests/3.9
INFO[0000] User-Agent: containerd/1.6.28
INFO[0000]
DEBU[0000] DNS lookup                                    host=europe-west3-docker.pkg.dev
DEBU[0000] DNS lookup complete                           coalesced=false result=173.194.76.82
DEBU[0000] Connection successful                         remote_addr="173.194.76.82:443" reused=false
INFO[0000] HTTP/1.1 403 Forbidden
INFO[0000] Transfer-Encoding: chunked
INFO[0000] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
INFO[0000] Content-Type: text/html; charset=UTF-8
INFO[0000] Date: Wed, 13 Mar 2024 13:02:46 GMT
INFO[0000]
INFO[0000]
DEBU[0000] fetch response received                       host=registry.k8s.io response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000" response.header.content-type="text/html; charset=UTF-8" response.header.date="Wed, 13 Mar 2024 13:02:46 GMT" response.status="403 Forbidden" url="https://registry.k8s.io/v2/pause/manifests/3.9"
ctr: failed to resolve reference "registry.k8s.io/pause:3.9": unexpected status from HEAD request to https://europe-west3-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9: 403 Forbidden

How it differs from #783 is that the cluster was able do provision and spinup a node.

[Despire]

I wasn't able to setup a HTTPS proxy, spend 2 days.

To use the Proxy it needs to handle HTTPS traffic on port 443 but also on 6443.
it also needs to handle HTTP traffic (which was the easy step that worked)

During the HTTPS proxy setup I stumbled upon random issues like 502 badgateway, SSL CONNECT errors or some TLS1.3 errors

You will also need a valid certificate I used the one for claudie.dev on cloudflare

I tried to setup the kubeone HTTP proxy e.g. https://docs.kubermatic.com/kubeone/v1.7/guides/proxy/