v1.29/stable not working on hetzner

[ZhouJian26]

Summary

microk8s not working correctly after install on hetzner ubuntu 22.04LTS server. Downgrading to 1.28/stable works fine

Reproduction Steps

• get a hetzner server with ubuntu 22.04 LTS
• apt update && apt upgrade -y
• apt install snapd
• snap install microk8s --classic --channel=1.29/stable
• microk8s status --wait-ready (will hang forever)

Introspection Report

Inspecting system
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-kubelite is running
  Service snap.microk8s.daemon-k8s-dqlite is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy openSSL information to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy asnycio usage and limits to the final report tarball
  Copy inotify max_user_instances and max_user_watches to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting dqlite
  Inspect dqlite
cp: cannot stat '/var/snap/microk8s/6641/var/kubernetes/backend/localnode.yaml': No such file or directory

Building the report tarball
  Report tarball is at /var/snap/microk8s/6641/inspection-report-20240413_075258.tar.gz

[neoaggelos]

Hi @ZhouJian26 can you check the logs to see if this is similar to #4361?

[ZhouJian26]

Probably it was this problem here kubernetes/registry.k8s.io#261 (in short: some cloud provider ip got banned by google registry, u can verify it by describing the calico pod in kube-system)

Warning  FailedCreatePodSandBox  2m30s (x47 over 12m)  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "registry.k8s.io/pause:3.7": failed to pull image "registry.k8s.io/pause:3.7": failed to pull and unpack image "registry.k8s.io/pause:3.7": failed to resolve reference "registry.k8s.io/pause:3.7": unexpected status from HEAD request to https://registry.k8s.io/v2/pause/manifests/3.7: 403 Forbidden

It would be great if we could replace a specific domain registry with a self-hosted alternative, such as Harbor, AWS ECR, etc. Is this feasible? @neoaggelos

I attempted to investigate this, and it appears to involve some configuration with containerd, but unfortunately, I haven't had any success. reference: https://microk8s.io/docs/registry-private

[neoaggelos]

You can have a look at https://microk8s.io/docs/registry-private#configure-registry-mirrors-7

[NGarate]

I have the same exact error, in contabo.com with stable v1.29
inspection-report-20240428_175121.tar.gz

[neoaggelos]

Hi @NGarate, looks like you experience a different issue #4361. I see this repeated in the logs:

Apr 28 17:36:29 host1.ngarate.com microk8s.daemon-kubelite[2497430]: E0428 17:36:29.752440 2497430 kubelet.go:1542] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: root container [kubepods] doesn't exist"

We recently did a bugfix to address this, #4505. This should now be out in the 1.29/edge channel and will be promoted to stable soon.

Could you be so kind as to test that this works for you? Try installing microk8s with:

sudo snap install microk8s --classic --channel 1.29/edge

Thank you!