Registry service stuck in pending state

[rparth]

Summary

Registry pods under container-registry namespace is stuck in pending state. The microk8s vm was created on an aws instance using packer and converted to an AMI. Images were added to the image with registry service before AMI was created. When new instance is created using the AMI, the registry service gets stuck in pending state.

journalctl shows the issue as follows
May 11 13:58:42 ip-172-31-22-250 microk8s.daemon-kubelite[44501]: E0511 13:58:42.338492 44501 schedule_one.go:154] "Error selecting node for pod" err="nodeinfo not found for node name "ip-172-31-32-250"" pod="container-registry/registry-6c9fcc695f-lf4fc"
May 11 13:58:42 ip-172-31-22-250 microk8s.daemon-kubelite[44501]: E0511 13:58:42.338528 44501 schedule_one.go:1004] "Error scheduling pod; retrying" err="nodeinfo not found for node name "ip-172-31-32-250"" pod="container-registry/registry-6c9fcc695f-lf4fc"

When checking the nodes, the node IP is different as the new VM was created from the AMI image. Microk8s scheduler tries to connect to registry hostname from the original VM which was used to create the AMI.

Original IP : ip-172-31-32-250
New VM IP : ip-172-31-22-250

Refereces to the old ip exist in the following files inside /var/snap

/var/snap/microk8s/6750/credentials/kubelet.config: user: system:node:ip-172-31-32-250
/var/snap/microk8s/6750/credentials/kubelet.config:- name: system:node:ip-172-31-32-250
/var/snap/microk8s/6750/var/tmp/csr-dqlite.conf:DNS = ip-172-31-32-250

root@ip-172-31-22-250:~# kubectl get nodes -o yaml|kubectl neat
apiVersion: v1
items:

• apiVersion: v1
  kind: Node
  metadata:
  annotations:
  node.alpha.kubernetes.io/ttl: "0"
  projectcalico.org/IPv4Address: 172.31.22.250/20
  projectcalico.org/IPv4VXLANTunnelAddr: 10.1.183.196
  volumes.kubernetes.io/controller-managed-attach-detach: "true"
  labels:
  beta.kubernetes.io/arch: amd64
  beta.kubernetes.io/os: linux
  kubernetes.io/arch: amd64
  kubernetes.io/hostname: ip-172-31-22-250
  kubernetes.io/os: linux
  microk8s.io/cluster: "true"
  node.kubernetes.io/microk8s-controlplane: microk8s-controlplane
  name: ip-172-31-22-250
  kind: List
  metadata:
  resourceVersion: ""

What Should Happen Instead?

registry pods should initialize

Reproduction Steps

This is reproducible in all recent versions of microk8s I believe post 1.22. I have verified against 1.24-1.29

Introspection Report

Done

Can you suggest a fix?

No

Are you interested in contributing with a fix?

No
inspection-report-20240511_140434.tar.gz