User clusters do not get up after node restarts #12065

clickersmudge · 2023-03-16T10:49:25Z

clickersmudge
Mar 16, 2023

Hi,
Tonight one of the machines failed. After a reboot everything was back to normal except for the pods in the user cluster namespace.

NAMESPACE                  NAME                                                   READY   STATUS                  RESTARTS          AGE
cert-manager               cert-manager-74f7ffc9d-8nz9b                           1/1     Running                 0                 7h24m
cert-manager               cert-manager-cainjector-b8bc4d9cf-sslnq                1/1     Running                 0                 7h24m
cert-manager               cert-manager-webhook-db984c8b5-bp67t                   1/1     Running                 0                 7h24m
cluster-dzd4vpffff         apiserver-69c868f64c-6sk25                             0/2     ImagePullBackOff        0                 7h24m
cluster-dzd4vpffff         apiserver-69c868f64c-7kx7v                             0/2     ImagePullBackOff        0                 7h24m
cluster-dzd4vpffff         controller-manager-866d5bd449-p9nmc                    0/1     Running                 113 (59s ago)     7h24m
cluster-dzd4vpffff         etcd-0                                                 1/1     Running                 0                 99m
cluster-dzd4vpffff         etcd-1                                                 0/1     Init:ImagePullBackOff   0                 99m
cluster-dzd4vpffff         etcd-2                                                 1/1     Running                 0                 99m
cluster-dzd4vpffff         hcloud-cloud-controller-manager-5989d7bb75-q2r4s       0/1     CrashLoopBackOff        55 (2m4s ago)     7h24m
cluster-dzd4vpffff         kube-state-metrics-649d784dc9-g4txg                    0/1     CrashLoopBackOff        55 (47s ago)      7h24m
cluster-dzd4vpffff         kubernetes-dashboard-65fccbc445-4mzsk                  1/1     Running                 55 (8m2s ago)     7h24m
cluster-dzd4vpffff         kubernetes-dashboard-65fccbc445-gw7lr                  0/1     CrashLoopBackOff        55 (62s ago)      7h24m
cluster-dzd4vpffff         machine-controller-cc75b75f7-bdmh2                     0/1     CrashLoopBackOff        141 (27s ago)     7h24m
cluster-dzd4vpffff         machine-controller-webhook-8ccbfc5d4-fwzgh             0/1     Running                 113 (50s ago)     7h24m
cluster-dzd4vpffff         operating-system-manager-86d57d8458-wmd8g              0/1     CrashLoopBackOff        141 (2m40s ago)   7h24m
cluster-dzd4vpffff         operating-system-manager-webhook-5497b7798b-wbxkr      0/1     Running                 113 (40s ago)     7h24m
cluster-dzd4vpffff         prometheus-0                                           1/1     Running                 1 (100m ago)      31h
cluster-dzd4vpffff         scheduler-684d8f6847-6vvwb                             0/1     CrashLoopBackOff        113 (32s ago)     7h24m
cluster-dzd4vpffff         usercluster-controller-868ccd9d9b-d2kph                0/1     CrashLoopBackOff        55 (47s ago)      7h24m
cluster-dzd4vpffff         usercluster-webhook-6549b9cf57-6h496                   0/1     Error                   55 (8m33s ago)    7h24m
cluster-gsg7dzntvb         apiserver-7c8c6c5848-4twr9                             0/2     ImagePullBackOff        0                 7h24m
cluster-gsg7dzntvb         apiserver-7c8c6c5848-zpvvs                             0/2     ImagePullBackOff        0                 7h24m
cluster-gsg7dzntvb         controller-manager-68686c9b8c-672w4                    0/1     Running                 112 (5m18s ago)   7h24m
cluster-gsg7dzntvb         etcd-0                                                 1/1     Running                 0                 99m
cluster-gsg7dzntvb         etcd-1                                                 0/1     Init:ImagePullBackOff   0                 99m
cluster-gsg7dzntvb         etcd-2                                                 1/1     Running                 0                 98m
cluster-gsg7dzntvb         hcloud-cloud-controller-manager-5b75c49564-lxgc4       0/1     CrashLoopBackOff        55 (2m23s ago)    7h24m
cluster-gsg7dzntvb         kube-state-metrics-59b95b4967-jj5cg                    0/1     Error                   55 (8m37s ago)    7h24m
cluster-gsg7dzntvb         kubernetes-dashboard-5f9fc55bc7-jtkc6                  0/1     CrashLoopBackOff        55 (117s ago)     7h24m
cluster-gsg7dzntvb         kubernetes-dashboard-5f9fc55bc7-vbjhc                  0/1     CrashLoopBackOff        55 (17s ago)      7h24m
cluster-gsg7dzntvb         machine-controller-65f74cc6dd-vcfsj                    0/1     CrashLoopBackOff        141 (69s ago)     7h24m
cluster-gsg7dzntvb         machine-controller-webhook-7979bd8978-gmmcf            0/1     Running                 113 (0s ago)      7h24m
cluster-gsg7dzntvb         operating-system-manager-55c7f774-k4dn7                0/1     CrashLoopBackOff        141 (2m8s ago)    7h24m
cluster-gsg7dzntvb         operating-system-manager-webhook-878d899bc-fc9nb       0/1     Running                 112 (6m17s ago)   7h24m
cluster-gsg7dzntvb         prometheus-0                                           1/1     Running                 1 (100m ago)      31h
cluster-gsg7dzntvb         scheduler-6fdcc5cb8f-8wqj5                             0/1     Running                 113 (60s ago)     7h24m
cluster-gsg7dzntvb         usercluster-controller-847469b9c-nvb2p                 0/1     CrashLoopBackOff        55 (60s ago)      7h24m
cluster-gsg7dzntvb         usercluster-webhook-6974489f65-bgpgg                   0/1     CrashLoopBackOff        55 (67s ago)      7h24m
cluster-udr5n6mzrs         apiserver-7fcd9f8c48-5wv4z                             0/2     ErrImagePull            0                 7h24m
cluster-udr5n6mzrs         apiserver-7fcd9f8c48-p5vww                             0/2     ErrImagePull            0                 7h24m
cluster-udr5n6mzrs         controller-manager-844bf7fcd9-cb5pg                    0/1     Running                 112 (6m1s ago)    7h24m
cluster-udr5n6mzrs         etcd-0                                                 1/1     Running                 0                 99m
cluster-udr5n6mzrs         etcd-1                                                 0/1     Init:ImagePullBackOff   0                 99m
cluster-udr5n6mzrs         etcd-2                                                 1/1     Running                 0                 99m
cluster-udr5n6mzrs         hcloud-cloud-controller-manager-69979bcddf-vlpmt       0/1     CrashLoopBackOff        55 (2m25s ago)    7h24m
cluster-udr5n6mzrs         kube-state-metrics-597b97cfd6-fn776                    0/1     CrashLoopBackOff        55 (32s ago)      7h24m
cluster-udr5n6mzrs         kubernetes-dashboard-dff5c8d77-gg994                   1/1     Running                 55 (7m55s ago)    7h24m
cluster-udr5n6mzrs         kubernetes-dashboard-dff5c8d77-q68hj                   0/1     CrashLoopBackOff        55 (21s ago)      7h24m
cluster-udr5n6mzrs         machine-controller-769f9558cc-xwsvb                    0/1     CrashLoopBackOff        141 (74s ago)     7h24m
cluster-udr5n6mzrs         machine-controller-webhook-b5d7c6499-ddzlh             0/1     Running                 112 (6m23s ago)   7h24m
cluster-udr5n6mzrs         operating-system-manager-7d6b7f894d-9vwf5              0/1     Running                 141 (4s ago)      7h24m
cluster-udr5n6mzrs         operating-system-manager-webhook-6489f968df-qbmvz      0/1     Running                 113 (18s ago)     7h24m
cluster-udr5n6mzrs         prometheus-0                                           1/1     Running                 1 (100m ago)      31h
cluster-udr5n6mzrs         scheduler-795b977697-kxtbp                             0/1     Running                 112 (6m24s ago)   7h24m
cluster-udr5n6mzrs         usercluster-controller-7646c9d98-cb9jk                 0/1     Running                 55 (8m17s ago)    7h24m
cluster-udr5n6mzrs         usercluster-webhook-c9677cf6f-2x285                    0/1     Running                 55 (8m14s ago)    7h24m
kube-system                calico-kube-controllers-7fbd9845c6-x2wgh               1/1     Running                 0                 4d15h
kube-system                canal-6v4hj                                            2/2     Running                 0                 4d15h
kube-system                canal-mtwjt                                            2/2     Running                 0                 7h24m
kube-system                canal-nbfb6                                            2/2     Running                 2 (100m ago)      31h
kube-system                cluster-autoscaler-86d7758dc9-jkktw                    1/1     Running                 0                 4d15h
kube-system                coredns-8467c695b7-6ghf8                               1/1     Running                 0                 4d15h
kube-system                coredns-8467c695b7-gkn84                               1/1     Running                 0                 4d15h
kube-system                etcd-kubeone-control-plane-1                           1/1     Running                 0                 4d15h
kube-system                hcloud-cloud-controller-manager-84dd748bc4-w6q8v       1/1     Running                 0                 4d15h
kube-system                hcloud-csi-controller-6cbf4fd8f6-nsz6p                 5/5     Running                 0                 4d15h
kube-system                hcloud-csi-node-vcxgz                                  3/3     Running                 3 (100m ago)      31h
kube-system                hcloud-csi-node-w7xxb                                  3/3     Running                 0                 7h24m
kube-system                hcloud-csi-node-whgdw                                  3/3     Running                 0                 4d15h
kube-system                kube-apiserver-kubeone-control-plane-1                 1/1     Running                 0                 4d15h
kube-system                kube-controller-manager-kubeone-control-plane-1        1/1     Running                 0                 4d15h
kube-system                kube-proxy-9tt6v                                       1/1     Running                 0                 7h24m
kube-system                kube-proxy-cptvt                                       1/1     Running                 0                 4d15h
kube-system                kube-proxy-xjktt                                       1/1     Running                 1 (100m ago)      31h
kube-system                kube-scheduler-kubeone-control-plane-1                 1/1     Running                 0                 4d15h
kube-system                kured-92jks                                            1/1     Running                 0                 4d15h
kube-system                kured-dtfpj                                            1/1     Running                 0                 7h23m
kube-system                kured-vvv2h                                            1/1     Running                 1 (100m ago)      31h
kube-system                machine-controller-75596d6bb4-hvdpv                    1/1     Running                 0                 4d15h
kube-system                machine-controller-webhook-bf64669c-plgp4              1/1     Running                 0                 4d15h
kube-system                metrics-server-5dbd4db586-sg59x                        1/1     Running                 0                 4d15h
kube-system                node-local-dns-8nrp4                                   1/1     Running                 0                 7h24m
kube-system                node-local-dns-bbmwc                                   1/1     Running                 0                 4d15h
kube-system                node-local-dns-slj49                                   1/1     Running                 1 (100m ago)      31h
kube-system                operating-system-manager-74dcdb469f-fzmnx              1/1     Running                 0                 4d15h
kube-system                operating-system-manager-webhook-6df59799bb-w5g88      1/1     Running                 0                 4d15h
kube-system                unattended-upgrades-install-ncvrd                      1/1     Running                 0                 7h23m
kube-system                unattended-upgrades-install-rp7v2                      1/1     Running                 2 (100m ago)      31h
kube-system                unattended-upgrades-install-z9lr7                      1/1     Running                 0                 4d15h
kubermatic                 kubermatic-api-666d57969f-9whzv                        1/1     Running                 0                 7h24m
kubermatic                 kubermatic-api-666d57969f-skgbx                        1/1     Running                 0                 7h24m
kubermatic                 kubermatic-dashboard-6b858c4dc9-2lhdf                  1/1     Running                 0                 7h24m
kubermatic                 kubermatic-dashboard-6b858c4dc9-vtgmh                  1/1     Running                 0                 7h24m
kubermatic                 kubermatic-master-controller-manager-8bb975658-xwdj5   1/1     Running                 0                 7h24m
kubermatic                 kubermatic-operator-7b4cfc68fd-thwqz                   1/1     Running                 0                 7h24m
kubermatic                 kubermatic-seed-controller-manager-7df4bb4678-lxqrs    1/1     Running                 0                 7h24m
kubermatic                 kubermatic-webhook-74466ccd7-xc99v                     1/1     Running                 0                 7h24m
kubermatic                 nodeport-proxy-envoy-d9bc475bd-6nhsl                   2/2     Running                 0                 7h24m
kubermatic                 nodeport-proxy-envoy-d9bc475bd-nrgmz                   2/2     Running                 0                 7h24m
kubermatic                 nodeport-proxy-envoy-d9bc475bd-r9tjn                   2/2     Running                 0                 7h24m
kubermatic                 nodeport-proxy-updater-77fd77c4d6-crb6x                1/1     Running                 0                 7h24m
kubermatic                 seed-proxy-kubermatic-7cd4b589db-wt44l                 1/1     Running                 0                 7h24m
nginx-ingress-controller   nginx-ingress-controller-6cc4f98d7c-2m4cg              1/1     Running                 0                 7h24m
nginx-ingress-controller   nginx-ingress-controller-6cc4f98d7c-j977s              1/1     Running                 0                 7h24m
nginx-ingress-controller   nginx-ingress-controller-6cc4f98d7c-qmmvs              1/1     Running                 0                 7h24m
oauth                      dex-69b8444cbf-gw4xg                                   1/1     Running                 0                 7h24m
oauth                      dex-69b8444cbf-ttwvf                                   1/1     Running                 0                 7h24m
reboot-coordinator         flatcar-linux-update-operator-78697ff4cf-kljfb         0/1     Pending                 0                 4d15h
velero                     node-agent-2q7zt                                       1/1     Running                 0                 7h23m
velero                     node-agent-vpv4f                                       1/1     Running                 1 (100m ago)      31h
velero                     velero-56447459b-m8kt2                                 1/1     Running                 0                 7h24m

kubectl describe pod -n cluster-dzd4vpffff         apiserver-69c868f64c-6sk25 
kubectl describe pod -n cluster-dzd4vpffff         apiserver-69c868f64c-6sk25 
Name:             apiserver-69c868f64c-6sk25
Namespace:        cluster-dzd4vpffff
Priority:         0
Service Account:  default
Node:             kubeone-pool1-565c47dcfc-stwxl/192.168.0.8
Start Time:       Thu, 16 Mar 2023 04:18:32 +0100
Labels:           adm-control-configmap-revision=1407083
                  apiserver-etcd-client-certificate-secret-revision=38720
                  apiserver-proxy-client-certificate-secret-revision=38711
                  apiserver-tls-secret-revision=38721
                  app=apiserver
                  app.kubernetes.io/version=1.26.1
                  audit-config-configmap-revision=38808
                  ca-bundle-configmap-revision=38817
                  ca-secret-revision=38703
                  cloud-config-secret-revision=38699
                  cluster=dzd4vpffff
                  front-proxy-ca-secret-revision=38708
                  konnectivity-kubeconfig-secret-revision=38779
                  konnectivityproxy-tls-secret-revision=38777
                  kube-apiserver-egress-configmap-revision=38838
                  kubelet-client-certificates-secret-revision=38724
                  pod-template-hash=69c868f64c
                  service-account-key-secret-revision=38728
                  tokens-secret-revision=38763
Annotations:      cni.projectcalico.org/containerID: d17f9e83bb97ad735e82e8c20176cfe990818a1fd182c67f182dbbe9ef54256d
                  cni.projectcalico.org/podIP: 10.244.9.55/32
                  cni.projectcalico.org/podIPs: 10.244.9.55/32
                  prometheus.io/path: /metrics
                  prometheus.io/port: 30385
                  prometheus.io/scrape_with_kube_cert: true
Status:           Pending
IP:               10.244.9.55
IPs:
  IP:           10.244.9.55
Controlled By:  ReplicaSet/apiserver-69c868f64c
Init Containers:
  etcd-running:
    Container ID:  containerd://ee77a7ed7abe5195246cd7e730bc4db6b8c313317f937aacb5fa96986672667b
    Image:         gcr.io/etcd-development/etcd:v3.5.6
    Image ID:      gcr.io/etcd-development/etcd@sha256:28cb0630cb8536504f9bd547c3e63e608242c40dbffb1464c892d8d59fd3da44
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
      for i in $(seq 1 100); do if ETCDCTL_API=3 /usr/local/bin/etcdctl --cacert=/etc/etcd/pki/client/ca.crt --cert=/etc/etcd/pki/client/apiserver-etcd-client.crt --key=/etc/etcd/pki/client/apiserver-etcd-client.key --dial-timeout=2s --endpoints='https://etcd-0.etcd.cluster-dzd4vpffff.svc.cluster.local.:2379,https://etcd-1.etcd.cluster-dzd4vpffff.svc.cluster.local.:2379,https://etcd-2.etcd.cluster-dzd4vpffff.svc.cluster.local.:2379' put kubermatic/quorum-check something; then echo "etcd ready"; exit 0; fi; echo "waiting for etcd. retry=$i/100"; sleep 2; done; echo "error: etcd not ready"; exit 1;
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 16 Mar 2023 09:58:54 +0100
      Finished:     Thu, 16 Mar 2023 10:03:56 +0100
    Ready:          True
    Restart Count:  29
    Environment:    <none>
    Mounts:
      /etc/etcd/pki/client from apiserver-etcd-client-certificate (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-csjnq (ro)
Containers:
  konnectivity-server:
    Container ID:  
    Image:         eu.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-server:v0.0.35
    Image ID:      
    Ports:         8132/TCP, 8133/TCP, 8134/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /proxy-server
    Args:
      --logtostderr=true
      -v=3
      --cluster-key=/etc/kubernetes/pki/konnectivityproxy-tls.key
      --cluster-cert=/etc/kubernetes/pki/konnectivityproxy-tls.crt
      --uds-name=/etc/kubernetes/konnectivity-server/konnectivity-server.socket
      --kubeconfig=/etc/kubernetes/kubeconfig/konnectivity-server.conf
      --server-count=2
      --mode=grpc
      --server-port=0
      --agent-port=8132
      --admin-port=8133
      --health-port=8134
      --agent-namespace=kube-system
      --agent-service-account=system-konnectivity-agent
      --delete-existing-uds-file=true
      --authentication-audience=system:konnectivity-server
      --proxy-strategies=default
      --keepalive-time=1m
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  200Mi
    Requests:
      cpu:        10m
      memory:     10Mi
    Liveness:     http-get http://:8134/healthz delay=15s timeout=15s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/kubernetes/konnectivity-server from konnectivity-uds (rw)
      /etc/kubernetes/kubeconfig from konnectivity-kubeconfig (ro)
      /etc/kubernetes/pki/ from konnectivityproxy-tls (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-csjnq (ro)
  apiserver:
    Container ID:  
    Image:         registry.k8s.io/kube-apiserver:v1.26.1
    Image ID:      
    Port:          30385/TCP
    Host Port:     0/TCP
    Command:
      /usr/local/bin/kube-apiserver
    Args:
      --advertise-address
      167.233.12.173
      --secure-port
      30385
      --etcd-servers
      https://etcd-0.etcd.cluster-dzd4vpffff.svc.cluster.local.:2379,https://etcd-1.etcd.cluster-dzd4vpffff.svc.cluster.local.:2379,https://etcd-2.etcd.cluster-dzd4vpffff.svc.cluster.local.:2379
      --etcd-cafile
      /etc/etcd/pki/client/ca.crt
      --etcd-certfile
      /etc/etcd/pki/client/apiserver-etcd-client.crt
      --etcd-keyfile
      /etc/etcd/pki/client/apiserver-etcd-client.key
      --storage-backend
      etcd3
      --enable-admission-plugins
      DefaultStorageClass,DefaultTolerationSeconds,EventRateLimit,LimitRanger,MutatingAdmissionWebhook,NamespaceLifecycle,NodeRestriction,Priority,ResourceQuota,ServiceAccount,ValidatingAdmissionWebhook
      --admission-control-config-file
      /etc/kubernetes/adm-control/admission-control.yaml
      --authorization-mode
      Node,RBAC
      --external-hostname
      dzd4vpffff.kubermatic.kubermatic.example.com
      --token-auth-file
      /etc/kubernetes/tokens/tokens.csv
      --enable-bootstrap-token-auth
      --service-account-key-file
      /etc/kubernetes/service-account-key/sa.key
      --service-cluster-ip-range
      10.240.16.0/20
      --service-node-port-range
      30000-32767
      --allow-privileged
      --audit-log-maxage
      30
      --audit-log-maxbackup
      3
      --audit-log-maxsize
      100
      --audit-log-path
      /var/log/kubernetes/audit/audit.log
      --tls-cert-file
      /etc/kubernetes/tls/apiserver-tls.crt
      --tls-cipher-suites
      TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
      --tls-private-key-file
      /etc/kubernetes/tls/apiserver-tls.key
      --proxy-client-cert-file
      /etc/kubernetes/pki/front-proxy/client/apiserver-proxy-client.crt
      --proxy-client-key-file
      /etc/kubernetes/pki/front-proxy/client/apiserver-proxy-client.key
      --client-ca-file
      /etc/kubernetes/pki/ca/ca.crt
      --kubelet-client-certificate
      /etc/kubernetes/kubelet/kubelet-client.crt
      --kubelet-client-key
      /etc/kubernetes/kubelet/kubelet-client.key
      --kubelet-certificate-authority
      /etc/kubernetes/pki/ca/ca.crt
      --requestheader-client-ca-file
      /etc/kubernetes/pki/front-proxy/ca/ca.crt
      --requestheader-allowed-names
      apiserver-aggregator
      --requestheader-extra-headers-prefix
      X-Remote-Extra-
      --requestheader-group-headers
      X-Remote-Group
      --requestheader-username-headers
      X-Remote-User
      --endpoint-reconciler-type
      none
      --profiling=false
      --service-account-issuer
      https://dzd4vpffff.kubermatic.kubermatic.example.com:30385
      --service-account-signing-key-file
      /etc/kubernetes/service-account-key/sa.key
      --api-audiences
      https://dzd4vpffff.kubermatic.kubermatic.example.com:30385,system:konnectivity-server
      --kubelet-preferred-address-types
      InternalIP,ExternalIP
      --egress-selector-config-file
      /etc/kubernetes/konnectivity/egress-selector-configuration.yaml
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   http-get https://:30385/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Readiness:  http-get https://:30385/healthz delay=0s timeout=15s period=5s #success=1 #failure=3
    Environment:
      SSL_CERT_FILE:  /etc/kubernetes/pki/ca-bundle/ca-bundle.pem
    Mounts:
      /etc/etcd/pki/client from apiserver-etcd-client-certificate (ro)
      /etc/kubernetes/adm-control from adm-control (ro)
      /etc/kubernetes/audit from audit-config (ro)
      /etc/kubernetes/cloud from cloud-config (ro)
      /etc/kubernetes/konnectivity from kube-apiserver-egress (ro)
      /etc/kubernetes/konnectivity-server from konnectivity-uds (rw)
      /etc/kubernetes/kubelet from kubelet-client-certificates (ro)
      /etc/kubernetes/pki/ca from ca (ro)
      /etc/kubernetes/pki/ca-bundle from ca-bundle (ro)
      /etc/kubernetes/pki/front-proxy/ca from front-proxy-ca (ro)
      /etc/kubernetes/pki/front-proxy/client from apiserver-proxy-client-certificate (ro)
      /etc/kubernetes/service-account-key from service-account-key (ro)
      /etc/kubernetes/tls from apiserver-tls (ro)
      /etc/kubernetes/tokens from tokens (ro)
      /var/log/kubernetes/audit from audit-log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-csjnq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  apiserver-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  apiserver-tls
    Optional:    false
  tokens:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  tokens
    Optional:    false
  kubelet-client-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubelet-client-certificates
    Optional:    false
  ca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ca
    Optional:    false
  ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ca-bundle
    Optional:  false
  service-account-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  service-account-key
    Optional:    false
  cloud-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloud-config
    Optional:    false
  apiserver-etcd-client-certificate:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  apiserver-etcd-client-certificate
    Optional:    false
  apiserver-proxy-client-certificate:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  apiserver-proxy-client-certificate
    Optional:    false
  front-proxy-ca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  front-proxy-ca
    Optional:    false
  audit-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      audit-config
    Optional:  false
  audit-log:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  adm-control:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      adm-control
    Optional:  false
  konnectivity-kubeconfig:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  konnectivity-kubeconfig
    Optional:    false
  konnectivityproxy-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  konnectivityproxy-tls
    Optional:    false
  konnectivity-uds:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-apiserver-egress:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-apiserver-egress
    Optional:  false
  kube-api-access-csjnq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason   Age                     From     Message
  ----    ------   ----                    ----     -------
  Normal  BackOff  2m28s (x439 over 102m)  kubelet  Back-off pulling image "eu.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-server:v0.0.35"

curl eu.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-server:v0.0.35
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://eu.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-server:v0.0.35">here</A>.
</BODY></HTML>

Answered by clickersmudge

Mar 16, 2023

@ahmedwaleedmalik @embik @xmudrii Thank you.

Thanks to the infrmation on Slack, we fixed this in the following way.

kubectl get machinedeployment --all-namespaces
kubectl edit machinedeployment --namespace kube-system   kubeone-pool1
	spec.template.spec.providerSpec.value.operatingSystemSpec.distUpgradeOnBoot: true
forceRestartAnnotations="{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"forceRestart\":\"$(date +%s)\"}}}}}"
kubectl patch machinedeployment --namespace kube-system   kubeone-pool1 --patch $forceRestartAnnotations --type merge
watch kubectl get nodes

View full answer

ahmedwaleedmalik · 2023-03-16T10:53:06Z

ahmedwaleedmalik
Mar 16, 2023

Hi @clickersmudge, which KKP version and OS are you using? This is a KKP installation on top of a KubeOne cluster if I'm not wrong?

If you are using KubeOne then what's the KubeOne version? Do you have OSM enabled or disabled?

A lot of questions :D but that'll help us understand and answer this in the best way possible.

4 replies

clickersmudge Mar 16, 2023
Author

KubeOne 1.6.0, KKP 2.22.0. Exactly " installation on top of a KubeOne cluster".

ahmedwaleedmalik Mar 16, 2023

Can you please check this thread to see if you are facing something similar? It seems like an issue with Ubuntu 22.04 and the latest systemd update.

clickersmudge Mar 16, 2023
Author

Unfortunately I do not have access to this workspace. Don’t have an @loodse.com email address?
Contact the workspace administrator at Kubermatic Community for an invitation. Can you grant me access?

embik Mar 16, 2023
Collaborator

This is a public Slack workspace for the community, you can find the link to join on https://docs.kubermatic.com/kubermatic/v2.22/how-to-contribute-to-kkp/#communication-channels.

clickersmudge · 2023-03-16T11:19:09Z

clickersmudge
Mar 16, 2023
Author

@ahmedwaleedmalik It seems to me that there are two problems here.

First

Init Containers:
  etcd-running:
    Container ID:  containerd://ee77a7ed7abe5195246cd7e730bc4db6b8c313317f937aacb5fa96986672667b
    Image:         gcr.io/etcd-development/etcd:v3.5.6
    Image ID:      gcr.io/etcd-development/etcd@sha256:28cb0630cb8536504f9bd547c3e63e608242c40dbffb1464c892d8d59fd3da44

curl gcr.io/etcd-development/etcd:v3.5.6
curl gcr.io/etcd-development/etcd:v3.5.6
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://gcr.io/etcd-development/etcd:v3.5.6">here</A>.

It appears that this image does not exist or has been moved.

2 replies

xmudrii Mar 16, 2023
Collaborator

@clickersmudge Do you have option to SSH to the node and try to pull image from there (e.g. with crictl pull)? Are you running on Hetzner, by the way?

clickersmudge Mar 16, 2023
Author

I apologise for the lack of information. Hetzner. I will check immediately and let you know.

clickersmudge · 2023-03-16T12:35:04Z

clickersmudge
Mar 16, 2023
Author

@xmudrii crictl pull

kubectl describe pod -n cluster-dzd4vpffff         etcd-1
Name:             etcd-1
Namespace:        cluster-dzd4vpffff
Priority:         0
Service Account:  etcd-launcher
Node:             kubeone-pool1-565c47dcfc-stwxl/192.168.0.8
Start Time:       Thu, 16 Mar 2023 10:02:38 +0100
Labels:           apiserver-etcd-client-certificate-secret-revision=38720
                  app=etcd
                  ca-secret-revision=38703
                  cluster=dzd4vpffff
                  controller-revision-hash=etcd-79ccd674d
                  etcd-tls-certificate-secret-revision=38714
                  statefulset.kubernetes.io/pod-name=etcd-1
Annotations:      cni.projectcalico.org/containerID: 441b2ef9a466cf33744e2ec8e91375383210c0bed37047655e122747fb4c8aca
                  cni.projectcalico.org/podIP: 10.244.9.93/32
                  cni.projectcalico.org/podIPs: 10.244.9.93/32
                  etcd.kubermatic.k8c.io/tls-peer-enabled: 
Status:           Pending
IP:               10.244.9.93
IPs:
  IP:           10.244.9.93
Controlled By:  StatefulSet/etcd
Init Containers:
  etcd-launcher-init:
    Container ID:  
    Image:         quay.io/kubermatic/etcd-launcher:v2.22.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/cp
      /etcd-launcher
      /opt/bin/
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /opt/bin/ from launcher (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tw22l (ro)
Containers:
  etcd:
    Container ID:  
    Image:         gcr.io/etcd-development/etcd:v3.5.6
    Image ID:      
    Ports:         2379/TCP, 2380/TCP, 2381/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /opt/bin/etcd-launcher
      -cluster
      dzd4vpffff
      -pod-name
      $(POD_NAME)
      -pod-ip
      $(POD_IP)
      -api-version
      $(ETCDCTL_API)
      -token
      $(TOKEN)
      -enable-corruption-check
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:      50m
      memory:   256Mi
    Liveness:   http-get http://:2378/health delay=5s timeout=10s period=30s #success=1 #failure=3
    Readiness:  exec [/usr/local/bin/etcdctl --command-timeout 10s endpoint health] delay=5s timeout=10s period=15s #success=1 #failure=3
    Environment:
      POD_NAME:                 etcd-1 (v1:metadata.name)
      POD_IP:                    (v1:status.podIP)
      NAMESPACE:                cluster-dzd4vpffff (v1:metadata.namespace)
      TOKEN:                    dzd4vpffff
      ENABLE_CORRUPTION_CHECK:  true
      ETCDCTL_API:              3
      ETCDCTL_CACERT:           /etc/etcd/pki/ca/ca.crt
      ETCDCTL_CERT:             /etc/etcd/pki/client/apiserver-etcd-client.crt
      ETCDCTL_KEY:              /etc/etcd/pki/client/apiserver-etcd-client.key
      PEER_TLS_MODE:            strict
    Mounts:
      /etc/etcd/pki/ca from ca (rw)
      /etc/etcd/pki/client from apiserver-etcd-client-certificate (ro)
      /etc/etcd/pki/tls from etcd-tls-certificate (rw)
      /opt/bin/ from launcher (rw)
      /var/run/etcd from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tw22l (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-etcd-1
    ReadOnly:   false
  etcd-tls-certificate:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  etcd-tls-certificate
    Optional:    false
  ca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ca
    Optional:    false
  apiserver-etcd-client-certificate:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  apiserver-etcd-client-certificate
    Optional:    false
  launcher:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-tw22l:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason   Age                      From     Message
  ----    ------   ----                     ----     -------
  Normal  Pulling  23m (x41 over 3h28m)     kubelet  Pulling image "quay.io/kubermatic/etcd-launcher:v2.22.0"
  Normal  BackOff  3m15s (x909 over 3h28m)  kubelet  Back-off pulling image "quay.io/kubermatic/etcd-launcher:v2.22.0"

root@kubeone-pool1-565c47dcfc-stwxl:~# crictl pull quay.io/kubermatic/etcd-launcher:v2.22.0
E0316 12:33:38.480713  783740 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"quay.io/kubermatic/etcd-launcher:v2.22.0\": failed to resolve reference \"quay.io/kubermatic/etcd-launcher:v2.22.0\": failed to do request: Head \"https://quay.io/v2/kubermatic/etcd-launcher/manifests/v2.22.0\": dial tcp: lookup quay.io: Temporary failure in name resolution" image="quay.io/kubermatic/etcd-launcher:v2.22.0"
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "quay.io/kubermatic/etcd-launcher:v2.22.0": failed to resolve reference "quay.io/kubermatic/etcd-launcher:v2.22.0": failed to do request: Head "https://quay.io/v2/kubermatic/etcd-launcher/manifests/v2.22.0": dial tcp: lookup quay.io: Temporary failure in name resolution 
root@kubeone-pool1-565c47dcfc-stwxl:~#

0 replies

clickersmudge · 2023-03-16T12:49:40Z

clickersmudge
Mar 16, 2023
Author

Isn't it strange that all pods in namespace "cluster-*" not working, but rest KubeOne,KKP, pod running?

2 replies

xmudrii Mar 16, 2023
Collaborator

Based on your previous comment, it seems like there are issues with the DNS resolution on that node:

failed to do request: Head "https://quay.io/v2/kubermatic/etcd-launcher/manifests/v2.22.0": dial tcp: lookup quay.io: Temporary failure in name resolution

Such issues are very environment and infrastructure specific, so it's hard to give a general answer to this. I recommend going through the upstream guide on resolving DNS issues: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

I also heavily recommend making sure that you're not affected by the issue that @ahmedwaleedmalik mentioned: #12065 (reply in thread). We'll soon have patch releases for KubeOne that are going to include a fix for this issue. There are already KKP patch releases that include this fix.

As one option, you might also want to consider rotating that broken node, i.e. replacing it with a new one. If it's managed by machine-controller, you can do that as described here: https://docs.kubermatic.com/kubeone/v1.6/cheat-sheets/rollout-machinedeployment/

Isn't it strange that all pods in namespace "cluster-*" not working, but rest KubeOne,KKP, pod running?

As of this question, it's hard to tell, but it might be that other images were already existing on node, so they are not pulled.

clickersmudge Mar 16, 2023
Author

Thanks everyone for helping. We read this issue on Slack and it is probably him.

clickersmudge · 2023-03-16T13:38:48Z

clickersmudge
Mar 16, 2023
Author

@ahmedwaleedmalik @embik @xmudrii Thank you.

Thanks to the infrmation on Slack, we fixed this in the following way.

kubectl get machinedeployment --all-namespaces
kubectl edit machinedeployment --namespace kube-system   kubeone-pool1
	spec.template.spec.providerSpec.value.operatingSystemSpec.distUpgradeOnBoot: true
forceRestartAnnotations="{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"forceRestart\":\"$(date +%s)\"}}}}}"
kubectl patch machinedeployment --namespace kube-system   kubeone-pool1 --patch $forceRestartAnnotations --type merge
watch kubectl get nodes

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User clusters do not get up after node restarts #12065

{{title}}

Replies: 5 comments 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

User clusters do not get up after node restarts #12065

clickersmudge Mar 16, 2023

Replies: 5 comments · 8 replies

ahmedwaleedmalik Mar 16, 2023

clickersmudge Mar 16, 2023 Author

ahmedwaleedmalik Mar 16, 2023

clickersmudge Mar 16, 2023 Author

embik Mar 16, 2023 Collaborator

clickersmudge Mar 16, 2023 Author

xmudrii Mar 16, 2023 Collaborator

clickersmudge Mar 16, 2023 Author

clickersmudge Mar 16, 2023 Author

clickersmudge Mar 16, 2023 Author

xmudrii Mar 16, 2023 Collaborator

clickersmudge Mar 16, 2023 Author

clickersmudge Mar 16, 2023 Author

clickersmudge
Mar 16, 2023

Replies: 5 comments 8 replies

ahmedwaleedmalik
Mar 16, 2023

clickersmudge Mar 16, 2023
Author

clickersmudge Mar 16, 2023
Author

embik Mar 16, 2023
Collaborator

clickersmudge
Mar 16, 2023
Author

xmudrii Mar 16, 2023
Collaborator

clickersmudge Mar 16, 2023
Author

clickersmudge
Mar 16, 2023
Author

clickersmudge
Mar 16, 2023
Author

xmudrii Mar 16, 2023
Collaborator

clickersmudge Mar 16, 2023
Author

clickersmudge
Mar 16, 2023
Author