Topology updater pod in crashloopbackoff

[urbaman]

Hi,

In my 6-nodes cluster, I tryed to deploy the topology updater, but 5 out of 6 pods are in crashloopbackoff state. Is there a way to fix this?

kubectl get pods -n node-feature-discovery
NAME                                                      READY   STATUS             RESTARTS        AGE
nfd-node-feature-discovery-master-79978fb9c-9s9rp         1/1     Running            0               8m15s
nfd-node-feature-discovery-topology-gc-85b5f78f89-lhwm4   1/1     Running            0               8m15s
nfd-node-feature-discovery-topology-updater-5v9t2         1/1     Running            0               8m15s
nfd-node-feature-discovery-topology-updater-5xm6b         0/1     CrashLoopBackOff   6 (2m25s ago)   8m15s
nfd-node-feature-discovery-topology-updater-8d6xw         0/1     CrashLoopBackOff   6 (2m15s ago)   8m15s
nfd-node-feature-discovery-topology-updater-8tlxj         0/1     CrashLoopBackOff   6 (2m29s ago)   8m15s
nfd-node-feature-discovery-topology-updater-g8shx         0/1     CrashLoopBackOff   6 (2m19s ago)   8m15s
nfd-node-feature-discovery-topology-updater-znbnl         0/1     CrashLoopBackOff   6 (2m16s ago)   8m15s
nfd-node-feature-discovery-worker-2fg9x                   1/1     Running            1 (7m49s ago)   8m15s
nfd-node-feature-discovery-worker-9dk8h                   1/1     Running            0               8m15s
nfd-node-feature-discovery-worker-rcb4z                   1/1     Running            1 (7m50s ago)   8m15s
nfd-node-feature-discovery-worker-vdl8r                   1/1     Running            0               8m15s
nfd-node-feature-discovery-worker-wtl76                   1/1     Running            1 (7m54s ago)   8m15s
nfd-node-feature-discovery-worker-xbqfc                   1/1     Running            1 (7m51s ago)   8m15s

Logs from one of the failing pods

nfd-node-feature-discovery-topology-updater-5v9t2         1/1     Running            0               8m15s
nfd-node-feature-discovery-topology-updater-5xm6b         0/1     CrashLoopBackOff   6 (2m25s ago)   8m15s
nfd-node-feature-discovery-topology-updater-8d6xw         0/1     CrashLoopBackOff   6 (2m15s ago)   8m15s
nfd-node-feature-discovery-topology-updater-8tlxj         0/1     CrashLoopBackOff   6 (2m29s ago)   8m15s
nfd-node-feature-discovery-topology-updater-g8shx         0/1     CrashLoopBackOff   6 (2m19s ago)   8m15s
nfd-node-feature-discovery-topology-updater-znbnl         0/1     CrashLoopBackOff   6 (2m16s ago)   8m15s

[marquiz]

What k8s version are you running? What do the pod logs say?

[urbaman]

Hi, here's my cluster kube version:

kubectl get nodes
NAME    STATUS   ROLES           AGE   VERSION
k8cp1   Ready    control-plane   90d   v1.28.1
k8cp2   Ready    control-plane   90d   v1.28.1
k8cp3   Ready    control-plane   90d   v1.28.1
k8w1    Ready    <none>          90d   v1.28.1
k8w2    Ready    <none>          90d   v1.28.1
k8w3    Ready    <none>          90d   v1.28.1

The logs from one of the failing pds are:

kubectl logs -n node-feature-discovery nfd-node-feature-discovery-topology-updater-5bbc7
Defaulted container "topology-updater" out of: topology-updater, k8tz (init)
F0828 16:27:21.614249       1 main.go:87] failed to get kubelet config from configz endpoint: failed to unmarshal json for kubelet config: json: cannot unmarshal string into Go struct field LoggingConfiguration.kubeletconfig.logging.flushFrequency of type time.Duration

Probably because of the kubernetes version, as the only working pod was on a worker still at 1.27.4 version? Now the worker is updated and the topology updater is not working on that node as well

[marquiz]

Interesting 🧐 And not nice. Looks like Kubernetes v1.28 broke compatibility wrt to the FlushFrequency field (i.e. older k8s version cannot marshal data generated from a newer version as the type has changed). We need to fix this somehow, I think the simplest would be simply to bump k8s version to v1.28 in the release-0.13 branch (and cut a new patch release). @PiotrProkop @ArangoGutierrez any thoughts on this?

[PiotrProkop]

Let's do it!

[marquiz]

Fixed