In the event of a kubernetes node failure..

[bshepyard]

In the event of a kubernetes node failure how is it possible for a strimzi operator to recover a kafka broker or zookeeper Pod that was running on that node. Currently the pods just go into terminating state and the operator fails to reconcile the failed broker and zookeeper pods by timing out. This is a bare metal kubernetes setup and I'm using ephemeral storage for the strimzi cluster.

[scholzj]

It depends on the exact state of the node and the pod. You can always force delete the pod to continue.

PS: Ephemeral storage is not suitable for any kind of prodution-grade use-cases etc.

[bshepyard]

I am testing the failure of a node so I am powering it off. After a while all deployments and stateful sets restart new pods and the old ones are set to Terminating except the sonar simulator operator which fails to restart as the liveliness and readiness probes fail. And any Kafka broker or zookeeper pod just sits in a Terminating state indefinitely with no new pods created.
strimzi-pods.txt

[scholzj]

What Strimzi version are you using? What is the full YAML of the Pod?

[bshepyard]

I installed the operator with helm strimzi-kafka-operator-0.40.0
Then I downloaded from the Strimzi site a yaml file to create the cluster see attached.
kafka-cluster.txt

[scholzj]

But that is not Pod YAML, so it does not explain the state of the Pod. Strimzi will delete the pod once it terminates -> when it is in failed or succeeded state. There is not much to know about its actual state when it is not terminated. If we delete it, we might cause data loss or other things.

[bshepyard]

It wont terminate the Pod until the node comes back online or its manually force deleted!