[Bug]: Strimzi operator unable to spin up kafka cluster on gke

[wali97]

Bug Description

Hi,
I've installed strimzi operator version 0.40.0 and trying to setup kafka version 3.7 on my kubernetes cluster running with version 1.28.5. I'm using helm for strimzi installation and kind:kafka manifest for deploying kafka cluster,as soon as i apply the manifest i run into an error " BlockedThreadChecker: - Thread Thread[kubernetes-ops-pool-4,5,main] has been blocked for 121444 ms, time limit is 120000 ms io.vertx.core.VertxException: Thread blocked". I've attached the full log file of strimzi operator pod. Not sure as to what is causing this error.

Steps to reproduce

No response

Expected behavior

The strimzi operator should be able to deploy the kafka cluster.

Strimzi version

0.40.0

Kubernetes version

GKE 1.28.5

Installation method

Helm Chart

Infrastructure

Google Kubernetes Engine

Configuration files and logs

The strimzi operator pod log file strimzi-cluster-operator.log

The helm cmd i'm using to install strimzi on my cluster,
"helm -n v0-kafka install strimzi-kafka-operator strimzi/strimzi-kafka-operator --version 0.40.0"

The kafka cluster manifest which strimzi operator will deploy,
kafka-cluster.txt

Additional context

No response

[scholzj]

The log suggests that some operation on your Kubernetes API server is timing out. An error like this could have many reasons from insufficient resources, networking issues, up to for example some misbehaving admission controller etc. A debug log from the operator might show what exactly is the operation. But from the log it looks to me that it is creating of a Secret.

So you would need to investigate on your cluster what the problem is.

[wali97]

Hi, @scholzj
I highly suspect that this could be kubernetes API server issue,since i've 100+ microservices running within the cluster and autoscaling is enabled so memory & cpu resources is also out of the picture.
I've attached the debug level logs too
strimzi-cluster-operator-debug.log

[scholzj]

Right, yeah. That confirms that it is the Secret creation request that times out. Unfortunately, it is not always easy to debug. It could be the API server itself. But also some Admission Controler or even something around networking.

[wali97]

can suggest any fix for this?

[scholzj]

No, I'm afraid I cannot. As I said, it is not easy to figure out what might be the issue. Checking Kubernetes logs might be a good place to start I guess.