Mutiny is creating zombie thread on delayIt() which are not doing anything after certain time

[mitkama]

We have retry mechanism in our application which waits for some time (2 seconds) and then do the retry.

If we use this in our application throughput is getting dropped to absolute zero after 2 -3 days, thread are going into unresponsive state.

Attached screenshots of our JVM dashboard.

Code block which we have in our mutiny pipeline is as below.

return
          Uni.combine().all().unis(delayUni, refreshedCredentialsUni)
              .asTuple()
              .onItem()
              .delayIt()
              .by(Duration.ofSeconds(Math.min(backOffDuration, getMaxBackoff())))
              .onItem()
              .transformToUni(r -> cspResponseUni)

I have found similar thread here --> #786

[mitkama]

Once we disable this delay, our throughput reduction problem is solved and we are not seeing this issue anymore.

But we need mechanism wherein the above code should work seam lessly.

[cescoffier]

Are you running in Quarkus? There were a leak in the Jboss scheduler used in Quarkus.

As a workaround, you could define your own executor to execute the delay (instead of using the default one).

[mitkama]

@cescoffier
Yes, we are running in Quarkus.

Despite supplying an external thread pool we are still observing the same behavior.

engineExecutorService = Executors .newFixedThreadPool(engineThreadPoolSize, new DefaultThreadFactory("worker"));

emitting on below thread pool

.emitOn(EngineThreadPool.getEngineExecutorService());

[cescoffier]

Ah no, sorry, not using emitOn, but onItem().delayIt().on(executor).by(..)

[mitkama]

@cescoffier

We haven't supplied an external pool in that pipeline.