HTTP connection stuck on tcp state CLOSE_WAIT, Cats-effect backend

[altrack]

Hey folks,

I'm running sttp stable 3.9.0 with the cat-effect backend with the latest JVM 17.
Occasionally, the server closes the connections, and the call stuck in CLOSE_WAIT state.
According to the info I found regarding this issue, it shouldn't happen and the client should close the connection.

Thanks

[adamw]

Is that the sttp.client3.httpclient.cats.HttpClientCatsBackend backend?

[altrack]

Is that the sttp.client3.httpclient.cats.HttpClientCatsBackend backend?

That's right sbt dependency
com.softwaremill.sttp.client3 :: cats

package sttp.client3.httpclient.cats
HttpClientCatsBackend

basicRequest
      .contentType(MediaType.ApplicationJson)
      .httpVersion(HttpVersion.HTTP_1_1)
....

Seems to have the same issue in 3.8.15

[altrack]

Dumping Cats-effect engine getting this suspicious trace:

cats.effect.IOFiber@118ae5d4 BLOCKED
 ? delay @ sttp.client3.impl.cats.CatsMonadError.eval(CatsMonadError.scala:19)
 ? map @ sttp.client3.impl.cats.CatsMonadError.map(CatsMonadError.scala:9)
 ? flatMap @ sttp.client3.impl.cats.CatsMonadError.flatMap(CatsMonadError.scala:12)
 ? map @ sttp.client3.impl.cats.CatsMonadError.map(CatsMonadError.scala:9)
 ? map @ sttp.client3.impl.cats.CatsMonadError.map(CatsMonadError.scala:9)
 ? delay @ sttp.client3.impl.cats.CatsMonadAsyncError.$anonfun$async$1(CatsMonadAsyncError.scala:10)
 ? map @ sttp.client3.impl.cats.CatsMonadAsyncError.$anonfun$async$1(CatsMonadAsyncError.scala:10)
 ? async @ sttp.client3.impl.cats.CatsMonadAsyncError.async(CatsMonadAsyncError.scala:10)
 ? map @ sttp.client3.impl.cats.CatsMonadError.map(CatsMonadError.scala:9)
 ? delay @ sttp.client3.impl.cats.CatsMonadError.eval(CatsMonadError.scala:19)
 ? flatMap @ sttp.client3.impl.cats.CatsMonadError.flatMap(CatsMonadError.scala:12)
 ? recoverWith @ sttp.client3.impl.cats.CatsMonadError.handleWrappedError(CatsMonadError.scala:17)

[adamw]

I think it's rather something in the java's HttpClient configuration, or the way you use the backend, rather then the specific cats integration.

Can you share some more details as to how you use the sttp client and the backend (when and how you create the backend, is it reused), and what do you mean exactly by "the call is stuck"? Does some method call block, while it shouldn't?

A reproducing example would be best, but I suspect it might be hard to create. But something close to it would help diagnose the issue.

[altrack]

I swapped the backend to sttp.client3.armeria.cats.ArmeriaCatsBackend and this resolved the issue. No other change in the code was done.

My use case is a batching of 4000 http requests in parallel and waiting until completion. The response from the server (Python's Uvicorn) can take up to a minute or so. After 20-40 minutes the application doesn't continue to the next batch.
Entering the application pod I see some socket are not able to close.
ss --tcp state CLOSE-WAIT
The amount of CLOSE-WAIT connections is equal to the amount of threads in Cats-effect engine, so this entirely blocks the application. Cats-effect fiber dump for one of them is above.

Unfortunately, I don't have a test to reproduce the issue.