grpc error "transport is closing"

[Renset]

Describe the bug
After migrating from micro v1 to microv3, even under a small load, some grpc connections break with error:
file=grpc/grpc.go:237 level=error grpc handler got error: rpc error: code = Unknown desc = connection error: desc = "transport is closing"
After that, an affected request is held until the timeout is reached.

No additional information can be found in the logs.
The same microservices with micro v1 worked well without such messages in the output.

To Reproduce
Steps to reproduce the behavior:

1. Run the micro server without any additional params
2. Run ~10 microservices (micro clients). Adding more clients doesn't affect the result.
3. Generate 10-20 requests per second

Environment:
Same results in Ubuntu and Windows Server

Are there any ideas about what could happen? Or maybe I can add some flags to micro to debug this deeper?

[iamnp]

I have the same error

[asim]

The transport in v1 was a custom protocol. v3 uses grpc. I don't have more details on the reported error. It requires investigation. This is likely somewhere in the client. Starting point might be to look at the logs of your service and see if there's errors there, secondly in the logs for the network service.

[Renset]

Thanks for your suggestions @asim .
Unfortunately, there are no additional logs on the client side.
However, I've investigated the TCP dump in WireShark and found out that just before transport is closing error message, the micro proxy closes the connection (sends packet with FIN flag) and opens a new one. Are there some actions I could perform for further investigation on the server side?

)

[asim]

How are you querying these services? Via cli? Via a custom client?

[Renset]

Custom clients.

BTW I found the workaround solution: when starting proxy with MUCP protocol it works well:

micro service proxy --protocol mucp

Thus, looks like this issue can be closed