The Operation has timed out client error - On Rabbitmq 3.12.13 64 bit env.

[hantst]

Hello All,

We have an rabbitmq environment with 3 nodes running behind load balancer. HA-ALL policy is applied and its running with default configurations.

We installed RabbitMQ 3.12.10 & Erlang 26.0.2 but since we have the below error thrown in random hours, thinking it could be a bug we upgraded rabbitmq to 3.12.13 but error still continoues. Can anyone help us regarding the issue ?

Kind regards,

|RABBITMQNOTIFY|System.TimeoutException: The operation has timed out.
at RabbitMQ.Util.BlockingCell1.WaitForValue(TimeSpan timeout) at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout) at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body) at RabbitMQ.Client.Framing.Impl.Model._Private_ChannelOpen(String outOfBand) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.CreateNonRecoveringModel() at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.CreateModel() at Peak.App.ESB.EventProcessing.RabbitMQ.SyncAckModelEnlistment.<>c.<.ctor>b__12_0() at System.Lazy1.CreateValue()
at System.Lazy`1.LazyInitValue()
at Peak.App.ESB.EventProcessing.RabbitMQ.SyncAckModelEnlistment.Commit()
at Peak.App.ESB.EventProcessing.TransactionScopeAttachment.NotifyEnlistments(TransactionScopeAttachment attachment, TransactionStatus status)

[michaelklishin]

This exception means that a channel.open response did not arrive on time. There can be a lot of reasons for that, including a genuine network delay.

See RabbitMQ node logs for clues.

[lukebakken]

@hantst - we need to know what is logged by RabbitMQ when these exceptions happen. Other questions:

• Has your application worked correctly with RabbitMQ for some period of time, or have you always seen this issue?
• Can you identify when these exceptions started?

[hantst]

@michaelklishin thank you for response. Well, i checked rabbitmq logs and all i see is lots and lots of these outputs. Can't see ant other thing in rabbitmq logs at specific time when client gave that error.

2024-03-06 02:04:42.981000+03:00 [info] <0.232907.0> accepting AMQP connection <0.232907.0> (10.30.xx.xx8228 -> 10.30.xx.xx:5672)
2024-03-06 02:04:43.091000+03:00 [info] <0.232907.0> connection <0.232907.0> (10.30.xx.xx:8228 -> 10.30.xx.xx:5672): user 'AdminRMQ' authenticated and granted access to vhost '/'
2024-03-06 04:00:05.575000+03:00 [warning] <0.232907.0> closing AMQP connection <0.232907.0> (10.30.xx.xx:8228 -> 10.xx.xx.xx:5672, vhost: '/', user: 'AdminRMQ'):
2024-03-06 04:00:05.575000+03:00 [warning] <0.232907.0> client unexpectedly closed TCP connection

@lukebakken Hello, thank you for response.

Well before rabbitmq upgrade we had 3.7.X version and it was fine. Upgrading rabbitmq requires also revision on the client app, our developers did that. But we see this error at random times after upgrade and events for that time frame takes long time to process (30 seconds).

[hantst]

@lukebakken I installed new rabbitmq & erlang version on top of old one ( old one was 3.11.XX ) . Same servers and everything. I installed exe on same home path and continue. (I did 3.12.0 upgrade first, checked feauture flags and so on) other than that nothing changed.

I've put the logs in the attachments section.

rabbitmq_logs.zip

[lukebakken]

other than that nothing changed.

Your application changed. Please roll back and see if the issue continues to happen in this environment.

[hantst]

@lukebakken Thank you my friend. I will talk with developers. Were you able to examine the logs i sent you.

[lukebakken]

No, I am working on higher-priority issues. Free support comes last.

[lukebakken]

Did you edit these log files to remove node names and other information?

I see very strange errors from the HTTP API (cowboy) but I don't see anything else of concern. You are using classic mirrored queues, which are deprecated, and should seriously consider using quorum queues. You are also mirroring to all nodes, which is unnecessary.

[hantst]

here is some screenshots from rabbitmq console: