RabbitMQ.Client.Exceptions.AlreadyClosedException

[nikriaz]

Publisher connection seems to be switched to some invalid state. Since then I was getting an exception on the each publish attempt:

2024-05-10 00:03:30.5579|ERROR|AFapiCore.Rabbit.RabbitService|RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=200, text='Timed out waiting for acks', classId=0, methodId=0, exception=System.IO.IOException: timed out waiting for acks
   at RabbitMQ.Client.Impl.SessionBase.Transmit(OutgoingCommand& cmd)
   at RabbitMQ.Client.Impl.ModelBase.BasicPublish(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, ReadOnlyMemory`1 body)

It happened in very random ordinary moment. No high load, no idling, just nothing special. Restarting publisher helped, consumer was okay, broker was okay.

In the broker log at the moment of the first problem occurrence:
2024-05-09 13:52:15.244000+03:00 [warning] <0.43641.0> Channel is stopping with 1 pending publisher confirms

Any ideas what does it mean? Do we have something like error id descriptions (for code = 200)? Any ideas how to properly recover from such situations in the publisher code?

[michaelklishin]

We do not guess in this community. Either provide more complete logs and relevant snippets of your code, or direct your question to a fortune teller.

"Timeout waiting for acks" suggests you may be using a "wait for confirms or die" kind of API functions , a.k.a. "publish-and-wait", which is inefficient and often unnecessary.

2024-05-09 13:52:15.244000+03:00 [warning] <0.43641.0> Channel is stopping with 1 pending publisher confirms

is fairly specific. If a channel has to terminate for any reason, whether it was cleanly closed or due to a channel-level protocol exception, and it had a pending publisher confirm ack on it, it will log this warning because it might (but won't necessarily, it depends on how your publishers handle retries) lead to the loss of that published and unconfirmed message.

[nikriaz]

My code is fairly the same as C# tutorials and there is really no more logs: only multiple repetition of these lines.

But you're absolutely right. I added channel.WaitForConfirmsOrDie for experimental purposes. It seems like one confirm was missed, then this fired an exception that turned channel to the stopping state that was never recovered. It was unexpected: I thought "Die" in "WaitForConfirmsOrDie" means message dying but it looks like a channel. I also thought that AutomaticRecoveryEnabled = true is something that should handle such things but seems like no.

I re-read tutorial and see that "wait for confirms or die" is not what we want and implemented asynchronous confirm handling through events callbacks as per tutorial.

I also added recovery like this, which is called if exception is throwing:

            _channel?.Close();
            _channel?.Dispose();
            _channel = null;

            _connection?.Close();
            _connection?.Dispose();
            _connection = null; 
                        
            Initialize(); //Same init is called in the constructor at service startup

I'm not sure whether it's correct or no. Fully operational example with all these details included would be highly appreciated.
Anyway, thank you very much for your answer! It was actually very informative and helpful.

[lukebakken]

Thanks for the follow-up. There's not much I can add without seeing your code. FWIW, we do have the following example:

https://github.com/rabbitmq/rabbitmq-tutorials/blob/main/dotnet/PublisherConfirms/PublisherConfirms.cs

I also thought that AutomaticRecoveryEnabled = true is something that should handle such things but seems like no

No, because missing a confirmation isn't really something that the library can automatically recover from - its up to your application to decide what to do.