CQ with no consumers not expiring messages (via message-ttl) but getting a single message via WebUI reproducibly triggers an expire

[frittentheke]

Describe the bug

I am running a RabbitMQ cluster with 3 nodes on Ubuntu Jammy (22.04 LTS) wíth RabbitMQ at 3.12.13 and Erlang 26.2.2.
The cluster is used for an OpenStack installation, with the usual queue structure. We do use Quorum Queues where possible (so all non "transient" queues).
One quorum queue in particular, notifications.info, shows a very strange, suspected buggy behavior:

The queue receives messages from various clients, but there might be no consumers for a longer period of time.
Therefore we configured a policy Notifications@/ to apply a message-ttl (and also a queue expire time) in order to only hold messages from the last 4 hours and expire those that become older:

expires:	604800000
message-ttl:	14400000

The policy actually does become effective for the queue:

but no messages are expired and they are building up over the configured message expire period.
We purged the queue once to avoid having any "old" non ttled messages in the queue, but that did not seem help (or only for a while, see first few hours of Prometheus graph below).

Strangely though, if one uses the WebUI to get a single message, the queue applied the message-ttl and expires all the older messages. This can be seen "nicely" in the WebUI and Prometheus metrics:

Around the time the issue started, RabbitMQ started logging rabbit_sysmon_handler busy_dist_port at different times and intervals across the three nodes:

Mar 07 06:48:45 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.5752.0> [{initial_call,{rabbit_prequeue,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:49:04 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.6817.0> [{initial_call,{rabbit_prequeue,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:49:04 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.6820.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:49:22 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.5752.0> [{initial_call,{rabbit_prequeue,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:49:27 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.439024.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:49:43 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.5752.0> [{initial_call,{rabbit_prequeue,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:49:45 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.3277.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:49:52 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.5752.0> [{initial_call,{rabbit_prequeue,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:50:27 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.439024.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:50:46 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.3277.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:52:21 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.8435.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:52:28 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.3277.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:52:47 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.6817.0> [{initial_call,{rabbit_prequeue,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:52:47 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.6820.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:53:02 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.7884.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:53:22 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.9510.0> [{initial_call,{rabbit_prequeue,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:53:22 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.9515.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:53:25 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.3277.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:54:05 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.3277.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:54:07 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.3277.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:54:30 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.439024.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:56:29 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.7884.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:56:49 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.7884.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:57:21 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.7884.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:57:29 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.7884.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:57:30 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.7884.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:57:50 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.8435.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:58:01 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.9510.0> [{initial_call,{rabbit_prequeue,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:58:01 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.9515.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}
Mar 07 06:58:22 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.3277.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.624>,unknown}
Mar 07 06:58:30 server01 rabbitmq-server[924545]: rabbit_sysmon_handler busy_dist_port <0.7884.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.1488>,unknown}

Mar 06 16:20:23 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:25:24 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:26:25 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:27:26 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:32:33 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:35:36 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:39:40 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:40:39 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:42:42 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:45:42 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:46:44 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:47:45 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:48:45 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 16:55:53 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 17:06:05 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 17:09:05 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 17:16:16 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 17:21:26 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 17:26:30 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 17:27:31 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 17:29:31 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}
Mar 06 17:30:32 server02 rabbitmq-server[2341771]: rabbit_sysmon_handler busy_dist_port <0.341155.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.770>,unknown}

Mar 07 06:16:14 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:22:19 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:24:20 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:29:25 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:31:27 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:35:33 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:37:35 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:38:35 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:41:38 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:42:40 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:45:42 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:46:44 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:47:45 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:51:49 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}
Mar 07 06:53:52 server03 rabbitmq-server[2149712]: rabbit_sysmon_handler busy_dist_port <0.289654.0> [{initial_call,{gm,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.16>,unknown}

I found and read https://www.rabbitmq.com/docs/runtime#distribution-buffer, but did not make any adjustments yet.

Reproduction steps

We tried to reproduce the issue in another environment, but were not really successful.
Maybe there is some factor that we missed and did not apply to the test-setup the incoming message pattern / sizes make a difference.

1. Apply a policy with message TTL of e.g. 4 hours to a quorum queue
2. Constantly send messages to this queue, which has no consumers
3. See messages older than set message-ttl to NOT expire
   ...

Expected behavior

Applying a message-ttl if 4 hours to a (quorum) queue is expected for that queue to expire them, no matter if there are consumers or not.

Additional context

1. We already saw this issue with RabbitMQ 3.12.11 and just did an update to 3.12.13.
   But the issue remained through the update and rolling restart of the cluster.
2. Certainly we could try and delete the queue to have it recreate, but we first wanted to report this bug to be able to deliver more debug data with the issue present. So if there is any more diagnosis we can do and provide, please let me know.

[michaelklishin]

This is a documented behavior, messages are expired when they reach the head of the queue and a consumption.

There are no plans to change this. Expired messages will not be delivered to consumers, as documented, and that's the key objective. Queues without consumers is a scenario RabbitMQ does not optimize for. If you need to store data for a period of time with exclusively time-based expiration and with sporadic consumer availability, streams are a better option.

Inter-node communication buffer is completely irrelevant to message TTL.

[frittentheke]

Thanks @michaelklishin for taking the time to answer this question. Embarrassingly I actually read (briefly) through the referenced documentation (https://www.rabbitmq.com/docs/ttl#quorum-queues), but unfortunately did not understand the consequence of "when it reaches the head of the queue" explanation properly.

We switched from Classic to Quorum queues a while back and I am quite certain that the message-ttl actually did work without consumers before. At least classic queues seems to do the expiry also when a policy is changed, https://www.rabbitmq.com/docs/ttl#classic-queues, is this not the case for quorum queues?

Let me also state that I understand that expiring messages from queues which are not actually used is not THE key feature for RabbitMQ or any message broker. But from an operational perspective it's always good to have "timeouts" configured, as high as they might be, to ensure no message will live forever. So is there any way to manually "trigger" a queue expire (via API or rabbitmqctl)? Or is purging a queue or fetching a message the only option?

[frittentheke]

@michaelklishin may I nag you again, about an idea for any "administrative" action via API or similar to trigger message expiration?

[michaelklishin]

@frittentheke if your environment has queues that have no consumers and should eventually delete all their data, I suggest that you set the TTL on the queue itself.

Triggering expiration at a different moment or on a schedule with certain conditions could be an option, you are welcome to look into how that might work and what it might take R&D effort-wide. This is open source software after all.

An "administrative action" that would allow a user to potentially trigger an expensive operation on thousands or tens of thousands of queues does not sound like a good idea to me.

[michaelklishin]

As someone on the team correctly points out, timer-based expiration will violate strict FIFO guarantees of, say, quorum queues.

And a "middle ground" option like expiring periodically up to the first non-expired message may or may not make much difference in practice. But using queue TTL instead for queues that do not have consumers for a long time is guaranteed to work and achieve the end goal of not allowing applications to leak resources.

This idea needs an RFC-style design document, not a single sentence description.

[frittentheke]

@michaelklishin First of all thanks again for your time and thoughts on this matter!

For clarity, the use case I have is OpenStack (oslo.messaging), but could be any other conglomerate of services, that has optional components and in case they are not installed / used certain messages to certain queues are simply not consumed. Yes, it might be best to stop sending them, which is possible, but I'd also like to configure safety net on the broker side that ensures no message lives forever (and requires me to purge queues at some point to get rid of them).

@frittentheke if your environment has queues that have no consumers and should eventually delete all their data, I suggest that you set the TTL on the queue itself.

But the queues in question might constantly receive messages. Will a queue TTL then work at all?

As someone on the team correctly points out, timer-based expiration will violate strict FIFO guarantees of, say, quorum queues.
And a "middle ground" option like expiring periodically up to the first non-expired message may or may not make much difference in practice. But using queue TTL instead for queues that do not have consumers for a long time is guaranteed to work and achieve the end goal of not allowing applications to leak resources. This idea needs an RFC-style design document, not a single sentence description.

Yes my idea was just to trigger the existing message-TTL expiry mechanism that apparently is also working if I fetch a single message (as per my initially reported issue). I was rather looking for something I could do via rabbitmqctl or the API, let's say once a week ... somewhat of a housekeeping action.

Another point which I'd like to clarify is the wording in the documentation as you agreeably receive questions about message-ttl not working as expected more often than necessary. So https://www.rabbitmq.com/docs/ttl#message-ttl-dead-lettering states:

Quorum queues dead letter expired messages when they reach the head of the queue.

What does that mean in relation to having no consumers? Will a quorum queue reliably expire messages which all have a message-ttl set from the beginning (so not retroactively)? Cause we have a cluster that seems to happily expire messages with no consumers for weeks now. But I did open the issue because it stopped working somewhere sometime. So I'd just like to clarify the conditions on message expiry and what head of the queue means. Is it that the first message in the queue has to have an expired ttl in order for message expiry work without an active consumer?