Quorum queue can not be scaled out accordingly as the rabbitmq cluster has been scaled out from 3 nodes to 5 nodes

[xixiangzouyibian]

Hello team,
As title mentioned, I deployed 3-node rabbitmq cluster by rabbitmq operator on kubernetes environment and created serveral quorum queues, each quorum queue state is 1 master with 2 followers. Now I patch the replicas to rabbtimqcluster (CRD), then rabbitmq statefulset scales out to 5 nodes. But quorum queues are still 3. What I expect is quorum queues are also scale out to 5.

I checked the document: https://www.rabbitmq.com/quorum-queues.html#replica-management
But from my understanding, the mentioned steps should effect on bare physical environment while not effect on kubernetes since we have rabbitmq operator. I suppose the operator should do this works. I'm not sure if the case is expected by design, or I missed some something.

Could you please provide the solution how can I do for quorum queues as the cluster has been scaled out.

[mkuratczyk]

It is perfectly valid to have a 5-node cluster with 3-node quorum queues, so the Operator doesn't make the assumption that you want to scale out all the quorum queues, just because you added nodes. What do you want to achieve by adding followers? You might be interested in the feature coming in 3.13, where quorum queues membership is evaluated periodically and the queues could grow automatically (but first, make sure adding followers is what you need). pull request: rabbitmq/rabbitmq-server#8218 discussed in: https://www.youtube.com/watch?v=_klVJpiimIU Best,

[xixiangzouyibian]

Hello Michal, thanks for support.

What I want to achieve is quorum queue can be scaled out automatically according to the cluster node so that the quorum queue's FTT (Failure To Tolerance) can be promoted from 1 to 2 to defense 2 nodes loss case.
As the number of physical k8s nodes increases, rabbitmq replicas may increase dynamically, such as from 3 to 5.
Without this function, all quorum queues will always be 3 (depends on initial group size), that is, 1 leader and 2 followers. At this time, for the cluster, if two pods are lost, the cluster can still be accessed. However, if the two lost pods happen to have the current quorum queue, then the quorum queue has lost its majority and cannot provide services. At this time, the availability of cluster becomes irrelevant.

I' m glad to hear that the functionalities could be provided in 3.13, the feature is very important to us. When can I use it, when 3.13 rabbitmq can be integrated into the rabbtimq operator, and how can I configure it ?

[mkuratczyk]

For testing, you can use it now, just use this image:
pivotalrabbitmq/rabbitmq:v3.13.0-beta.6-otp-min-bazel
or
pivotalrabbitmq/rabbitmq:main-otp-min-bazel (a new beta release will be built soon based on main)

I don't know if more docs has been written already but the PR and the video should get you started.

[xixiangzouyibian]

Got it, Thanks Michal.

[xixiangzouyibian]

So for now, do I need to manually grow the quorum queues as the cluster scales out ?

[mkuratczyk]

yes

[xixiangzouyibian]

@mkuratczyk Greetings to you.

I noticed that 3.13 has been officially released, but I have not found the configuration instructions related to automatically grow.
From the posted PR, I can see that there is one parameter that needs to be configured:
quorum_queue.continuous_membership_reconciliation.target_group_size

This appears to be a new configuration, so one question is: if it is already supported in the current 2.7.0 rabbitmq-operator ?

Also, if my rabbitmq replicas number changes from 3 to 5, and I expect all quorum queues to automatically grow from 1 primary and 2 followers to 1 primary and 4 followers, do I need to set "target_group_size" to 5 at the beginning of deployment manifest file and does this configuration impact replica 3 case ?

[mkuratczyk]

Cluster Operator is not really involved in this. It just passes the config you specify in rabbitmqCluster definition to a configmap that is used by RabbitMQ.

I don't understand the rest of the question. I suggest you experiment with this and share the answer here. It's a new feature, so it'd be great to get your feedback from using/testing it.

[xixiangzouyibian]

Got you. So the configuration should be added into "spec.rabbitmq.additionalConfig" in rabbitmqCluster CRD, right ?

Yes, I did the tests and suffered in the other problems.
I used the official rabbitmq 3.13.0-management image and 2.7.0-debian-12-r8 RabbitMQ cluster operator.
When I deploy 3 nodes, I found that the 3 RabbitMQ pods do not form a cluster (Verified by rabbitmqctl cluster_status), it was the same situation every time.
So I am not sure whether it is a compatibility problem or some other problem, which blocks my automatically grow testing.

[mkuratczyk]

yes, like all other configuration

of course if the cluster is not formed correctly, that's a major issue and quorum queue membership reconciliation can't work

There were changes to peer discovery in 3.13. We are aware of issues in IPv6 environments, but if that's not your case, please create an issue with reproduction steps.

[xixiangzouyibian]

Filed the issue: #1602.
Please feel free to check

[xixiangzouyibian]

Filed the issue: #1602.
Please feel free to check

Since the issue was resolved, move on the progress of functional testing. My test steps:

1. Deployed 3-node rabbitmq cluster, and specify below configuration in spec.rabbitmq.additionlConfig (Pay attention: the origin replicas number is 3)：

      quorum_queue.continuous_membership_reconciliation.enabled = true
      quorum_queue.continuous_membership_reconciliation.interval = 5
      quorum_queue.continuous_membership_reconciliation.target_group_size = 5

2. After deployment complete, create test quorum queue:
   rabbitmqadmin declare queue name=test-queue durable=true arguments='{"x-queue-type": "quorum"}' -u guest -p guest
   At this time , the quorum queue status is 1 leader and 2 followers

3. Enlarge the cluster: kubectl edit rmq rabbitmq -n default, modify replicas: 3 to 5
   The cluster will do rolling upgrade, and 2 more rabbitmq pods are scheduled by k8s, cluster status is: 5/5
   Check the cluster status, got a lot of network partition issues, and the quorum queue status is not meet the expectation.
   It looks the quorum queue got one more follower, but didn't reach the expectation (expect: 1 leader and 4 followers), maybe impact by network partition issue. But to a certain extent, the above configuration has played a role.

---

I know that the new rabbitmq version may require more testing to ensure its stability, so the following is just a discussion, not asking work immediately.

• The rabbitmq operator may need to deal with the change of replicas specially. If only increase the values of replicas, Only need to schedule the delta pods and join to the existing cluster instead of rolling upgrade for all of rabbitmq pods. In this way, we can not only save the time of cluster changes, but also possible to reduce the risk of network partition.
• Regarding the configuration of this new function, more documentation is needed in further release, otherwise the user needs to check the source code to know the specific configuration items.
• From my observation, once the rabbitmq network partition occurs, the cluster cannot be restored again, and manual intervention is required, such as manually deleting the pods in the partition list. Not sure if there is a better way to address the network partition issue.

[Zerpet]

Hi @xixiangzouyibian, thank you for testing our new feature. Your observations are very welcome ⭐

One quick observation is that you are using RabbitMQ 4.0 in your tests, which is an unreleased version, and it may not be stable yet for general consumption. However, if you can reproduce this issue consistently, it will be great if you open a new discussion in https://github.com/rabbitmq/rabbitmq-server/discussions because this looks rabbitmq-core specific, and unrelated to the kubernetes Operator.

Also note that this question was marked as "answered" earlier. I'd recommend that you open a new "follow up" discussion if you find issues in 3.13, and mention this discussion #1471 as background. Sometimes we miss updates on issues that are closed, or discussions that are answered.

[xixiangzouyibian]

Sure, I'll keep observing on the cluster network partition problem. If the problem can be reproduced stably, I will open a new discussion in rabbitmq-server.
But there are still some questions about rabbitmq cluster operator:

• First, I only modified the replicas of the rabbitmqcluster, but each pod to be upgraded rollingly, which significantly increases the time of cluster changes. Is this an expected behavior?
• Second, what is the best practice to restore the cluster in the event of a network partition ?

[mkuratczyk]

The rolling restart triggered by scaling out is a bug that we should fix. It wasn't like that in the past. Most likely it got accidentally introduced by #1113

[xixiangzouyibian]

Thanks for confirmation.

[xixiangzouyibian]

@mkuratczyk
I tested against: 3.13.2-management RMQ image, when scale out from 3 to 5 replicas, I found the pods startup history:

It looks pods are still rolling restart. I'm not sure if the fix: #1610 has been thoroughly addressed the issue.

[mkuratczyk]

please provide commands I can copy-paste to see this, starting from a fresh kubernetes cluster (eg. kind). we did test it successfully

[xixiangzouyibian]

1. Deploy RMQ cluster

2. After the deployment, scale out the cluster:
   kubectl patch rmq rabbitmq-test --type='merge' -p '{"spec":{"replicas":5}}'

3. Watch the changes
   kubectl get pods -l app.kubernetes.io/name=rabbitmq-test --watch

[mkuratczyk]

These are not reproduction steps starting from a fresh cluster. Plus, there's a lot of irrelevant stuff you are asking me to go through.
I've checked again and everything works as expected for me. Please put some effort into reporting an issue (and don't forget you can also debug and fix the issue - this is an open source project)