Support vMotion when RabbitMQ is hosted on virtual machines in vSphere

[jonnepmyra]

Is your feature request related to a problem? Please describe.

We're encountering issues with our RabbitMQ clusters hosted on virtual servers, particularly during virtual machine migration (vMotion in vmware vSphere). The RabbitMQ documentation advises against suspending VMs during operation, which aligns very well with our experiences 😅 .

Describe the solution you'd like

One potential approach could involve exploring the integration of vMotion Notifications introduced in vSphere 8 into RabbitMQ. These notifications, as highlighted in the vSphere 8 release notes, aim to improve interoperability with applications sensitive to latency or clustered environments.

By considering the feasibility of detecting "Migration Start" and "Migration Completed" events, RabbitMQ instances could potentially be gracefully stopped and started accordingly, potentially helping to mitigate interruptions during VM migrations.

Upon receiving a "Migration Start" event:

1. Gracefully stop the RabbitMQ instance to prevent data corruption.
2. Allow migration to complete.

Once the "Migration Completed" event arrives:

3. Start the RabbitMQ instance on the new host.

Describe alternatives you've considered

As a workaround, we've resorted to restricting VM migration by deactivating vMotion and DRS on specific hosts. However, this solution isn't ideal as it necessitates separate hardware configurations solely for RabbitMQ's requirements in a virtualized environment.

Another option we've explored is developing a custom tool to manage RabbitMQ instances during VM migrations. This tool would detect vMotion events and adjust RabbitMQ instances accordingly.

However, integrating this functionality directly into RabbitMQ would not only address our needs but also benefit other vSphere users facing similar challenges, streamlining deployment and maintenance processes within this specific virtualization platform.

Additional context

We understand that integrating RabbitMQ into specific virtualized environments may not be a primary focus for the RabbitMQ product 😃 but there's potential for synergies between RabbitMQ and VMware's virtualization technologies that could enhance the overall ecosystem and benefit users leveraging these platforms.

https://core.vmware.com/blog/whats-new-vsphere-8-vmotion
https://core.vmware.com/resource/vsphere-vmotion-notifications#sec21823-sub1

[michaelklishin]

@jonnepmyra there is nothing vSphere-specific in RabbitMQ. vMotion side effects has been discussed many times on our team, below is a reasonably brief explanation of what can (in fact, has been) or cannot be improved.

Its effects are very difficult to cope with for many distributed data services. A moved node disappears for the rest of the cluster but the node itself is not aware of it. This is a very curious case of what Team RabbitMQ calls a partial network partition, where node A detects unavailability of B but B can still communicate just fine with A, or, in the case of vMotion, does not observe the failure for other reasons (it is "frozen", or "stunned" in vMotion terms).

Certain features and implementation details in RabbitMQ 3.x have a very hard time dealing with that.
Even though all modern distributed Raft-based features (quorum queues, streams, Khepri in 3.13)
would recover safely and predictably from such an event much better, what vMotion does to a node inevitably will have
side effects:

• A moved (RabbitMQ) node will be detected as failed (or restarted), which means quorum queue and stream/superstream leader elections will happen
• As a result of a leader election, some queues and streams will potentially have to move some (not all) of its data for absolutely no reason
• In environments with a lot of quorum queues and streams, mass leader re-elections will waste a noticeable amount of bandwidth and CPU resources

So even in 4.x vMotion cannot be side-effect free for RabbitMQ and, I would argue, pretty much any distributed data service. Raft provides enough safety guarantees for this case but what vMotion does will be treated as leader failure or a follower falling behind and having to "re-join" (reinstall the missing portion of its log from the current leader).

And in 3.x, certain features are not guaranteed recover safely from a partial network partition of this kind.

But at least in 4.x the effects of vMotion will be mostly safe.

How RabbitMQ can "integrate" with vMotion events is an open ended question, and likely won't yield any improvements because a "frozen" node is a form of failure in many distributed system algorithms.

[michaelklishin]

Starting and stopping nodes in response to events can cause more trouble than it will be worth:

1. Stopping a node can take time because depending on the features used, a node might have to move a non-trivial amount of data to disk (this is true for classic queues and transient messages even in 3.13). vMotion likely will not wait for more than N seconds
2. Starting a node can take time, again, in particular with classic queues but also quorum queues (until 4.0: use Ra checkpoints in rabbit_fifo for sub-linear time recovery of QQs on boot #10487 is finished). vMotion should not have any restrictions around node restart

RabbitMQ nodes pausing themselves in reaction to certain events (certain peer failures) is a feature
in RabbitMQ 3.x that we would really like to remove completely. It does more harm than good,
if the last ≈ 15 years of RabbitMQ with replicated queue types are any indication.

Modern versions have a maintenance mode that can be activated in response to a vMotion event. It won't change much: some replicas
will be transferred away, all client connections will be dropped (which sounds like something
that a vMotion user may find surprising and misinterpret). In theory this can also take longer
than the vMotion "post notification" grace period.

With a range of improvements planned for 4.x, all of those effects will be less of a problem, or maybe won't affect most deployments at all. My point is that reaction to vMotion events per se does not change the safety and side effects equation much.

[jonnepmyra]

@michaelklishin Thank you for your thorough answers!

I noticed that the vMotion Notification timeout can be quite lengthy, with the documentation citing an example of 5 minutes. So, theoretically, there's plenty of time for a graceful shutdown, at least in theory :) However, I'm curious, does this change your perspective on the matter? Would putting the node into maintenance mode as it's being "vMoved" be a better approach than simply stopping it? Could this potentially prevent the cluster from experiencing a network partition?

Regardless, I have a sense that developing a custom tool to respond to vMotion events and manage RabbitMQ might not be the preferred route. It seems we might run into situations where we cause more harm than good.

Out of curiosity, would you be willing to speculate on a potential release date for version 4.0? This year? Next year?

[michaelklishin]

Let's assume that the preparation window is 5 minutes. That's enough for a shutdown in the absolute majority of cases.

Then the scenario looks very similar to what maintenance mode seeks to do: a more controlled transfer
of responsibilities to other cluster members, before shutting down.

If that's the solution you are looking for, an external tool that would react to vMotion notifications and first enable maintenance mode, then shut the node down, should in theory be enough, and require no RabbitMQ changes. In fact, it would be relatively easy to prototype and compare different workloads.

With the adoption of Raft-based Khepri in 3.13 and in the future, and the removal of classic queue mirroring, RabbitMQ behavior in response to vMotion should only get more predictable, and in theory require very few to no changes to the aforementioned notification handler.

Note that some folks believe that maintenance mode does not yield a lot of benefit once you
use quorum queues and streams, and enable Khepri, because a node shutdown should be enough. I don't agree with that but for the sake of this discussion, that'd also be an option: the tool can shut nodes down and start them back up, then await complete startup with rabbitmqctl await_startup.

[lukebakken]

Another option we've explored is developing a custom tool to manage RabbitMQ instances during VM migrations. This tool would detect vMotion events and adjust RabbitMQ instances accordingly.

I was about to suggest doing this. If your tool works as expected, sharing the code with the RabbitMQ community would be a great way to give back.