Skip to content

Commit

Permalink
Merge pull request #1819 from rabbitmq/mk-doc-updates-for-the-3.13.x-era
Browse files Browse the repository at this point in the history 
More doc updates for the post 3.13.0 era
  • Loading branch information
@michaelklishin
michaelklishin committed Mar 4, 2024
2 parents 28fac5d + 7fc41b8 commit 670443a
Show file tree
Hide file tree
Showing 3 changed files with 59 additions and 146 deletions.
160 changes: 40 additions & 120 deletions docs/clustering.md
View file
Expand Up @@ -31,6 +31,7 @@ This guide covers fundamental topics related to RabbitMQ clustering:
* [How clusters are formed](#cluster-formation)
* How nodes [authenticate to each other](#erlang-cookie) (and with CLI tools)
* Why it's important to [use an odd number of nodes](#node-count) and **two-cluster nodes are strongly discouraged**
* [Queue and stream leader replica placement](#replica-placement) strategies
* [Node restarts](#restarting) and how nodes rejoin their cluster
* [Node readiness probes](#restarting-readiness-probes) and how they can affect rolling cluster restarts
* How to [remove a cluster node](#removing-nodes)
Expand Down Expand Up @@ -427,6 +428,42 @@ however.
See [Connecting to Streams](https://blog.rabbitmq.com/posts/2021/07/connecting-to-streams/#well-behaved-clients)
to learn more.

### Queue and Stream Leader Replica Placement {#replica-placement}

Every queue and srream in RabbitMQ has a primary replica. That replica is called
_the leader_. All publishing operations on queues and streams go through the leader
replica first and then are replicated to followers (secondary replicas). This is necessary to
guarantee FIFO ordering of messages.

To avoid some nodes in a cluster hosting a significant majority of queue leader
replicas and thus handling most of the load, queue leaders should
be reasonably evenly distributed across cluster nodes.

Queue leaders can be distributed between nodes using several
strategies. Which strategy is used is controlled in three ways,
namely, using the `x-queue-master-locator` [optional queue argument](./queues#optional-arguments), setting the `queue-master-locator`
policy key or by defining the `queue_master_locator`
key in [`the configuration file`](./configure#configuration-files).

There are two options available:

* `balanced`, the default strategy, uses the data on how many replicas peer nodes host,
when there are relatively few (say, no more than 1000) queues in the cluster; it falls back
to a more efficient strategy of picking a random node when there are many queues
* `client-local` will always pick the node the client is connected to

The following example sets the `queue_leader_locator` setting in `rabbitmq.conf` to its default value:

``` ini
queue_leader_locator = balanced
```

The client-provided queue argument takes presedence when both are used.

Note that all Raft-based features, namely quorum queues and streams, use this value as a suggestion.
Raft leader election algorithm involves a degree of randomness, therefore the selected recommended
node will have a replica placed on it but it will not always be the leader replica.


## Clustering and Observability {#clustering-and-observability}

Expand Down Expand Up @@ -1158,9 +1195,9 @@ Learn more in the [section on ports](#ports) above and dedicated [RabbitMQ Netwo

## Erlang Versions Across the Cluster {#erlang}

All nodes in a cluster are *highly recommended* to run the same major [version of Erlang](./which-erlang): `22.2.0`
and `22.2.8` can be mixed but `21.3.6` and `22.2.6` can potentially introduce breaking changes in
inter-node communication protocols. While such breaking changes are relatively rare, they are possible.
All nodes in a cluster are *highly recommended* to run the same major [version of Erlang](./which-erlang): `26.2.0`
and `26.1.2` can be mixed but `25.3.2.8` and `26.2.0` can potentially introduce breaking changes in
inter-node communication protocols. While such breaking changes are rare, they are possible.

Incompatibilities between patch releases of Erlang/OTP versions
are very rare.
Expand Down Expand Up @@ -1191,120 +1228,3 @@ In general, this aspect of managing the
connection to nodes within a cluster is beyond the scope of
this guide, and we recommend the use of other
technologies designed specifically to address these problems.


## Disk and RAM Nodes {#cluster-node-types}

A node can be a <em>disk node</em> or a <em>RAM node</em>.
(<b>Note:</b> <i>disk</i> and <i>disc</i> are used
interchangeably). RAM nodes store internal database tables
in RAM only. This does not include messages, message store
indices, queue indices and other node state.

In the vast majority of cases you want all your nodes to be
disk nodes; RAM nodes are a special case that can be used
to improve the performance clusters with high queue,
exchange, or binding churn. RAM nodes do not provide
higher message rates. When in doubt, use
disk nodes only.

Since RAM nodes store internal database tables in RAM only, they must sync
them from a peer node on startup. This means that a cluster must contain
at least one disk node. It is therefore not possible to manually remove
the last remaining disk node in a cluster.


## Clusters with RAM nodes {#ram-nodes}

RAM nodes keep their metadata only in memory. As RAM nodes
don't have to write to disc as much as disc nodes, they can
perform better. However, note that since persistent queue
data is always stored on disc, the performance improvements
will affect only resource management (e.g. adding/removing
queues, exchanges, or vhosts), but not publishing or
consuming speed.

RAM nodes are an advanced use case; when setting up your
first cluster you should simply not use them. You should
have enough disc nodes to handle your redundancy
requirements, then if necessary add additional RAM nodes for
scale.

A cluster containing only RAM nodes would be too volatile; if the
cluster stops you will not be able to start it again and
**will lose all data**. RabbitMQ will prevent the creation of a
RAM-node-only cluster in many situations, but it can't
absolutely prevent it.

The examples here show a cluster with one disc and one RAM
node for simplicity only; such a cluster is a poor design
choice.

### Creating RAM nodes {#creating-ram}

We can declare a node as a RAM node when it first joins
the cluster. We do this with
`rabbitmqctl join_cluster` as before, but passing the
`--ram` flag:

```bash
# on rabbit2
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit2 ...done.

rabbitmqctl join_cluster --ram rabbit@rabbit1
# => Clustering node rabbit@rabbit2 with [rabbit@rabbit1] ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit2 ...done.
```

RAM nodes are shown as such in the cluster status:

```bash
# on rabbit1
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit1 ...
# => [{nodes,[{disc,[rabbit@rabbit1]},{ram,[rabbit@rabbit2]}]},
# => {running_nodes,[rabbit@rabbit2,rabbit@rabbit1]}]
# => ...done.

# on rabbit2
rabbitmqctl cluster_status
# => Cluster status of node rabbit@rabbit2 ...
# => [{nodes,[{disc,[rabbit@rabbit1]},{ram,[rabbit@rabbit2]}]},
# => {running_nodes,[rabbit@rabbit1,rabbit@rabbit2]}]
# => ...done.
```

### Changing node types {#change-type}

We can change the type of a node from ram to disc and vice
versa. Say we wanted to reverse the types of
`rabbit@rabbit2` and `rabbit@rabbit1`, turning
the former from a ram node into a disc node and the latter from a
disc node into a ram node. To do that we can use the
`change_cluster_node_type` command. The node must be
stopped first.

```bash
# on rabbit2
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit2 ...done.

rabbitmqctl change_cluster_node_type disc
# => Turning rabbit@rabbit2 into a disc node ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit2 ...done.

# on rabbit1
rabbitmqctl stop_app
# => Stopping node rabbit@rabbit1 ...done.

rabbitmqctl change_cluster_node_type ram
# => Turning rabbit@rabbit1 into a ram node ...done.

rabbitmqctl start_app
# => Starting node rabbit@rabbit1 ...done.
```
9 changes: 9 additions & 0 deletions docs/configure.md
View file
Expand Up @@ -844,6 +844,15 @@ disk_free_limit.absolute = 50MB
</p>
</td>
</tr>

<tr>
<td>`queue_leader_locator`</td>
<td>
Controls the [strategy used when selecting a node](./clustering#replica-placement) to host the leader replica of a newly declared
queue or stream.
</td>
</tr>

<tr>
<td>`log.file.level`</td>
<td>
Expand Down
36 changes: 10 additions & 26 deletions docs/ha.md
View file
Expand Up @@ -18,17 +18,17 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Classic Queue Mirroring
# Classic Queue Mirroring (Deprecated)

## Wait, There's a Better Way: Next Generation Highly Available Queues and Streams {#interstitial}
## Wait, There's a Better Way: Next Replicated Queues and Streams {#interstitial}

This guide covers a [**deprecated and scheduled for removal feature**](https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/): mirroring (queue contents replication) of classic queues.
[Quorum queues](./quorum-queues) and/or [streams](./streams) should be used instead of mirrored classic queues.

Quorum queues are a more advanced queue type, which offers high availability using replication and focuses on data safety. From RabbitMQ 3.10, quorum queues [support message TTL](https://blog.rabbitmq.com/posts/2022/05/rabbitmq-3.10-release-overview/) and provide [higher throughput and more stable latency](https://blog.rabbitmq.com/posts/2022/05/rabbitmq-3.10-performance-improvements/) compared to mirrored classic queues. You can [Migrate your RabbitMQ Mirrored Classic Queues to Quorum Queues](./migrate-mcq-to-qq) now.
Quorum queues are a more advanced queue type, which offers high availability using replication and focuses on data safety. Quorum queues [support message TTL](https://blog.rabbitmq.com/posts/2022/05/rabbitmq-3.10-release-overview/) and provide [higher throughput and more stable latency](https://blog.rabbitmq.com/posts/2022/05/rabbitmq-3.10-performance-improvements/) compared to mirrored classic queues. Please [migrate from Mirrored Classic Queues to Quorum Queues](./migrate-mcq-to-qq) now.

[Streams](./streams) is a messaging data structure available as of [RabbitMQ 3.9](./changelog),
and is also replicated.
[Streams](./streams) is an [alternative messaging data structure](https://blog.rabbitmq.com/blog/2021/07/13/rabbitmq-streams-overview) supported by RabbitMQ.
Just like quorum queues, streams are replicated.

Quorum queues should be the **default choice** for a replicated queue type.
Classic queue mirroring will be **removed in a future version** of RabbitMQ:
Expand All @@ -44,7 +44,7 @@ using policies before changing the default version in the configuration.

Topics covered in this guide include

* [Next generation replicated queue type](#interstitial) and why it should be preferred over classic queue mirroring.
* [Next generation replicated queues and streams](#interstitial), and why they should be preferred over classic queue mirroring
* What is [classic queue mirroring](#what-is-mirroring) and how it works
* How to [enable it](#ways-to-configure)
* What [mirroring settings are available](#mirroring-arguments)
Expand All @@ -62,7 +62,8 @@ This guide assumes general familiarity with [RabbitMQ clustering](./clustering).
## What is Queue Mirroring {#what-is-mirroring}

**Important**: mirroring of classic queues will be **removed in a future version** of RabbitMQ.
Consider using [quorum queues](./quorum-queues) or a non-replicated classic queue instead.
Consider using [quorum queues](./quorum-queues), [streams](./streams), or a non-replicated classic queue
v2 instead.

By default, contents of a queue within a RabbitMQ cluster are located on
a single node (the node on which the queue was
Expand Down Expand Up @@ -263,27 +264,10 @@ See [Runtime Parameters and Policies](./parameters#policies) to learn more.


## Queue Leader Replicas, Leader Migration, Data Locality {#leader-migration-data-locality}

### Queue Leader Location {#queue-leader-location}

Every queue in RabbitMQ has a primary replica. That replica is called
_queue leader_ (originally "queue master"). All queue operations go through the leader
replica first and then are replicated to followers (mirrors). This is necessary to
guarantee FIFO ordering of messages.

To avoid some nodes in a cluster hosting the majority of queue leader
replicas and thus handling most of the load, queue leaders should
be reasonably evenly distributed across cluster nodes.

Queue leaders can be distributed between nodes using several
strategies. Which strategy is used is controlled in three ways,
namely, using the `x-queue-master-locator` [optional queue argument](./queues#optional-arguments), setting the `queue-master-locator`
policy key or by defining the `queue_master_locator`
key in [`the configuration file`](./configure#configuration-files). Here are the possible strategies and how to set them:

* Pick the node hosting the minimum number of leaders of the same queue type:
`balanced`
* Pick the node the client that declares the queue is
connected to: `client-local`
This section has been moved to the [Clustering guide](./clustering#replica-placement).

### "nodes" Policy and Migrating Leaders {#fixed-leader-promotion}

Expand Down

0 comments on commit 670443a

Please sign in to comment.