Feature flags need quality of life improvements

[dumbbell]

Why

Since the introduction of the first required feature flags, it became more painful for users to upgrade if they did not pay attention to the feature flags states. Things like:

• A node refuses to start because it requires a feature flag which was still disabled before the upgrade.
• It's difficult to downgrade if a package system was used because there is little chance the previous package is available easily.
• It's difficult to add a newer node to a cluster using an older version.

There is room for improvement in the current subsystem and I would like to follow two routes:

1. better communicate from RabbitMQ itself that users have to enable feature flags
2. make changes to the subsystem to handle common situations that are problematic today
3. prevent foot-shooting when upgrading RabbitMQ using our packages (Debian and RPM)

How

Here is a list of improvements that I plan to make:

• Log a warning during start, stop and perhaps on a regular basis that list stable feature flags that are still disabled
• Display a warning in the management UI to invite users to pay attention to the Feature flags section
• Highlight stable feature flags that are still disabled in the management UI's Feature flags section
• Improve the compatibility check in join_cluster to take into account the fact that the node will be reset. There should be no need to mess with $RABBITMQ_FEATURE_FLAGS because the joining node's feature flags states will be aligned with the remote cluster anyway.
  See:
  • rabbit_db: Copy feature states early when joining a cluster #9682
  • Relax feature flag compat check during join cluster #9729
• When a clustered node is upgraded to a version that requires some feature flags, it should be possible to enable them remotely in the cluster and then proceed with the start of the local node.
• When a node is upgraded, users could configure RabbitMQ to automatically enable all stable feature flags as soon as possible. This could be an opt-in or opt-out behavior.
• Improve Debian and RPM packages to verify that RabbitMQ can be upgraded. This requires that a preinst script can compare the list of feature flags from the installed version and the new one.

[wast]

I'd like to add my 2 cents. There's an error when using 1 replica in a RabbitMQ Cluster Kubernetes operator:
"Feature flags: refuse to enable feature flags while clustered nodes are missing, stopped or unreachable"

[dumbbell]

I'd like to add my 2 cents. There's an error when using 1 replica in a RabbitMQ Cluster Kubernetes operator: "Feature flags: refuse to enable feature flags while clustered nodes are missing, stopped or unreachable"

All RabbitMQ nodes in a cluster need to run before a feature flag can be enabled. Could you please expand on your use case?

[wast]

"All nodes" in my scenario is 1 single node (as defined in yaml: replicas: 1), so why is it expecting more?

[michaelklishin]

@wast please start a separate GitHub Discussion, we will not let well defined issues to be turned into open ended discussions and troubleshooting sessions.

[michaelklishin]

"All nodes" in my scenario is 1 single node (as defined in yaml: replicas: 1), so why is it expecting more?

Most likely because there were more nodes in the cluster at some point and existing nodes still have knowledge of their prior peers. The Cluster Operator does not support shrinking the cluster, at least not in all cases, IIRC. There is a certain workaround but in general, shrinking member count should not be considered a supported operation.

This is a topic for a separate discussion, this issue has well defined and specific scope.