Proposal: Migration Path

[cipherboy]

(Edit 01/25/2024: Updated with policy proposal to sync with).

Definitions

From a consumption perspective, there are three personas we can define:

1. Bao Operators. These are the people tasked with deploying updates to
   the bao binary, handling outages, &c. They might typically access /sys,
   setup new mounts, get high-privileged root tokens temporarily to setup
   additional clusters or change quotas, &c, but aren't (for this discussion)
   direct consumers of many secrets methods. They likely also interface with
   OpenBao's configuration, logging, and auditing interfaces, and may interact
   briefly (during outages &c) with storage directly. They'd be in charge
   of handling any migration which needs to occur during an upgrade or change
   event from upstream to OpenBao. While a breaking change, these are most
   likely to have a flexible enough deployment & workflow to accept change.

2. Connection Operators. These are individuals who deploy applications
   which talk to OpenBao. They might directly interact with bao's API
   namespace, to set up new accounts, services, potentially (if delegated by
   Bao operators) creating new secrets engines, and configuration any existing
   secrets engines in their purview. They may do this directly (via CLI or
   UI) or indirectly (via a centralized management service like OpenTofu).
   These types of users can accept a moderate amount of change, but changes
   to APIs (especially unexpectedly) can cause issues.

3. Applications & end-users. These are individuals (broadly speaking)
   which only interface with the API of authentication and secrets engines to
   get secrets (again, broadly speaking -- a PKI certificate is not a
   traditional secret nor does Transit expose a similar interface that grants
   leases). These typically have the least flexible workflow and are hesitant
   to change.

We can also talk about compatibility on several layers:

1. Seal compatibility. Is encrypted data drop-in compatible with
   upstream, for a given combination of (seal mechansim, storage provider, plugin). Can this storage layout be read (i.e., is the disk tree similar)
   by core and can the resulting plugin be loaded and function equivalently
   to API consumers? This implies no to minimal migration necessary.

2. Storage compatibility. For unencrypted data (i.e., using an unencrypted
   backup/one-time migration bundle or using the unsupported-but-useful
   sys/raw interface), can this be migrated from an upstream instance
   directly into downstream at an equivalent path without rewriting the
   data itself. This means all the rest of the core (minus the initial
   encryption layer and potentially meaning a different physical storage
   backend) are compatible with upstream.

3. API compatibility. For consumers of upstream's secret's or auth
   plugin's API only, would they be able to point interchangeably at an
   upstream or OpenBao instance without caring about which, assuming suitable
   data & plugins are available in both locations. This usage does not extend
   to non-plugin APIs (e.g., under /sys), but should include the GRPC
   external plugin communication mechanism.

(In this hierarchy it is implied that 1 implies 2 implies 3, i.e., a seal
or storage compatible fork would imply API compatibility, and likewise
seal compatibility implies storage compatibility).

Observations

Notably, API compatibility is most aligned with the connection operators' and
application & end-users' goals: minimize API change. While seal and storage
compatibility are cool from a Bao operator's perspective, breaking either
(w.r.t. upstream -- likely ahead of initial GA) results in a one-time
migration, but difficult from a Bao development community perspective. This
type of compatibility helps cross-adoption but becomes hard to support,
especially as several key portions are impossible to do without reverse
engineering Vault Enterprise (e.g., auto-unseal, seal wrapping, replication,
certain secret engine features, ...).

Proposal

Aim for API compatibility only, with limited seal compatibility when using remaining
seal, storage, and plugins as discussed in that proposal.

This allows us to prune unnecessary features, create alternative,
non-compatible implementations of various upstream, Enterprise features, and
decreases maintenance burdens. This allows us to create a more healthy,
separate but welcoming community from upstream.

It also recognizes that OpenBao (by discussing removing support for already
deprecated changes or removing plugins that no maintainer has stepped up to
support) will inevitably, intentionally or otherwise, deviate from upstream.
It provides support commitment to portions of the organization most resistant
to change, and allows Bao operators to decide whether or not to make the
migration. It also recognizes that upstream may or may not make other decisions
and potentially backwards-incompatible changes of their own, allowing us to
take a different approach if necessary.

This still allows us to take advantage of the broader secrets & auth engine
ecosystem, as GRPC compatibility is still encouraged. And third-party client
tooling should continue to function between the two, unless they were using
unsupported, internal implementation details and crossing the API boundary.

However, it walks a fine line that, when a user's existing usage of Vault aligns
with OpenBao's goals, the two should be drop-in compatible at the initial release.
This then allows a smoother migration between the two projects.

Migration Path

This means that some operators will need to make a conscious, one-time migration
from a compatible Vault version to an OpenBao version. This likely warrants
the creation of documentation and tools to help aid this migration, on a
per-plugin basis. It means that OpenBao's server will not be drop-in binary
equivalent from upstream's, and that we should likely place some marker to
detect and prevent incompatible, unsupported usage.

The remaining operators (who use a supported subset) will be able to
drop-in replace like a regular upgrade.

[JanMa]

I think we should target seal compatibility for the initial release, so OpenBao can serve as a drop-in replacement for Vault 1.4.x. After our initial release we can then commit to keeping only API compatibility.

Doing it this way will probably greatly increase adoption and seeing as we mostly rename stuff and maybe fix a few bugs in our initial release this should not be too hard for us to do. This is also the strategy OpenSearch and OpenTofu followed, and it seemed to work well for them :-)

[cipherboy]

Seal compatibility isn't possible between OpenBao and Enterprise. None of that code is public, so for many, OpenBao cannot reach parity in its initial release anyways (and later, only with careful reverse-engineering -- but more likely, if OpenBao introduces any similar features, it will deviate substantially). Only for those that were using Vault Community could seal compatibility theoretically be achieved...

On top of that, there's the issue of versions: only users who stopped on or before the last MPL licensed version (1.14.8) could use OpenBao with seal compatibility: if you upgraded past (to 1.15+), you cannot drop in OpenBao due to it (technically) being a downgrade. A best practice in this case would either be to revert to a previous backup (prior to upgrading to 1.15, and thus requiring you recreate any new data) or doing a manual migration from Vault to OpenBao anyways.

There's also diminishing returns to seal compatibility. There may be a one-time migration (where you're not using raft or using a an auto-unseal mechanism we remove) for OpenBao/Vault operators, but for the majority of consumers and producers of secrets, no change will be expected (unless they relied on removed plugins).

Lastly, this is a consistent approach: any Vault version (even later ones) can be migrated to any OpenBao version using a manual migration. It is consistent between versions (Vault <=1.14.8, OpenBao 0) or (OpenBao 0, OpenBao n>=1) or even (Vault >= 1.15, OpenBao n), &c, so users of OpenBao won't expect seal compatibility going forward from our initial release. And, users won't expect future OpenBao versions to be seal compatible with Vault 1.15+ &c.

In terms of maintenance burden on us, it is much easier, IMHO, to do API compatibility only. Yes, it negatively impacts a small subset of users one time, but for the majority of interactions with OpenBao, it does not matter.

---

OpenTofu also doesn't guarantee exact 1:1 parity: https://opentofu.org/docs/language/v1-compatibility-promises/ says they'll be mostly backwards compatible, but outlined places where they intend to deviate (https://opentofu.org/docs/language/v1-compatibility-promises/#commands-that-might-change). We're doing the same: identifying areas where we'll plan to deviate but for the majority of users, the same stable interface will still be presented.

[JanMa]

I wonder if Vault Enterprise users are really the target audience for OpenBao? As you've said, we will not support any Enterprise features in the initial release, maybe at some point in the future.

Regarding your point about downgrading Vault, I guess you're right. I don't know if it is possible to safely downgrade from Vault 1.15 or 1.16 to 1.14 in order to perform an in-place migration. I'm guessing it will probably just work, but it's not supported in any way by HashiCorp. It would be worth a shot to test this IMO.

Do you already have an idea how a one-time migration might work? Could we for example pull a snapshot from the Vault cluster and recover it in OpenBao, even if Vault is already running on 1.15+? Or do we need to write some custom tooling which performs the migration by copying over existing data from a live cluster?

[cipherboy]

@JanMa Agreed, re: enterprise users.

Duly noted about migration; for the subset that use existing features, I've clarified that seal compatibility would be preferable for these users.

Migration would work like it does in other cases that upstream requires migration: copying data over explicitly. Sometimes this is via sys/raw, sometimes this is via using TF to re-configure stuff in the new instance, ... &c. It depends on the organization and their capabilities. I'm not sure off the top of my head if we could make snapshots work, I thought they were mostly aimed for the case where its a drop-in migration, and not recommended in certain scenarios (e.g., migrating to FIPS mode, as all the keys need to be regenerated).

Edit to add: For the subset of plugins that are no longer supported by OpenBao as built-in (but moved externally due to community interest in supporting them), the new version of the plugin can first be loaded into Vault as an external plugin that shadows the builtin version, and then the Vault binary can be swapped with an OpenBao binary, and I believe (though, needs to be tested!) that this should allow a wider seal compatibility, even for newly-external plugins.

[JanMa]

Thanks for the update @cipherboy, I like the compromise :-) So to summarize:

Seal compatibility (aka drop-in replacement) in the initial release for the plugins we officially support, API compatibility for all others. If community supported plugins are used, the user has to perform some (to be defined) migration procedure. For subsequent releases we commit to API compatibility only also for officially supported plugins.

[cipherboy]

Yep, w.r.t. upstream, noting this is a migration policy and not an update policy (i.e., only applies from Vault to Bao migrations and not earlier Bao->later Bao).

Presumably future Bao releases will be drop-in compatible with prior ones, though this has yet to be discussed and can be punted to later. :-)

[JoeMurray]

Just a +1 to support @JanMa 's point which seems to be accepted by @cipherboy : initial version of OpenBao would aim to be a drop-in replacement of upstream's last Community Edition.

[cipherboy]

@JoeMurray note that it's subtly different: for the subset of plugins supported by the community, it will be such.

Initial proposal here: #64 -- I think @JanMa has privately expressed interest in maintaining a Nomad plugin externally, but so far that's the only one mention. Are there plugins there you rely on, that are otherwise removed, and would you be interested in helping to maintain them as external plugins?

[DanGhita]

Hello,

Just to say that I'm OK with the initial proposal (API compatibility), but, indeed, it would be even better to have initial seal compatibility (of course, for the supported plugins).

[naphelps]

Proposal accepted 2024-02-08. OpenBao will provide api compatibility at the minimum, and for the limited initial plugins this community supports, the project has the goal of providing seal compatibility.