Pre-warmed cluster upgrades

[ecordell]

Right now, the operator rolls out new versions of SpiceDB by updating a deployment.

New pods become available as soon as they are connected to the datastore and the dispatch ring, which means that cache is lost during upgrades. Depending on the queries SpiceDB is handling, this could cause a significant increase in latency.

Some options worth exploring:

1. Slowly introducing new pods so that only some % of a cluster loses its cache at a time. This is probably the simplest option, but requires all dispatch API changes to be fully backwards-compatible.
2. Traffic mirroring via external routing (similar to how flagger provides generic blue/green mirroring). Currently, spicedb-operator operates "below" the level that most of these tools work, so the scope would need to increase dramatically to include more networking/ingress concerns.
3. Traffic mirroring via SpiceDB itself. We could introduce mirroring flags into SpiceDB itself, so that incoming traffic can be forwarded to a parallel set of nodes to fill their cache. This would require old and new clusters to be exposed under different service objects so that their hashrings don't collide.
4. Saving and restoring the cache. Currently, SpiceDB caches exist only in memory. We could switch to a cache that syncs to the filesystem or provide apis for dumping the cache (either over the network or to disk), and the operator could ensure the caches come back in the new pods. We would likely want to switch to a StatefulSet if we try this.