Firecracker Patch Limitations & The Flintlock Update API

[jmickey]

Firecracker Patch Limitations & The Flintlock Update API

• Status: proposed
• Date: 2021-11-08
• Authors: @jmickey
• Deciders: @jmickey @richardcase @Callisto13 @yitsushi

Context

Users of Flintlock may want to make changes to MicroVMs that are already provisioned and running. In order to support MicroVM updates and provide users with an API to achieve this we need to understand what the limitations are within the MicroVM providers. Flintlocks initial MicroVM provider is Firecracker.

Firecracker provides Swagger API definitions that outlines the different operations that are possible, and when those operations are valid.

Unfortunately, very few operations are valid after a MicroVM has been started. These include:

• Network interface rate limiting.
• Volume rate limiting, and "hot-swapping" the underlying raw block device on the host. However, there are limitations with this:
  • The guest must not have mounted the device.
  • The guest must not read or write from the raw block device during the update sequence.
• MicroVM metadata service data store.
• Balloon devices.

As a result of discovering these limitations, the flintlock update gRPC API and update implementation code was removed in #222 and the need to revisit how we would handle updates moving forward was identified.

Moving forward there are a few possible directions that could be taken:

1. Provide no support for in-place updates. The primary consumer of the flintlock API will be the MicroVM Cluster API (CAPI) provider, which doesn't support updates. As a result it would be perfectly valid for us to decide that we won't support any in-place update capabilities, and instead all changes to MicroVM specs will result in recreation.
2. Implement in-place updates for valid operations. Rate limiting, limited unmounted volume changes, metadata, and balloon devices are able to updated in-place.
3. Only update metadata. The metadata service data store is probably the most logical place where in-place updates would make sense within the context of possible consumers (CAPI).

Decision

Consequences

Having to recreate MicroVMs in order to support spec updates could possibly lead to issues where the host does not have sufficient resources in order to support CreateBeforeDelete style updates, where a new MicroVM is created, workloads are migrated, and the old MicroVM is removed. However, this is less of a consequence of this decision, and more a consequence of the inherent limitations of the Firecracker MicroVM provider.

[richardcase]

With the Liquid Metal use case the main client of Flintlock will be our Cluster API provider. In Cluster API Machines are immutable:

From the perspective of Cluster API, all Machines are immutable: once they are created, they are never updated (except for labels, annotations and status), only deleted.

source

So we won't need any update functionality to support.

[jmickey]

Yep, that is basically option 1. Basically ignore updates completely.

[richardcase]

Yes it is 👍

[Callisto13]

good clear write up 👍 opt 1 sounds legit despite the consequences. if people complain about those later then we can reconsider and it becomes a problem for future us (those suckers)

[jmickey]

ADR has been merged into the repo. Locking discussion