architecture.md

Technical overview

Solo5 is a sandboxed execution environment primarily intended for, but not limited to, running applications built using various unikernels (a.k.a. library operating systems).

Conceptually, if you think of a unikernel as a user process, albeit one that does more in "userspace", Solo5 re-defines the interface between the process and its host operating system or hypervisor in a way that is designed to:

• be as "legacy-free" and "thin"(*) as possible, thus having a minimal attack surface, making it much easier to reason about the isolation properties of Solo5-based applications,

  (*) compared to existing interfaces, such as that exposed to a Linux process by the kernel, or to a full-system virtual machine by QEMU

• make it easy to implement new targets, targeting a variety of different sandboxing technologies (e.g. hardware virtualization, Linux seccomp, Intel SGX), host operating systems and hypervisors. Solo5-based applications can then be straightforwardly deployed on any supported target without the need to modify their source code.

It turns out that simplifying this interface has the following desirable properties:

• removal of interrupts implies more deterministic behaviour of applications, allowing for efficient record/replay and debugging,
• fast "boot" times, comparable to loading a standard user process, suitable for "function as a service" use-cases,
• ease of porting existing and future unikernels to run on top of the Solo5 interface.

Additionally, Solo5 introduces the concept of a tender, which, for targets that make use of one, is the component responsible for "tending to" the guest at load and/or run time:

• In the case of hvt ("hardware virtualized tender"): the tender is loosely equivalent to QEMU in a typical KVM/QEMU setup. The solo5-hvt tender is designed to be modular from the outset, and is several orders of magnitude smaller in code size than QEMU. See the original ukvm paper from Hotcloud 2016 for a detailed comparison.

• In the case of spt ("sandboxed process tender"): the solo5-spt tender is responsible for loading the guest into memory, installing a seccomp sandbox and passing control to the guest. The seccomp filter installed by spt is a strict whitelist allowing access to the minimal set of system calls required to run the guest (as of this writing, 7 distinct system calls in total if all devices are configured). See our SoCC'18 paper for a discussion of the security properties of the initial implementation of this concept.

Note: Both the hvt tender and target were formerly known as the "ukvm monitor", or just ukvm. As of the Solo5 0.4.0 release, these have been renamed to reflect that they are no longer specific to the Linux/KVM hypervisor, and to allow for development of further tenders such as spt.

Goals

The Solo5 implementation strives to be minimalist and correct, with an emphasis on working out good external, internal and user/operator interfaces in preference to (prematurely) adding features or scope creep.

Non-goals

• SMP is not supported. This is intentional, Solo5-based applications should scale out by running multiple instances of the application,
• no interrupts or preemptive scheduling, in fact no scheduling at all is provided at the Solo5 level. This reduces complexity and removes the problem of "scheduling a scheduler" while fitting in with the prevalent model of current unikernels, which use co-operative scheduling internally,
• Solo5 does not run on bare metal (without a host operating system or hypervisor).

Current status and limitations

Solo5 has been in development since late 2015, and is suitable for early adopters. A variety of different targets are supported. Production deployments of Solo5-based unikernels exist on Linux/KVM, FreeBSD/vmm and Google Compute Engine.

The current interface and its implementations have not been designed for high I/O performance. Proposed prototypes show that I/O performance comparable or better than that of virtual machines using virtio-style devices is possible.

The Solo5 interfaces and ABI are not yet stable, and will change in the future. We expect to provide a stable ABI (binary compatibility) in the future.

Architecture

Application manifest

Solo5 introduces the concept of an application manifest, which is defined by the developer at unikernel build time, using the following JSON format and customarily named manifest.json:

{
  "type": "solo5.manifest",
  "version": 1,
  "devices": [
    { "name": "NAME", "type": "TYPE" }
    // ... up to 63 user-specified devices ...
  ]
}

NAME is a unique logical identifier (a.k.a. "pet name") for the device. The intention is that the name conveys some meaning of the intended use ("wiring") of the device, eg. frontend for a network or storage for a block device.

NAME must be composed of alphanumeric characters only, and within 1..67 characters in length.

TYPE is the type of device being declared, currently BLOCK_BASIC or NET_BASIC.

Note that there is a maximum limit of 63 user-specified devices in the manifest.

At unikernel build time, manifest.json is pre-processed by solo5-elftool, generating a C source file with a binary representation. This source file is then compiled using cc from the Solo5 toolchain, and linked into the unikernel binary, where it is represented as an ELF "NOTE".

Additionally, solo5-elftool provides the query-manifest subcommand to extract the binary manifest from a unikernel binary and display it as JSON.

Public API, Tenders and Bindings

The bindings implement the Solo5 public ("unikernel-facing") API defined in include/solo5/solo5.h for a particular target.

The hvt and spt targets make use of a tender which serves as a loader for the unikernel within the host OS, sets up access to host resources declared in the application manifest and, in the case of hvt, mediates guest access to those resources.

Code organisation

The main components of Solo5 are:

• toolchain/: wrappers that appropriate the host system's GCC or Clang compiler toolchain, producing the Solo5 toolchain used to build bindings and unikernels.
• bindings/: the Solo5 bindings to (implementation of) the unikernel-facing interface for the various supported targets, as defined in solo5.h.
• tenders/common/: common code shared by tender implementations, notably including the ELF loader and application manifest validation routines. The latter is also indirectly included by the bindings implementation.
• tenders/hvt/: the tender implementation for the hvt target, with tender-internal interfaces defined in hvt.h and the internal "hypercall" ABI to Solo5 defined in hvt_abi.h.
• tenders/spt: the tender implementation for the spt target. Tender-internal interfaces are defined in spt.h and internal Solo5-facing ABIs in spt_abi.h.
• elftool/: a tool for application manifest generation and extracting information from Solo5 binaries.
• tests/: self tests used as part of our CI system.
• scripts/: extra tooling and scripts (mainly to support the virtio target).
• opam/: package definitions and pkg-config files for integration with OPAM and MirageOS.

The code is architected to split processor architecture and/or host operating system (target)-specific modules from shared functionality where possible. However, it is acceptable to make minimal, judicious use of #ifdef where doing otherwise would result in excessive code duplication.

Due to the need to run automated tests on bare metal, our CI system is a custom solution, the details of which can be found in a dedicated repository. The scripts run by the CI system on every GitHub pull request can be found in build.sh. We also use Travis CI as a "backup", with the corresponding configuration in .travis.yml.