Skip to content

Releases: cloud-hypervisor/cloud-hypervisor

v15.0

29 Apr 15:27
@github-actions github-actions
v15.0
30a0127
Compare
Choose a tag to compare
v15.0

This release has been tracked through the v15.0 project.

Highlights for cloud-hypervisor version v15.0 include:

Version numbering and stability guarantees

This release is the first in a new version numbering scheme to represent that
we believe Cloud Hypervisor is maturing and entering a period of stability.
With this new release we are beginning our new stability guarantees:

  • The API (including command line options) will not be removed or changed in a
    breaking way without a minimum of 2 releases notice. Where possible warnings
    will be given about the use of deprecated functionality and the deprecations
    will be documented in the release notes.
  • Point releases will be made between individual releases where there are
    substantial bug fixes or security issues that need to be fixed.

Currently the following items are not guaranteed across updates:

  • Snapshot/restore is not supported across different versions
  • Live migration is not supported across different versions
  • The following features are considered experimental and may change
    substantially between releases: TDX, SGX.

Network device rate limiting

Building on our existing support for rate limiting block activity the network
device also now supports rate limiting. Full details of the controls are in the
IO throttling documentation.

Support for runtime control of virtio-net guest offload

The guest is now able to change the offload settings for the virtio-net
device. As well as providing a useful control this mitigates an issue in the
Linux kernel where the guest will attempt to reprogram the offload settings
even if they are not advertised as configurable (#2528).

--api-socket supports file descriptor parameter

The --api-socket can now take an fd= parameter to specify an existing file
descriptor to use. This is particularly beneficial for frameworks that need to
programmatically control Cloud Hypervisor.

Bug fixes

  • A workaround has been put in place to mitigate a Linux kernel issues that
    results in the CPU thread spinning at 100% when using virtio-pmem (#2277).
  • PCI BARs are now correctly aligned removing the need for the guest to
    reprogram them (#1797,#1798)
  • Handle TAP interface not being writable within virtio-net (due to the buffer
    exhaustion on the host) (#2517)
  • The recommended Linux kernel is now v5.12.0 as it contains a fix that
    prevents snapshot & restore working (#2535)

Deprecations

Deprecated features will be removed in a subsequent release and users should plan to use alternatives

  • Support for booting with the "LinuxBoot" protocol for ELF and bzImage
    binaries has been deprecated. When using direct boot users should configure
    their kernel with CONFIG_PVH=y. Will be removed in v16.0.

Contributors

Many thanks to everyone who has contributed to our release including some new faces.

Assets 5
4 Join discussion

v0.14.1

31 Mar 16:11
@github-actions github-actions
v0.14.1
bb02c28
Compare
Choose a tag to compare
v0.14.1

Bug fix release branched off the v0.14.0 release. The following bugs
were fixed in this release:

  • CPU hotplug on Windows failed due to misreported CPU state information and
    the lack of HyperV CPUID bit enabled (#2437, #2449, #2436)
  • A seccomp rule was missing that was triggered on CPU unplug (#2455)
  • A bounds check in VIRTIO queue validation was erroneously generating
    DescriptorChainTooShort errors in certain circumstances (#2450, #2424)
Assets 5

v0.14.0

26 Mar 12:01
@github-actions github-actions
v0.14.0
84b3992
Compare
Choose a tag to compare
v0.14.0

This release has been tracked through the 0.14.0 project.

Highlights for cloud-hypervisor version 0.14.0 include:

Structured event monitoring

A new option was added to the VMM --event-monitor which reports structured
events (JSON) over a file or file descriptor at key events in the lifecycle of
the VM. The list of events is limited at the moment but will be further
extended over subsequent releases. The events exposed form part of the Cloud
Hypervisor API surface.

MSHV improvements

Basic support has been added for running Windows guests atop the MSHV
hypervisor as an alternative to KVM and further improvements have been made to
the MSHV support.

Improved aarch64 platform

The aarch64 platform has been enhanced with more devices exposed to the running
VM including an enhanced serial UART.

Updated hotplug documentation

The documentation for the hotplug support has been updated to reflect the use
of the ch-remote tool and to include details of virtio-mem based hotplug as
well as documenting hotplug of paravirtualised and VFIO devices.

PTY control for serial and virtio-console

The --serial and --console parameters can now direct the console to a PTY
allowing programmatic control of the console from another process through the
PTY subsystem.

Block device rate limiting

The block device performance can now be constrained as part of the VM
configuration allowing rate limiting. Full details of the controls are in the
IO throttling doumentation.

Deprecations

Deprecated features will be removed in a subsequent release and users should plan to use alternatives

  • Support for booting with the "LinuxBoot" protocol for ELF and bzImage
    binaries has been deprecated. When using direct boot users should configure
    their kernel with CONFIG_PVH=y.

Contributors

Many thanks to everyone who has contributed to our 0.14.0 release including
some new faces.

Assets 5

v0.13.0

12 Feb 11:02
@github-actions github-actions
v0.13.0
f58b6f8
Compare
Choose a tag to compare
v0.13.0 Pre-release
Pre-release

This release has been tracked through the 0.13.0 project.

Highlights for cloud-hypervisor version 0.13.0 include:

Wider VFIO device support

It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI
devices that do not support MSI or MSI-X and instead rely on INTx interrupts.
Most notably this widens the support to most NVIDIA cards with the proprietary
drivers.

Improve huge page support

Through the addition of hugepage_size on --memory it is now possible to
specify the desired size of the huge pages used when allocating the guest
memory. The user is required to ensure they have sufficient pages of the
desired size in their pool.

MACvTAP support

It is now possible to provide file descriptors using the fd parameter to
--net which point at TAP devices that have already been opened by the user.
This aids integration with libvirt but also permits the use of MACvTAP
support. This is documented in dedicated macvtap documentation.

VHD disk image support

It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw
disk image with Cloud Hypervisor.

Improved Virtio device threading

Device threads are now derived from the main VMM thread which allows more
restrictive seccomp filters to be applied to them. The threads also have a
predictable name derived from the device id.

Clean shutdown support via synthetic power button

It is now possible to request that the guest VM shut itself down by triggering
a synthetic ACPI power button press from the VMM. If the guest is listening for
such an event (e.g. using systemd) then it will process the event and cleanly
shut down. This functionality is exposed through the HTTP API and can be
triggered via ch-remote --api-socket=<API socket> power-button.

Contributors

Many thanks to everyone who has contributed to our 0.13.0 release including
some new faces.

Assets 5

v0.12.0

10 Dec 18:56
@github-actions github-actions
v0.12.0
3eb9e61
Compare
Choose a tag to compare
v0.12.0 Pre-release
Pre-release

This release has been tracked through the 0.12.0 project.

Highlights for cloud-hypervisor version 0.12.0 include:

ARM64 enhancements

The use of --watchdog is now fully supported as is the ability to reboot the
VM from within the guest when running Cloud Hypervisor on an ARM64 system.

Removal of vhost-user-net and vhost-user-block self spawning

In order to use vhost-user-net or vhost-user-block backends the user is now
responsible for starting the backend and providing the socket for the VMM to
use. This functionality was deprecated in the last release and how now been
removed.

Migration of vhost-user-fs backend

The vhost-user-fs backend is no longer included in Cloud Hypervisor and it is
instead hosted in it's own
repository

Enhanced "info" API

The vm.info HTTP API endpoint has been extended to include the details of the
devices used by the VM including any VFIO devices used.

Contributors

Many thanks to everyone who has contributed to our 0.12.0 release:

Assets 5

v0.11.0

29 Oct 17:55
@github-actions github-actions
v0.11.0
7dcd3af
Compare
Choose a tag to compare
v0.11.0 Pre-release
Pre-release

v0.11.0

This release has been tracked through the 0.11.0 project.

Highlights for cloud-hypervisor version 0.11.0 include:

io_uring support by default for virtio-block

Provided that the host OS supports it (Linux kernel 5.8+) then io_uring will
be used for a significantly higher performance block device.

Windows Guest Support

This is the first release where we officially support Windows running as a
guest. Full details of how to setup the image and run Cloud Hypervisor with a
Windows guest can be found in the dedicated Windows
documentation.

vhost-user "Self Spawning" Deprecation

Automatically spawning a vhost-user-net or vhost-user-block backend is now
deprecated. Users of this functionality will receive a warning and should make
adjustments. The functionality will be removed in the next release.

virtio-mmmio Removal

Support for using the virtio-mmio transport, rather than using PCI, has been
removed. This has been to simplify the code and significantly
reduce the testing burden of the project.

Snapshot/Restore support for ARM64

When running on the ARM64 architecture snapshot and restore has now been
implemented.

Improved Linux Boot Time

The time to boot the Linux kernel has been significantly improved by the
identifying some areas of delays around PCI bus probing, IOAPIC programming and
MPTABLE issues. Full details can be seen in #1728.

SIGTERM/SIGINT Interrupt Signal Handling

When the VMM process receives the SIGTERM or SIGINT signals then it will
trigger the VMM process to cleanly deallocate resources before exiting. The
guest VM will not be cleanly shutdown but the VMM process will clean up its
resources.

Default Log Level Changed

The default logging level was changed to include warnings which should make it
easier to see potential issues. New logging
documentation was also added.

New --balloon Parameter Added

Control of the setup of virtio-balloon has been moved from --memory to its
own dedicated parameter. This makes it easier to add more balloon specific
controls without overloading --memory.

Experimental virtio-watchdog Support

Support for using a new virtio-watchdog has been added which can be used to
have the VMM reboot the guest if the guest userspace fails to ping the
watchdog. This is enabled with --watchdog and requires kernel support.

Notable Bug Fixes

  • MTRR bit was missing from CPUID advertised to guest
  • "Return" key could not be used under CMD.EXE under Windows SAC (#1170)
  • CPU identification string is now exposed to the guest
  • virtio-pmem withdiscard_writes=on no longer marks the guest memory as
    read only so avoids excessive VM exits (#1795)
  • PCI device hotplug after an unplug was fixed (#1802)
  • When using the ACPI method to resize the guest memory the full reserved size
    can be used (#1803)
  • Snapshot and restore followed by a second snapshot and restore now works
    correctly
  • Snapshot and restore of VMs with more than 2GiB in one region now work
    correctly

Contributors

Many thanks to everyone who has contributed to our 0.11.0 release including some new faces.

Assets 5

v0.10.0

18 Sep 13:42
@github-actions github-actions
v0.10.0
198bd55
Compare
Choose a tag to compare
v0.10.0 Pre-release
Pre-release

This release has been tracked through the 0.10.0 project.

Highlights for cloud-hypervisor version 0.10.0 include:

virtio-block Support for Multiple Descriptors

Some virtio-block device drivers may generate requests with multiple descriptors and support has been added for those drivers.

Memory Zones

Support has been added for fine grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the memory documentation.

Seccomp Sandbox Improvements

All the remaining threads and devices are now isolated within their own seccomp filters. This provides a layer of sandboxing and enhances the security model of cloud-hypervisor.

Preliminary KVM HyperV Emulation Control

A new option (kvm_hyperv) has been added to --cpus to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices.

Notable Bug Fixes

  • When using ch-remote to resize the VM parameter now accepts the standard sizes suffices (#1596)
  • cloud-hypervisor no longer panics when started with --memory hotplug_method=virtio-mem and no hotplug_size (#1564)
  • After a reboot memory can remove when using --memory hotplug_method=virtio-mem (#1593)
  • --version shows the version for released binaries (#1669)
  • Errors generated by worker threads for virtio devices are now printed out (#1551)

Contributors

Many thanks to everyone who has contributed to our 0.10.0 release including some new faces.

Assets 5

v0.9.0

06 Aug 14:30
@github-actions github-actions
v0.9.0
d714cf8
Compare
Choose a tag to compare
v0.9.0 Pre-release
Pre-release

This release has been tracked through the 0.9.0 project.

Highlights for cloud-hypervisor version 0.9.0 include:

io_uring Based Block Device Support

If the io_uring feature is enabled and the host kernel supports it then io_uring will be used for block devices. This results a very significant performance improvement.

Block and Network Device Statistics

Statistics for activity of the virtio network and block devices is now exposed through a new vm.counters HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM.

HTTP API Responses

The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF.

CPU Topology

A topology parameter has been added to --cpus which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core.

Release Build Optimization

Our release build is now built with LTO (Link Time Optimization) which results in a ~20% reduction in the binary size.

Hypervisor Abstraction

A new abstraction has been introduced, in the form of a hypervisor crate so as to enable the support of additional hypervisors beyond KVM.

Snapshot/Restore Improvements

Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs.

Virtio Memory Ballooning Support

A virtio-balloon device has been added, controlled through the resize control, which allows the reclamation of host memory by resizing a memory balloon inside the guest.

Enhancements to ARM64 Support

The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting.

Intel SGX Support

The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated SGX documentation.

Seccomp Sandbox Improvements

The most frequently used virtio devices are now isolated with their own seccomp filters. It is also now possible to pass --seccomp=log which result in the logging of requests that would have otherwise been denied to further aid development.

Notable Bug Fixes

  • Our virtio-vsock implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes.
  • CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338)
  • A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543).
  • virtio-mmio based devices are now more widely tested (#275).
  • Multiple issues have been fixed with virtio device configuration (#1217)
  • Console input was wrongly consumed by both virtio-console and the serial. (#1521)

Contributors

Many thanks to everyone who has contributed to our 0.9.0 release including some new faces.

Assets 5

v0.8.0

11 Jun 17:48
@github-actions github-actions
v0.8.0
52b8396
Compare
Choose a tag to compare
v0.8.0 Pre-release
Pre-release

v0.8.0

This release has been tracked through the 0.8.0 project.

Highlights for cloud-hypervisor version 0.8.0 include:

Experimental Snapshot and Restore Support

This release includes the first version of the snapshot and restore feature.
This allows a VM to be paused and then subsequently snapshotted. At a later
point that snapshot may be restored into a new running VM identical to the
original VM at the point it was paused.

This feature can be used for offline migration from one VM host to another, to
allow the upgrading or rebooting of the host machine transparently to the guest
or for templating the VM. This is an experimental feature and cannot be used on
a VM using passthrough (VFIO) devices. Issues with SMP have also been observed
(#1176).

Experimental ARM64 Support

Included in this release is experimental support for running on ARM64.
Currently only virtio-mmio devices and a serial port are supported. Full
details can be found in the ARM64 documentation.

Support for Using 5-level Paging in Guests

If the host supports it the guest is now enabled for 5-level paging (aka LA57).
This works when booting the Linux kernel with a vmlinux, bzImage or firmware
based boot. However booting an ELF kernel built with CONFIG_PVH=y does not
work due to current limitations in the PVH boot process.

Virtio Device Interrupt Suppression for Network Devices

With virtio-net and vhost-user-net devices the guest can suppress
interrupts from the VMM by using the VIRTIO_RING_F_EVENT_IDX feature. This
can lead to an improvement in performance by reducing the number of interrupts
the guest must service.

vhost_user_fs Improvements

The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with seccomp.

Notable Bug Fixes

  • VMs that have not yet been booted can now be deleted (#1110).
  • By creating the tap device ahead of creating the VM it is not required to
    run the cloud-hypervisor binary with CAP_NET_ADMIN (#1273).
  • Block I/O via virtio-block or vhost-user-block now correctly adheres to
    the specification and synchronizes to the underlying filesystem as required
    based on guest feature negotiation. This avoids potential data loss (#399,
    #1216).
  • When booting with a large number of vCPUs then the ACPI table would be
    overwritten by the SMP MPTABLE. When compiled with the acpi feature the
    MPTABLE will no longer be generated (#1132).
  • Shutting down VMs that have been paused is now supported (#816).
  • Created socket files are deleted on shutdown (#1083).
  • Trying to use passthrough devices (VFIO) will be rejected on mmio builds
    (#751).

Command Line and API Changes

This is non exhaustive list of HTTP API and command line changes:

  • All user visible socket parameters are now consistently called socket
    rather than sock in some cases.
  • The ch-remote tool now shows any error message generated by the VMM
  • The wce parameter has been removed from --disk as the feature is always
    offered for negotiation.
  • --net has gained a host_mac option that allows the setting of the MAC
    address for the tap device on the host.

Contributors

Many thanks to everyone who has contributed to our 0.8.0 release including some new faces.

Assets 5

v0.7.0

30 Apr 15:52
@github-actions github-actions
v0.7.0
96be822
Compare
Choose a tag to compare
v0.7.0 Pre-release
Pre-release

This release has been tracked through the 0.7.0 project.

Highlights for cloud-hypervisor version 0.7.0 include:

Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug

Further to our effort to support modifying a running guest we now support
hotplug and unplug of the following virtio backed devices: block, network,
pmem, virtio-fs and vsock. This functionality is available on the (default) PCI
based tranport and is exposed through the HTTP API. The ch-remote utility
provides a CLI for adding or removing these device types after the VM has
booted. User can use the id parameter on the devices to choose names for
devices to ease their removal.

Alternative libc Support

Cloud Hypervisor can now be compiled with the musl C library and this release
contains a static binary compiled using that toolchain.

Multithreaded Multi Queued vhost-user Backends

The vhost-user backends for network and block support that are shipped by
Cloud Hypervisor have been enhanced to support multiple threads and queues to
improve throughput. These backends are used automatically if vhost_user=true
is passed when the devices are created.

Initial RamFS Support

By passing the --initramfs command line option the user can specify a file to
be loaded into the guest memory to be used as the kernel initial filesystem.
This is usually used to allow the loading of drivers needed to be able to
access the real root filesystem but it can also be used standalone for a very
minimal image.

Alternative Memory Hotplug: virtio-mem

As well as supporting ACPI based hotplug Cloud Hypervisor now supports using
the virtio-mem hotplug alternative. This can be controlled by the
hotplug_method parameter on the --memory command line option. It currently
requires kernel patches to be able to support it.

Seccomp Sandboxing

Cloud Hypervisor now has support for restricting the system calls that the
process can use via the seccomp security API. This on by default and is
controlled by the --seccomp command line option.

Updated Distribution Support

With the release of Ubuntu 20.04 we have added that to the list of supported
distributions and is part of our regular testing programme.

Command Line and API Changes

This is non exhaustive list of HTTP API and command line changes

  • New id fields added for devices to allow them to be named to ease removal.
    If no name is specified the VMM chooses one.
  • Use --memory's shared and hugepages controls for determining backing
    memory instead of providing a path.
  • The --vsock parameter only takes one device as the Linux kernel only
    supports a single Vsock device. The REST API has removed the vector for this
    option and replaced it with a single optional field.
  • There is enhanced validation of the command line and API provided
    configurations to ensure that the provided options are compatible e.g. that
    shared memory is in use if any attempt is made to used a vhost-user backed
    device.
  • ch-remote has added add-disk, add-fs, add-net, add-pmem and
    add-vsock subcommands. For removal remove-device is used. The REST API
    has appropriate new HTTP endpoints too.
  • Specifying a size with --pmem is no longer required and instead the size
    will be obtained from the file. A discard_writes option has also been added
    to provide the equivalent of a read-only file.
  • The parameters to --block-backend have been changed to more closely align
    with those used by --disk.

Contributors

Many thanks to everyone who has contributed to our 0.7.0 release including some new faces.

Assets 5