This release has been tracked in our roadmap
project as iteration
v39.0. The following user visible changes have been made:

Variable Sizing of PCI Apertures for Segments

It is now possible to use --pci-segment to adjust the aperture size that
devices 32-bit and 64-bit PCI device BARs will be allocated from. Previously
the address space was equally distributed across all the segments which may
leave insufficient space for devices that require a large 32-bit space. With
this change the weighting per segment can be adjusted. (#6387)

Direct Booting with bzImages

Support for directly booting Linux from bzImages has been added.(#6200)

Support for NVIDIA GPUDirect P2P Support

The x_nv_gpudirect_clique option was added to --device to allow the
configuration of device P2P support with NVIDIA GPUs. (#6235)

Guest NMI Injection Support

A new API endpoint and ch-remote option added for injecting an NMI into the
guest. (#6047)

Notable Bug Fixes

• Workaround for kernel bug affecting guest IRQ masking on AMD (#6353)
• Correctly cleanup sigwinch_listener process (#6208)
• Graceful shutdown of HTTP API thread (#6248, #6247)
• Fix queue_affinity option in OpenAPI metadata (#6268)
• Fix documentation to indicate only stream mode is supported by virtio-vsock
  (#6306)
• Fix virtio-fs tag validation (#6358, #6359)
• Add missing pvpanic device to OpenAPI metadata (#6372)
• Fixes for nested virtualization with VFIO devices (#6110, #6298, #6297,
  #6319)
• Fix for backing file for virtio-mem regions with snapshot/restore (#6337,
  #6338)
• Explicitly mark FDs used for network devices as invalid across
  snapshot/restore (#6332, #6286)
• Improve event-monitor events around reboot (#6277, #6274)
• Fix potential deadlock around paused devices during live migration (#6293)
• Fix panic when running ch-remote with no subcommand (#6230)
• Fix hotplug of virtio devices after snapshot/restore and live migration
  (#6326, #6265)

Contributors

Many thanks to everyone who has contributed to our release:

• Alexandru Matei [email protected]
• Andrew Carp [email protected]
• Bo Chen [email protected]
• Bouke van der Bijl [email protected]
• Chris Webb [email protected]
• Jinank Jain [email protected]
• Lucas Jacques [email protected]
• Muminul Islam [email protected]
• Nuno Das Neves [email protected]
• Ravi kumar Veeramally [email protected]
• Rob Bradford [email protected]
• Ruslan Mstoi [email protected]
• Stefan Nuernberger [email protected]
• Thomas Barrett [email protected]
• Wei Liu [email protected]
• Yi Wang [email protected]

This is a bug fix release. The following issues have been addressed:

• Fix several security advisories from dependencies (#6134, #6141)
• Enable HTT flag to avoid crashing cpu topology enumeration software
  such as hwloc in the guest (#6146)
• Enable nested virtualization on AMD if supported (#6106)
• Handle non-power-of-two CPU topology properly (#6062)
• Various bug fixes around virtio-vsock(#6080, #6091, #6095)
• Align VFIO devices PCI BARs naturally (#6196)

This release has been tracked in our roadmap
project as iteration
v38.0. The following user visible changes have been made:

Group Rate Limiter on Block Devices

Users now can throttle a group of block devices with the new
--rate-limiter-group option. Details can be found from the I/O
Throttling documentation

CPU Pinning Support for Block Device Worker Thread

Users now have the option to pin virt-queue threads for block devices
to specific host cpus.

Optimized Boot Time with Parallel Memory Prefault

The boot time with prefault option enabled is optimized via parallel
memory prefault.

New 'debug-console' Device

A 'debug-console' device is added to provide a user-configurable debug
port for logging guest information. Details can be found from the Debug
IO Ports documentation.

Improved VFIO Device Support

All non-emulated MMIO regions of VFIO devices are now mapped to the VFIO
container, allowing PCIe P2P between all VFIO devices on the same
VM. This is required for a wide variety of multi-GPU workloads involving
GPUDirect P2P (DMA between two GPUs), GPUDirect RDMA (DMA between a GPU
and an IB device).

Extended CPU Affinity Support

Users now can set the vcpu affinity to a host CPU with index larger
than 255.

Notable Bug Fixes

• Enable HTT flag to avoid crashing cpu topology enumeration software
  such as hwloc in the guest (#6146)
• Fix several security advisories from dependencies (#6134, #6141)
• Handle non-power-of-two CPU topology properly (#6062)
• Various bug fixes around virtio-vsock(#6080, #6091, #6095)
• Enable nested virtualization on AMD if supported (#6106)
• Align VFIO devices PCI BARs naturally (#6196)

Contributors

Many thanks to everyone who has contributed to our release:

• Alyssa Ross [email protected]
• Bo Chen [email protected]
• Daniel Farina [email protected]
• Jinank Jain [email protected]
• Muminul Islam [email protected]
• Peteris Rudzusiks [email protected]
• Philipp Schuster [email protected]
• Ravi kumar Veeramally [email protected]
• Rob Bradford [email protected]
• Ruslan Mstoi [email protected]
• Sean Banko [email protected]
• Thomas Barrett [email protected]
• Wei Liu [email protected]
• Yi Wang [email protected]
• acarp [email protected]

This is a bug fix release. The following issues have been addressed:

• Fix VFIO on platforms with non-4k page size (#5450)
• Fix seccomp filter lists related to virtio-console, serial and pty
  (#5506, #5524)
• Populate APIC ID properly (#5512)
• Ignore and warn TAP FDs in more situations (#5522)
• Disallow concurrent CPU resizing (#5668)
• Use wrapping add for memory offset from instruction emulator (#5719)
• Replace unsound static mut with once_cell (#5772)
• Fix a deadlock when TDX is enabled (#5845)
• Bug fix to OpenAPI specification file (#5967)
• Error out early for live migration when TDX is enabled (#6025)

This is a bug fix release. The following issues have been addressed:

• Fix aio backend behavior for block devices when writeback cache
  disabled (#5930)
• Fix PvPanic device PCI BAR alignment (#5956)
• Bug fix to OpenAPI specification file (#5967)
• Error out early for live migration when TDX is enabled (#6025)

This is a bug fix release. The following issues have been addressed:

• Fix a deadlock when TDX is enabled (#5845)
• Correct default value for vCPU topology on AArch64 (#5893)
• Fix PvPanic device PCI BAR alignment (#5956)
• Bug fix to OpenAPI specification file (#5967)
• Error out early for live migration when TDX is enabled (#6025)

This release has been tracked in our roadmap project as iteration
v37.0. The following user visible changes have been made:

Long Term Support (LTS) Release

This release is a LTS release. Point releases for bug fixes will be made
for the next 18 months; live migration and live upgrade will be
supported between the point releases of the LTS.

Multiple PCI segments Support for 32-bit VFIO Devices

Now VFIO devices with 32-bit memory BARs can be attached to non-zero PCI
segments on the guest, allowing users to have more 32-bit devices and
assign such devices to appropriate NUMA nodes for better performance.

Configurable Named TAP Devices

Named TAP devices now accepts IP configuration from users, such as IP
and MAC address, as long as the named TAP device is created by Cloud
Hypervisor (e.g. not existing TAP devices).

TTY Output from Both Serial Device and Virtio Console

Now legacy serial device and virtio console can be set as TTY mode as
the same time. This allows users to capture early boot logs with the
legacy serial device without losing performance benefits of using
virtio-console, when appropriate kernel configuration is used (such as
using kernel command-line console=hvc0 earlyprintk=ttyS0 on x86).

Faster VM Restoration from Snapshots

The speed of VM restoration from snapshots is improved with a better
implementation of deserializing JSON files.

Notable Bug Fixes

• Fix aio backend behavior for block devices when writeback cache
  disabled (#5930)
• Fix PvPanic device PCI BAR alignment (#5956)
• Bug fix to OpenAPI specification file (#5967)
• Error out early for live migration when TDX is enabled (#6025)

Contributors

Many thanks to everyone who has contributed to our release:

• Bo Chen [email protected]
• Jinank Jain [email protected]
• Markus Sütter [email protected]
• Michael Zhao [email protected]
• Muminul Islam [email protected]
• Rob Bradford [email protected]
• Rui Chang [email protected]
• Ruslan Mstoi [email protected]
• Thomas Barrett [email protected]
• Wei Liu [email protected]
• Yi Wang [email protected]
• Yong He [email protected]

This release has been tracked in our roadmap project as iteration
v36.0. The following user visible changes have been made:

Command Line Changes

We switched back to use the clap crate to create our command line,
since the argh crate is barely maintained. There were several syntax
changes:

• All --option value commands now are --option=value.
• The --disk DISK1 --disk DISK2 command now is --disk DISK1 DISK2.
• The -v -v -vcommand now is -vvv.

Note: the released binary size increased around 0.3M due to this change.

Enabled Features Reported via API Endpoint and CLI

Now the enabled (Cargo) features of the running Cloud Hypervisor
instance can be queried via API endpoint (/vmm.ping) and CLI
(--version -v).

NUMA Support for PCI segments

The --numa command is augmented with a new option pci_segment=, so
that users can define the relationship between PCI segments and NUMA
nodes. Examples can be found from the memory documentation

CPU Topology Support on AMD Platforms

Now the CPU topology on x86_64 platforms supports multiple vendors.

Unix Socket Backend for Serial Port

The --serial command is augmented with a new option socket=, allowing
users to access the serial port using a Unix socket.

AIO Backend for Block Devices

An AIO backend is added for virtio-block devices to improve block
device performance when the io_uring feature is not supported by the
host Operating System.

Documentation Improvements

• New documentation for collecting coverage data
• Various typo fixes

Notable Bug Fixes

• Fix a deadlock when TDX is enabled (#5845)
• Only advertise AMX feature bits to guest when the AMX cpu feature is
  enabled (#5834)
• Correct default value for vCPU topology on AArch64 (#5893)

Contributors

Many thanks to everyone who has contributed to our release:

• Anatol Belski [email protected]
• Bo Chen [email protected]
• Dario Nieuwenhuis [email protected]
• Jianyong Wu [email protected]
• Jinank Jain [email protected]
• Muminul Islam [email protected]
• Praveen K Paladugu [email protected]
• Ravi kumar Veeramally [email protected]
• Rob Bradford [email protected]
• Thomas Barrett [email protected]
• Wei Liu [email protected]
• Yi Wang [email protected]
• dom.song [email protected]

This release has been tracked in our roadmap project as iteration v35.0.
The following user visible changes have been made:

virtio-vsock Support for Linux Guest Kernel v6.3+

Since kernel v6.3, a vsock packet can be included in a single descriptor,
instead of being split over two descriptors. Our virtio-vsock implementation
now support both situations.

User Specified Serial Number for virtio-block

A new option serial is added to the --block command that allows users to
specify a serial number for block devices which will be guest visible.

vCPU TSC Frequency Included in Migration State

This ensures migration works correctly between hosts that have different TSC
frequencies if the guest is running with TSC as the source of timekeeping.

Notable Bug Fixes

• Disallow concurrent CPU resizing (#5668)
• Handle APIC EOI message for MSHV (#5681)
• Use wrapping add for memory offset from instruction emulator (#5719)
• Add global spell check via the 'typos' GitHub action (#5720)
• Ensure probing reads are block size aligned (#5727)
• Multiple bug fixes around the latency counter for block devices
  (#5712, #5750, #5762, #5763)
• Replace unsound static mut with once_cell (#5772)

Contributors

Many thanks to everyone who has contributed to our release:

• Alyssa Ross [email protected]
• Anatol Belski [email protected]
• Bo Chen [email protected]
• Christian Blichmann [email protected]
• Jianyong Wu [email protected]
• Jinank Jain [email protected]
• Julian Stecklina [email protected]
• Omer Faruk Bayram [email protected]
• Philipp Schuster [email protected]
• Rob Bradford [email protected]
• Ruslan Mstoi [email protected]
• Thomas Barrett [email protected]
• Wei Liu [email protected]
• Yi Wang [email protected]
• zhongbingnan [email protected]

This release has been tracked in our roadmap project as iteration v34.0. The following user visible changes have been made:

Paravirtualised Panic Device Support

A new device has been added that can communicate when the guest kernel has panicked and share those details with the VMM. This is controlled with a new --pvpanic command line option and JSON API change equivalent. (#5526)

Improvements to VM Core Dump

Requesting to dump the guest memory as core dump will now transparently pause the VM if required; returning to the original state after. (#5604)

QCOW2 Support for Backing Files

The support for QCOW2 files has been enhanced to include support for using backing files. (#5573)

Minimum Host Kernel Bump

The minimum supported host kernel is now 5.13 in order to incorporate a bug fix for KVM_FEATURE_ASYNC_PF_INT functionality. (#5626)

Notable Bug Fixes

• The x86 emulator is only compiled in if MSHV is compiled in (the kernel carries out this job with KVM) (#5561).
• A regression has been fixed in VFIO support for devices that use MSI rather than MSI-X (#5658).
• When triggering a VM shutdown or reset via I/O the vCPU thread will be blocked until that asynchronous event has been received (#5645).
• Pausing a VM is now a synchronous action with the request only completing when all vCPUs are paused (#5611).
• Event monitor support now correctly supports concurrent access (#5633).
• Bug fixes for the QCOW2 file support (#5573).

Contributors

Many thanks to everyone who has contributed to our release:

• Alyssa Ross [email protected]
• Anatol Belski [email protected]
• Bo Chen [email protected]
• Changyuan Lyu [email protected]
• Christian Blichmann [email protected]
• Manish Goregaokar [email protected]
• Omer Faruk Bayram [email protected]
• Philipp Schuster [email protected]
• Praveen K Paladugu [email protected]
• Rob Bradford [email protected]
• Ruslan Mstoi [email protected]
• Yi Wang [email protected]
• Yong He [email protected]
• Yu Li [email protected]
• dom.song [email protected]

This is a bug fix release. The following issues have been addressed:

• Report errors explicitly to users when VM failed to boot (#5453)
• Fix VFIO on platforms with non-4k page size (#5450, #5469)
• Fix TDX initialization (#5454)
• Ensure all guest memory regions are page-size aligned (#5496)
• Fix seccomp filter lists related to virtio-console, serial and pty
  (#5506, #5524)
• Populate APIC ID properly (#5512)
• Ignore and warn TAP FDs in more situations (#5522)

This is a bug fix release. The following issues have been addressed:

• The number of vCPUs is capped at the hypervisor maximum (#5357)
• Fixes for TTY reset (#5414)
• CPU topology fixes on MSHV (#5325)
• Seccomp fixes for older distributions (#5397)
• Report errors explicitly to users when VM failed to boot (#5453)
• Fix VFIO on platforms with non-4k page size (#5450, #5469)
• Fix TDX initialization (#5454)
• Ensure all guest memory regions are page-size aligned (#5496)
• Fix seccomp filter lists related to virtio-console, serial and pty
  (#5506, #5524)
• Populate APIC ID properly (#5512)
• Ignore and warn TAP FDs in more situations (#5522)

v33.0

This release has been tracked in our roadmap project as iteration
v33.0. The following user visible changes have been made:

D-Bus based API

A D-Bus based API has been added as an alternative to the existing REST
API. This feature is gated by the dbus_api feature. Details can be
found in the API documentation.

Expose Host CPU Cache Details for AArch64

Now the CPU cache information on the host is properly exposed to the
guest on AArch64.

Notable Bug Fixes

• Report errors explicitly to users when VM failed to boot (#5453)
• Fix VFIO on platforms with non-4k page size (#5450, #5469)
• Fix TDX initialization (#5454)
• Ensure all guest memory regions are page-size aligned (#5496)
• Fix seccomp filter lists related to virtio-console, serial and pty
  (#5506, #5524)
• Populate APIC ID properly (#5512)
• Ignore and warn TAP FDs in more situations (#5522)

Contributors

Many thanks to everyone who has contributed to our release:

• Alyssa Ross [email protected]
• Anatol Belski [email protected]
• Bo Chen [email protected]
• Jianyong Wu [email protected]
• Omer Faruk Bayram [email protected]
• Rafael Mendonca [email protected]
• Ravi kumar Veeramally [email protected]
• Rob Bradford [email protected]
• Ruslan Mstoi [email protected]
• Yu Li [email protected]
• zhongbingnan [email protected]

This release has been tracked in our roadmap project as iteration v32.0. The following user visible changes have been made:

Increased PCI Segment Limit

The maximum number of PCI segments that can be used is now 96 (up from 16).

API Changes

• The VmmPingResponse now includes the PID as well as the build details. (#5348)

Notable Bug Fixes

• Ignore and warn TAP FDs sent via the HTTP request body (#5350)
• Properly preserve and close valid FDs for TAP devices (#5373)
• Only use KVM_ARM_VCPU_PMU_V3 if available (#5360)
• Only touch the tty flags if it's being used (#5343)
• Fix seccomp filter lists for vhost-user devices (#5361)
• The number of vCPUs is capped at the hypervisor maximum (#5357)
• Fixes for TTY reset (#5414)
• CPU topology fixes on MSHV (#5325)
• Seccomp fixes for older distributions (#5397)

Contributors

Many thanks to everyone who has contributed to our release:

• Alyssa Ross [email protected]
• Anatol Belski [email protected]
• Bo Chen [email protected]
• Hao Xu [email protected]
• Muminul Islam [email protected]
• Omer Faruk Bayram [email protected]
• Rafael Mendonca [email protected]
• Rob Bradford [email protected]
• Ruslan Mstoi [email protected]
• Smit Gardhariya [email protected]
• Wei Liu [email protected]

This is a bug fix release. The following issues have been addressed:

• Ignore and warn TAP FDs sent via the HTTP request body (#5350)
• Properly preserve and close valid FDs for TAP devices (#5373)
• Only use KVM_ARM_VCPU_PMU_V3 if available (#5360)
• Fix seccomp filter lists for vhost-user devices (#5361)
• Fix the offset setting while removing the entire mapping of
  vhost-user FS client (#5235)
• Set vcpu thread status properly and signal exit_evt upon thread exit (#5211)
• Populate CPUID leaf 0x4000_0010 (TSC frequency) (#5179)
• Bug fix to OpenAPI specification file (#5186)

This is a bug fix release. The following issues have been addressed:

• Ignore and warn TAP FDs sent via the HTTP request body (#5350)
• Properly preserve and close valid FDs for TAP devices (#5373)
• Only use KVM_ARM_VCPU_PMU_V3 if available (#5360)
• Only touch the tty flags if it's being used (#5343)
• Fix seccomp filter lists for vhost-user devices (#5361)

This is a bug fix release. The following issues have been addressed:

• Ignore and warn TAP FDs sent via the HTTP request body (#5350)
• Properly preserve and close valid FDs for TAP devices (#5373)
• Only use KVM_ARM_VCPU_PMU_V3 if available (#5360)
• Only touch the tty flags if it's being used (#5343)
• Fix seccomp filter lists for vhost-user devices (#5361)
• Fix the offset setting while removing the entire mapping of
  vhost-user FS client (#5235)
• Fix the ShutdownVmm and Shutdown commands to call the correct API
  endpoint (#5322)

This release has been tracked in our roadmap project as iteration
v31.0. The following user visible changes have been made:

Update to Latest acpi_tables

Adapted to the latest acpi_tables. There has been significant API changes in
the crate.

Update Reference Kernel to 6.2

Updated the recommended guest kernel version from 6.1.6 to 6.2.

Improvements on Console SIGWINCH Handler

A separate thread had been created to capture the SIGWINCH signal and resize
the guest console. Now the thread is skipped if the console is not resizable.

Two completely different code paths existed for handling console resizing, one
for tty and the other for pty. That makes the understanding of the console
handling code unnecessarily complicated. Now the code paths are unified. Both
tty and pty are supported in single SIGWINCH handler. And the new handler
can works with kernel versions earlier than v5.5.

Remove Directory Support from MemoryZoneConfig::file

Setting a directory to MemoryZoneConfig::file is no longer supported.

Before this change, user can set a directory to file of the --memory-zone
option. In that case, a temporary file will be created as the backing file for
the mmap(2) operation. This functionality has been unnecessary since we had
the native support for hugepages and allocating anonymous shared memory.

Documentation Improvements

• Various improvements in API document
• Improvements in Doc comments
• Updated Slack channel information in README

Notable Bug Fixes

• Fixed the offset setting while removing the entire mapping of vhost-user FS
  client.
• Fixed the ShutdownVmm and Shutdown commands to call the correct API
  endpoint.

Contributors

Many thanks to everyone who has contributed to our release:

• Alyssa Ross [email protected]
• Bo Chen [email protected]
• Daniel Farina [email protected]
• Dom [email protected]
• Hao Xu [email protected]
• Muminul Islam [email protected]
• Omer Faruk Bayram [email protected]
• Ravi kumar Veeramally [email protected]
• Rob Bradford [email protected]
• Ruslan Mstoi [email protected]
• Smit Gardhariya [email protected]
• Yang [email protected]
• Yong He [email protected]

This release has been tracked in our roadmap project as iteration
v30.0. The following user visible changes have been made:

Command Line Changes for Reduced Binary Size

The clap crate was replaced by the argh crate to create our command
line, which reduced our release binary size from 3.6MB to 3.3MB. There
were several syntax changes:

• All --option=value commands now are --option value.
• The --disk DISK1 DISK2 command now is --disk DISK1 --disk DISK2.
• The -vvv command now is -v -v -v

Basic vfio-user Server Support

Our vfio-user crate is extended to provide basic server side support
with an example of gpio vfio-user device. This crate now is moved to its
own repository under the
rust-vmm organization.

Heap Profiling Support

A new building target is added for profiling purposes with examples of
heap profiling using dhat gated by the dhat-heap feature.

Documentation Improvements

The documentation on Intel TDX is expanded with details of the building
and using TD-Shim,
references to TDX Tools, and
version information of guest/host kernel/TDVF/TDShim being tested. Also,
a new 'heap profiling' documentation is added with improvements on the
existing 'profiling' documentation.

Notable Bug Fixes

• Close FDs for TAP devices that are provided to VM (#5199, #5206)
• Set vcpu thread status properly and signal exit_evt upon thread exit (#5211)
• Populate CPUID leaf 0x4000_0010 (TSC frequency) (#5178, #5179)
• Inform the TPM guest driver upon failed TPM requests on the host (#5151)
• Bug fix to OpenAPI specification file (#5186)

Contributors

Many thanks to everyone who has contributed to our release:

• Anatol Belski [email protected]
• Anirudh Rayabharam [email protected]
• Bo Chen [email protected]
• Jinank Jain [email protected]
• Kaihang Zhang [email protected]
• Michael Zhao [email protected]
• Muminul Islam [email protected]
• Philipp Schuster [email protected]
• Praveen K Paladugu [email protected]
• Ravi kumar Veeramally [email protected]
• Rob Bradford [email protected]
• Ruslan Mstoi [email protected]
• Sebastien Boeuf [email protected]
• Wei Liu [email protected]
• Yong He [email protected]
• Yu Li [email protected]

This is a bug fix release. The following issues have been addressed:

• Fix QCOW2 refcount table size (#5034)
• Fix unpause support on MSHV in dual binary (#5037)
• Threads inside virtio devices are now shutdown on reboot (#5095)