This is unreleased documentation for SUSE® Virtualization v1.9 (Dev).

Storage Performance Options

Starting with SUSE Virtualization v1.9.1, you can tune virtual machine disk I/O performance directly from the SUSE Virtualization UI. These options expose KubeVirt disk and I/O thread settings that previously required manual edits to the virtual machine YAML manifest.

The options are categorized into two scopes:

Per-volume options: Configure the cache mode, I/O mode, and dedicated I/O thread assignments for individual disks.
VM-wide options: Configure Virtio Block Multi-Queue and the I/O thread policy across all disks attached to the virtual machine.

These options are not required. If unconfigured, KubeVirt and the underlying hypervisor apply the default values, leaving the virtual machine specification unchanged.

Prerequisites

SUSE Virtualization v1.9.1 or later
Storage supporting direct I/O for Cache Mode none and I/O Mode native (including the High Performance profile), requiring volumes provisioned with the Block volume mode
Disks using the virtio bus for Dedicated I/O Thread and Virtio Block Multi-Queue
Sufficient unallocated CPU capacity on the host node when using the Supplemental Pool I/O threads policy, which adds CPU requests to the virtual machine launcher pod

Limitations

These options are applied when the virtual machine starts. Modifications to a running virtual machine require a restart to take effect.
I/O Mode native requires Cache Mode none. For more information, see I/O Mode.
Dedicated I/O threads are not supported on disks using the scsi bus.
If a storage backend does not support direct I/O on Filesystem volumes, a virtual machine configured with Cache Mode none fails to start.
You cannot verify Virtio Block Multi-Queue from inside the guest operating system. For more information, see Virtio Block Multi-Queue.

Performance impacts vary by cluster configuration. Always benchmark these options in a staging environment before modifying production workloads. For testing instructions, see Measuring the effect.

Configuring Per-Volume Options

Go to Virtual Machines, and then perform one of the following actions:
- To create a virtual machine, click Create.
- To edit an existing virtual machine, select ⋮ → Edit Config.
On the Volumes tab, locate the target disk, and then click Show high-performance options.

The Storage Performance Options section is available for image, new, existing, and container volumes. If the disk already has any of these options configured, the section is expanded by default.

Select a Performance Profile.

Profile Description

Profile	Description
Default (Balanced)	Applies default hypervisor and KubeVirt settings. No custom options are configured on the virtual machine manifest.
High Performance	Preconfigures the disk for heavy I/O workloads by applying the `virtio` bus, Cache Mode `none`, I/O Mode `native`, and a Dedicated I/O Thread. Use this profile for volumes provisioned with `Block` volume mode.
Custom	Allows you to configure Cache Mode, I/O Mode, and Dedicated I/O Thread independently.

Default (Balanced)

Applies default hypervisor and KubeVirt settings. No custom options are configured on the virtual machine manifest.

High Performance

Preconfigures the disk for heavy I/O workloads by applying the virtio bus, Cache Mode none, I/O Mode native, and a Dedicated I/O Thread. Use this profile for volumes provisioned with Block volume mode.

Custom

Allows you to configure Cache Mode, I/O Mode, and Dedicated I/O Thread independently.

(Optional) If you selected Custom, configure the Cache Mode, IO Mode, and Dedicated I/O Thread settings.
Click Create or Save.

If the virtual machine is running, the Restart Virtual Machine dialog appears. Click Save and Restart to apply the changes immediately. If you do not restart the virtual machine, the changes are applied on the next restart.

You can also configure these options during volume creation. Go to Volumes, click Create, and configure the Storage Performance Options section. The selected options are saved as defaults on the volume and are automatically applied whenever it is attached to a virtual machine.

Per-volume options are fields of each disk in the VirtualMachine object.

spec:
  template:
    spec:
      domain:
        devices:
          disks:
            - name: disk-0
              bootOrder: 1
              disk:
                bus: virtio
              cache: none                   # none, writeback or writethrough
              io: native                    # native or threads
              dedicatedIOThread: true

For information about each field, see Disks and Volumes in the KubeVirt user guide.

Cache mode

The cache mode determines whether host page caching is used between the guest operating system and the storage backend.

Cache Mode Description Trade-offs Recommended Use

Cache Mode	Description	Trade-offs	Recommended Use
Default (Unset)	KubeVirt automatically selects `none` if the underlying storage supports direct I/O. Otherwise, it selects `writethrough`.	The behavior depends on storage capabilities.	General-purpose workloads with standard I/O requirements
None (no host cache)	Guest I/O bypasses the host page cache. Reads and writes interact directly with the storage backend (direct I/O). Guest-level cache and flush requests remain active.	The storage backend must support direct I/O. If unsupported (common on `Filesystem` volumes), the virtual machine fails to start.	High-throughput I/O workloads on `Block` volumes. This mode is required when using I/O Mode `native`. KubeVirt describes this mode as generally the best choice.
WriteBack	Guest I/O is cached on the host. A write completes as soon as the data reaches the host page cache, and the data is written to the storage when the guest operating system issues a flush.	The host page cache consumes host memory. This mode is safe only if the guest operating system flushes its disk caches correctly. Otherwise, a host crash or power loss can corrupt data in the guest.	Guests whose operating system reliably flushes its disk caches, on storage that does not support direct I/O, when writes must complete without waiting for the storage.
WriteThrough	Guest reads are cached on the host. A write completes only after the data is written to the storage.	Write performance is significantly lower than with the other modes because each write waits for the storage.	Guests that do not flush their disk caches correctly, when write durability is more important than write performance.

Default (Unset)

KubeVirt automatically selects none if the underlying storage supports direct I/O. Otherwise, it selects writethrough.

The behavior depends on storage capabilities.

General-purpose workloads with standard I/O requirements

None (no host cache)

Guest I/O bypasses the host page cache. Reads and writes interact directly with the storage backend (direct I/O). Guest-level cache and flush requests remain active.

The storage backend must support direct I/O. If unsupported (common on Filesystem volumes), the virtual machine fails to start.

High-throughput I/O workloads on Block volumes. This mode is required when using I/O Mode native. KubeVirt describes this mode as generally the best choice.

WriteBack

Guest I/O is cached on the host. A write completes as soon as the data reaches the host page cache, and the data is written to the storage when the guest operating system issues a flush.

The host page cache consumes host memory. This mode is safe only if the guest operating system flushes its disk caches correctly. Otherwise, a host crash or power loss can corrupt data in the guest.

Guests whose operating system reliably flushes its disk caches, on storage that does not support direct I/O, when writes must complete without waiting for the storage.

WriteThrough

Guest reads are cached on the host. A write completes only after the data is written to the storage.

Write performance is significantly lower than with the other modes because each write waits for the storage.

Guests that do not flush their disk caches correctly, when write durability is more important than write performance.

I/O Mode

The I/O mode determines how QEMU submits disk I/O to the host. The following table describes each I/O mode:

I/O mode Description Trade-offs Recommended use

I/O mode	Description	Trade-offs	Recommended use
Default (not set)	The hypervisor selects the mode. On block volumes with cache mode `none`, KubeVirt selects `native`.		Most virtual machines.
Native (AIO)	QEMU submits I/O through the native asynchronous I/O interface of the Linux kernel.	This mode requires Cache Mode `none`. When you select this mode, the UI sets Cache Mode to `none` and disables the other cache modes.	Disks with Cache Mode `none` on Block volumes, as in the High Performance profile.
Threads	QEMU submits I/O from a pool of host worker threads.		Disks with Cache Mode `writeback` or `writethrough`, which the `native` mode does not support.

Default (not set)

The hypervisor selects the mode. On block volumes with cache mode none, KubeVirt selects native.

Most virtual machines.

Native (AIO)

QEMU submits I/O through the native asynchronous I/O interface of the Linux kernel.

This mode requires Cache Mode none. When you select this mode, the UI sets Cache Mode to none and disables the other cache modes.

Disks with Cache Mode none on Block volumes, as in the High Performance profile.

Threads

QEMU submits I/O from a pool of host worker threads.

Disks with Cache Mode writeback or writethrough, which the native mode does not support.

I/O Mode native requires Cache Mode none. The UI enforces this requirement. However, if a virtual machine is created or edited through YAML or the API with io: native and a different cache mode, the virtual machine is accepted but fails to start. The virtual machine remains in the Starting state, and its Synchronized condition reports unsupported configuration: io='native' needs either no disk cache or directsync cache mode.

Dedicated I/O thread

An I/O thread is a host thread that processes disk I/O. The Dedicated I/O Thread option assigns the disk an I/O thread that is never shared with other disks. Without this option, the disk shares I/O threads with other disks according to the I/O threads policy.

Recommended use: A disk with heavy I/O whose processing must not compete with the other disks of the virtual machine. For example, a database data disk.
Requirements: The disk must use the virtio bus. Dedicated I/O threads are not supported on the SCSI bus. If a disk with performance options uses a different bus, the UI displays a tip.
Effect on the virtual machine: Enabling this option on any disk also enables the I/O threads policy of the virtual machine. If no policy is selected, the UI sets the policy to Shared.

Configuring virtual machine-wide options

These options apply to all volumes of the virtual machine and complement the per-volume options.

Go to Virtual Machines, and then perform one of the following actions:
- To create a virtual machine, click Create.
- To edit an existing virtual machine, select ⋮ → Edit Config.
On the Volumes tab, locate the High Performance (I/O Threads and Multi-Queue) section below the disks, and then click Show VM-wide high-performance options.

The section is already expanded if the virtual machine uses one of these options.
(Optional) Select Virtio Block Multi-Queue.

At least one disk must use the virtio bus.
(Optional) Select an I/O threads policy.

If you select Supplemental Pool, specify the I/O Thread Count.
Click Create or Save.

If the virtual machine is running, the Restart Virtual Machine dialog appears. Click Save and Restart to apply the changes immediately. If you do not restart the virtual machine, the changes are applied on the next restart.

VM-wide options are fields of the domain in the VirtualMachine object.

spec:
  template:
    spec:
      domain:
        ioThreadsPolicy: supplementalPool   # shared, auto or supplementalPool
        ioThreads:
          supplementalPoolThreadCount: 2    # only with supplementalPool
        devices:
          blockMultiQueue: true

For information about each field, see Disks and Volumes in the KubeVirt user guide.

Virtio Block Multi-Queue

Virtio Block Multi-Queue maps the I/O of each virtio disk to multiple queues, one per vCPU, so that disk I/O can be processed on several CPUs in parallel.

Queue count: The number of queues automatically matches the number of vCPUs of the virtual machine, as libvirt recommends. You do not need to configure the queue count.
Requirements: At least one disk must use the virtio bus, and the virtual machine must have a specific CPU allocation. SUSE Virtualization virtual machines always have a specific CPU allocation.
Recommended use: Virtual machines with multiple vCPUs that run disk I/O from many threads or processes concurrently.

You cannot verify multi-queue from inside the guest by counting the queues of a disk (for example, by running ls /sys/block/vda/mq). By default, QEMU already assigns one queue per vCPU to virtio-blk disks, so the guest reports the same number of queues regardless of whether the option is enabled. To verify the setting, check the virtual machine YAML for blockMultiQueue: true, as described in Verifying the settings.

I/O threads policy

The I/O threads policy determines the number of I/O threads of the virtual machine and how disks are assigned to them. The following table describes each policy:

Policy Description Recommended Use

Policy	Description	Recommended Use
Disabled (default)	No I/O threads are used.	Most virtual machines.
Shared	All disks share one I/O thread. Disks with Dedicated I/O Thread enabled receive their own thread in addition.	Virtual machines with a small number of disks, or with one or two disks that require a dedicated thread. The UI selects this policy when a disk requests a dedicated thread and no policy is set.
Auto	KubeVirt creates a pool of I/O threads and distributes the disks across the pool. The pool is limited to twice the number of vCPUs.	Virtual machines with many disks, so that the disks do not all share one thread.
Supplemental Pool	Creates the number of I/O threads specified in I/O Thread Count (default: `2`) and adds the same number of CPUs to the pod of the virtual machine. With dedicated CPU placement, each I/O thread is pinned to one of the added CPUs, so that I/O threads and vCPUs run on separate physical CPUs.	Workloads that require explicit control over the number of I/O threads. For example, virtual machines that use CPU pinning. Account for the added CPUs when sizing nodes.

Disabled (default)

No I/O threads are used.

Most virtual machines.

Shared

All disks share one I/O thread. Disks with Dedicated I/O Thread enabled receive their own thread in addition.

Virtual machines with a small number of disks, or with one or two disks that require a dedicated thread. The UI selects this policy when a disk requests a dedicated thread and no policy is set.

Auto

KubeVirt creates a pool of I/O threads and distributes the disks across the pool. The pool is limited to twice the number of vCPUs.

Virtual machines with many disks, so that the disks do not all share one thread.

Supplemental Pool

Creates the number of I/O threads specified in I/O Thread Count (default: 2) and adds the same number of CPUs to the pod of the virtual machine. With dedicated CPU placement, each I/O thread is pinned to one of the added CPUs, so that I/O threads and vCPUs run on separate physical CPUs.

Workloads that require explicit control over the number of I/O threads. For example, virtual machines that use CPU pinning. Account for the added CPUs when sizing nodes.

Verifying the settings

Go to Virtual Machines.

If the saved options differ from the options of the running virtual machine, the UI displays a Pending Restart notification that lists the virtual machine, and the state of the virtual machine indicates that a restart is required for the configuration change to take effect.
Click the name of the target virtual machine, and then go to the Volumes tab.

The Storage Performance Options section of each disk and the High Performance (I/O Threads and Multi-Queue) section display the saved options. The High Performance (I/O Threads and Multi-Queue) section appears only if the virtual machine uses one of these options.

Inspect the disks and domain settings of the virtual machine. The VirtualMachine object shows the saved settings, and the VirtualMachineInstance object shows the settings of the running virtual machine. If the two differ, the virtual machine has not been restarted since the change, and the VirtualMachine object has the RestartRequired condition.

# Saved settings
kubectl get vm <vm-name> -n <namespace> -o yaml | grep -E 'cache:|io:|dedicatedIOThread|blockMultiQueue|ioThreadsPolicy|supplementalPoolThreadCount'

# Settings of the running virtual machine
kubectl get vmi <vm-name> -n <namespace> -o yaml | grep -E 'cache:|io:|dedicatedIOThread|blockMultiQueue|ioThreadsPolicy|supplementalPoolThreadCount'

Measuring the effect

To measure the effect, run the same benchmark in the guest before and after you change the options. For example, the following fio command runs a random read/write test against a 1 GiB test file in a Linux guest. Use a test file rather than a raw device to avoid overwriting data, and delete the file afterwards.

fio --name=randrw --filename=/var/tmp/fio-test --size=1G \
    --rw=randrw --bs=4k --ioengine=libaio --direct=1 \
    --iodepth=32 --numjobs=4 --runtime=60 --time_based --group_reporting
rm /var/tmp/fio-test

Compare the IOPS and latency in the output. The results depend on the storage, the workload, and the number of vCPUs, so test with an I/O pattern that resembles your workload.