Virtualization Best Practices

Changing the configuration of the VM Guest or the VM Host Server can lead to data loss or an unstable state. It is really important that you do backups of files, data, images, etc. before making any changes. Without backups you cannot restore the original state after a data loss or a misconfiguration. Do not perform tests or experiments on production systems.

The efficiency of a virtualization environment depends on many factors. This guide provides a reference for helping to make good choices when configuring virtualization in a production environment. Nothing is carved in stone. Hardware, workloads, resource capacity, etc. should all be considered when planning, testing, and deploying your virtualization infrastructure. Testing your virtualized workloads is vital to a successful virtualization implementation.

SUSE strongly recommends using the libvirt framework to configure, manage, and operate VM Host Servers and VM Guest. It offers a single interface (GUI and shell) for all supported virtualization technologies and therefore is easier to use than the hypervisor-specific tools.

We do not recommend using libvirt and hypervisor-specific tools at the same time, because changes done with the hypervisor-specific tools may not be recognized by the libvirt tool set. See Book “Virtualization Guide”, Chapter 8 “Starting and stopping libvirtd” for more information on libvirt.

Similar to real 64-bit PC hardware, qemu-system-x86_64 supports VM Guests running a 32-bit or a 64-bit operating system. Because qemu-system-x86_64 usually also provides better performance for 32-bit guests, SUSE generally recommends using qemu-system-x86_64 for both 32-bit and 64-bit VM Guests on KVM. Scenarios where qemu-system-i386 is known to perform better are not supported by SUSE.

Xen also uses binaries from the qemu package but prefers qemu-system-i386, which can be used for both 32-bit and 64-bit Xen VM Guests. To maintain compatibility with the upstream Xen Community, SUSE encourages using qemu-system-i386 for Xen VM Guests.

Linux manages memory in units called pages. On most systems the default page size is 4 KB. Linux and the CPU need to know which pages belong to which process. That information is stored in a page table. If a lot of processes are running, it takes more time to find where the memory is mapped, because of the time required to search the page table. To speed up the search, the TLB (Translation Lookaside Buffer) was invented. But on a system with a lot of memory, the TLB is not enough. To avoid any fallback to normal page table (resulting in a cache miss, which is time consuming), huge pages can be used. Using huge pages will reduce TLB overhead and TLB misses (pagewalk). A host with 32 GB (32*1014*1024 = 33,554,432 KB) of memory and a 4 KB page size has a TLB with 33,554,432/4 = 8,388,608 entries. Using a 2 MB (2048 KB) page size, the TLB only has 33554432/2048 = 16384 entries, considerably reducing TLB misses.

4.1.1 Configuring the VM Host Server and the VM Guest to use huge pages #

  

The AMD64/Intel 64 CPU architecture supports larger pages than 4 KB: huge pages. To determine the size of huge pages available on your system (could be 2 MB or 1 GB), check the flags line in the output of /proc/cpuinfo for occurrences of pse and/or pdpe1gb.

Table 1: Determine the available huge pages size #

  

CPU flag	Huge pages size available
Empty string	No huge pages available
pse	2 MB
pdpe1gb	1 GB

Using huge pages improves performance of VM Guests and reduces host memory consumption.

By default the system uses THP. To make huge pages available on your system, activate it at boot time with hugepages=1, and—optionally—add the huge pages size with, for example, hugepagesz=2MB.

Note: 1 GB huge pages

1 GB pages can only be allocated at boot time and cannot be freed afterward.

To allocate and use the huge page table (HugeTlbPage) you need to mount hugetlbfs with correct permissions.

Note: Restrictions of huge pages

Even if huge pages provide the best performance, they do come with some drawbacks. You lose features such as Memory ballooning (see Section 6.1.3, “virtio balloon”), KSM (see Section 4.1.4, “KSM and page sharing”), and huge pages cannot be swapped.

Procedure 1: Configuring the use of huge pages #

  

1. Mount hugetlbfs to /dev/hugepages:

   > sudo mount -t hugetlbfs hugetlbfs /dev/hugepages

2. To reserve memory for huge pages use the sysctl command. If your system has a huge page size of 2 MB (2048 KB), and you want to reserve 1 GB (1,048,576 KB) for your VM Guest, you need 1,048,576/2048=512 pages in the pool:

   > sudo sysctl vm.nr_hugepages=512

   The value is written to /proc/sys/vm/nr_hugepages and represents the current number of persistent huge pages in the kernel's huge page pool. Persistent huge pages will be returned to the huge page pool when freed by a task.

3. Add the memoryBacking element in the VM Guest configuration file (by running virsh edit CONFIGURATION).

   <memoryBacking>
     <hugepages/>
   </memoryBacking>

4. Start your VM Guest and check on the host whether it uses hugepages:

   > cat /proc/meminfo | grep HugePages_
   HugePages_Total:1     512
   HugePages_Free:2       92
   HugePages_Rsvd:3        0
   HugePages_Surp:4        0

   1	Size of the pool of huge pages
   2	Number of huge pages in the pool that are not yet allocated
   3	Number of huge pages for which a commitment to allocate from the pool has been made, but no allocation has yet been made
   4	Number of huge pages in the pool above the value in /proc/sys/vm/nr_hugepages. The maximum number of surplus huge pages is controlled by /proc/sys/vm/nr_overcommit_hugepages

4.1.2 Transparent huge pages #

  

Transparent huge pages (THP) provide a way to dynamically allocate huge pages with the khugepaged kernel thread, rather than manually managing their allocation and use. Workloads with contiguous memory access patterns can benefit greatly from THP. A 1000 fold decrease in page faults can be observed when running synthetic workloads with contiguous memory access patterns. Conversely, workloads with sparse memory access patterns (like databases) may perform poorly with THP. In such cases it may be preferable to disable THP by adding the kernel parameter transparent_hugepage=never, rebuild your grub2 configuration, and reboot. Verify if THP is disabled with:

> cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

If disabled, the value never is shown in square brackets like in the example above.

Note: Xen

THP is not available under Xen.

4.1.4 KSM and page sharing #

  

Kernel Samepage Merging is a kernel feature that reduces memory consumption on the VM Host Server by sharing blocks of memory that VM Guests have in common. The KSM daemon ksmd periodically scans user memory looking for pages with identical contents, which can be replaced by a single write-protected page. To enable the KSM service, first make sure that the package qemu-ksm is installed, then run the command:

> sudo systemctl enable --now ksm.service

Alternatively, it can also be started by running the command:

#  echo 1 > /sys/kernel/mm/ksm/run

One advantage of using KSM from a VM Guest's perspective is that all guest memory is backed by host anonymous memory. You can share pagecache, tmpfs or any kind of memory allocated in the guest.

KSM is controlled by sysfs. You can check KSM's values in /sys/kernel/mm/ksm/:

• pages_shared: The number of shared pages that are being used (read-only).

• pages_sharing: The number of sites sharing the pages (read-only).

• pages_unshared: The number of pages that are unique and repeatedly checked for merging (read-only).

• pages_volatile: The number of pages that are changing too fast to be considered for merging (read-only).

• full_scans: The number of times all mergeable areas have been scanned (read-only).

• sleep_millisecs: The number of milliseconds ksmd should sleep before the next scan. A low value will overuse the CPU, consuming CPU time that could be used for other tasks. We recommend a value greater than 1000.

• pages_to_scan: The number of present pages to scan before ksmd goes to sleep. A high value will overuse the CPU. We recommend to start with a value of 1000, and then adjust as necessary based on the KSM results observed while testing your deployment.

• merge_across_nodes: By default the system merges pages across NUMA nodes. Set this option to 0 to disable this behavior.

Note: Use cases

KSM is a good technique to over-commit host memory when running multiple instances of the same application or VM Guest. When applications and VM Guest are heterogeneous and do not share any common data, it is preferable to disable KSM. In order to do that, run:

> sudo systemctl disable --now ksm.service

Alternatively, it can also be disabled by running the command:

#  echo 0 > /sys/kernel/mm/ksm/run

In a mixed heterogeneous and homogeneous environment, KSM can be enabled on the host but disabled on a per VM Guest basis. Use virsh edit to disable page sharing of a VM Guest by adding the following to the guest's XML configuration:

<memoryBacking>
   <nosharepages/>
</memoryBacking>

Warning: Avoid out-of-memory conditions

KSM can free up some memory on the host system, but the administrator should reserve enough swap to avoid out-of-memory conditions if that shareable memory decreases. If the amount of shareable memory decreases, the use of physical memory is increased.

Warning: Memory access latencies

By default, KSM will merge common pages across NUMA nodes. If the merged, common page is now located on a distant NUMA node (relative to the node running the VM Guest vCPUs), this may degrade VM Guest performance. If increased memory access latencies are noticed in the VM Guest, disable cross-node merging with the merge_across_nodes sysfs control:

#  echo 0 > /sys/kernel/mm/ksm/merge_across_nodes

<maxMemory1 slots='16'2 unit='KiB'>209715203</maxMemory>
  <memory4 unit='KiB'>1048576</memory>
<currentMemory5 unit='KiB'>1048576</currentMemory>

<memory model='dimm'>
  <target>
  <size unit='KiB'>524287</size>
  <node>0</node>
  </target>
</memory>

> virsh attach-device vm-name mem-dev.xml

Swap is usually used by the system to store underused physical memory (low usage, or not accessed for a long time). To prevent the system running out of memory, setting up a minimum swap is highly recommended.

#  echo 35 > /proc/sys/vm/swappiness

> free -h
total       used       free     shared    buffers     cached
Mem:      24616680    4991492   19625188     167056     144340    2152408
-/+ buffers/cache:    2694744   21921936
Swap:      6171644          0    6171644

vm.swappiness = 35

<memtune>1
  <hard_limit unit='G'>1</hard_limit>2
  <soft_limit unit='M'>128</soft_limit>3
  <swap_hard_limit unit='G'>2</swap_hard_limit>4
</memtune>

Storage space for VM Guests can either be a block device (for example, a partition on a physical disk), or an image file on the file system:

For detailed information about image formats and maintaining images refer to Section 5, “VM Guest images”.

If your image is stored on an NFS share, you should check some server and client parameters to improve access to the VM Guest image.

nfs_server:/exported/vm_images1 /mnt/images2 nfs3 rw4,hard5,sync6, rsize=81927,wsize=81928 0 0

> sudo nfsstat -rc
Client rpc stats:
calls      retrans    authrefrsh
6401066    198          0          0

Host CPU “components” will be “translated” to virtual CPUs in a VM Guest when being assigned. These components can either be:

4.5.2 VM Guest CPU configuration #

  

This section describes how to choose and configure a CPU type for a VM Guest. You will also learn how to pin virtual CPUs to physical CPUs on the host system. For more information about virtual CPU configuration and tuning parameters refer to the libvirt documentation at https://libvirt.org/formatdomain.html#elementsCPU.

4.5.2.1 Virtual CPU models and features #

  

The CPU model and topology can be specified individually for each VM Guest. Configuration options range from selecting specific CPU models to excluding certain CPU features. Predefined CPU models are listed in files in the directory /usr/share/libvirt/cpu_map/. A CPU model and topology that is similar to the host generally provides the best performance. The host system CPU model and topology can be displayed by running virsh capabilities.

Note that changing the default virtual CPU configuration will require a VM Guest shutdown when migrating it to a host with different hardware. More information on VM Guest migration is available at Book “Virtualization Guide”, Chapter 11 “Basic VM Guest management”, Section 11.7 “Migrating VM Guests”.

To specify a particular CPU model for a VM Guest, add a respective entry to the VM Guest configuration file. The following example configures a Broadwell CPU with the invariant TSC feature:

<cpu mode='custom' match='exact'>
  <model>Broadwell</model>
  <feature name='invtsc'/>
  </cpu>

For a virtual CPU that most closely resembles the host physical CPU, <cpu mode='host-passthrough'> can be used. Note that a host-passthrough CPU model may not exactly resemble the host physical CPU, since by default KVM will mask any non-migratable features. For example invtsc is not included in the virtual CPU feature set. Changing the default KVM behavior is not directly supported through libvirt, although it does allow arbitrary pass-through of KVM command line arguments. Continuing with the invtsc example, you can achieve pass-through of the host CPU (including invtsc) with the following command line pass-through in the VM Guest configuration file:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
     <qemu:commandline>
     <qemu:arg value='-cpu'/>
     <qemu:arg value='host,migratable=off,+invtsc'/>
     </qemu:commandline>
     ...
     </domain>

Note: The host-passthrough mode

Since host-passthrough exposes the physical CPU details to the virtual CPU, migration to dissimilar hardware is not possible. See Section 4.5.2.3, “Virtual CPU migration considerations” for more information.

4.5.2.2 Virtual CPU pinning #

  

Virtual CPU pinning is used to constrain virtual CPU threads to a set of physical CPUs. The vcpupin element specifies the physical host CPUs that a virtual CPU can use. If this element is not set and the attribute cpuset of the vcpu element is not specified, the virtual CPU is free to use any of the physical CPUs.

CPU intensive workloads can benefit from virtual CPU pinning by increasing the physical CPU cache hit ratio. To pin a virtual CPU to a specific physical CPU, run the following commands:

> virsh vcpupin DOMAIN_ID --vcpu vCPU_NUMBER
VCPU: CPU Affinity
----------------------------------
0: 0-7
# virsh vcpupin SLE15 --vcpu 0 0 --config

The last command generates the following entry in the XML configuration:

<cputune>
   <vcpupin vcpu='0' cpuset='0'/>
</cputune>

Note: Virtual CPU pinning on NUMA nodes

To confine a VM Guest's CPUs and its memory to a NUMA node, you can use virtual CPU pinning and memory allocation policies on a NUMA system. See Section 4.6, “NUMA tuning” for more information related to NUMA tuning.

Warning: Virtual CPU pinning and live migration

Even though vcpupin can improve performance, it can complicate live migration. See Section 4.5.2.3, “Virtual CPU migration considerations” for more information on virtual CPU migration considerations.

4.5.2.3 Virtual CPU migration considerations #

  

Selecting a virtual CPU model containing all the latest features may improve performance of a VM Guest workload, but often at the expense of migratability. Unless all hosts in the cluster contain the latest CPU features, migration can fail when a destination host lacks the new features. If migratability of a virtual CPU is preferred over the latest CPU features, a normalized CPU model and feature set should be used. The virsh cpu-baseline command can help define a normalized virtual CPU that can be migrated across all hosts. The following command, when run on each host in the migration cluster, illustrates collection of all hosts' capabilities in all-hosts-caps.xml.

> sudo virsh capabilities >> all-hosts-cpu-caps.xml

With the capabilities from each host collected in all-hosts-caps.xml, use virsh cpu-baseline to create a virtual CPU definition that will be compatible across all hosts.

> sudo virsh cpu-baseline all-hosts-caps.xml

The resulting virtual CPU definition can be used as the cpu element in VM Guest configuration file.

At a logical level, virtual CPU pinning is a form of hardware pass-through. Pinning couples physical resources to virtual resources, and can also be problematic for migration. For example, the migration will fail if the requested physical resources are not available on the destination host, or if the source and destination hosts have different NUMA topologies. For more recommendations about Live Migration see Book “Virtualization Guide”, Chapter 11 “Basic VM Guest management”, Section 11.7.1 “Migration requirements”.

NUMA is an acronym for Non Uniform Memory Access. A NUMA system has multiple physical CPUs, each with local memory attached. Each CPU can also access other CPUs' memory, known as “remote memory access”, but it is much slower than accessing local memory. NUMA systems can negatively impact VM Guest performance if not tuned properly. Although ultimately tuning is workload dependent, this section describes controls that should be considered when deploying VM Guests on NUMA hosts. Always consider your host topology when configuring and deploying VMs.

SUSE Linux Enterprise Server contains a NUMA auto-balancer that strives to reduce remote memory access by placing memory on the same NUMA node as the CPU processing it. In addition, standard tools such as cgset and virtualization tools such as libvirt provide mechanisms to constrain VM Guest resources to physical resources.

numactl is used to check for host NUMA capabilities:

> sudo numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 72 73 74 75 76 77 78
79 80 81 82 83 84 85 86 87 88 89
node 0 size: 31975 MB
node 0 free: 31120 MB
node 1 cpus: 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 90 91 92 93
94 95 96 97 98 99 100 101 102 103 104 105 106 107
node 1 size: 32316 MB
node 1 free: 31673 MB
node 2 cpus: 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 108 109 110
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
node 2 size: 32316 MB
node 2 free: 31726 MB
node 3 cpus: 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 126 127 128
129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
node 3 size: 32314 MB
node 3 free: 31387 MB
node distances:
node   0   1   2   3
0:  10  21  21  21
1:  21  10  21  21
2:  21  21  10  21
3:  21  21  21  10

The numactl output shows this is a NUMA system with 4 nodes or cells, each containing 36 CPUs and approximately 32G memory. virsh capabilities can also be used to examine the systems NUMA capabilities and CPU topology.

4.6.1 NUMA balancing #

  

On NUMA machines, there is a performance penalty if remote memory is accessed by a CPU. Automatic NUMA balancing scans a task's address space and unmaps pages. By doing so, it detects whether pages are properly placed or whether to migrate the data to a memory node local to where the task is running. In defined intervals (configured with numa_balancing_scan_delay_ms), the task scans the next scan size number of pages (configured with numa_balancing_scan_size_mb) in its address space. When the end of the address space is reached the scanner restarts from the beginning.

Higher scan rates cause higher system overhead as page faults must be trapped and data needs to be migrated. However, the higher the scan rate, the more quickly a task's memory is migrated to a local node when the workload pattern changes. This minimizes the performance impact caused by remote memory accesses. These sysctl directives control the thresholds for scan delays and the number of pages scanned:

> sudo sysctl -a | grep numa_balancing
kernel.numa_balancing = 11
kernel.numa_balancing_scan_delay_ms = 10002
kernel.numa_balancing_scan_period_max_ms = 600003
kernel.numa_balancing_scan_period_min_ms = 10004
kernel.numa_balancing_scan_size_mb = 2565

1	Enables/disables automatic page fault-based NUMA balancing
2	Starting scan delay used for a task when it initially forks
3	Maximum time in milliseconds to scan a task's virtual memory
4	Minimum time in milliseconds to scan a task's virtual memory
5	Size in megabytes' worth of pages to be scanned for a given scan

For more information see Book “System Analysis and Tuning Guide”, Chapter 11 “Automatic Non-Uniform Memory Access (NUMA) balancing”.

The main goal of automatic NUMA balancing is either to reschedule tasks on the same node's memory (so the CPU follows the memory), or to copy the memory's pages to the same node (so the memory follows the CPU).

Warning: Task placement

There are no rules to define the best place to run a task, because tasks could share memory with other tasks. For best performance, it is recommended to group tasks sharing memory on the same node. Check NUMA statistics with # cat /proc/vmstat | grep numa_.

4.6.2 Memory allocation control with the CPUset controller #

  

The cgroups cpuset controller can be used confine memory used by a process to a NUMA node. There are three cpuset memory policy modes available:

• interleave: This is a memory placement policy which is also known as round-robin. This policy can provide substantial improvements for jobs that need to place thread local data on the corresponding node. When the interleave destination is not available, it will be moved to another node.

• bind: This will place memory only on one node, which means in case of insufficient memory, the allocation will fail.

• preferred: This policy will apply a preference to allocate memory to a node. If there is not enough space for memory on this node, it will fall back to another node.

You can change the memory policy mode with the cgset tool from the libcgroup-tools package:

> sudo cgset -r cpuset.mems=NODE sysdefault/libvirt/qemu/KVM_NAME/emulator

To migrate pages to a node, use the migratepages tool:

> migratepages PID FROM-NODE TO-NODE

To check everything is fine. use: cat /proc/PID/status | grep Cpus.

Note: Kernel NUMA/cpuset memory policy

For more information see Kernel NUMA memory policy and cpusets memory policy. Check also the Libvirt NUMA Tuning documentation.

4.6.3 VM Guest: NUMA related configuration #

  

libvirt allows to set up virtual NUMA and memory access policies. Configuring these settings is not supported by virt-install or virt-manager and needs to be done manually by editing the VM Guest configuration file with virsh edit.

4.6.3.1 VM Guest virtual NUMA topology #

  

Creating a VM Guest virtual NUMA (vNUMA) policy that resembles the host NUMA topology can often increase performance of traditional large, scale-up workloads. VM Guest vNUMA topology can be specified using the numa element in the XML configuration:

<cpu>
...
  <numa>
    <cell1 id="0"2 cpus='0-1'3 memory='512000' unit='KiB'/>
    <cell id="1" cpus='2-3' memory='256000'4
    unit='KiB'5 memAccess='shared'6/>
  </numa>
  ...
</cpu>

1	Each cell element specifies a vNUMA cell or node
2	All cells should have an id attribute, allowing to reference the cell in other configuration blocks. Otherwise cells are assigned ids in ascending order starting from 0.
3	The CPU or range of CPUs that are part of the node
4	The node memory
5	Units in which node memory is specified
6	Optional attribute which can control whether the memory is to be mapped as shared or private. This is valid only for hugepages-backed memory.

To find where the VM Guest has allocated its pages. use: cat /proc/PID/numa_maps and cat /sys/fs/cgroup/memory/sysdefault/libvirt/qemu/KVM_NAME/memory.numa_stat.

Warning: NUMA specification

The libvirt VM Guest NUMA specification is currently only available for QEMU/KVM.

4.6.3.2 Memory allocation control with libvirt #

  

If the VM Guest has a vNUMA topology (see Section 4.6.3.1, “VM Guest virtual NUMA topology”), memory can be pinned to host NUMA nodes using the numatune element. This method is currently only available for QEMU/KVM guests. See Important: Non-vNUMA VM Guest for how to configure non-vNUMA VM Guests.

<numatune>
    <memory mode="strict"1 nodeset="1-4,^3"2/>
    <memnode3 cellid="0"4 mode="strict" nodeset="1"/>
    <memnode cellid="2" placement="strict"5 mode="preferred" nodeset="2"/>
</numatune>

1	Policies available are: interleave (round-robin like), strict (default) or preferred.
2	Specify the NUMA nodes.
3	Specify memory allocation policies for each guest NUMA node (if this element is not defined then this will fall back and use the memory element).
4	Addresses the guest NUMA node for which the settings are applied.
5	The placement attribute can be used to indicate the memory placement mode for a domain process, the value can be auto or strict.

Important: Non-vNUMA VM Guest

On a non-vNUMA VM Guest, pinning memory to host NUMA nodes is done like in the following example:

<numatune>
   <memory mode="strict" nodeset="0-1"/>
</numatune>

In this example, memory is allocated from the host nodes 0 and 1. In case these memory requirements cannot be fulfilled, starting the VM Guest will fail. virt-install also supports this configuration with the --numatune option.

Warning: Memory and CPU across NUMA nodes

You should avoid allocating VM Guest memory across NUMA nodes, and prevent virtual CPUs from floating across NUMA nodes.

Certain storage formats which QEMU recognizes have their origins in other virtualization technologies. By recognizing these formats, QEMU can leverage either data stores or entire guests that were originally targeted to run under these other virtualization technologies. Some formats are supported only in read-only mode. To use them in read/write mode, convert them to a fully supported QEMU storage format (using qemu-img). Otherwise they can only be used as read-only data store in a QEMU guest.

Use qemu-img info VMGUEST.IMG to get information about an existing image, such as: the format, the virtual size, the physical size, snapshots if available.

Note: Performance

It is recommended to convert the disk images to either raw or qcow2 to achieve good performance.

Warning: Encrypted images cannot be compressed

When you create an image, you cannot use compression (-c) in the output file together with the encryption option (-e).

The qcow2 and qed formats provide a way to create a base image (also called backing file) and overlay images on top of the base image. A backing file is useful to be able to revert to a known state and discard the overlay. If you write to the image, the backing image will be untouched and all changes will be recorded in the overlay image file. The backing file will never be modified unless you use the commit monitor command (or qemu-img commit).

To create an overlay image:

# qemu-img create -o1backing_file=vmguest.raw2,backing_fmt=raw3\
     -f4 qcow2 vmguest.cow5

Warning: Backing image path

You should not change the path to the backing image, otherwise you will need to adjust it. The path is stored in the overlay image file. To update the path, you should make a symbolic link from the original path to the new path and then use the qemu-img rebase option.

# ln -sf /var/lib/images/OLD_PATH/vmguest.raw  \
 /var/lib/images/NEW_PATH/vmguest.raw
# qemu-img rebase1 -u2 -b3 \
 /var/lib/images/OLD_PATH/vmguest.raw /var/lib/images/NEW_PATH/vmguest.cow

1	The rebase subcommand tells qemu-img to change the backing file image.
2	The -u option activates the unsafe mode (see note below). There are two different modes in which rebase can operate: Safe: This is the default mode and performs a real rebase operation. The safe mode is a time-consuming operation. Unsafe: The unsafe mode (-u) only changes the backing files name and the format of the file name without making any checks on the files contents. You should use this mode to rename or moving a backing file.
3	The backing image to be used is specified with -b and the image path is the last argument of the command.

A common use is to initiate a new guest with the backing file. Let's assume we have a sle15_base.img VM Guest ready to be used (fresh installation without any modification). This will be our backing file. Now you need to test a new package, on an updated system and on a system with a different kernel. We can use sle15_base.img to instantiate the new SUSE Linux Enterprise VM Guest by creating a qcow2 overlay file pointing to this backing file (sle15_base.img).

In our example we will use sle15_updated.qcow2 for the updated system, and sle15_kernel.qcow2 for the system with a different kernel.

To create the two thin provisioned systems use the qemu-img command line with the -b option:

# qemu-img create -b /var/lib/libvirt/sle15_base.img -f qcow2 \
/var/lib/libvirt/sle15_updated.qcow2
Formatting 'sle15_updated.qcow2', fmt=qcow2 size=17179869184
backing_file='sle15_base.img' encryption=off cluster_size=65536
lazy_refcounts=off nocow=off
# qemu-img create -b /var/lib/libvirt/sle15_base.img -f qcow2 \
/var/lib/libvirt/sle15_kernel.qcow2
Formatting 'sle15_kernel.qcow2', fmt=qcow2 size=17179869184
backing_file='vmguest-sle15_base.img' encryption=off cluster_size=65536
lazy_refcounts=off nocow=off

The images are now usable, and you can do your test without touching the initial sle15_base.img backing file. All changes will be stored in the new overlay images. Additionally, you can also use these new images as a backing file, and create a new overlay.

# qemu-img create -b sle15_kernel.qcow2 -f qcow2 sle15_kernel_TEST.qcow2

When using qemu-img info with the option --backing-chain, it will return all information about the entire backing chain recursively:

# qemu-img info --backing-chain
/var/lib/libvirt/images/sle15_kernel_TEST.qcow2
image: sle15_kernel_TEST.qcow2
file format: qcow2
virtual size: 16G (17179869184 bytes)
disk size: 196K
cluster_size: 65536
backing file: sle15_kernel.qcow2
Format specific information:
compat: 1.1
lazy refcounts: false

image: sle15_kernel.qcow2
file format: qcow2
virtual size: 16G (17179869184 bytes)
disk size: 196K
cluster_size: 65536
backing file: SLE15.qcow2
Format specific information:
compat: 1.1
lazy refcounts: false

image: sle15_base.img
file format: qcow2
virtual size: 16G (17179869184 bytes)
disk size: 16G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: true

Figure 1: Understanding image overlay #

  

To access the file system of an image, use the guestfs-tools. If you do not have this tool installed on your system you can mount an image with other Linux tools. Avoid accessing an untrusted or unknown VM Guest's image system because this can lead to security issues (for more information, read D. Berrangé's post).

5.3.1 Opening a raw image #

  

Procedure 4: Mounting a raw image #

  

1. To be able to mount the image, find a free loop device. The following command displays the first unused loop device, /dev/loop1 in this example.

   # losetup -f
   /dev/loop1

2. Associate an image (SLE15.raw in this example) with the loop device:

   # losetup /dev/loop1 SLE15.raw

3. Check whether the image has successfully been associated with the loop device by getting detailed information about the loop device:

   # losetup -l
   NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
   /dev/loop1         0      0         0  0    /var/lib/libvirt/images/SLE15.raw

4. Check the image's partitions with kpartx:

   # kpartx -a1 -v2 /dev/loop1
   add map loop1p1 (254:1): 0 29358080 linear /dev/loop1 2048

   1	Add partition device mappings.
   2	Verbose mode.

5. Now mount the image partition(s) (to /mnt/sle15mount in the following example):

   # mkdir /mnt/sle15mount
   # mount /dev/mapper/loop1p1 /mnt/sle15mount

Note: Raw image with LVM

If your raw image contains an LVM volume group you should use LVM tools to mount the partition. Refer to Section 5.3.3, “Opening images containing LVM”.

Procedure 5: Unmounting a raw image #

  

1. Unmount all mounted partitions of the image, for example:

   # umount /mnt/sle15mount

2. Delete partition device mappings with kpartx:

   # kpartx -d /dev/loop1

3. Detach the devices with losetup

   # losetup -d /dev/loop1

5.3.3 Opening images containing LVM #

  

Warning

If your VM Host Server uses VG name system, and the guest image also uses VG name system, LVM will complain during its activation. A workaround is to temporarily rename the guest VG, while a correct approach is to use different VG names for the guests than for the VM Host Server.

Procedure 8: Mounting images containing LVM #

  

1. To check images for LVM groups, use vgscan -v. If an image contains LVM groups, the output of the command looks like the following:

   # vgscan -v
   Wiping cache of LVM-capable devices
   Wiping internal VG cache
   Reading all physical volumes.  This may take a while...
   Finding all volume groups
   Finding volume group "system"
   Found volume group "system" using metadata type lvm2

2. The system LVM volume group has been found on the system. You can get more information about this volume with vgdisplay VOLUMEGROUPNAME (in our case VOLUMEGROUPNAME is system). You should activate this volume group to expose LVM partitions as devices so the system can mount them. Use vgchange:

   # vgchange -ay -v
   Finding all volume groups
   Finding volume group "system"
   Found volume group "system"
   activation/volume_list configuration setting not defined: Checking only
   host tags for system/home
   Creating system-home
   Loading system-home table (254:0)
   Resuming system-home (254:0)
   Found volume group "system"
   activation/volume_list configuration setting not defined: Checking only
   host tags for system/root
   Creating system-root
   Loading system-root table (254:1)
   Resuming system-root (254:1)
   Found volume group "system"
   activation/volume_list configuration setting not defined: Checking only
   host tags for system/swap
   Creating system-swap
   Loading system-swap table (254:2)
   Resuming system-swap (254:2)
   Activated 3 logical volumes in volume group system
       3 logical volume(s) in volume group "system" now active

3. All partitions in the volume group will be listed in the /dev/mapper directory. You can simply mount them now.

   # ls /dev/mapper/system-*
   /dev/mapper/system-home  /dev/mapper/system-root /dev/mapper/system-swap
   
   # mkdir /mnt/system-root
   # mount  /dev/mapper/system-root /mnt/system-root
   
   # ls /mnt/system-root/
   bin   dev  home  lib64       mnt  proc        root  sbin     srv  tmp  var
   boot  etc  lib   lost+found  opt  read-write  run   selinux  sys  usr

Procedure 9: Unmounting images containing LVM #

  

1. Unmount all partitions (with umount)

   # umount /mnt/system-root

2. Deactivate the LVM volume group (with vgchange -an VOLUMEGROUPNAME)

   # vgchange -an -v system
   Using volume group(s) on command line
   Finding volume group "system"
   Found volume group "system"
   Removing system-home (254:0)
   Found volume group "system"
   Removing system-root (254:1)
   Found volume group "system"
   Removing system-swap (254:2)
   Deactivated 3 logical volumes in volume group system
   0 logical volume(s) in volume group "system" now active

3. Now you have two choices:

   • In case of a qcow2 image, proceed as described in Step 2 (qemu-nbd -d /dev/nbd0).

   • In case of a raw image, proceeds as described in Step 2 (kpartx -d /dev/loop1; losetup -d /dev/loop1).

   

   Important: Check for a successful unmount

   You should double-check that unmounting succeeded by using a system command like losetup, qemu-nbd, mount or vgscan. If this is not the case you may have trouble using the VM Guest because its system image is used in different places.

You can access a host directory in the VM Guest using the filesystem element. In the following example we will share the /data/shared directory and mount it in the VM Guest. Note that the accessmode parameter only works with type='mount' for the QEMU/KVM driver (most other values for type are exclusively used for the LXC driver).

<filesystem type='mount'1 accessmode='mapped'2>
   <source dir='/data/shared'3>
   <target dir='shared'4/>
</filesystem>

To mount the shared directory on the VM Guest, use the following commands: Under the VM Guest now you need to mount the target dir='shared':

# mkdir /opt/mnt_shared
# mount shared -t 9p /opt/mnt_shared -o trans=virtio

See libvirt File System and QEMU 9psetup for more information.

To increase VM Guest performance it is recommended to use paravirtualized drivers within the VM Guests. The virtualization standard for such drivers for KVM are the virtio drivers, which are designed for running in a virtual environment. Xen uses similar paravirtualized device drivers (like VMDP in a Windows* guest).

6.1.1 virtio blk #

  

virtio_blk is the virtio block device for disk. To use the virtio blk driver for a block device, specify the bus='virtio' attribute in the disk definition:

<disk type='....' device='disk'>
    ....
    <target dev='vda' bus='virtio'/>
</disk>

Important: Disk device names

virtio disk devices are named /dev/vd[a-z][1-9]. If you migrate a Linux guest from a non-virtio disk you need to adjust the root= parameter in the GRUB configuration, and regenerate the initrd file. Otherwise the system cannot boot. On VM Guests with other operating systems, the boot loader may need to be adjusted or reinstalled accordingly, too.

Important: Using virtio disks with qemu-system-ARCH

When running qemu-system-ARCH, use the -drive option to add a disk to the VM Guest. See Book “Virtualization Guide”, Chapter 34 “Guest installation”, Section 34.1 “Basic installation with qemu-system-ARCH” for an example. The -hd[abcd] option will not work for virtio disks.

<interface type='network'>
    ...
    <model type='virtio' />
</interface>

<memory unit='KiB'>16777216</memory>
    <currentMemory unit='KiB'>1048576</currentMemory>
    [...]
    <devices>
        <memballoon model='virtio'/>
    </devices>

> virsh setmem DOMAIN_ID MEMORY in KB

> find /sys/devices/ -name virtio*
/sys/devices/pci0000:00/0000:00:06.0/virtio0
/sys/devices/pci0000:00/0000:00:07.0/virtio1
/sys/devices/pci0000:00/0000:00:08.0/virtio2

> find /sys/devices/ -name virtio* -print  -exec ls {}/block 2>/dev/null \;
/sys/devices/pci0000:00/0000:00:06.0/virtio0
/sys/devices/pci0000:00/0000:00:07.0/virtio1
/sys/devices/pci0000:00/0000:00:08.0/virtio2
vda

> udevadm info -p /sys/devices/pci0000:00/0000:00:08.0/virtio2
P: /devices/pci0000:00/0000:00:08.0/virtio2
E: DEVPATH=/devices/pci0000:00/0000:00:08.0/virtio2
E: DRIVER=virtio_blk
E: MODALIAS=virtio:d00000002v00001AF4
E: SUBSYSTEM=virtio

> find /sys/devices/ -name virtio* -print  -exec ls -l {}/driver 2>/dev/null \;
/sys/devices/pci0000:00/0000:00:06.0/virtio0
lrwxrwxrwx 1 root root 0 Jun 17 15:48 /sys/devices/pci0000:00/0000:00:06.0/virtio0/driver -> ../../../../bus/virtio/drivers/virtio_console
/sys/devices/pci0000:00/0000:00:07.0/virtio1
lrwxrwxrwx 1 root root 0 Jun 17 15:47 /sys/devices/pci0000:00/0000:00:07.0/virtio1/driver -> ../../../../bus/virtio/drivers/virtio_balloon
/sys/devices/pci0000:00/0000:00:08.0/virtio2
lrwxrwxrwx 1 root root 0 Jun 17 14:35 /sys/devices/pci0000:00/0000:00:08.0/virtio2/driver -> ../../../../bus/virtio/drivers/virtio_blk

> qemu-system-x86_64 -device virtio-net,help
virtio-net-pci.ioeventfd=on/off
virtio-net-pci.vectors=uint32
virtio-net-pci.indirect_desc=on/off
virtio-net-pci.event_idx=on/off
virtio-net-pci.any_layout=on/off
.....

To get 16-bit color, high compatibility and better performance it is recommended to use the cirrus video driver.

Note: libvirt

libvirt ignores the vram value because video size has been hardcoded in QEMU.

<video>
   <model type='cirrus' vram='9216' heads='1'/>
</video>

Virtio RNG (random number generator) is a paravirtualized device that is exposed as a hardware RNG device to the guest. On the host side, it can be wired up to one of several sources of entropy (including a real hardware RNG device and the host's /dev/random) if hardware support does not exist. The Linux kernel contains the guest driver for the device from version 2.6.26 and higher.

The system entropy is collected from various non-deterministic hardware events and is mainly used by cryptographic applications. The virtual random number generator device (paravirtualized device) allows the host to pass through entropy to VM Guest operating systems. This results in a better entropy in the VM Guest.

To use Virtio RNG, add an RNG device in virt-manager or directly in the VM Guest's XML configuration:

<devices>
   <rng model='virtio'>
       <backend model='random'>/dev/random</backend>
   </rng>
</devices>

The host now should used /dev/random:

> lsof /dev/random
qemu-syst 4926 qemu    6r   CHR    1,8      0t0 8199 /dev/random

On the VM Guest, the source of entropy can be checked with:

> cat /sys/devices/virtual/misc/hw_random/rng_available

The current device used for entropy can be checked with:

> cat /sys/devices/virtual/misc/hw_random/rng_current
virtio_rng.0

You should install the rng-tools package on the VM Guest, enable the service, and start it. Under SUSE Linux Enterprise Server 15, do the following:

# zypper in rng-tools
# systemctl enable rng-tools
# systemctl start rng-tools

Per host, use one virtualization technology only. For example, do not use KVM and Xen on the same host. Otherwise, you may find yourself with a reduced amount of available resources, increased security risk and a longer software update queue. Even when the amount of resources allocated to each of the technologies is configured carefully, the host may suffer from reduced overall availability and degraded performance.

Minimize the amount of software and services available on hosts. Most default installations of operating systems are not optimized for VM usage. Install what you really need and remove all other components in the VM Guest.

Windows* Guest:

Linux Guest:

QEMU machine types define details of the architecture that are particularly relevant for migration and session management. As changes or improvements to QEMU are made, new machine types are added. Old machine types are still supported for compatibility reasons, but to take advantage of improvements, we recommend to always migrate to the latest machine type when upgrading.

Changing the guest's machine type for a Linux guest will mostly be transparent. For Windows* guests, we recommend to take a snapshot or backup of the guest—in case Windows* has issues with the changes it detects and subsequently the user decides to revert to the original machine type the guest was created with.

Note: Changing the machine type

Refer to Book “Virtualization Guide”, Chapter 15 “Configuring virtual machines with virsh”, Section 15.2 “Changing the machine type” for documentation.

The ability to change a VM Guest's state heavily depends on the operating system. It is very important to test this feature before any use of your VM Guests in production. For example, most Linux operating systems disable this capability by default, so this requires you to enable this operation (mostly through Polkit).

ACPI must be enabled in the guest for a graceful shutdown to work. To check if ACPI is enabled, run:

> virsh dumpxml VMNAME | grep acpi

If nothing is printed, ACPI is not enabled for your machine. Use virsh edit to add the following XML under <domain>:

<features>
   <acpi/>
</features>

If ACPI was enabled during a Windows Server* guest installation, it is not sufficient to turn it on in the VM Guest configuration only. For more information, see https://support.microsoft.com/en-us/kb/309283.

Regardless of the VM Guest's configuration, a graceful shutdown is always possible from within the guest operating system.

Though it is possible to specify the keyboard layout from a qemu-system-ARCH command, it is recommended to configure it in the libvirt XML file. To change the keyboard layout while connecting to a remote VM Guest using vnc, you should edit the VM Guest XML configuration file. For example, to add an en-us keymap, add in the <devices> section:

<graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>

Check the vncdisplay configuration and connect to your VM Guest:

> virsh vncdisplay sles15 127.0.0.1:0

If no network interface other than lo is assigned an IPv4 address on the host, the default address on which the spice server listens will not work. An error like the following one will occur:

> virsh start sles15
error: Failed to start domain sles15
error: internal error: process exited while connecting to monitor: ((null):26929): Spice-Warning **: reds.c:2330:reds_init_socket: getaddrinfo(127.0.0.1,5900): Address family for hostname not supported
2015-08-12T11:21:14.221634Z qemu-system-x86_64: failed to initialize spice server

To fix this, you can change the default spice_listen value in /etc/libvirt/qemu.conf using the local IPv6 address ::1. The spice server listening address can also be changed on a per VM Guest basis, use virsh edit to add the listen XML attribute to the graphics type='spice' element:

<graphics type='spice' listen='::1' autoport='yes'/>>

Sometimes it could be useful to get the QEMU command line to launch the VM Guest from the XML file.

> virsh domxml-to-native1 qemu-argv2 SLE15.xml3

> sudo virsh domxml-to-native qemu-argv /etc/libvirt/qemu/SLE15.xml
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
   QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name SLE15 -machine \
   pc-i440fx-2.3,accel=kvm,usb=off -cpu SandyBridge -m 4048 -realtime \
   mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 8616d00f-5f05-4244-97cc-86aeaed8aea7 \
   -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/SLE15.monitor,server,nowait \
   -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew \
   -global kvm-pit.lost_tick_policy=discard -no-hpet \
   -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 \
   -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 \
   -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 \
   -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 \
   -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 \
   -drive file=/var/lib/libvirt/images/SLE15.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none \
   -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 \
   -drive if=none,id=drive-ide0-0-1,readonly=on,format=raw  \
   -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,id=hostnet0  \
   -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:28:04:a9,bus=pci.0,addr=0x3,bootindex=1 \
   -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 \
   -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
   -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

To create a new VM Guest based on an XML file, you can specify the QEMU command line using the special tag qemu:commandline. For example, to add a virtio-balloon-pci, add this block at the end of the XML configuration file (before the </domain> tag):

<qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='virtio-balloon-pci,id=balloon0'/>
</qemu:commandline>

Some virtualization environments allow adding or removing CPUs while the virtual machine is running.

For the safe removal of CPUs, deactivate them first by executing

# echo 0 > /sys/devices/system/cpu/cpuX/online

Replace X with the CPU number. To bring a CPU back online, execute

# echo 1 > /sys/devices/system/cpu/cpuX/online