Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / SUSE Linux Enterprise Server Documentation / Virtualization Guide / Managing Virtual Machines with libvirt / Configuring Virtual Machines
Applies to SUSE Linux Enterprise Server 12 SP5

14 Configuring Virtual Machines

Virtual Machine Manager's Details view offers in-depth information about the VM Guest's complete configuration and hardware equipment. Using this view, you can also change the guest configuration or add and modify virtual hardware. To access this view, open the guest's console in Virtual Machine Manager and either choose View › Details from the menu, or click Show virtual hardware details in the toolbar.

Details View of a VM Guest
Figure 14.1: Details View of a VM Guest

The left panel of the window lists VM Guest overview and already installed hardware. After clicking an item in the list, you can access its detailed settings in the details view. You can change the hardware parameters to match your needs, then click Apply to confirm them. Some changes take effect immediately, while others need a reboot of the machine—and virt-manager warns you about that fact.

To remove installed hardware from a VM Guest, select the appropriate list entry in the left panel and then click Remove in the bottom right of the window.

To add new hardware, click Add Hardware below the left panel, then select the type of the hardware you want to add in the Add New Virtual Hardware window. Modify its parameters and confirm with Finish.

The following sections describe configuration options for the specific hardware type being added. They do not focus on modifying an existing piece of hardware as the options are identical.

14.1 Machine Setup

This section describes the setup of the virtualized processor and memory hardware. These components are vital to a VM Guest, therefore you cannot remove them. It also shows how to view the overview and performance information, and how to change boot options.

14.1.1 Overview

Overview shows basic details about VM Guest and the hypervisor.

Overview details
Figure 14.2: Overview details

Name, Title, and Description are editable and help you identify VM Guest in the Virtual Machine Manager list of machines.

VM Guest Title and Description
Figure 14.3: VM Guest Title and Description

UUID shows the universally unique identifier of the virtual machine, while Status shows its current status—Running, Paused, or Shutoff.

The Hypervisor Details section shows the hypervisor type, CPU architecture, used emulator, and chipset type. None of the hypervisor parameters can be changed.

14.1.2 Performance

Performance shows regularly updated charts of CPU and memory usage, and disk and network I/O.

Performance
Figure 14.4: Performance
Tip
Tip: Enabling Disabled Charts

Not all the charts in the Graph view are enabled by default. To enable these charts, go to File › View Manager, then select Edit › Preferences › Polling, and check the charts that you want to see regularly updated.

Statistics Charts
Figure 14.5: Statistics Charts

14.1.3 Processor

Processor includes detailed information about VM Guest processor configuration.

Processor View
Figure 14.6: Processor View

In the CPUs section, you can configure several parameters related to the number of allocated CPUs.

Logical host CPUs

The real number of CPUs installed on VM Host Server.

Current allocation

The number of currently allocated CPUs. You can hotplug more CPUs by increasing this value up to the Maximum allocation value.

Maximum allocation

Maximum number of allocatable CPUs for the current session. Any change to this value will take effect after the next VM Guest reboot.

The Configuration section lets you configure the CPU model and topology.

When activated, the Copy host CPU configuration option uses the host CPU model for VM Guest. Otherwise you need to specify the CPU model from the drop-down box.

After you activate Manually set CPU topology, you can specify a custom number of sockets, cores and threads for the CPU.

14.1.4 Memory

Memory contains information about the memory that is available to VM Guest.

Memory View
Figure 14.7: Memory View
Total host memory

Total amount of memory installed on VM Host Server.

Current allocation

The amount of memory currently available to VM Guest. You can hotplug more memory by increasing this value up to the value of Maximum allocation.

Maximum allocation

The maximum value to which you can hotplug the currently available memory. Any change to this value will take effect after the next VM Guest reboot.

14.1.5 Boot Options

Boot Options introduces options affecting the VM Guest boot process.

Boot Options
Figure 14.8: Boot Options

In the Autostart section, you can specify whether the virtual machine should automatically start during the VM Host Server boot phase.

In the Boot device order, activate the devices that will be used for booting VM Guest. You can change their order with the up and down arrow buttons on the right side of the list. To choose from a list of bootable devices on VM Guest start, activate Enable boot menu.

To boot a different kernel than the one on the boot device, activate Enable direct kernel boot and specify the paths to the alternative kernel and initrd placed on the VM Host Server file system. You can also specify kernel arguments that will be passed to the loaded kernel.

14.2 Storage

This section gives you a detailed description of configuration options for storage devices. It includes both hard disks and removable media, such as USB or CD-ROM drives.

Procedure 14.1: Adding a New Storage Device
  1. Click Add Hardware below the left panel, then select Storage from the Add New Virtual Hardware window.

    Add a New Storage
    Figure 14.9: Add a New Storage
  2. To create a qcow2 disk image in the default location, activate Create a disk image for the virtual machine and specify its size in gigabytes.

    To gain more control over the disk image creation, activate Select or create custom storage and click Manage to manage storage pools and images. The window Choose Storage Volume opens which has almost identical functionality as the Storage tab described in Section 12.1, “Managing Storage with Virtual Machine Manager”.

    Tip
    Tip: Supported Storage Formats

    SUSE only supports the following storage formats: raw, qcow2, and qed.

  3. After you manage to create and specify the disk image file, specify the Device type. It can be one of the following options:

    • Disk device

    • CDROM device: Does not allow using Create a disk image for the virtual machine.

    • Floppy device: Does not allow using Create a disk image for the virtual machine.

    • LUN Passthrough: Required to use an existing SCSI storage directly without adding it into a storage pool.

  4. Select the Bus type for your device. The list of available options depends on the device type you selected in the previous step. The types based on VirtIO use paravirtualized drivers.

  5. In the Advanced options section, select the preferred Cache mode. For more information on cache modes, see Chapter 15, Disk Cache Modes.

  6. Confirm your settings with Finish. A new storage device appears in the left panel.

14.3 Controllers

This section focuses on adding and configuring new controllers.

Procedure 14.2: Adding a New Controller
  1. Click Add Hardware below the left panel, then select Controller from the Add New Virtual Hardware window.

    Add a New Controller
    Figure 14.10: Add a New Controller
  2. Select the type of the controller. You can choose from IDE, Floppy, SCSI, SATA, VirtIO Serial (paravirtualized), USB, or CCID (smart card devices).

  3. Optionally, in the case of a USB or SCSI controller, select a controller model.

  4. Confirm your settings with Finish. A new controller appears in the left panel.

14.4 Networking

This section describes how to add and configure new network devices.

Procedure 14.3: Adding a New Network Device
  1. Click Add Hardware below the left panel, then select Network from the Add New Virtual Hardware window.

    Add a New Controller
    Figure 14.11: Add a New Controller
  2. From the Network source list, select the source for the network connection. The list includes VM Host Server's available physical network interfaces, network bridges, or network bonds. You can also assign the VM Guest to an already defined virtual network. See Chapter 13, Managing Networks for more information on setting up virtual networks with Virtual Machine Manager.

  3. Specify a MAC address for the network device. While Virtual Machine Manager pre-fills a random value for your convenience, it is recommended to supply a MAC address appropriate for your network environment to avoid network conflicts.

  4. Select a device model from the list. You can either leave the Hypervisor default, or specify one of e1000, rtl8139, or virtio models. Note that virtio uses paravirtualized drivers.

  5. Confirm your settings with Finish. A new network device appears in the left panel.

14.5 Enabling Seamless and Synchronized Mouse Pointer Movement

When you click within a VM Guest's console with the mouse, the pointer is captured by the console window and cannot be used outside the console unless it is explicitly released (by pressing AltCtrl). To prevent the console from grabbing the key and to enable seamless pointer movement between host and guest instead, add a tablet to the VM Guest.

Adding a tablet has the additional advantage of synchronizing the mouse pointer movement between VM Host Server and VM Guest when using a graphical environment on the guest. With no tablet configured on the guest, you will often see two pointers with one dragging behind the other.

  1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

  2. Click Add Hardware and choose Input and then EvTouch USB Graphics Tablet in the pop-up window. Proceed with Finish.

  3. If the guest is running, you will be asked whether to enable the tablet after the next reboot. Confirm with Yes.

  4. When you start or restart the VM Guest, the tablet becomes available in the VM Guest.

14.6 Adding a CD/DVD-ROM Device with Virtual Machine Manager

KVM supports CD or DVD-ROMs in VM Guest either by directly accessing a physical drive on the VM Host Server or by accessing ISO images. To create an ISO image from an existing CD or DVD, use dd:

dd if=/dev/CD_DVD_DEVICE of=my_distro.iso bs=2048

To add a CD/DVD-ROM device to your VM Guest, proceed as follows:

  1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

  2. Click Add Hardware and choose Storage in the pop-up window.

  3. Change the Device Type to IDE CDROM.

  4. Select Select or create custom storage.

    1. To assign the device to a physical medium, enter the path to the VM Host Server's CD/DVD-ROM device (for example, /dev/cdrom) next to Manage. Alternatively, use Manage to open a file browser and then click Browse Local to select the device. Assigning the device to a physical medium is only possible when the Virtual Machine Manager was started on the VM Host Server.

    2. To assign the device to an existing image, click Manage to choose an image from a storage pool. If the Virtual Machine Manager was started on the VM Host Server, alternatively choose an image from another location on the file system by clicking Browse Local. Select an image and close the file browser with Choose Volume.

  5. Save the new virtualized device with Finish.

  6. Reboot the VM Guest to make the new device available. For more information, see Section 14.8, “Ejecting and Changing Floppy or CD/DVD-ROM Media with Virtual Machine Manager”.

14.7 Adding a Floppy Device with Virtual Machine Manager

Currently KVM only supports the use of floppy disk images—using a physical floppy drive is not supported. Create a floppy disk image from an existing floppy using dd:

dd if=/dev/fd0 of=/var/lib/libvirt/images/floppy.img

To create an empty floppy disk image use one of the following commands:

Raw Image
dd if=/dev/zero of=/var/lib/libvirt/images/floppy.img bs=512 count=2880
FAT Formatted Image
mkfs.msdos -C /var/lib/libvirt/images/floppy.img 1440

To add a floppy device to your VM Guest, proceed as follows:

  1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

  2. Click Add Hardware and choose Storage in the pop-up window.

  3. Change the Device Type to Floppy Disk.

  4. Choose Select or create custom storage and click Manage to choose an existing image from a storage pool. If Virtual Machine Manager was started on the VM Host Server, alternatively choose an image from another location on the file system by clicking Browse Local. Select an image and close the file browser with Choose Volume.

  5. Save the new virtualized device with Finish.

  6. Reboot the VM Guest to make the new device available. For more information, see Section 14.8, “Ejecting and Changing Floppy or CD/DVD-ROM Media with Virtual Machine Manager”.

14.8 Ejecting and Changing Floppy or CD/DVD-ROM Media with Virtual Machine Manager

Whether you are using the VM Host Server's physical CD/DVD-ROM device or an ISO/floppy image: Before you can change the media or image of an existing device in the VM Guest, you first need to disconnect the media from the guest.

  1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

  2. Choose the Floppy or CD/DVD-ROM device and eject the medium by clicking Disconnect.

  3. To insert a new medium, click Connect.

    1. If using the VM Host Server's physical CD/DVD-ROM device, first change the media in the device (this may require unmounting it on the VM Host Server before it can be ejected). Then choose CD-ROM or DVD and select the device from the drop-down box.

    2. If you are using an ISO image, choose ISO image Location and select an image by clicking Manage. When connecting from a remote host, you may only choose images from existing storage pools.

  4. Click OK to finish. The new media can now be accessed in the VM Guest.

14.9 Editing VM Configuration with virsh

The configuration of a VM is stored in an XML file in /etc/libvirtd/qemu/ and looks like this:

Example 14.1: Example XML Configuration File
<domain type='kvm'>
  <name>sles15</name>
  <uuid>ab953e2f-9d16-4955-bb43-1178230ee625</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.11'>hvm</type>
  </os>
  <features>...</features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Skylake-Client-IBRS</model>
  </cpu>
  <clock>...</clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>...</disk>
  </devices>
  ...
</domain>

If you want to edit the configuration of a VM Guest, check if it is offline:

tux > sudo virsh list --inactive

If your VM Guest is in this list, you can safely edit its configuration:

tux > sudo virsh edit NAME_OF_VM_GUEST

Before saving the changes, virsh validates your input against a RelaxNG schema.

14.10 Changing the Machine Type with virsh

By default, when installing with the virt-install tool, the machine type for VM Guest is pc-i440fx. The machine type is stored in the VM Guest's xml configuration file in /etc/libvirt/qemu/ in the tag type:

<type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>

As an example, the following procedure shows how to change this value to the machine type q35. q35 is an Intel* chipset. It includes PCIe, supports up to 12 USB ports, and has support for SATA and IOMMU. IRQ routing has also been improved.

  1. Check whether your VM Guest is inactive:

    virsh list --inactive
    Id    Name                           State
    ----------------------------------------------------
    -     sles11                         shut off
  2. Edit the configuration for this VM Guest:

    virsh edit sles11
  3. Change the value of the machine attribute:

    <type arch='x86_64' machine='pc-q35-2.0'>hvm</type>
  4. Restart the VM Guest.

    root # virsh start sles11
  5. Check that the machine type has changed. Log in to the VM Guest as root and run the following command:

    root # dmidecode | grep Product
    Product Name: Standard PC (Q35 + ICH9, 2009)
Tip
Tip: Machine Type Update Recommendations

Whenever the QEMU version on the host system is upgraded (for example, when upgrading the VM Host Server to a new service pack), upgrade the machine type of the VM Guests to the latest available version. To check, use the command qemu-system-x86_64 -M help on the VM Host Server.

The default machine type pc-i440fx, for example, is regularly updated. If your VM Guest still runs with a machine type of pc-i440fx-1.X, an update to pc-i440fx-2.X is strongly recommended. This allows taking advantage of the most recent updates and corrections in machine definitions, and ensures better future compatibility.

14.11 Assigning a Host PCI Device to a VM Guest

You can directly assign host-PCI devices to guests (PCI pass-through). When the PCI device is assigned to one VM Guest, it cannot be used on the host or by another VM Guest unless it is re-assigned. A prerequisite for this feature is a VM Host Server configuration as described in Important: Requirements for VFIO and SR-IOV.

14.11.1 Adding a PCI Device with Virtual Machine Manager

The following procedure describes how to add a PCI device to a VM Guest using Virtual Machine Manager:

  1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

  2. Click Add Hardware and choose the PCI Host Device category in the left panel. A list of available PCI devices appears in the right part of the window.

    Adding a PCI Device
    Figure 14.12: Adding a PCI Device
  3. From the list of available PCI devices, choose the one you want to pass to the guest. Confirm with Finish.

Tip
Tip: Assigning a PCI Device Requires a VM Guest Shutdown

Although it is possible to assign a PCI device to a running VM Guest as described above, the device will not become available until you shut down the VM Guest and reboot it afterward.

14.11.2 Adding a PCI Device with virsh

To assign a PCI device to VM Guest with virsh, follow these steps:

  1. Identify the host PCI device to assign to the guest. In the following example, we are assigning a DEC network card to the guest:

    tux > sudo lspci -nn
    [...]
    03:07.0 Ethernet controller [0200]: Digital Equipment Corporation DECchip \
    21140 [FasterNet] [1011:0009] (rev 22)
    [...]

    Note down the device ID (03:07.0 in this case).

  2. Gather detailed information about the device using virsh nodedev-dumpxml ID. To get the ID, you need to replace colon and period in the device ID (03:07.0) with underscore and prefix the result with pci_0000_ (pci_0000_03_07_0).

    tux > virsh nodedev-dumpxml pci_0000_03_07_0
    <device>
      <name>pci_0000_03_07_0</name>
      <path>/sys/devices/pci0000:00/0000:00:14.4/0000:03:07.0</path>
      <parent>pci_0000_00_14_4</parent>
      <driver>
        <name>tulip</name>
      </driver>
      <capability type='pci'>
        <domain>0</domain>
        <bus>3</bus>
        <slot>7</slot>
        <function>0</function>
        <product id='0x0009'>DECchip 21140 [FasterNet]</product>
        <vendor id='0x1011'>Digital Equipment Corporation</vendor>
        <numa node='0'/>
      </capability>
    </device>

    Note down the values for domain, bus, and function.

  3. Detach the device from the host system prior to attaching it to VM Guest.

    tux > virsh nodedev-detach pci_0000_03_07_0
      Device pci_0000_03_07_0 detached
    Tip
    Tip: Multi-Function PCI Devices

    When using a multi-function PCI device that does not support FLR (function level reset) or PM (power management) reset, you need to detach all its functions from the VM Host Server. The whole device must be reset for security reasons. libvirt will refuse to assign the device if one of its functions is still in use by the VM Host Server or another VM Guest.

  4. Convert the domain, bus, slot, and function value from decimal to hexadecimal, and prefix with 0x to tell the system that the value is hexadecimal. In our example, domain = 0, bus = 3, slot = 7, and function = 0. Their hexadecimal values are:

    tux > printf %x 0
    0
    tux > printf %x 3
    3
    tux > printf %x 7
    7

    This results in domain = 0x0000, bus = 0x03, slot = 0x07 and function = 0x00.

  5. Run virsh edit on your domain, and add the following device entry in the <devices> section using the values from the previous step:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x07' function='0x00'/>
      </source>
    </hostdev>
    Tip
    Tip: managed Compared to unmanaged

    libvirt recognizes two modes for handling PCI devices: they can be either managed or unmanaged. In the managed case, libvirt handles all details of unbinding the device from the existing driver if needed, resetting the device, binding it to vfio-pci before starting the domain, etc. When the domain is terminated or the device is removed from the domain, libvirt will unbind from vfio-pci and rebind to the original driver in the case of a managed device. If the device is unmanaged, the user must ensure all of these management aspects of the device are done before assigning it to a domain, and after the device is no longer used by the domain.

    In the example above, the managed='yes' option means that the device is managed. To switch the device mode to unmanaged, set managed='no' in the listing above. If you do so, you need to take care of the related driver with the virsh nodedev-detach and virsh nodedev-reattach commands. That means you need to run virsh nodedev-detach pci_0000_03_07_0 prior to starting the VM Guest to detach the device from the host. In case the VM Guest is not running, you can make the device available for the host by running virsh nodedev-reattach pci_0000_03_07_0.

  6. Shut down the VM Guest and restart it to make the assigned PCI device available.

    Tip
    Tip: SELinux

    If you are running SELinux on your VM Host Server, you need to disable it prior to starting the VM Guest with

    setsebool -P virt_use_sysfs 1

14.12 Assigning a Host USB Device to a VM Guest

Analogous to assigning host PCI devices (see Section 14.11, “Assigning a Host PCI Device to a VM Guest”), you can directly assign host USB devices to guests. When the USB device is assigned to one VM Guest, it cannot be used on the host or by another VM Guest unless it is re-assigned.

14.12.1 Adding a USB Device with Virtual Machine Manager

To assign a host USB device to VM Guest using Virtual Machine Manager, follow these steps:

  1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

  2. Click Add Hardware and choose the USB Host Device category in the left panel. A list of available USB devices appears in the right part of the window.

    Adding a USB Device
    Figure 14.13: Adding a USB Device
  3. From the list of available USB devices, choose the one you want to pass to the guest. Confirm with Finish. The new USB device appears in the left pane of the Details view.

    Tip
    Tip: USB Device Removal

    To remove the host USB device assignment, click it in the left pane of the Details view and confirm with Remove.

14.12.2 Adding a USB Device with virsh

To assign a USB device to VM Guest using virsh, follow these steps:

  1. Identify the host USB device to assign to the guest:

    tux > sudo lsusb
    [...]
    Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon
    [...]

    Note down the vendor and product IDs. In our example, the vendor ID is 0557 and the product ID is 2221.

  2. Run virsh edit on your domain, and add the following device entry in the <devices> section using the values from the previous step:

    <hostdev mode='subsystem' type='usb'>
      <source startupPolicy='optional'>
       <vendor id='0557'/>
       <product id='2221'/>
      </source>
    </hostdev>
    Tip
    Tip: Vendor/Product or Device's Address

    Instead of defining the host device with <vendor/> and <product/> IDs, you can use the <address/> element as described for host PCI devices in Section 14.11.2, “Adding a PCI Device with virsh.

  3. Shut down the VM Guest and restart it to make the assigned USB device available.

    Tip
    Tip: SELinux

    If you are running SELinux on your VM Host Server, you need to disable it prior to starting the VM Guest with

    setsebool -P virt_use_sysfs 1

14.13 Adding SR-IOV Devices

Single Root I/O Virtualization (SR-IOV) capable PCIe devices can replicate their resources, so they appear to be multiple devices. Each of these "pseudo-devices" can be assigned to a VM Guest.

SR-IOV is an industry specification that was created by the Peripheral Component Interconnect Special Interest Group (PCI-SIG) consortium. It introduces physical functions (PF) and virtual functions (VF). PFs are full PCIe functions used to manage and configure the device. PFs also can move data. VFs lack the configuration and management part—they only can move data and a reduced set of configuration functions. Since VFs do not have all PCIe functions, the host operating system or the Hypervisor must support SR-IOV to be able to access and initialize VFs. The theoretical maximum for VFs is 256 per device (consequently the maximum for a dual-port Ethernet card would be 512). In practice this maximum is much lower, since each VF consumes resources.

14.13.1 Requirements

The following requirements must be met to be able to use SR-IOV:

  • An SR-IOV-capable network card (as of SUSE Linux Enterprise Server 12 SP5, only network cards support SR-IOV)

  • An AMD64/Intel 64 host supporting hardware virtualization (AMD-V or Intel VT-x), see Section 7.3, “KVM Hardware Requirements” for more information

  • A chipset that supports device assignment (AMD-Vi or Intel VT-d)

  • libvirt-0.9.10 or better

  • SR-IOV drivers must be loaded and configured on the host system

  • A host configuration that meets the requirements listed at Important: Requirements for VFIO and SR-IOV

  • A list of the PCI addresses of the VF(s) that will be assigned to VM Guests

Tip
Tip: Checking if a Device is SR-IOV-Capable

The information whether a device is SR-IOV-capable can be obtained from its PCI descriptor by running lspci. A device that supports SR-IOV reports a capability similar to the following:

Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
Note
Note: Adding an SR-IOV Device at VM Guest Creation

Before adding an SR-IOV device to a VM Guest when initially setting it up, the VM Host Server already needs to be configured as described in Section 14.13.2, “Loading and Configuring the SR-IOV Host Drivers”.

14.13.2 Loading and Configuring the SR-IOV Host Drivers

To be able to access and initialize VFs, an SR-IOV-capable driver needs to be loaded on the host system.

  1. Before loading the driver, make sure the card is properly detected by running lspci. The following example shows the lspci output for the dual-port Intel 82576NS network card:

    tux > sudo /sbin/lspci | grep 82576
    01:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
    01:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
    04:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
    04:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)

    In case the card is not detected, it is likely that the hardware virtualization support in the BIOS/EFI has not been enabled.

  2. Check whether the SR-IOV driver is already loaded by running lsmod. In the following example a check for the igb driver (for the Intel 82576NS network card) returns a result. That means the driver is already loaded. If the command returns nothing, the driver is not loaded.

    tux > sudo /sbin/lsmod | egrep "^igb "
    igb                   185649  0
  3. Skip this step if the driver is already loaded.

    If the SR-IOV driver is not yet loaded, the non-SR-IOV driver needs to be removed first, before loading the new driver. Use rmmod to unload a driver. The following example unloads the non-SR-IOV driver for the Intel 82576NS network card:

    sudo /sbin/rmmod igbvf

    Load the SR-IOV driver subsequently using the modprobe command—the VF parameter (max_vfs) is mandatory:

    sudo /sbin/modprobe igb max_vfs=8

    Or load the driver via SYSFS:

    Find the PCI ID of the physical NIC by listing Ethernet devices:

    tux > sudo lspci | grep Eth
    06:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
    06:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)

    To enable VFs, echo the number of desired VFs to load to the sriov_numvfs parameter:

    tux > sudo echo 1 > /sys/bus/pci/devices/0000:06:00.1/sriov_numvfs

    Verify that the VF NIC was loaded:

    tux > sudo lspci | grep Eth
    06:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
    06:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
    06:08.0 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)

    Obtain the maximum number of VFs available:

    tux > sudo lspci -vvv -s 06:00.1 | grep 'Initial VFs'
                           Initial VFs: 32, Total VFs: 32, Number of VFs: 0,
    Function Dependency Link: 01
  4. Create a before.service file which loads VF via SYSFS on boot:

    [Unit]
    Before=
    After=network-online.target
    [Service]
    Type=oneshot
    RemainAfterExit=true
    ExecStart=/bin/bash -c "echo 1 > /sys/bus/pci/devices/0000:06:00.1/sriov_numvfs"
    # beware, executable is run directly, not through a shell, check the man pages
    # systemd.service and systemd.unit for full syntax
    [Install]
    # target in which to start the service
    WantedBy=multi-user.target
    #WantedBy=graphical.target

    And copy it to /etc/systemd/system.

    Additionally, it is required to create another service file (after-local.service) pointing to /etc/init.d/after.local script that detaches the NIC prior to starting the VM, otherwise the VM would fail to start:

    [Unit]
    Description=/etc/init.d/after.local Compatibility
    After=libvirtd.service
    Requires=libvirtd.service
    [Service]
    Type=oneshot
    ExecStart=/etc/init.d/after.local
    RemainAfterExit=true
    
    [Install]
    WantedBy=multi-user.target

    And copy it to /etc/systemd/system.

    #! /bin/sh
    #
    # Copyright (c) 2010 SuSE LINUX Products GmbH, Germany.  All rights reserved.
    # ...
    virsh nodedev-detach pci_0000_06_08_0

    Then save it as /etc/init.d/after.local.

  5. Reboot the machine and check if the SR-IOV driver is loaded by re-running the lspci command from the first step of this procedure. If the SR-IOV driver was loaded successfully you should see additional lines for the VFs:

    01:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
    01:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
    01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    01:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    [...]
    04:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
    04:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
    04:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    04:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    04:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
    [...]

14.13.3 Adding a VF Network Device to an Existing VM Guest

When the SR-IOV hardware is properly set up on the VM Host Server, you can add VFs to VM Guests. To do so, you need to collect some data first.

Note: The following procedure is using example data. Make sure to replace it by appropriate data from your setup.

  1. Use the virsh nodedev-list command to get the PCI address of the VF you want to assign and its corresponding PF. Numerical values from the lspci output shown in Section 14.13.2, “Loading and Configuring the SR-IOV Host Drivers” (for example 01:00.0 or 04:00.1) are transformed by adding the prefix "pci_0000_" and by replacing colons and dots with underscores. So a PCI ID listed as "04:00.0" by lspci is listed as "pci_0000_04_00_0" by virsh. The following example lists the PCI IDs for the second port of the Intel 82576NS network card:

    tux > sudo virsh nodedev-list | grep 0000_04_
    pci_0000_04_00_0
    pci_0000_04_00_1
    pci_0000_04_10_0
    pci_0000_04_10_1
    pci_0000_04_10_2
    pci_0000_04_10_3
    pci_0000_04_10_4
    pci_0000_04_10_5
    pci_0000_04_10_6
    pci_0000_04_10_7
    pci_0000_04_11_0
    pci_0000_04_11_1
    pci_0000_04_11_2
    pci_0000_04_11_3
    pci_0000_04_11_4
    pci_0000_04_11_5

    The first two entries represent the PFs, whereas the other entries represent the VFs.

  2. Get more data that will be needed by running the command virsh nodedev-dumpxml on the PCI ID of the VF you want to add:

    tux > sudo virsh nodedev-dumpxml pci_0000_04_10_0
    <device>
      <name>pci_0000_04_10_0</name>
      <parent>pci_0000_00_02_0</parent>
      <capability type='pci'>
        <domain>0</domain>
        <bus>4</bus>
        <slot>16</slot>
        <function>0</function>
        <product id='0x10ca'>82576 Virtual Function</product>
        <vendor id='0x8086'>Intel Corporation</vendor>
        <capability type='phys_function'>
          <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
        </capability>
      </capability>
    </device>

    The following data is needed for the next step:

    • <domain>0</domain>

    • <bus>4</bus>

    • <slot>16</slot>

    • <function>0</function>

  3. Create a temporary XML file (for example /tmp/vf-interface.xml containing the data necessary to add a VF network device to an existing VM Guest. The minimal content of the file needs to look like the following:

    <interface type='hostdev'>1
     <source>
      <address type='pci' domain='0' bus='11' slot='16' function='0'2/>2
     </source>
    </interface>

    1

    VFs do not get a fixed MAC address; it changes every time the host reboots. When adding network devices the traditional way with <hostdev>, it would require to reconfigure the VM Guest's network device after each reboot of the host, because of the MAC address change. To avoid this kind of problem, libvirt introduced the interface type='hostdev' directive, which sets up network-specific data before assigning the device.

    2

    Specify the data you acquired in the previous step here.

  4. In case a device is already attached to the host, it cannot be attached to a guest. To make it available for guests, detach it from the host first:

    virsh nodedev-detach pci_0000_04_10_0
  5. Last, add the VF interface to an existing VM Guest:

    virsh attach-device GUEST /tmp/vf-interface.xml --OPTION

    GUEST needs to be replaced by the domain name, ID or UUID of the VM Guest and --OPTION can be one of the following:

    --persistent

    This option will always add the device to the domain's persistent XML. In addition, if the domain is running, it will be hotplugged.

    --config

    This option will only affect the persistent XML, even if the domain is running. The device will only show up in the guest on next boot.

    --live

    This option will only affect a running domain. If the domain is inactive, the operation will fail. The device is not persisted in the XML and will not be available in the guest on next boot.

    --current

    This option affects the current state of the domain. If the domain is inactive, the device is added to the persistent XML and will be available on next boot. If the domain is active, the device is hotplugged but not added to the persistent XML.

    To detach a VF interface, use the virsh detach-device command, which also takes the options listed above.

14.13.4 Dynamic Allocation of VFs from a Pool

If you define the PCI address of a VF into a guest's configuration statically as described in Section 14.13.3, “Adding a VF Network Device to an Existing VM Guest”, it is hard to migrate such guest to another host. The host must have identical hardware in the same location on the PCI bus, or the guest configuration must be modified prior to each start.

Another approach is to create a libvirt network with a device pool that contains all the VFs of an SR-IOV device. The guest then references this network, and each time it is started, a single VF is dynamically allocated to it. When the guest is stopped, the VF is returned to the pool, available for another guest.

14.13.4.1 Defining Network with Pool of VFs on VM Host Server

The following example of network definition creates a pool of all VFs for the SR-IOV device with its physical function (PF) at the network interface eth0 on the host:

<network>
  <name>passthrough</name>
    <forward mode='hostdev' managed='yes'>
      <pf dev='eth0'/>
    </forward>
  </network>

To use this network on the host, save the above code to a file, for example /tmp/passthrough.xml, and execute the following commands. Remember to replace eth0 with the real network interface name of your SR-IOV device's PF:

virsh net-define /tmp/passthrough.xml
virsh net-autostart passthrough
virsh net-start passthrough

14.13.4.2 Configuring VM Guest to Use VF from the Pool

The following example of guest device interface definition uses a VF of the SR-IOV device from the pool created in Section 14.13.4.1, “Defining Network with Pool of VFs on VM Host Server”. libvirt automatically derives the list of all VFs associated with that PF the first time the guest is started.

<interface type='network'>
  <source network='passthrough'>
</interface>

To verify the list of associated VFs, run virsh net-dumpxml passthrough on the host after the first guest that uses the network with the pool of VFs starts.

<network connections='1'>
  <name>passthrough</name>
  <uuid>a6a26429-d483-d4ed-3465-4436ac786437</uuid>
  <forward mode='hostdev' managed='yes'>
    <pf dev='eth0'/>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x5'/>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x7'/>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x1'/>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x3'/>
    <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x5'/>
  </forward>
  </network>

14.14 Using Macvtap to Share VM Host Server Network Interfaces

Macvtap provides direct attachment of a VM Guest virtual interface to a host network interface. The macvtap-based interface extends the VM Host Server network interface and has its own MAC address on the same Ethernet segment. Typically, this is used to make both the VM Guest and the VM Host Server show up directly on the switch that the VM Host Server is connected to.

Note
Note: Macvtap Cannot Be Used with a Linux Bridge

Macvtap cannot be used with network interfaces already connected to a Linux bridge. Before attempting to create the macvtap interface, remove the interface from the bridge.

Note
Note: VM Guest to VM Host Server Communication with Macvtap

When using macvtap, a VM Guest can communicate with other VM Guests, and with other external hosts on the network. But it cannot communicate with the VM Host Server on which the VM Guest runs. This is the defined behavior of macvtap, because of the way the VM Host Server's physical Ethernet is attached to the macvtap bridge. Traffic from the VM Guest into that bridge that is forwarded to the physical interface cannot be bounced back up to the VM Host Server's IP stack. Similarly, traffic from the VM Host Server's IP stack that is sent to the physical interface cannot be bounced back up to the macvtap bridge for forwarding to the VM Guest.

Virtual network interfaces based on macvtap are supported by libvirt by specifying an interface type of direct. For example:

<interface type='direct'>
  <mac address='aa:bb:cc:dd:ee:ff'/>
  <source dev='eth0' mode='bridge'/>
  <model type='virtio'/>
  </interface>

The operation mode of the macvtap device can be controlled with the mode attribute. The following list shows its possible values and a description for each:

  • vepa: All VM Guest packets are sent to an external bridge. Packets whose destination is a VM Guest on the same VM Host Server as where the packet originates from are sent back to the VM Host Server by the VEPA capable bridge (today's bridges are typically not VEPA capable).

  • bridge: Packets whose destination is on the same VM Host Server as where they originate from are directly delivered to the target macvtap device. Both origin and destination devices need to be in bridge mode for direct delivery. If either one of them is in vepa mode, a VEPA capable bridge is required.

  • private: All packets are sent to the external bridge and will only be delivered to a target VM Guest on the same VM Host Server if they are sent through an external router or gateway and that device sends them back to the VM Host Server. This procedure is followed if either the source or destination device is in private mode.

  • passthrough: A special mode that gives more power to the network interface. All packets will be forwarded to the interface, allowing virtio VM Guests to change the MAC address or set promiscuous mode to bridge the interface or create VLAN interfaces on top of it. Note that a network interface is not shareable in passthrough mode. Assigning an interface to a VM Guest will disconnect it from the VM Host Server. For this reason SR-IOV virtual functions are often assigned to the VM Guest in passthrough mode.

14.15 Managing Guest Memory Allocation (Xen only)

libvirt now includes support for adjusting memory allocation per guest with virsh. Xen paravirtual devices connect to the xenbus controller, which is analogous to a physical device bus such as a PCI controller. Xen's max_grant_frames attribute sets how many frames, which are analogous to memory, are allocated to the xenbus controller for each guest.

The default is 32, and this can be increased as needed, up to the amount set for dom0, or decreased. How much is enough? This depends on the number of devices and workload demand, such as a saturated network interface or heavy I/O. Use xen-diag to see your current used and maximum max_grant_frames values for dom0 and your guests. The guests must be running:

tux > sudo virsh list
 Id   Name             State
--------------------------------
 0    Domain-0         running
 3    sle15sp1         running

 tux > sudo xen-diag gnttab_query_size 0
domid=0: nr_frames=1, max_nr_frames=256

tux > sudo xen-diag gnttab_query_size 3
domid=3: nr_frames=3, max_nr_frames=32

The sle15sp1 guest is using only 3 frames out of 32. If you are seeing performance issues, and log entries that point to insufficient frames, increase the value with virsh. Look for the <controller type='xenbus' line in the guest's configuration file and add the maxGrantFrames control element:

tux > sudo virsh edit sle15sp1
 <controller type='xenbus' index='0' maxGrantFrames='40'/>

Save your changes and restart the guest. Now it should show your change:

tux > sudo xen-diag gnttab_query_size 3
domid=3: nr_frames=3, max_nr_frames=40

See the Controllers section of the libvirt Domain XML format manual at https://libvirt.org/formatdomain.html#elementsControllers for more information.

14.16 Disabling a Memory Balloon Device

Memory Balloon has become a default option for KVM. The device will be added to the VM Guest explicitly, so you do not need to add this element in the VM Guest's XML configuration. However, if you want to disable Memory Balloon in the VM Guest for any reason, you need to set model='none' as shown below:

<devices>
   <memballoon model='none'/>
</device>

14.17 Configuring Multiple Monitors (Dual Head)

libvirt supports a dual head configuration to display the video output of the VM Guest on multiple monitors.

Important
Important: No Support for Xen

The Xen hypervisor does not support dual head configuration.

Procedure 14.4: Configuring Dual Head
  1. While the virtual machine is running, verify that the xf86-video-qxl package is installed in the VM Guest:

    tux > rpm -q xf86-video-qxl
  2. Shut down the VM Guest and start editing its configuration XML as described in Section 14.9, “Editing VM Configuration with virsh.

  3. Verify that the model of the virtual graphics card is 'qxl':

    <video>
     <model type='qxl' ... />
  4. Increase the heads parameter in the graphics card model specification from the default 1 to 2, for example:

    <video>
     <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='2' primary='yes'/>
     <alias name='video0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
  5. Configure the virtual machine to use the Spice display instead of VNC:

    <graphics type='spice' port='5916' autoport='yes' listen='0.0.0.0'>
     <listen type='address' address='0.0.0.0'/>
    </graphics>
  6. Start the virtual machine and connect to its display with virt-viewer, for example:

    tux > virt-viewer --connect qemu+ssh://USER@VM_HOST/system
  7. From the list of VMs, select the one whose configuration you have modified and confirm with Connect.

  8. After the graphical subsystem (Xorg) loads in the VM Guest, select View › Displays › Display 2 to open a new window with the second monitor's output.