Assigning a PCI device to a virtual machine

Publication Date: 22 Sep 2023

1 Environment #

This document applies to the following product and product versions:

SUSE Linux Enterprise Server 15 SP3, 15 SP2, 15 SP1, 15 GA, 12 SP5, 12 SP4, 12 SP3

2 Introduction #

libvirt is a unified interface to various virtualization solutions. libvirt virtual machines are typically managed by one or the other of two tools: virsh, a command-line utility, or Virtual Machine Manager, an application with a graphical user interface.

Assigning a VM Host Server's PCI device to a VM Guest—also known as PCI Pass-Through—dedicates the PCI device's resources to the VM Guest. After the PCI device is assigned, it cannot be used by the VM Host Server or by any other VM Guest on the same VM Host Server.

In this article, you will learn how to assign VM Host Server's PCI device to a VM Guest using both virsh and Virtual Machine Manager.

3 Requirements #

A SUSE Linux Enterprise-based client machine with libvirt-client and virt-manager installed.
At least one connection to a valid VM Host Server configured by Virtual Machine Manager or virsh.
An existing libvirt-based virtual machine on the VM Host Server.

4 Assigning a PCI device with `virsh` #

To assign a VM Host Server's PCI device to a VM Guest with virsh, follow these steps:

Identify the host PCI device to assign to the VM Guest. The following example identifies a DEC network card:
```
> sudo lspci -nn
[...]
03:07.0 Ethernet controller [0200]: Digital Equipment Corporation DECchip \
21140 [FasterNet] [1011:0009] (rev 22)
[...]
```
Write down the device ID (03:07.0 in this case).

Gather detailed information about the device using virsh nodedev-dumpxml ID. To get the ID, replace the colon and the period in the device ID (03:07.0) with underscores. Prefix the result with “pci_0000_”: pci_0000_03_07_0.

> sudo virsh nodedev-dumpxml pci_0000_03_07_0
<device>
  <name>pci_0000_03_07_0</name>
  <path>/sys/devices/pci0000:00/0000:00:14.4/0000:03:07.0</path>
  <parent>pci_0000_00_14_4</parent>
  <driver>
    <name>tulip</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>3</bus>
    <slot>7</slot>
    <function>0</function>
    <product id='0x0009'>DECchip 21140 [FasterNet]</product>
    <vendor id='0x1011'>Digital Equipment Corporation</vendor>
    <numa node='0'/>
  </capability>
</device>

Write down the values for domain, bus, and function.

Detach the device from the host system:
```
> sudo virsh nodedev-detach pci_0000_03_07_0
  Device pci_0000_03_07_0 detached
```
Tip: Multi-function PCI devices
When using a multi-function PCI device that does not support FLR (function level reset) or PM (power management) reset, you need to detach all its functions from the VM Host Server. The whole device must be reset for security reasons. libvirt will refuse to assign the device if one of its functions is still in use by the VM Host Server or another VM Guest.
Convert the domain, bus, slot, and function values from decimal to hexadecimal. In this example, domain = 0, bus = 3, slot = 7, and function = 0. Ensure that the values are inserted in the right order:
```
> printf "<address domain='0x%x' bus='0x%x' slot='0x%x' function='0x%x'/>\n" 0 3 7 0
```
This results in:
```
<address domain='0x0' bus='0x3' slot='0x7' function='0x0'/>
```
Run virsh edit on your domain, and add the following device entry in the <devices> section using the result from the previous step:
```
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0' bus='0x03' slot='0x07' function='0x0'/>
  </source>
</hostdev>
```
Tip: managed compared to unmanaged
libvirt recognizes two modes for handling PCI devices: managed or unmanaged.
If the device is managed, libvirt handles all of the details of adding or removing the device. Before starting the domain, libvirt unbinds the device from the existing driver if needed, resets the device, and binds it to vfio-pci. When the domain is terminated or the device is removed from the domain, libvirt unbinds the device from vfio-pci and rebinds it to the original driver.
If the device is unmanaged, you must manually manage these tasks before assigning the device to a domain, and after the device is no longer used by the domain.
In the example above, the managed='yes' option means that the device is managed. To switch the device mode to unmanaged, set managed='no'. If you do so, you need to take care of the related driver with the virsh nodedev-detach and virsh nodedev-reattach commands. Prior to starting the VM Guest you need to detach the device from the host by running
```
> sudo virsh nodedev-detach pci_0000_03_07_0
```
When the VM Guest is not running, you can make the device available for the host by running
```
> sudo virsh nodedev-reattach pci_0000_03_07_0
```
Shut down the VM Guest:
```
> sudo shutdown -h now
```
If SELinux is running on the VM Host Server, you need to allow managing the PCI device configuration:
```
> sudo setsebool -P virt_use_sysfs 1
```
Start your VM Guest to make the assigned PCI device available:
```
> sudo virsh start VM_NAME
```

Important: SLES11 SP4 KVM guests

On a newer QEMU machine type (pc-i440fx-2.0 or higher) with SLES11 SP4 KVM guests, the acpiphp module is not loaded by default in the guest. This module must be loaded to enable hotplugging of disk and network devices. To load the module manually, use the command modprobe acpiphp. It is also possible to autoload the module by adding install acpiphp /bin/true to the /etc/modprobe.conf.local file.

Important: KVM guests using QEMU Q35 machine type

KVM guests using the QEMU Q35 machine type have a PCI topology that includes a pcie-root controller and seven pcie-root-port controllers. The pcie-root controller does not support hotplugging. Each pcie-root-port controller supports hotplugging a single PCIe device. PCI controllers cannot be hotplugged, so plan accordingly and add more pcie-root-ports if more than seven PCIe devices will be hotplugged. A pcie-to-pci-bridge controller can be added to support hotplugging legacy PCI devices. See https://libvirt.org/pci-hotplug.html for more information about PCI topology between QEMU machine types.

5 Assigning a PCI device with Virtual Machine Manager #

To assign a VM Host Server's PCI device to a VM Guest with Virtual Machine Manager, follow these steps:

Double-click a VM Guest entry in the Virtual Machine Manager to open its console, then switch to the Details view with View › Details.
Click Add Hardware and choose the PCI Host Device category in the left panel. A list of available PCI devices appears in the right part of the window.
Figure 1: Adding a PCI device #
From the list of available PCI devices, choose the one you want to pass to the guest. Confirm with Finish.

6 Summary #

After you successfully assign a VM Host Server's PCI device to a VM Guest, the VM Guest can exclusively use its resources.

7 Troubleshooting #

PCI device assignment fails.: Verify that the device is not assigned to another VM Guest and that it is properly detached from the VM Host Server.