Applies to SUSE Linux Enterprise Server 15 SP3

23 Setting up a virtual machine host #

This section documents how to set up and use SUSE Linux Enterprise Server 15 SP3 as a virtual machine host.

Usually, the hardware requirements for the Dom0 are the same as those for the SUSE Linux Enterprise Server operating system. Additional CPU, disk, memory, and network resources should be added to accommodate the resource demands of all planned VM Guest systems.

Tip: Resources

Remember that VM Guest systems, like physical machines, perform better when they run on faster processors and have access to more system memory.

The virtual machine host requires several software packages and their dependencies to be installed. To install all necessary packages, run YaST Software Management, select View › Patterns and choose Xen Virtual Machine Host Server for installation. The installation can also be performed with YaST using the module Virtualization › Install Hypervisor and Tools.

After the Xen software is installed, restart the computer and, on the boot screen, choose the newly added option with the Xen kernel.

Updates are available through your update channel. To be sure to have the latest updates installed, run YaST Online Update after the installation has finished.

23.1 Best practices and suggestions #

When installing and configuring the SUSE Linux Enterprise Server operating system on the host, be aware of the following best practices and suggestions:

If the host should always run as Xen host, run YaST System › Boot Loader and activate the Xen boot entry as default boot section.
- In YaST, click System > Boot Loader.
- Change the default boot to the Xen label, then click Set as Default.
- Click Finish.
For best performance, only the applications and processes required for virtualization should be installed on the virtual machine host.
If you intend to use a watchdog device attached to the Xen host, use only one at a time. It is recommended to use a driver with actual hardware integration over a generic software one.

Note: Hardware monitoring

The Dom0 kernel is running virtualized, so tools like irqbalance or lscpu do not reflect the real hardware characteristics.

Important: Trusted boot not supported by Xen

Trusted boot (Tboot) is not supported by Xen. To ensure that the Xen host boots correctly, verify that the Enable Trusted Boot Support option is deactivated in the GRUB 2 configuration dialog.

23.2 Managing Dom0 memory #

In previous versions of SUSE Linux Enterprise Server, the default memory allocation scheme of a Xen host was to allocate all host physical memory to Dom0 and enable auto-ballooning. Memory was automatically ballooned from Dom0 when additional domains were started. This behavior has always been error prone and disabling it was strongly encouraged. Starting with SUSE Linux Enterprise Server 15 SP1, auto-ballooning has been disabled by default and Dom0 is given 10% of host physical memory + 1 GB. For example, on a host with 32 GB of physical memory, 4.2 GB of memory is allocated for Dom0.

The use of the dom0_mem Xen command line option in /etc/default/grub is still supported and encouraged. You can restore the old behavior by setting dom0_mem to the host physical memory size and enabling the autoballoon setting in /etc/xen/xl.conf.

Warning: Insufficient memory for Dom0

The amount of memory reserved for Dom0 is a function of the number of VM Guests running on the host since Dom0 provides back-end network and disk I/O services for each VM Guest. Other workloads running in Dom0 should also be considered when calculating Dom0 memory allocation. Memory sizing of Dom0 should be determined like any other virtual machine.

23.2.1 Setting Dom0 memory allocation #

Determine memory allocation required for Dom0.
At Dom0, type xl info to view the amount of memory that is available on the machine. Dom0's current memory allocation can be determined with the xl list command.
Edit /etc/default/grub and adjust the GRUB_CMDLINE_XEN option so that it includes dom0_mem=MEM_AMOUNT. Replace MEM_AMOUNT with the maximum amount of memory to allocate to Dom0. Add K, M, or G, to specify the size unit. For example:
```
GRUB_CMDLINE_XEN="dom0_mem=2G"
```
Restart the computer to apply the changes.

Tip

Refer to Section 14.2.2, “The file /etc/default/grub” for more details about Xen-related boot configuration options.

Warning: Xen Dom0 memory

When using the XL tool stack and the dom0_mem= option for the Xen hypervisor in GRUB 2 you need to disable xl autoballoon in etc/xen/xl.conf. Otherwise launching VMs will fail with errors about not being able to balloon down Dom0. So add autoballoon=0 to xl.conf if you have the dom0_mem= option specified for Xen. Also see Xen dom0 memory

23.3 Network card in fully virtualized guests #

In a fully virtualized guest, the default network card is an emulated Realtek network card. However, it also possible to use the split network driver to run the communication between Dom0 and a VM Guest. By default, both interfaces are presented to the VM Guest, because the drivers of some operating systems require both to be present.

When using SUSE Linux Enterprise Server, only the paravirtualized network cards are available for the VM Guest by default. The following network options are available:

emulated

To use an emulated network interface like an emulated Realtek card, specify type=ioemu in the vif device section of the domain xl configuration. An example configuration would look like:

vif = [ 'type=ioemu,mac=00:16:3e:5f:48:e4,bridge=br0' ]

Find more details about the xl configuration in the xl.conf man page man 5 xl.conf.

paravirtualized

When you specify type=vif and do not specify a model or type, the paravirtualized network interface is used:

vif = [ 'type=vif,mac=00:16:3e:5f:48:e4,bridge=br0,backen=0' ]

emulated and paravirtualized

If the administrator should be offered both options, simply specify both type and model. The xl configuration would look like:

vif = [ 'type=ioemu,mac=00:16:3e:5f:48:e4,model=rtl8139,bridge=br0' ]

In this case, one of the network interfaces should be disabled on the VM Guest.

23.4 Starting the virtual machine host #

If virtualization software is correctly installed, the computer boots to display the GRUB 2 boot loader with a Xen option on the menu. Select this option to start the virtual machine host.

Warning

When booting a Xen system, you might observe error messages in the /var/log/messages log file or systemd journal of dom0 similar to following:

isst_if_mbox_pci: probe of 0000:ff:1e.1 failed with error -5
isst_if_pci: probe of 0000:fe:00.1 failed with error -5

Ignore them as they are harmless and are caused by the fact that the ISST driver does not provide any power or frequency scaling feature for virtual machines.

Note: Xen and Kdump

In Xen, the hypervisor manages the memory resource. If you need to reserve system memory for a recovery kernel in Dom0, this memory needs to be reserved by the hypervisor. Thus, it is necessary to add crashkernel=size to the GRUB_CMDLINE_XEN_DEFAULT variable in the /etc/dfault/grub file, save it and run the following command:

> sudo grub2-mkconfig -o /boot/grub2/grub.cfg

For more information on the crashkernel parameter, see Section 20.4, “Calculating crashkernel allocation size”.

If the Xen option is not on the GRUB 2 menu, review the steps for installation and verify that the GRUB 2 boot loader has been updated. If the installation has been done without selecting the Xen pattern, run the YaST Software Management, select the filter Patterns and choose Xen Virtual Machine Host Server for installation.

After booting the hypervisor, the Dom0 virtual machine starts and displays its graphical desktop environment. If you did not install a graphical desktop, the command line environment appears.

Tip: Graphics problems

Sometimes it may happen that the graphics system does not work properly. In this case, add vga=ask to the boot parameters. To activate permanent settings, use vga=mode-0x??? where ??? is calculated as 0x100 + VESA mode from https://en.wikipedia.org/wiki/VESA_BIOS_Extensions, for example, vga=mode-0x361.

Before starting to install virtual guests, make sure that the system time is correct. To do this, configure NTP (Network Time Protocol) on the controlling domain:

In YaST select Network Services › NTP Configuration.
Select the option to automatically start the NTP daemon during boot. Provide the IP address of an existing NTP time server, then click Finish.

Note: Time services on virtual guests

Hardware clocks commonly are not very precise. All modern operating systems try to correct the system time compared to the hardware time by means of an additional time source. To get the correct time on all VM Guest systems, also activate the network time services on each respective guest or make sure that the guest uses the system time of the host. For more about Independent Wallclocks in SUSE Linux Enterprise Server see Section 19.2, “Xen virtual machine clock settings”.

For more information about managing virtual machines, see Chapter 25, Managing a virtualization environment.

23.5 PCI Pass-Through #

To take full advantage of VM Guest systems, it is sometimes necessary to assign specific PCI devices to a dedicated domain. When using fully virtualized guests, this functionality is only available if the chipset of the system supports this feature, and if it is activated from the BIOS.

This feature is available from both AMD* and Intel*. For AMD machines, the feature is called IOMMU; in Intel speak, this is VT-d. Note that Intel-VT technology is not sufficient to use this feature for fully virtualized guests. To make sure that your computer supports this feature, ask your supplier specifically to deliver a system that supports PCI Pass-Through.

Limitations #

Certain graphics drivers use highly optimized ways to access DMA. This is not supported, and thus using graphics cards may be difficult.
When accessing PCI devices behind a PCIe bridge, all the PCI devices must be assigned to a single guest. This limitation does not apply to PCIe devices.
Guests with dedicated PCI devices cannot be migrated live to a different host.

The configuration of PCI Pass-Through is twofold. First, the hypervisor must be informed at boot time that a PCI device should be available for reassigning. Second, the PCI device must be assigned to the VM Guest.

23.5.1 Configuring the hypervisor for PCI Pass-Through #

Select a device to reassign to a VM Guest. To do this, run lspci -k, and read the device number and the name of the original module that is assigned to the device:
```
06:01.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05)
        Subsystem: Dell Device 0617
        Kernel driver in use: e1000e
        Kernel modules: e1000e
```
In this case, the PCI number is (06:01.0) and the dependent kernel module is e1000e.
Specify a module dependency to ensure that xen_pciback is the first module to control the device. Add the file named /etc/modprobe.d/50-e1000e.conf with the following content:
```
install e1000e /sbin/modprobe xen_pciback ; /sbin/modprobe \
 --first-time --ignore-install e1000e
```
Instruct the xen_pciback module to control the device using the 'hide' option. Edit or create /etc/modprobe.d/50-xen-pciback.conf with the following content:
```
options xen_pciback hide=(06:01.0)
```
Reboot the system.
Check if the device is in the list of assignable devices with the command
```
xl pci-assignable-list
```

23.5.1.1 Dynamic assignment with xl #

To avoid restarting the host system, you can use dynamic assignment with xl to use PCI Pass-Through.

Begin by making sure that dom0 has the pciback module loaded:

> sudo modprobe pciback

Then make a device assignable by using xl pci-assignable-add. For example, to make the device 06:01.0 available for guests, run the command:

> sudo xl pci-assignable-add 06:01.0

23.5.2 Assigning PCI devices to VM Guest systems #

There are several possibilities to dedicate a PCI device to a VM Guest:

Adding the device while installing:

During installation, add the pci line to the configuration file:

pci=['06:01.0']

Hotplugging PCI devices to VM Guest systems

The command xl can be used to add or remove PCI devices on the fly. To add the device with number 06:01.0 to a guest with name sles12 use:

xl pci-attach sles12 06:01.0

Adding the PCI device to Xend

To add the device to the guest permanently, add the following snippet to the guest configuration file:

pci = [ '06:01.0,power_mgmt=1,permissive=1' ]

After assigning the PCI device to the VM Guest, the guest system must care for the configuration and device drivers for this device.

23.5.3 VGA Pass-Through #

Xen 4.0 and newer supports VGA graphics adapter pass-through on fully virtualized VM Guests. The guest can take full control of the graphics adapter with high-performance full 3D and video acceleration.

Limitations #

VGA Pass-Through functionality is similar to PCI Pass-Through and as such also requires IOMMU (or Intel VT-d) support from the mainboard chipset and BIOS.
Only the primary graphics adapter (the one that is used when you power on the computer) can be used with VGA Pass-Through.
VGA Pass-Through is supported only for fully virtualized guests. Paravirtual guests (PV) are not supported.
The graphics card cannot be shared between multiple VM Guests using VGA Pass-Through—you can dedicate it to one guest only.

To enable VGA Pass-Through, add the following settings to your fully virtualized guest configuration file:

gfx_passthru=1
pci=['yy:zz.n']

where yy:zz.n is the PCI controller ID of the VGA graphics adapter as found with lspci -v on Dom0.

23.5.4 Troubleshooting #

In some circumstances, problems may occur during the installation of the VM Guest. This section describes some known problems and their solutions.

During boot, the system hangs

The software I/O translation buffer allocates a large chunk of low memory early in the bootstrap process. If the requests for memory exceed the size of the buffer it usually results in a hung boot process. To check if this is the case, switch to console 10 and check the output there for a message similar to

kernel: PCI-DMA: Out of SW-IOMMU space for 32768 bytes at device 000:01:02.0

In this case, you need to increase the size of the swiotlb. Add swiotlb=VALUE (where VALUE is specified as the number of slab entries) on the command line of Dom0. The number can be adjusted up or down to find the optimal size for the machine.

Note: swiotlb a PV guest

The swiotlb=force kernel parameter is required for DMA access to work for PCI devices on a PV guest. For more information about IOMMU and the swiotlb option see the file boot-options.txt from the package kernel-source.

23.5.5 More information #

There are several resources on the Internet that provide interesting information about PCI Pass-Through:

23.6 USB pass-through #

There are two methods for passing through individual host USB devices to a guest. The first is via an emulated USB device controller, the second is using PVUSB.

23.6.1 Identify the USB device #

Before you can pass through a USB device to the VM Guest, you need to identify it on the VM Host Server. Use the lsusb command to list the USB devices on the host system:

# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 0461:4d15 Primax Electronics, Ltd Dell Optical Mouse
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

To pass through the Dell mouse, for example, specify either the device tag in the form vendor_id:device_id (0461:4d15) or the bus address in the form bus.device (2.3). Remember to remove leading zeros, otherwise xl would interpret the numbers as octal values.

23.6.2 Emulated USB device #

In emulated USB, the device model (QEMU) presents an emulated USB controller to the guest. The USB device is then controlled from Dom0 while USB commands are translated between the VM Guest and the host USB device. This method is only available to fully virtualized domains (HVM).

Enable the emulated USB hub with the usb=1 option. Then specify devices among the list of devices in the configuration file along with other emulated devices by using host:USBID. For example:

usb=1
usbdevice=['tablet','host:2.3','host:0424:460']

23.6.3 Paravirtualized PVUSB #

PVUSB is a new high performance method for USB Pass-Through from dom0 to the virtualized guests. With PVUSB, there are two ways to add USB devices to a guest:

via the configuration file at domain creation time
via hotplug while the VM is running

PVUSB uses paravirtualized front- and back-end interfaces. PVUSB supports USB 1.1 and USB 2.0, and it works for both PV and HVM guests. To use PVUSB, you need usbfront in your guest OS, and usbback in dom0 or usb back-end in qemu. On SUSE Linux Enterprise Server, the USB back-end comes with qemu.

As of Xen 4.7, xl PVUSB support and hotplug support is introduced.

In the configuration file, specify USB controllers and USB host devices with usbctrl and usbdev. For example, in case of HVM guests:

usbctrl=['type=qusb,version=2,ports=4', 'type=qusb,version=1,ports=4', ]
usbdev=['hostbus=2, hostaddr=1, controller=0,port=1', ]

Note

It is important to specify type=qusb for the controller of HVM guests.

To manage hotpluggin PVUSB devices, use the usbctrl-attach, usbctrl-detach, usb-list, usbdev-attach and usb-detach subcommands. For example:

Create a USB controller which is version USB 1.1 and has 8 ports:

# xl usbctrl-attach test_vm version=1 ports=8 type=qusb

Find the first available controller:port in the domain, and attach USB device whose busnum:devnum is 2:3 to it; you can also specify controller and port:

# xl usbdev-attach test_vm hostbus=2 hostaddr=3

Show all USB controllers and USB devices in the domain:

# xl usb-list test_vm
Devid  Type   BE  state usb-ver ports
0      qusb   0   1     1       8
  Port 1: Bus 002 Device 003
  Port 2:
  Port 3:
  Port 4:
  Port 5:
  Port 6:
  Port 7:
  Port 8:

Detach the USB device under controller 0 port 1:

# xl usbdev-detach test_vm 0 1

Remove the USB controller with the indicated dev_id, and all USB devices under it:

# xl usbctrl-detach test_vm dev_id

For more information, see https://wiki.xenproject.org/wiki/Xen_USB_Passthrough.