C XM, XL Toolstacks and Libvirt framework #
C.1 Xen Toolstacks #
Since the early Xen 2.x releases, xend
has been
the de facto toolstack for managing Xen installations. In Xen
4.1, a new toolstack called libxenlight (also known as libxl) was
introduced with technology preview status. libxl is a small, low-level
library written in C. It has been designed to provide a simple API for
all client toolstacks
(XAPI,
libvirt
, xl). In Xen 4.2, libxl was promoted to supported status
and xend
was marked deprecated.
xend
has been included in the Xen 4.3 and 4.4
series to give users enough time to convert their tooling to libxl. It
has been removed from the upstream Xen project and will no longer be
provided starting with the Xen 4.5 series and SUSE Linux Enterprise Server
12 SP1.
Although SLES 11 SP3 contains Xen 4.2, SUSE
retained the xend
toolstack since making such an invasive
change in a service pack would be too disruptive for SUSE Linux Enterprise
customers. However, SLES 12 provides a suitable opportunity to move to the
new libxl toolstack and remove the deprecated, unmaintained
xend
stack. Starting with SUSE Linux Enterprise Server 12 SP1,
xend
is no longer supported.
One of the major differences between xend
and libxl is
that the former is stateful, while the latter is stateless. With
xend
, all client applications such as
xm
and libvirt
see the same system state.
xend
is responsible for maintaining state for the
entire Xen host. In libxl, client applications such as
xl
or libvirt
must maintain state. Thus domains
created with xl
or not visible or known to other libxl
applications such as libvirt
. Generally, it is discouraged to mix
and match libxl applications and is preferred that a single libxl
application be used to manage a Xen host. In SUSE Linux Enterprise Server, we
recommend to use libvirt
to manage Xen hosts. This allows
management of the Xen system through libvirt
applications such
as virt-manager
, virt-install
,
virt-viewer
,
libguestfs, etc. If xl
is used to manage the Xen
host, any virtual machines under its management will not be accessible
to libvirt
. Hence, they are not accessible to any of the libvirt
applications.
C.1.1 Upgrading from xend/xm to xl/libxl #
The xl
application, along with its configuration
format (see man xl.cfg
), was designed to be
backward-compatible with the xm
application and its
configuration format (see man xm.cfg
). Existing
xm
configuration should be usable with
xl
. Since libxl is stateless, and
xl
does not support the notion of managed domains,
SUSE recommends using libvirt
to manage Xen hosts.
SUSE has provided a tool called xen2libvirt
, which
provides a simple mechanism to import domains previously managed by
xend
into libvirt
. See
Section C.2, “Import Xen Domain Configuration into libvirt
” for more information on
xen2libvirt
.
C.1.2 XL design #
The basic structure of every xl
command is:
xl subcommand
OPTIONS
DOMAIN
DOMAIN is the numeric domain id, or the domain name (which will be internally translated to domain id), and OPTIONS are subcommand specific options.
Although xl/libxl was designed to be backward-compatible with xm/xend, there are a few differences that should be noted:
Managed or persistent domains.
libvirt
now provides this functionality.xl/libxl does not support Python code in the domain configuration files.
xl/libxl does not support creating domains from SXP format configuration files (
xm
create
-F).xl/libxl does not support sharing storage across DomU's via
w!
in domain configuration files.
xl/libxl is relatively new and under heavy development, hence a few features are still missing with regard to the xm/xend toolstack:
SCSI LUN/Host pass-through (PVSCSI)
USB pass-through (PVUSB)
Direct Kernel Boot for fully virtualized Linux guests for Xen
C.1.3 Checklist before Upgrade #
Before upgrading a SLES 11 SP4 Xen host to SLES 15:
You must remove any Python code from your xm domain configuration files.
It is recommended to capture the libvirt domain XML from all existing virtual machines using
virsh
dumpxml
DOMAIN_NAME DOMAIN_NAME.xml.It is recommended to do a backup of
/etc/xen/xend-config.sxp
and/boot/grub/menu.lst
files to keep references of previous parameters used for Xen.
Currently, live migrating virtual machines running on a SLES 11 SP4
Xen host to a SLES 15 Xen host is not supported. The
xend
and libxl toolstacks are not
runtime-compatible. Virtual machine downtime will be required to move
the virtual machines.
C.2 Import Xen Domain Configuration into libvirt
#
xen2libvirt
is a command line tool to import legacy
Xen domain configuration into the libvirt
virtualization library
(see The Virtualization book for more information on libvirt
).
xen2libvirt provides an easy way to import domains managed by the
deprecated xm
/xend tool stack into the new
libvirt
/libxl tool stack. Several domains can be imported at once
using its --recursive mode
xen2libvirt
is included in the
xen-tools
package. If needed, install it with
>
sudo
zypper install xen-tools
xen2libvirt
general syntax is
xen2libvirt <options> /path/to/domain/config
where options
can be:
-h
,--help
Prints short information about
xen2libvirt
usage.-c
,--convert-only
Converts the domain configuration to the
libvirt
XML format, but does not do the import tolibvirt
.-r
,--recursive
Converts and/or imports all domains configuration recursively, starting at the specified path.
-f
,--format
Specifies the format of the source domain configuration. Can be either
xm
, orsexpr
(S-expression format).-v
,--verbose
Prints more detailed information about the import process.
libvirt
#
Suppose you have a Xen domain managed with xm
with the following configuration saved in
/etc/xen/sle12.xm
:
kernel = "/boot/vmlinuz-2.6-xenU" memory = 128 name = "SLE12" root = "/dev/hda1 ro" disk = [ "file:/var/xen/sle12.img,hda1,w" ]
Convert it to libvirt
XML without importing it, and look at its
content:
>
sudo
xen2libvirt -f xm -c /etc/xen/sle12.xm > /etc/libvirt/qemu/sles12.xml # cat /etc/libvirt/qemu/sles12.xml <domain type='xen'> <name>SLE12</name> <uuid>43e1863c-8116-469c-a253-83d8be09aa1d</uuid> <memory unit='KiB'>131072</memory> <currentMemory unit='KiB'>131072</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='xenpv'>linux</type> <kernel>/boot/vmlinuz-2.6-xenU</kernel> </os> <clock offset='utc' adjustment='reset'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <disk type='file' device='disk'> <driver name='file'/> <source file='/var/xen/sle12.img'/> <target dev='hda1' bus='xen'/> </disk> <console type='pty'> <target type='xen' port='0'/> </console> </devices> </domain>
To import the domain into libvirt
, you can either run the same
xen2libvirt
command without the -c
option, or use the exported file
/etc/libvirt/qemu/sles12.xml
and define a new
Xen domain using virsh
:
>
sudo
virsh define /etc/libvirt/qemu/sles12.xml
C.3 Differences between the xm
and xl
Applications #
The purpose of this chapter is to list all differences between
xm
and xl
applications. Generally,
xl
is designed to be compatible with
xm
. Replacing xm
with
xl
in custom scripts or tools is usually sufficient.
You can also use the libvirt
framework using the
virsh
command. In this documentation only the first
OPTION for virsh
will be
shown. To get more help on this option do a:
virsh
help
OPTION
C.3.1 Notation Conventions #
To easily understand the difference between xl
and
xm
commands, the following notation is used in this
section:
Notation |
Meaning |
---|---|
(-) minus |
Option exists in |
(+) plus |
Option exists in |
C.3.2 New Global Options #
Options |
Task |
---|---|
(+) |
Verbose, increase the verbosity of the output |
(+) |
Dry run, do not actually execute the command |
(+) |
Force execution. |
C.3.3 Unchanged Options #
List of common options of xl
and
xm
, and their libvirt
equivalents.
Options |
Task |
|
---|---|---|
destroy DOMAIN |
Immediately terminate the domain. |
|
domid DOMAIN_NAME |
Convert a domain name to a DOMAIN_ID. |
|
domname DOMAIN_ID |
Convert a DOMAIN_ID to a DOMAIN_NAME. |
|
help |
Display the short help message (that is, common commands). |
|
pause DOMAIN_ID |
Pause a domain. When in a paused state, the domain will still consume allocated resources such as memory, but will not be eligible for scheduling by the Xen hypervisor. |
|
unpause DOMAIN_ID |
Move a domain out of the paused state. This will allow a previously paused domain to be eligible for scheduling by the Xen hypervisor. |
|
rename DOMAIN_ID NEW_DOMAIN_NAME |
Change the domain name of DOMAIN_ID to NEW_DOMAIN_NAME. |
|
sysrq DOMAIN <letter> |
Send a Magic System Request to the domain, each type of request is represented by a different letter. It can be used to send SysRq requests to Linux guests, see https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html for more information. It requires PV drivers to be installed in your guest OS. |
|
vncviewer OPTIONS DOMAIN |
Attach to domain's VNC server, forking a
|
|
|
Set the number of virtual CPUs for the domain in question. Like
|
|
vcpu-list DOMAIN_ID |
List VCPU information for a specific domain. If no domain is specified, VCPU information for all domains will be provided. |
|
vcpu-pin DOMAIN_ID <VCPU|all> <CPUs|all> |
Pin the VCPU to only run on the specific CPUs. The keyword all can be used to apply the CPU list to all VCPUs in the domain. |
|
|
Read the Xen message buffer, similar to dmesg on a Linux system. The buffer contains informational, warning, and error messages created during Xen's boot process. | |
|
Execute the |
|
|
Print the current uptime of the domains running. With the
| |
|
Send debug keys to Xen. It is the same as pressing the Xen conswitch (Ctrl-A by default) three times and then pressing "keys". | |
|
Move a domain specified by DOMAIN_ID or DOMAIN into a CPU_POOL. | |
|
Deactivate a cpu pool. This is possible only if no domain is active in the cpu-pool. | |
|
Detach a domain's virtual block device. devid
may be the symbolic name or the numeric device id given to the
device by Dom0. You will need to run |
|
|
Create a new network device in the domain specified by DOMAIN_ID. network-device describes the device to attach, using the same format as the vif string in the domain configuration file |
|
|
Hotplug a new pass-through PCI device to the specified domain. BDF is the PCI Bus/Device/Function of the physical device to be passed through. |
|
|
List pass-through PCI devices for a domain | |
|
Determine if the FLASK security module is loaded and enforcing its policy. | |
|
Enable or disable enforcing of the FLASK access controls. The default is permissive and can be changed using the flask_enforcing option on the hypervisor's command line. |
C.3.4 Removed Options #
List of xm
options
which are no more
available with the XL tool stack and a replacement solution if available.
C.3.4.1 Domain Management #
The list of Domain management removed command and their replacement.
Domain Management Removed Options | ||
---|---|---|
Options |
Task |
Equivalent |
(-) |
Print the Xend log. |
This log file can be found in
|
(-) |
Remove a domain from Xend domain management. The
|
|
(-) |
Adds a domain to Xend domain management |
|
(-) |
Start a Xend managed domain that was added using the
|
|
(-) |
Dry run - prints the resulting configuration in SXP but does not create the domain |
|
(-) |
Reset a domain |
|
(-) |
Show domain state |
|
(-) |
Proxy Xend XMLRPC over stdio | |
(-) |
Moves a domain out of the suspended state and back into memory |
|
(-) |
Suspend a domain to a state file so that it can be later resumed
using the |
|
C.3.4.2 USB Devices #
USB options
are not available with xl/libxl tool stack.
virsh
has the attach-device
and
detach-device
options but it does not work yet with
USB
.
USB Devices Management Removed Options | |
---|---|
Options |
Task |
(-) |
Add a new USB physical bus to a domain |
(-) |
Delete a USB physical bus from a domain |
(-) |
Attach a new USB physical bus to domain's virtual port |
(-) |
Detach a USB physical bus from domain's virtual port |
(-) |
List domain's attachment state of all virtual port |
(-) |
List all the assignable USB devices |
(-) |
Create a domain's new virtual USB host controller |
(-) |
Destroy a domain's virtual USB host controller |
C.3.4.3 CPU Management #
CPU management options has changed. New options are available, see:
Section C.3.5.10, “xl
cpupool-*
”
CPU Management Removed Options | |
---|---|
Options |
Task |
(-) |
Adds a CPU pool to Xend CPU pool management |
(-) |
Starts a Xend CPU pool |
(-) |
Removes a CPU pool from Xend management |
C.3.4.4 Other Options #
Other Removed Options | |
---|---|
Options |
Task |
(-) |
Launch an interactive shell |
(-) |
Change vnc password |
(-) |
List virtual TPM devices |
(-) |
Change block device configuration |
C.3.5 Changed Options #
C.3.5.1 create
#
xl
create
CONFIG_FILE OPTIONS
VARS
libvirt
Equivalent:
virsh
create
xl
create
Changed Options #
| |
---|---|
Options |
Task |
(*) -f=FILE, --defconfig=FILE |
Use the given configuration file |
xm
create
Removed Options #
| |
---|---|
Options |
Task |
(-) |
Skip DTD checking - skips checks on XML before creating |
(-) |
XML dry run |
(-) |
Use the given SXP formatted configuration script |
(-) |
Search path for configuration scripts |
(-) |
Print the available configuration variables (vars) for the configuration script |
(-) |
Dry run — prints the configuration in SXP but does not create the domain |
(-) |
Connect to the console after the domain is created |
(-) |
Quiet mode |
(-) |
Leave the domain paused after it is created |
xl
create
Added Options #
| |
---|---|
Options |
Task |
(+) |
Attach to domain's VNC server, forking a vncviewer process |
(+) |
Pass VNC password to vncviewer via stdin |
C.3.5.2 console
#
xl
console
OPTIONS DOMAIN
libvirt
Equivalent
virsh
console
xl
console
Added Options #
| |
---|---|
Option |
Task |
(+) |
Connect to a PV console or connect to an emulated serial console. PV consoles are the only consoles available for PV domains while HVM domains can have both |
C.3.5.3 info #
xl
info
xm
info
Removed Options #
| |
---|---|
Options |
Task |
(-) |
Numa info |
(-) |
List Xend configuration parameters |
C.3.5.4 dump-core
#
xl
dump-core
DOMAIN FILENAME
libvirt
Equivalent
virsh
dump
xm
dump-core
Removed Options #
| |
---|---|
Options |
Task |
(-) |
Dump core without pausing the domain |
(-) |
Crash domain after dumping core |
(-) |
Reset domain after dumping core |
C.3.5.5 list
#
xl list
options
DOMAIN
libvirt
Equivalent
virsh
list --all
xm
list
Removed Options #
| |
---|---|
Options |
Task |
(-) |
The output for |
(-) |
Output information for VMs in the specified state |
xl
list
Added Options #
| |
---|---|
Options |
Task |
(+) |
Also prints the security labels |
(+) |
Also prints the domain UUIDs, the shutdown reason and security labels |
C.3.5.6 mem-*
#
libvirt
Equivalent
virsh
setmem
virsh
setmaxmem
xl
mem-*
Changed Options #
| |
---|---|
Options |
Task |
|
Appending |
|
Set the domain's used memory using the balloon driver |
C.3.5.7 migrate
#
xl
migrate
OPTIONS DOMAIN
HOST
libvirt
Equivalent
virsh migrate --live hvm-sles11-qcow2 xen+
CONNECTOR://USER@IP_ADDRESS/
xm
migrate
Removed Options #
| |
---|---|
Options |
Task |
(-) |
Use live migration. This will migrate the domain between hosts without shutting down the domain |
(-) |
Set maximum Mbs allowed for migrating the domain |
(-) |
Change home server for managed domains |
(-)
|
Number of iterations before final suspend (default:30) |
(-)
|
Max amount of memory to transfer before final suspend (default: 3*RAM). |
(-)
|
Number of dirty pages before final suspend (default:50) |
(-) |
Abort migration instead of doing final suspend |
(-) |
Log progress of migration to |
(-) |
Use ssl connection for migration |
xl
migrate
Added Options #
| |
---|---|
Options |
Task |
(+) |
Use <sshcommand> instead of |
(+) |
On the new host, do not wait in the background (on <host>) for the death of the domain |
(+) |
Send <config> instead of the configuration file used when creating the domain |
C.3.5.8 Domain Management #
xl
reboot
OPTIONS DOMAIN
libvirt
Equivalent
virsh
reboot
xm
reboot
Removed Options #
| |
---|---|
Options |
Task |
(-) |
Reboot all domains |
(-) |
Wait for reboot to complete before returning. This may take a while, as all services in the domain need to be shut down cleanly |
xl
reboot
Added Options #
| |
---|---|
Option |
Task |
(+) |
Fallback to ACPI reset event for HVM guests with no PV drivers |
xl
save
OPTIONS DOMAIN
CHECK_POINT_FILE
CONFIG_FILE
libvirt
Equivalent
virsh
save
xl
save
Added Options #
| |
---|---|
Option |
Task |
(+) |
Leave domain running after creating the snapshot |
xl
restore
OPTIONS
CONFIG_FILE
CHECK_POINT_FILE
libvirt
Equivalent
virsh
restore
xl
restore
Added Options #
| |
---|---|
Options |
Task |
(+) |
Do not unpause domain after restoring it |
(+) |
Do not wait in the background for the death of the domain on the new host |
(+) |
Enable debug messages |
(+) |
Attach to domain's VNC server, forking a vncviewer process |
(+) |
Pass VNC password to vncviewer via stdin |
xl
shutdown
OPTIONS DOMAIN
libvirt
Equivalent
virsh
shutdown
xm
shutdown
Removed Options #
| |
---|---|
Options |
Task |
(-) |
Wait for the domain to complete shutdown before returning |
(-) |
Shutdown all guest domains |
(-) | |
(-) |
xl
shutdown
Added Options #
| |
---|---|
Option |
Task |
(+) |
If the guest does not support PV shutdown control then fallback to sending an ACPI power event |
xl
trigger
Changed Options #
| |
---|---|
Option |
Task |
|
Send a trigger to a domain. Only available for HVM domains |
C.3.5.9 xl
sched-*
#
xl
sched-credit
OPTIONS
libvirt
Equivalent
virsh
schedinfo
xm
sched-credit
Removed Options #
| |
---|---|
Options |
Task |
|
Domain |
|
A domain with a weight of 512 will get twice as much CPU as a domain with a weight of 256 on a contended host. Legal weights range from 1 to 65535 and the default is 256 |
|
The CAP optionally fixes the maximum amount of CPU a domain can consume |
xl
sched-credit
Added Options #
| |
---|---|
Options |
Task |
(+) |
Restrict output to domains in the specified cpupool |
(+) |
Specify to list or set pool-wide scheduler parameters |
(+) |
Timeslice tells the scheduler how long to allow VMs to run before pre-empting |
(+) |
Ratelimit attempts to limit the number of schedules per second |
xl
sched-credit2
OPTIONS
libvirt
Status
virsh
only supports credit scheduler, not credit2
scheduler
xm
sched-credit2
Removed Options #
| |
---|---|
Options |
Task |
|
Domain |
|
Legal weights range from 1 to 65535 and the default is 256 |
xl
sched-credit2
Added Options #
| |
---|---|
Option |
Task |
(+) |
Restrict output to domains in the specified cpupool |
xl
sched-sedf
OPTIONS
xm
sched-sedf
Removed Options #
| |
---|---|
Options |
Task |
|
The normal EDF scheduling usage in milliseconds |
|
The normal EDF scheduling usage in milliseconds |
|
Scaled period if domain is doing heavy I/O |
|
Flag for allowing domain to run in extra time (0 or 1) |
|
Another way of setting CPU slice |
xl
sched-sedf
Added Options #
| |
---|---|
Options |
Task |
(+) |
Restrict output to domains in the specified cpupool |
(+) |
Domain |
C.3.5.10 xl
cpupool-*
#
xl
cpupool-cpu-remove
CPU_POOL <CPU
nr>|node:<node nr>
xl
cpupool-list
[-c|--cpus]
CPU_POOL
xm
cpupool-list
Removed Options #
| |
---|---|
Option |
Task |
(-) |
Output all CPU pool details in SXP format |
xl
cpupool-cpu-add
CPU_POOL cpu-nr|node:node-nr
xl
cpupool-create
OPTIONS
CONFIG_FILE [Variable=Value ...]
xm
cpupool-create
Removed Options #
| |
---|---|
Options |
Task |
(-) |
Use the given Python configuration script. The configuration script is loaded after arguments have been processed |
(-) |
Dry run - prints the resulting configuration in SXP but does not create the CPU pool |
(-) |
Print the available configuration variables (vars) for the configuration script |
(-) |
Search path for configuration scripts. The value of PATH is a colon-separated directory list |
(-) |
CPU pool configuration to use (SXP) |
C.3.5.11 PCI and Block Devices #
xl
pci-detach
[-f]
DOMAIN_ID <BDF>
libvirt
Equivalent
virsh
detach-device
xl
pci-detach
Added Options #
| |
---|---|
Option |
Task |
(+) |
If |
xm
block-list
Removed Options #
| |
---|---|
Option |
Task |
(-) |
List virtual block devices for a domain |
Option |
|
---|---|
|
|
|
|
C.3.5.12 Network #
Option |
|
---|---|
|
|
|
|
|
|
xl
network-attach
Removed Options #
Removed Options | |
---|---|
Option |
Task |
(-) |
C.3.6 New Options #
Options |
Task |
---|---|
|
Update the saved configuration for a running domain. This has no immediate effect but will be applied when the guest is next restarted. This command is useful to ensure that runtime modifications made to the guest will be preserved when the guest is restarted |
| |
|
List count of shared pages.List specifically for that domain. Otherwise, list for all domains |
|
Prints information about guests. This list excludes information about service or auxiliary domains such as Dom0 and stubdoms |
|
Renames a cpu-pool to newname |
|
Splits up the machine into one cpu-pool per numa node |
cd-insert DOMAIN <VirtualDevice> <type:path> |
Insert a CD-ROM into a guest domain's existing virtual CD drive. The virtual drive must already exist but can be current empty |
|
Eject a CD-ROM from a guest's virtual CD drive. Only works with HVM domains |
|
List all the assignable PCI devices. These are devices in the system which are configured to be available for pass-through and are bound to a suitable PCI back-end driver in Dom0 rather than a real driver |
|
Make the device at PCI Bus/Device/Function BDF assignable to guests.This will bind the device to the pciback driver |
|
Make the device at PCI Bus/Device/Function BDF assignable to guests. This will at least unbind the device from pciback |
|
Load FLASK policy from the given policy file. The initial policy is provided to the hypervisor as a multiboot module; this command allows runtime updates to the policy. Loading new security policy will reset runtime changes to device labels |
C.4 External links #
For more information on Xen tool stacks refer to the following online resources:
- XL in Xen
xl
commandXL command line.
- xl.cfg
xl.cfg domain configuration file syntax.
- xl disk
xl disk configuration option.
- XL vs Xend
XL vs Xend feature comparison.
- BDF doc
- libvirt
virsh command.
C.5 Saving a Xen Guest Configuration in an xm
Compatible Format #
Although xl
is now the current toolkit for managing
Xen guests (apart from the preferred libvirt
), you may need to
export the guest configuration to the previously used
xm
format. To do this, follow these steps:
First export the guest configuration to a file:
>
virsh dumpxml guest_id > guest_cfg.xmlThen convert the configuration to the
xm
format:>
virsh domxml-to-native xen-xm guest_cfg.xml > guest_xm_cfg