Using KubeVirt on SUSE Linux Enterprise

KubeVirt consists of two RPM-based packages and six container images that provide the Kubernetes virtual machine management extension. The RPM packages include kubevirt-virtctl and kubevirt-manifests. The container images include virt-api, virt-controller, virt-handler, virt-launcher, and virt-operator, libguestfs-tools.

kubevirt-virtctl can be installed on any machine with administrator access to the cluster. It contains the virtctl tool, which provides syntactic sugar on top of the kubectl tool for virtual machine resources. Although the kubectl tool can be used to manage virtual machines, it is a bit awkward since, unlike standard Kubernetes resources, virtual machines maintain state. Migration is also unique to virtual machines. If a standard Kubernetes resource needs to be evacuated from a cluster node, it is destroyed and started again on an alternate node. Since virtual machines are stateful, they cannot be destroyed and must be live-migrated away if a node is under evacuation. The virtctl tool abstracts the complexity of managing virtual machines with kubectl. It can be used to stop, start, pause, unpause and migrate virtual machines. virtclt also provides access to the virtual machine's serial console and graphics server.

kubevirt-manifests contains the manifests, or recipes, for installing KubeVirt. The most interesting files are /usr/share/kube-virt/manifests/release/kubevirt-cr.yaml and /usr/share/kube-virt/manifests/release/kubevirt-operator.yaml. kubevirt-cr.yaml contains the KubeVirt Custom Resource definition that represents the KubeVirt service. kubevirt-operator.yaml is the recipe for deploying the KubeVirt operator, which deploys the KubeVirt service to the cluster and manages its' lifecycle.

virt-api is a cluster component that provides the Kubernetes API extension for virtual machine resources. Like virt-api, virt-controller is a cluster component that watches for new objects created via virt-api, or updates to existing objects, and takes action to ensure the object state matches the requested state. virt-handler is a DaemonSet and a node component that has the job of keeping the cluster-level virtual machine object in sync with the libvirtd domain running in virt-launcher. virt-handler can also perform node-centric operations like configuring networking and/or storage on the node per the virtual machine specification. virt-launcher is also a node component and has the job of running libvirt plus qemu to provide the virtual machine environment. virt-launcher is a lowly pod resource. libguestfs-tools is a component providing a set of utilities for accessing and modifying VM disk images.

virt-operator implements the Kubernetes operator pattern. Operators encode the human knowledge required to deploy, run and maintain an application. Operators are a Kubernetes Deployment resource type and are often used to manage the custom resources and custom controllers that together provide a more complex Kubernetes application such as KubeVirt.

KubeVirt can be installed on a Kubernetes cluster by installing the kubevirt-manifests package on an admin node, applying the virt-operator manifest, and creating the KubeVirt custom resource. For example, on a cluster admin node execute the following:

> sudo zypper install kubevirt-manifests
> kubectl apply -f /usr/share/kube-virt/manifests/release/kubevirt-operator.yaml
> kubectl apply -f /usr/share/kube-virt/manifests/release/kubevirt-cr.yaml

After creating the KubeVirt custom resource, virt-operator deploys the remaining KubeVirt components. Progress can be monitored by viewing the status of the resources in the kubevirt namespace:

> kubectl get all -n kubevirt

The cluster is ready to deploy virtual machines once virt-api, virt-controller, and virt-handler are READY with STATUS “Running”.

Alternatively it is possible to wait until KubeVirt custom resource becomes available:

> kubectl -n kubevirt wait kv kubevirt --for condition=Available

Some KubeVirt functionality is disabled by default and must be enabled via feature gates. For example, live migration and the use of HostDisk for virtual machine disk images are disabled. Enabling KubeVirt feature gates can be done by altering an existing KubeVirt custom resource and specifying the list of features to enable. For example, you can enable live migration and the use of HostDisks:

> kubectl edit kubevirt kubevirt -n kubevirt
    ...
    spec:
      configuration:
        developerConfiguration:
          featureGates:
            - HostDisk
            - LiveMigration

Note

The names of feature gates are case-sensitive.

Updating KubeVirt is similar to the initial installation. The updated operator manifest from the kubevirt-manifests package is applied to the cluster.

> sudo zypper update kubevirt-manifests
> kubectl apply -f /usr/share/kube-virt/manifests/release/kubevirt-operator.yaml

KubeVirt can be deleted from a cluster by deleting the custom resource and operator:

> kubectl delete -n kubevirt kubevirt kubevirt # or alternatively: kubectl delete -f /usr/share/kube-virt/manifests/release/kubevirt-cr.yaml
> kubectl delete -f /usr/share/kube-virt/manifests/release/kubevirt-operator.yaml

Note

It is important to delete the custom resource first otherwise it gets stuck in the Terminating state. To fix that the resource finalizer needs to be manually deleted:

> kubectl -n kubevirt patch kv kubevirt --type=json -p '[{ "op": "remove", "path": "/metadata/finalizers" }]'

After deleting the resources from Kubernetes cluster the installed KubeVirt RPMs can be removed from the system:

> sudo zypper rm kubevirt-manifests kubevirt-virtctl

Containerized Data Importer (CDI) is an add-on for Kubernetes focused on persistent storage management. It is primarily used for building and importing Virtual Machine Disks for KubeVirt.

> sudo zypper in containerized-data-importer-manifests
> kubectl apply -f /usr/share/cdi/manifests/release/cdi-operator.yaml
> kubectl apply -f /usr/share/cdi/manifests/release/cdi-cr.yaml

> sudo zypper update containerized-data-importer-manifests
> kubectl apply -f /usr/share/cdi/manifests/release/cdi-operator.yaml

> kubectl delete -f /usr/share/cdi/manifests/release/cdi-cr.yaml
> kubectl delete -f /usr/share/cdi/manifests/release/cdi-operator.yaml
> sudo zypper rm containerized-data-importer-manifests

Two of the most interesting custom resources provided by KubeVirt are VirtualMachine (VM) and VirtualMachineInstance (VMI). As the names imply, a VMI is a running instance of a VM. The lifecycle of a VMI can be managed independently from a VM, but long-lived, stateful virtual machines are managed as a VM. The VM is deployed to the cluster in a shutoff state, then activated by changing the desired state to running. Changing a VM resource state can be done with the standard Kubernetes client tool kubectl or with the client virtctl provided by KubeVirt.

The VM and VMI custom resources make up part of the KubeVirt API. To create a virtual machine, a VM or VMI manifest must be created that adheres to the API. The API supports setting a wide variety of the common virtual machine attributes, for example, model of vCPU, number of vCPUs, amount of memory, disks, network ports, etc. Below is a simple example of a VMI manifest for a virtual machine with one Nehalem CPU, 2G of memory, one disk, and one network interface:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  labels:
    special: vmi-host-disk
  name: sles15sp2
spec:
  domain:
    cpu:
      model: Nehalem-IBRS
    devices:
      disks:
      - disk:
          bus: virtio
        name: host-disk
      interfaces:
        - name: green
          masquerade: {}
          ports:
            - port: 80
    machine:
      type: ""
    resources:
      requests:
        memory: 2048M
  terminationGracePeriodSeconds: 0
  networks:
  - name: green
    pod: {}
  volumes:
  - hostDisk:
      path: /hostDisks/sles15sp2/disk.raw
      type: Disk
      shared: true
    name: host-disk

Applying this VMI manifest to the cluster creates a virt-launcher container running libvirt and qemu, providing the familiar KVM virtual machine environment.

> kubectl apply -f sles15sp2vmi.yaml
> kubectl get vmis

Similar to other Kubernetes resources, VMs and VMIs can be managed with the kubectl client tool. Any kubectl operation that works with resource types works with the KubeVirt custom resources, for example, describe, delete, get, log, patch, etc. VM resources are a bit more awkward to manage with kubectl. Since a VM resource can be in a shutoff state, turning it on requires patching the manifest to change the desired state to running. Find an example below:

> kubectl patch vm sles15sp2 --type merge -p '{"spec":{"running":true}}'

The virtctl tool included in the kubevirt-virtclt package provides syntactic sugar on top of kubectl for VM and VMI resources, allowing them to be stopped, started, paused, unpaused and migrated. virtctl also provides access to the virtual machine's serial console and graphics server. Find an example below:

> virtctl start VM
> virtctl console VMI
> virtctl stop VM
> virtctl pause VM|VMI
> virtctl unpause VM|VMI
> virtctl vnc VMI
> virtctl migrate VM

KubeVirt supports live migration of VMs. Though this functionality must first be activated by adding LiveMigration to the list of feature gates in the KubeVirt custom resource.

> kubectl edit kubevirt kubevirt -n kubevirt

spec:
  configuration:
    developerConfiguration:
      featureGates:
        - LiveMigration

> kubectl describe vmi sles15sp2

Status:
  Conditions:
    Status: True
    Type: LiveMigratable
  Migration Method: BlockMigration

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
  name: migration-job
spec:
  vmiName: sles15sp2

> kubectl apply -f migration-job.yaml

> virtctl migrate VM

> kubectl delete VirtualMachineInstanceMigration migration-job

KubeVirt allows hotplugging additional storage into a running VM. Both block and file system volume types are supported. The hotplug volumes feature can be activated via the HotplugVolumes feature gate:

> kubectl edit kubevirt kubevirt -n kubevirt

spec:
  configuration:
    developerConfiguration:
      featureGates:
        - HotplugVolumes

Assuming that hp-volume is an existing DataVolume or PVC, virtctl can be used to operate with the volume on a runnig VM:

> virtctl addvolume sles15sp2 --volume-name=hp-volume
> virtctl removevolume sles15sp2 --volume-name=hp-volume

The VMDP ISO is provided in the form of a container image which can be consumed by KubeVirt. To run a Windows VM with VMDP ISO attached, the corresponding containerDisk needs to be added to the VM definition:

spec:
  domain:
    devices:
      disks:
        - name: vmdp
          cdrom:
            bus: sata
volumes:
  - containerDisk:
      image: registry.suse.com/suse/vmdp/vmdp:latest
    name: vmdp

Note

The sequence in which the disks are defined affects the boot order. It is possible to specify the bootOrder explicitly or otherwise sort the disk items as needed.

If issues are encountered the following debug resources are available to help identify the problem.

The status of all KubeVirt resources can be examined with the kubectl get command:

> kubectl get all -n kubevirt

Resources with failed status can be further queried by examining their definition and expanded status information.

> kubectl describe deployment virt-operator
> kubectl get deployment virt-operator -o yaml -n kubevirt
> kubectl describe pod virt-handler-xbjkg -n kubevirt
> kubectl get pod virt-handler-xbjkg -o yaml -n kubevirt

Logs from the problematic KubeVirt pod can contain a wealth of information since stderr and service logging from within the pod is generally available via the Kubernetes log service:

> kubectl logs virt-operator-558c57bc4-mg68w -n kubevirt
  > kubectl logs virt-handler-xbjkg -n kubevirt

If the underlying pod is running but there are problems with the service running in it, a shell can be accessed to inspect the pod environment and poke at its service:

> kubectl -n kubevirt exec -it virt-handler-xbjkg -- /bin/bash