Documentation survey

This is unreleased documentation for SUSE® Virtualization v1.6 (Dev).

Upgrades

SUSE Virtualization is adopting a new lifecycle strategy that simplifies version management and upgrades. This strategy includes the following:

  • Four-month minor release cadence

  • Two-month patch release cadence

  • Component adoption policy

SUSE Virtualization does not support downgrades. This restriction helps prevent unexpected system behavior and issues associated with function incompatibility, deprecation, and removal.

Upgrade paths

The following table outlines the supported upgrade paths.

Installed Version Supported Upgrade Versions

v1.4.2 and v1.4.3

v1.5.0 and v1.5.1

v1.4.1 and v1.4.2

v1.4.3

v1.4.1

v1.4.2

v1.4.0

v1.4.1

v1.3.1

v1.3.2

v1.2.2 and v1.3.0

v1.3.1

v1.2.1

v1.2.2

v1.1.2, v1.1.3, and v1.2.0

v1.2.1

SUSE Virtualization v1.5.x and later versions allow the following:

  • Upgrading from one minor version to the next (for example, from v1.4.2 to v1.5.1) without needing to install the patches released in between the two versions. This is possible because SUSE Virtualization allows a maximum of one minor version upgrade for underlying components.

  • Upgrading to a later patch version (for example, from v1.5.0 to v1.5.1), assuming that the same component versions are used across the releases for a given minor version.

The following table outlines the components used in these versions:

Component SUSE Virtualization v1.4.2 and v1.4.3 SUSE Virtualization v1.5.x

KubeVirt

v1.3

v1.4

SUSE Storage

v1.7

v1.8

SUSE Rancher Prime

v2.10

v2.11

RKE2

v1.31

v1.32

Skipping multiple Kubernetes minor versions is not supported upstream and is a key reason behind the limited upgrade paths. For more information, see Version Skew Policy in the Kubernetes documentation.

Rancher upgrade

If you are using Rancher to manage your SUSE Virtualization cluster, you must upgrade Rancher before upgrading SUSE Virtualization.

The SUSE Virtualization and Rancher upgrade processes are independent of each other. During a Rancher upgrade, you can still access your SUSE Virtualization cluster using its virtual IP. SUSE Virtualization is not automatically upgraded.

When a Rancher version reaches its End of Maintenance (EOM) date, SUSE Virtualization only provides fixes for critical security-related issues that affect integration functions (Virtualization Management). For more information, see the Support Matrix.

Before starting an upgrade

Check out the available upgrade-config setting to tweak the upgrade strategies and behaviors that best suit your cluster environment.

Start an upgrade

  • Before you upgrade your SUSE Virtualization cluster, we highly recommend:

    • Back up your VMs if needed.

  • Do not operate the cluster during an upgrade. For example, creating new VMs, uploading new images, etc.

  • Make sure your hardware meets the preferred hardware requirements. This is due to there will be intermediate resources consumed by an upgrade.

  • Make sure each node has at least 30 GiB of free system partition space (df -h /usr/local/). If any node in the cluster has less than 30 GiB of free system partition space, the upgrade will be denied. Check free system partition space requirement for more information.

  • Run the pre-check script on a SUSE Virtualization control-plane node. Please pick a script according to your cluster’s version: https://github.com/harvester/upgrade-helpers/tree/main/pre-check.

  • A number of one-off privileged pods will be created in the harvester-system and cattle-system namespaces to perform host-level upgrade operations. If pod security admission is enabled, adjust these policies to allow these pods to run.

  • Make sure all nodes' times are in sync. Using an NTP server to synchronize time is recommended. If an NTP server is not configured during the installation, you can manually add an NTP server on each node:

      $ sudo -i

  # Add time servers
  $ vim /etc/systemd/timesyncd.conf
  [ntp]
  NTP=0.pool.ntp.org

  # Enable and start the systemd-timesyncd
  $ timedatectl set-ntp true

  # Check status
  $ sudo timedatectl status

  • NICs that connect to a PCI bridge might be renamed after an upgrade. Please check the knowledge base article for further information.

  1. On the SUSE Virtualization UI Dashboard screen, click Upgrade.

    The Upgrade button appears whenever a new version that you can upgrade to becomes available.

    If your environment does not have direct internet access, follow the instructions in Prepare an air-gapped upgrade, which provides an efficient approach to downloading the ISO.

    upgrade button

  2. Select the version that you want to upgrade to.

    If you require customizations, see Customize the version.

    upgrade select version

  3. Click the progress indicator (circular icon) to view the status of each related process.

    upgrade progress

Customize the version

  1. Download the version file (https://releases.rancher.com/harvester/{version}/version.yaml).

    Example:

    The v1.5.0 version file is downloaded as v1.5.0.yaml.

    apiVersion: harvesterhci.io/v1beta1
kind: Version
metadata:
  name: v1.5.0-customized # Changed, to avoid duplicated with the official version name
  namespace: harvester-system
spec:
  isoChecksum: 'df28e9bf8dc561c5c26dee535046117906581296d633eb2988e4f68390a281b6856a5a0bd2e4b5b988c695a53d0fc86e4e3965f19957682b74317109b1d2fe32'  # Don't change
  isoURL: https://releases.rancher.com/harvester/v1.5.0/harvester-v1.5.0-amd64.iso # Official ISO path by default
  releaseDate: '20250425'

  2. Create the version using the command kubectl create -f v1.5.0.yaml.

Prepare an air-gapped upgrade

Make sure to check Upgrade paths section first about upgradable versions.

Prepare the ISO file

  1. Download an ISO file from the Releases page.

  2. Save the ISO to a local HTTP server.

    Assume the file is hosted at http://10.10.0.1/harvester.iso.

Prepare the Version

  1. Download the version file (https://releases.rancher.com/harvester/{version}/version.yaml).

  2. Replace the isoURL value in the file.

      apiVersion: harvesterhci.io/v1beta1
  kind: Version
  metadata:
    name: v1.5.0
    namespace: harvester-system
  spec:
    isoChecksum: <SHA-512 checksum of the ISO>
    isoURL: http://10.10.0.1/harvester.iso  # change to local ISO URL
    releaseDate: '20250425'

    Assume the file is hosted at http://10.10.0.1/version.yaml. If you require customizations, see Customize the version.

  3. Access one of the control plane nodes via SSH and log in using the root account.

  4. Create a version object.

    rancher@node1:~> sudo -i
rancher@node1:~> kubectl create -f http://10.10.0.1/version.yaml

Start the upgrade

The Upgrade button appears on the Dashboard screen whenever a new version that you can upgrade to becomes available. Refresh the screen if the button does not appear.

Manually start an upgrade before the official upgrade becomes available

The Upgrade button does not appear on the UI immediately after a new version is released. If you want to upgrade your cluster before the option becomes available on the UI, follow the steps in Prepare an air-gapped upgrade.

In production environments, upgrading clusters via the UI is recommended.

Free system partition space requirement

SUSE Virtualization loads images on each node during upgrades. When disk usage exceeds the kubelet’s garbage collection threshold, the kubelet deletes unused images to free up space. This may cause issues in air-gapped environments because the images are not available on the node.

SUSE Virtualization includes checks that ensure nodes do not trigger garbage collection after loading new images.

When disk space is insufficient, SUSE Virtualization blocks the upgrade and returns an error similar to the following:

Node "harvester-node-0" will reach 92.84% storage space after loading new images. It's higher than kubelet image garbage collection threshold 85%.

If you want to try upgrading even if the free system partition space is insufficient on some nodes, you can update the harvesterhci.io/skipGarbageCollectionThresholdCheck: true annotation of the Upgrade object.

apiVersion: harvesterhci.io/v1beta1
kind: Upgrade
metadata:
  annotations:
    harvesterhci.io/skipGarbageCollectionThresholdCheck: true
  generateName: hvst-upgrade-
  namespace: harvester-system
spec:
  version: "1.6.0"
  logEnabled: true

Setting a smaller value than the pre-defined value may cause the upgrade to fail and is not recommended in a production environment.

The following sections describe solutions for issues related to this requirement.

Free system partition space manually

SUSE Virtualization attempts to remove unnecessary container images after an upgrade is completed. However, this automatic image cleanup may not be performed for various reasons. You can use a script to manually remove images. For more information, see issue #6620.

Set up a private container registry and skip image preloading

The system partition might still lack free space even after you remove images. To address this, set up a private container registry for both current and new images, and configure the setting upgrade-config with following value:

{"imagePreloadOption":{"strategy":{"type":"skip"}}, "restoreVM": false}

SUSE Virtualization skips the upgrade image preloading process. When the deployments on the nodes are upgraded, the container runtime loads the images stored in the private container registry.

Do not rely on the public container registry. Note any potential internet service interruptions and how close you are to reaching your Docker Hub rate limit. Failure to download any of the required images may cause the upgrade to fail and may leave the cluster in a middle state.

Certificate expiration check

SUSE Virtualization checks the validity period of certificates on each node. This check eliminates the possibility of certificates expiring while the upgrade is in progress. If a certificate will expire within 7 days, an error is returned. This behavior can be overridden by setting the harvesterhci.io/minCertsExpirationInDay annotation.

Example:

apiVersion: harvesterhci.io/v1beta1
kind: Upgrade
metadata:
  annotations:
    harvesterhci.io/minCertsExpirationInDay: "14"
  generateName: hvst-upgrade-
  namespace: harvester-system
spec:
  version: "1.6.0"
  logEnabled: true

When this annotation is added to the Upgrade object, SUSE Virtualization returns an error when it detects a certificate that will expire within 14 days.

For more information, see auto-rotate-rke2-certs.

Virtual Machine Backup Compatibility

You may encounter certain limitations when creating and restoring backups that involve external storage.

Longhorn Manager Crashes Due to Backing Image Eviction

When upgrading to SUSE Virtualization v1.4.x, Longhorn Manager may crash if the EvictionRequested flag is set to true on any node or disk. This issue is caused by a race condition between the deletion of a disk in the backing image spec and the updating of its status.

To prevent the issue from occurring, ensure that the EvictionRequested flag is set to false before you start the upgrade process.

Re-enable RKE2 ingress-nginx admission webhooks (CVE-2025-1974)

If you disabled the RKE2 ingress-nginx admission webhooks to mitigate CVE-2025-1974, you must re-enable the webhook after upgrading to SUSE Virtualization v1.5.0 or later.

  1. Verify that SUSE Virtualization is using nginx-ingress v1.12.1 or later.

    $ kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'
rancher/nginx-ingress-controller:v1.12.1-hardened1

  2. Run kubectl -n kube-system edit helmchartconfig rke2-ingress-nginx to remove the following configurations from the HelmChartConfig resource.

    • .spec.valuesContent.controller.admissionWebhooks.enabled: false

    • .spec.valuesContent.controller.extraArgs.enable-annotation-validation: true

  3. Verify that the new .spec.ValuesContent configuration is similar to the following example.

    apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      admissionWebhooks:
        port: 8444
      extraArgs:
        default-ssl-certificate: cattle-system/tls-rancher-internal
      config:
        proxy-body-size: "0"
        proxy-request-buffering: "off"
      publishService:
        pathOverride: kube-system/ingress-expose

    If the HelmChartConfig resource contains other custom ingress-nginx configuration, you must retain them when editing the resource.

  4. Exit the kubectl edit command execution to save the configuration.

    SUSE Virtualization automatically applies the change once the content is saved.

  5. Verify that the rke2-ingress-nginx-admission webhook configuration is re-enabled.

    $ kubectl get validatingwebhookconfiguration rke2-ingress-nginx-admission
NAME                           WEBHOOKS   AGE
rke2-ingress-nginx-admission   1          6s

  6. Verify that the ingress-nginx pods are restarted successfully.

    kubectl -n kube-system get po -lapp.kubernetes.io/instance=rke2-ingress-nginx
NAME                                  READY   STATUS    RESTARTS   AGE
rke2-ingress-nginx-controller-l2cxz   1/1     Running   0          94s