If you are using Rancher to manage your SUSE Virtualization cluster, we recommend upgrading your Rancher server first. For more information, please refer to the Rancher upgrade guide.

For the SUSE Virtualization & Rancher support matrix, please visit our website here.

Upgrading Rancher will not automatically upgrade your SUSE Virtualization cluster. You still need to upgrade your SUSE Virtualization cluster after upgrading Rancher.
Upgrading Rancher will not bring your SUSE Virtualization cluster down. You can still access your SUSE Virtualization cluster using its virtual IP.

Before starting an upgrade

Check out the available upgrade-config setting to tweak the upgrade strategies and behaviors that best suit your cluster environment.

Start an upgrade

Before you upgrade your SUSE Virtualization cluster, we highly recommend:
- Back up your VMs if needed.
Do not operate the cluster during an upgrade. For example, creating new VMs, uploading new images, etc.
Make sure your hardware meets the preferred hardware requirements. This is due to there will be intermediate resources consumed by an upgrade.
Make sure each node has at least 30 GiB of free system partition space (df -h /usr/local/). If any node in the cluster has less than 30 GiB of free system partition space, the upgrade will be denied. Check free system partition space requirement for more information.
Run the pre-check script on a SUSE Virtualization control-plane node. Please pick a script according to your cluster’s version: https://github.com/harvester/upgrade-helpers/tree/main/pre-check.
A number of one-off privileged pods will be created in the harvester-system and cattle-system namespaces to perform host-level upgrade operations. If pod security admission is enabled, adjust these policies to allow these pods to run.

Make sure all nodes' times are in sync. Using an NTP server to synchronize time is recommended. If an NTP server is not configured during the installation, you can manually add an NTP server on each node:

  $ sudo -i

  # Add time servers
  $ vim /etc/systemd/timesyncd.conf
  [ntp]
  NTP=0.pool.ntp.org

  # Enable and start the systemd-timesyncd
  $ timedatectl set-ntp true

  # Check status
  $ sudo timedatectl status

NICs that connect to a PCI bridge might be renamed after an upgrade. Please check the knowledge base article for further information.

On the SUSE Virtualization UI Dashboard screen, click Upgrade.

The Upgrade button appears whenever a new version that you can upgrade to becomes available.

If your environment does not have direct internet access, follow the instructions in Prepare an air-gapped upgrade, which provides an efficient approach to downloading the ISO.
Select the version that you want to upgrade to.

If you require customizations, see Customize the version.
Click the progress indicator (circular icon) to view the status of each related process.

Customize the version

Download the version file (https://releases.rancher.com/harvester/{version}/version.yaml).

Example:

The v1.5.0 version file is downloaded as v1.5.0.yaml.

apiVersion: harvesterhci.io/v1beta1
kind: Version
metadata:
  name: v1.5.0-customized # Changed, to avoid duplicated with the official version name
  namespace: harvester-system
spec:
  isoChecksum: 'df28e9bf8dc561c5c26dee535046117906581296d633eb2988e4f68390a281b6856a5a0bd2e4b5b988c695a53d0fc86e4e3965f19957682b74317109b1d2fe32'  # Don't change
  isoURL: https://releases.rancher.com/harvester/v1.5.0/harvester-v1.5.0-amd64.iso # Official ISO path by default
  releaseDate: '20250425'

Add the necessary annotations.
- minCertsExpirationInDay Annotation
- skipGarbageCollectionThresholdCheck Annotation
Create the version using the command kubectl create -f v1.5.0.yaml.

minCertsExpirationInDay Annotation

SUSE Virtualization checks the validity period of certificates on each node. This check eliminates the possibility of certificates expiring while the upgrade is in progress. If a certificate will expire within 7 days, an error is returned.

Example: harvesterhci.io/minCertsExpirationInDay: "14"

When this annotation is added, SUSE Virtualization returns an error when it detects a certificate that will expire within 14 days. For more information, see auto-rotate-rke2-certs.

skipGarbageCollectionThresholdCheck Annotation

SUSE Virtualization checks the disk space on each node to ensure that the kubelet’s image garbage collection threshold is not exceeded when the required images are loaded during upgrades.

Example: harvesterhci.io/skipGarbageCollectionThresholdCheck: true

When this annotation is added, SUSE Virtualization skips the check. For more information, see Free system partition space requirement.

Do not use this annotation in production environments. When the check is skipped, required images might be deleted, causing the upgrade to fail.

Prepare an air-gapped upgrade

Make sure to check Upgrade support matrix section first about upgradable versions.

Prepare the ISO file

Download an ISO file from the Releases page.
Save the ISO to a local HTTP server.

Assume the file is hosted at http://10.10.0.1/harvester.iso.

Prepare the Version

Download the version file (https://releases.rancher.com/harvester/{version}/version.yaml).

Replace the isoURL value in the file.

  apiVersion: harvesterhci.io/v1beta1
  kind: Version
  metadata:
    name: v1.5.0
    namespace: harvester-system
  spec:
    isoChecksum: <SHA-512 checksum of the ISO>
    isoURL: http://10.10.0.1/harvester.iso  # change to local ISO URL
    releaseDate: '20250425'

Assume the file is hosted at http://10.10.0.1/version.yaml. If you require customizations, see Customize the version.

Access one of the control plane nodes via SSH and log in using the root account.

Create a version object.

rancher@node1:~> sudo -i
rancher@node1:~> kubectl create -f http://10.10.0.1/version.yaml

Start the upgrade

The Upgrade button appears on the Dashboard screen whenever a new version that you can upgrade to becomes available. Refresh the screen if the button does not appear.

Manually start an upgrade before the official upgrade becomes available

The Upgrade button does not appear on the UI immediately after a new version is released. If you want to upgrade your cluster before the option becomes available on the UI, follow the steps in Prepare an air-gapped upgrade.

In production environments, upgrading clusters via the UI is recommended.

Free system partition space requirement

SUSE Virtualization loads images on each node during upgrades. When disk usage exceeds the kubelet’s garbage collection threshold, the kubelet deletes unused images to free up space. This may cause issues in air-gapped environments because the images are not available on the node.

SUSE Virtualization v1.5.0 includes checks that ensure nodes do not trigger garbage collection after loading new images.

If you want to try upgrading even if the free system partition space is insufficient on some nodes, you can update the harvesterhci.io/skipGarbageCollectionThresholdCheck: true annotation of the Version object.

apiVersion: harvesterhci.io/v1beta1
kind: Version
metadata:
  annotations:
    harvesterhci.io/skipGarbageCollectionThresholdCheck: true
  name: 1.5.0
  namespace: harvester-system
spec:
  isoChecksum: <SHA-512 checksum of the ISO>
  isoURL: http://192.168.0.181:8000/harvester-master-amd64.iso
  minUpgradableVersion: 1.4.1
  releaseDate: "20250630"

Setting a smaller value than the pre-defined value may cause the upgrade to fail and is not recommended in a production environment.

The following sections describe solutions for issues related to this requirement.

Free system partition space manually

SUSE Virtualization attempts to remove unnecessary container images after an upgrade is completed. However, this automatic image cleanup may not be performed for various reasons. You can use a script to manually remove images. For more information, see issue #6620.

Set up a private container registry and skip image preloading

The system partition might still lack free space even after you remove images. To address this, set up a private container registry for both current and new images, and configure the setting upgrade-config with following value:

{"imagePreloadOption":{"strategy":{"type":"skip"}}, "restoreVM": false}

SUSE Virtualization skips the upgrade image preloading process. When the deployments on the nodes are upgraded, the container runtime loads the images stored in the private container registry.

Do not rely on the public container registry. Note any potential internet service interruptions and how close you are to reaching your Docker Hub rate limit. Failure to download any of the required images may cause the upgrade to fail and may leave the cluster in a middle state.

Virtual Machine Backup Compatibility

You may encounter certain limitations when creating and restoring backups that involve external storage.

Longhorn Manager Crashes Due to Backing Image Eviction

When upgrading to SUSE Virtualization v1.4.x, Longhorn Manager may crash if the EvictionRequested flag is set to true on any node or disk. This issue is caused by a race condition between the deletion of a disk in the backing image spec and the updating of its status.

To prevent the issue from occurring, ensure that the EvictionRequested flag is set to false before you start the upgrade process.

Re-enable RKE2 ingress-nginx admission webhooks (CVE-2025-1974)

If you disabled the RKE2 ingress-nginx admission webhooks to mitigate CVE-2025-1974, you must re-enable the webhook after upgrading to SUSE Virtualization v1.5.0 or later.

Verify that SUSE Virtualization is using nginx-ingress v1.12.1 or later.

$ kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'
rancher/nginx-ingress-controller:v1.12.1-hardened1

Run kubectl -n kube-system edit helmchartconfig rke2-ingress-nginx to remove the following configurations from the HelmChartConfig resource.
- .spec.valuesContent.controller.admissionWebhooks.enabled: false
- .spec.valuesContent.controller.extraArgs.enable-annotation-validation: true

Verify that the new .spec.ValuesContent configuration is similar to the following example.

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      admissionWebhooks:
        port: 8444
      extraArgs:
        default-ssl-certificate: cattle-system/tls-rancher-internal
      config:
        proxy-body-size: "0"
        proxy-request-buffering: "off"
      publishService:
        pathOverride: kube-system/ingress-expose

If the HelmChartConfig resource contains other custom ingress-nginx configuration, you must retain them when editing the resource.

Exit the kubectl edit command execution to save the configuration.

SUSE Virtualization automatically applies the change once the content is saved.

Verify that the rke2-ingress-nginx-admission webhook configuration is re-enabled.

$ kubectl get validatingwebhookconfiguration rke2-ingress-nginx-admission
NAME                           WEBHOOKS   AGE
rke2-ingress-nginx-admission   1          6s

Verify that the ingress-nginx pods are restarted successfully.

kubectl -n kube-system get po -lapp.kubernetes.io/instance=rke2-ingress-nginx
NAME                                  READY   STATUS    RESTARTS   AGE
rke2-ingress-nginx-controller-l2cxz   1/1     Running   0          94s