Upgrade from v1.2.1 to v1.2.2

General Information

An Upgrade button appears on the Dashboard screen whenever a new Harvester version that you can upgrade to becomes available. For more information, see Start an upgrade.

For air-gapped environments, see Prepare an air-gapped upgrade.

Known Issues

1. Upgrade stuck in the Upgrading System Service state

Expired certificates of certain system services may cause an upgrade to become stuck in the Upgrading System Service state for an extended period. To resolve this issue, perform the following steps:

  1. Identify the name of the apply-manifest job.

    kubectl get jobs -n harvester-system -l harvesterhci.io/upgradeComponent=manifest

    Example of output:

    NAME                                 COMPLETIONS   DURATION   AGE
hvst-upgrade-9gmg2-apply-manifests   0/1           46s        46s

  2. Check the logs of the apply-manifest job.

     kubectl logs jobs/hvst-upgrade-9gmg2-apply-manifests -n harvester-system

    If the following messages appear in the log, continue to the next step.

    Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 30259)...

  3. Check the state of the Cluster API (CAPI) cluster.

    kubectl get clusters.provisioning.cattle.io local -n fleet-local -o yaml

    If the output is similar to the following example, the issue likely exists in the cluster.

        - lastUpdateTime: "2023-01-17T16:26:48Z"
      message: 'configuring bootstrap node(s) custom-24cb32ce8387: waiting for probes:
        kube-controller-manager, kube-scheduler'
      reason: Waiting
      status: Unknown
      type: Updated

  4. Identify the hostname of the affected node, and implement the workaround to see if service certificates expire.

    kubectl get machines.cluster.x-k8s.io -n fleet-local <machine_name> -o yaml | yq .status.nodeRef.name

    Replace <machine_name> with the name of the affected node (as stated in the output of the previous step).

    If multiple nodes joined the cluster around the same time, you must implement the workaround on all those nodes.

Related issue: #3863 (See workaround)

2. The harvester-seeder add-on is not upgraded

When upgrading Harvester clusters from v1.2.1 to v1.2.2, the harvester-seeder add-on is not fully upgraded because the Helm chart versions do not match. The harvester-seeder chart bundled with Harvester v1.2.2 is of v0.1.1, while the harvester-seeder add-on uses the v0.1.0 chart.

Error: chart "harvester-seeder" matching 0.1.0 not found in harvester-seeder index. (try 'helm repo update'): no chart version found for harvester-seeder-0.1.0

If you enable the add-on after upgrading to Harvester v1.2.2, installing the chart will also not be possible because of the mentioned mismatch.

The workaround is to manually update the harvester-seeder add-on manifest using the following command. You must disable the add-on before patching.

kubectl -n harvester-system patch addons.harvesterhci harvester-seeder --type=json -p '[{"op": "replace", "path": "/spec/version", "value": "0.1.1"}]'