Upgrade from v1.3.2 to v1.4.0

General Information

An Upgrade button appears on the Dashboard screen whenever a new Harvester version that you can upgrade to becomes available. For more information, see Start an upgrade.

For air-gapped environments, see Prepare an air-gapped upgrade.

Known Issues

1. The upgrade process becomes stuck in the "Pre-draining" state.

A virtual machine with a container disk cannot be migrated because of a limitation of the live migration feature. This causes the upgrade process to become stuck in the "Pre-draining" state.

Manually stop the virtual machines to continue the upgrade process.

For more information, see Issue #7005.

2. The upgrade process becomes stuck on waiting for the Harvester bundle to become ready.

This issue is caused by a race condition when the Fleet agent (fleet-agent) is redeployed. The following error messages indicate that the issue exists.

> kubectl get bundles -n fleet-local
NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
mcc-harvester                                 0/1                       ErrApplied(1) [Cluster fleet-local/local: encountered 2 deletion errors. First is: admission webhook "validator.harvesterhci.io" denied the request: Internal error occurred: no route match found for DELETE /v1, Kind=Secret harvester-system/sh.helm.release.v1.harvester.v2]
mcc-harvester-crd                             0/1                       ErrApplied(1) [Cluster fleet-local/local: admission webhook "validator.harvesterhci.io" denied the request: Internal error occurred: no route match found for DELETE /v1, Kind=Secret harvester-system/sh.helm.release.v1.harvester-crd.v1]
shell

You can run the following script to fix the issue.

#!/bin/bash

patch_fleet_bundle() {
  local bundleName=$1
  local generation=$(kubectl get -n fleet-local bundle ${bundleName} -o jsonpath='{.spec.forceSyncGeneration}')
  local new_generation=$((generation+1))
  patch_manifest="$(mktemp)"
  cat > "$patch_manifest" <<EOF
{
  "spec": {
    "forceSyncGeneration": $new_generation
  }
}
EOF
  echo "patch bundle to new generation: $new_generation"
  kubectl patch -n fleet-local bundle ${bundleName}  --type=merge --patch-file $patch_manifest
  rm -f $patch_manifest
}

echo "removing harvester validating webhook"
kubectl delete validatingwebhookconfiguration harvester-validator

for bundle in mcc-harvester-crd mcc-harvester
do
  patch_fleet_bundle ${bundle}
done

echo "removing longhorn services"
kubectl delete svc longhorn-engine-manager -n longhorn-system --ignore-not-found=true
kubectl delete svc longhorn-replica-manager -n longhorn-system --ignore-not-found=true
shell

3. Upgrade stuck on waiting for Fleet

When upgrading from v1.3.2 to v1.4.0, the upgrade process may become stuck on waiting for Fleet to become ready. This issue is caused by a race condition when Rancher is redeployed.

Check the Harvester logs and Fleet history for the following indicators:

  • The manifest pod is stuck in the deployed status.

  • The upgrade is pending with a chart version that has been deployed.

Example:

> kubectl logs -n harvester-system -l harvesterhci.io/upgradeComponent=manifest
wait helm release cattle-fleet-system fleet fleet-104.0.2+up0.10.2 0.10.2 deployed

> helm history -n cattle-fleet-system fleet
REVISION	UPDATED                 	STATUS         	CHART                	APP VERSION	DESCRIPTION
26      	Tue Dec 10 03:09:13 2024	superseded     	fleet-103.1.5+up0.9.5	0.9.5      	Upgrade complete
27      	Sun Dec 15 09:26:54 2024	superseded     	fleet-103.1.5+up0.9.5	0.9.5      	Upgrade complete
28      	Sun Dec 15 09:27:03 2024	superseded     	fleet-103.1.5+up0.9.5	0.9.5      	Upgrade complete
29      	Mon Dec 16 05:57:03 2024	deployed       	fleet-103.1.5+up0.9.5	0.9.5      	Upgrade complete
30      	Mon Dec 16 05:57:13 2024	pending-upgrade	fleet-103.1.5+up0.9.5	0.9.5      	Preparing upgrade
shell

You can run the following command to fix the issue.

helm rollback fleet -n cattle-fleet-system <last-deployed-revision>
shell