24 Upgrading Your Cluster and Updating Software Packages #
This chapter covers two different scenarios: upgrading a cluster to another version of SUSE Linux Enterprise High Availability (either a major release or a service pack) as opposed to updating individual packages on cluster nodes. See Section 24.2, “Upgrading your Cluster to the Latest Product Version” versus Section 24.3, “Updating Software Packages on Cluster Nodes”.
If you want to upgrade your cluster, check Section 24.2.1, “Supported Upgrade Paths for SLE HA and SLE HA Geo” and Section 24.2.2, “Required Preparations Before Upgrading” before starting to upgrade.
24.1 Terminology #
In the following, find definitions of the most important terms used in this chapter:
- Major Release, General Availability (GA) Version
The Major Release of SUSE Linux Enterprise (or any software product) is a new version that brings new features and tools, decommissions previously deprecated components and comes with backward incompatible changes.
- Cluster Offline Upgrade
If a new product version includes major changes that are backward incompatible, the cluster needs to be upgraded by a cluster offline upgrade. You need to take all nodes offline and upgrade the cluster as a whole, before you can bring all nodes back online.
- Cluster Rolling Upgrade
In a cluster rolling upgrade, one cluster node at a time is upgraded while the rest of the cluster is still running. You take the first node offline, upgrade it and bring it back online to join the cluster. Then you continue one by one until all cluster nodes are upgraded to a major version.
- Service Pack (SP)
Combines several patches into a form that is easy to install or deploy. Service packs are numbered and usually contain security fixes, updates, upgrades, or enhancements of programs.
- Update
Installation of a newer minor version of a package.
- Upgrade
Installation of a newer major version of a package or distribution, which brings new features. See also Cluster Offline Upgrade versus Cluster Rolling Upgrade.
24.2 Upgrading your Cluster to the Latest Product Version #
Which upgrade path is supported, and how to perform the upgrade depends both on the current product version and on the target version you want to migrate to.
SUSE Linux Enterprise High Availability has the same supported upgrade paths as the underlying base system. For a complete overview, see the section Supported Upgrade Paths to SUSE Linux Enterprise Server 12 SP5 in the SUSE Linux Enterprise Server Deployment Guide.
In addition, the following rules apply, as the High Availability cluster stack offers two methods for upgrading the cluster:
Cluster Rolling Upgrade. A cluster rolling upgrade is only supported within the same major release (from one service pack to the next, or from the GA version of a product to SP1).
Cluster Offline Upgrade. A cluster offline upgrade is required to upgrade from one major release to the next (for example, from SLE HA 11 to SLE HA 12) or from a service pack within one major release to the next major release (for example, from SLE HA 11 SP3 to SLE HA 12).
Section 24.2.1 list the supported upgrade paths and methods for SLE HA (Geo), moving from one version to the next. The column For Details lists the specific upgrade documentation you should refer to (including also the base system and Geo Clustering for SUSE Linux Enterprise High Availability). This documentation is available from:
Mixed clusters running on SUSE Linux Enterprise High Availability 11/SUSE Linux Enterprise High Availability 12 are not supported.
After the upgrade process to product version 12, reverting back to product version 11 is not supported.
24.2.1 Supported Upgrade Paths for SLE HA and SLE HA Geo #
Upgrade From ... To |
Upgrade Path |
For Details |
---|---|---|
SLE HA 11 SP3 to SLE HA (Geo) 12 |
Cluster Offline Upgrade |
|
SLE HA (Geo) 11 SP4 to SLE HA (Geo) 12 SP1 |
Cluster Offline Upgrade |
|
SLE HA (Geo) 12 to SLE HA (Geo) 12 SP1 |
Cluster Rolling Upgrade |
|
SLE HA (Geo) 12 SP1 to SLE HA (Geo) 12 SP2 |
Cluster Rolling Upgrade |
|
SLE HA (Geo) 12 SP2 to SLE HA (Geo) 12 SP3 |
Cluster Rolling Upgrade |
|
SLE HA (Geo) 12 SP3 to SLE HA (Geo) 12 SP4 |
Cluster Rolling Upgrade |
|
SLE HA (Geo) 12 SP4 to SLE HA (Geo) 12 SP5 |
Cluster Rolling Upgrade |
|
24.2.2 Required Preparations Before Upgrading #
- Backup
Ensure that your system backup is up to date and restorable.
- Testing
Test the upgrade procedure on a staging instance of your cluster setup first, before performing it in a production environment.
This gives you an estimation of the time frame required for the maintenance window. It also helps to detect and solve any unexpected problems that might arise.
24.2.3 Cluster Offline Upgrade #
This section applies to the following scenarios:
Upgrading from SLE HA 11 SP3 to SLE HA 12
Upgrading from SLE HA 11 SP4 to SLE HA 12 SP1
If your cluster is still based on an older product version than the ones listed above, first upgrade it to a version of SUSE Linux Enterprise Server and SUSE Linux Enterprise High Availability that can be used as a source for upgrading to the desired target version.
SUSE Linux Enterprise High Availability 12 cluster stack comes with major changes in various
components (for example, /etc/corosync/corosync.conf
, disk formats of OCFS2).
Therefore, a cluster rolling upgrade
from any SUSE Linux Enterprise High Availability
11 version is not supported. Instead, all cluster nodes must be offline
and the cluster needs to be upgraded as a whole as described below.
Log in to each cluster node and stop the cluster stack with:
#
rcopenais
stopFor each cluster node, perform an upgrade to the desired target version of SUSE Linux Enterprise Server and SUSE Linux Enterprise High Availability. If you have an existing Geo cluster setup and want to upgrade it, see the additional instructions in the Geo Clustering for SUSE Linux Enterprise High Availability Geo Clustering Quick Start. To find the details for the individual upgrade processes, see Section 24.2.1, “Supported Upgrade Paths for SLE HA and SLE HA Geo”.
After the upgrade process has finished, reboot each node with the upgraded version of SUSE Linux Enterprise Server and SUSE Linux Enterprise High Availability.
If you use OCFS2 in your cluster setup, update the on-device structure by executing the following command:
#
o2cluster
--update PATH_TO_DEVICEIt adds additional parameters to the disk which are needed for the updated OCFS2 version that is shipped with SUSE Linux Enterprise High Availability 12 and 12 SPx.
To update
/etc/corosync/corosync.conf
for Corosync version 2:Log in to one node and start the YaST cluster module.
Switch to the Procedure 4.1, “Defining the First Communication Channel (Multicast)” or Procedure 4.2, “Defining the First Communication Channel (Unicast)”, respectively.
category and enter values for the following new parameters: and . For details, seeIf YaST should detect any other options that are invalid or missing according to Corosync version 2, it will prompt you to change them.
Confirm your changes in YaST. YaST will write them to
/etc/corosync/corosync.conf
.If Csync2 is configured for your cluster, use the following command to push the updated Corosync configuration to the other cluster nodes:
#
csync2
-xv
For details on Csync2, see Section 4.7, “Transferring the Configuration to All Nodes”.
Alternatively, synchronize the updated Corosync configuration by manually copying
/etc/corosync/corosync.conf
to all cluster nodes.
Log in to each node and start the cluster stack with:
#
systemctl
start pacemakerCheck the cluster status with
crm status
or with Hawk2.Configure the following services to start at boot time:
#
systemctl enable pacemaker#
systemctl enable hawk#
systemctl enable sbd
Tags (for grouping resources) and some ACL features only work with the
CIB syntax version pacemaker-2.0
or higher. (To
check your version, use the cibadmin -Q |grep
validate-with
command.) If you have upgraded from SUSE Linux Enterprise High Availability 11
SPx, your CIB version will not be upgraded by
default. To manually upgrade to the latest CIB version use one of the
following commands:
#
cibadmin
--upgrade --force
or
#
crm
configure upgrade force
24.2.4 Cluster Rolling Upgrade #
This section applies to the following scenarios:
Upgrading from SLE HA 12 to SLE HA 12 SP1
Upgrading from SLE HA 12 SP1 to SLE HA 12 SP2
Upgrading from SLE HA (Geo) 12 SP2 to SLE HA (Geo) 12 SP3
Upgrading from SLE HA (Geo) 12 SP3 to SLE HA (Geo) 12 SP4
Upgrading from SLE HA (Geo) 12 SP4 to SLE HA (Geo) 12 SP5
Use one of the following procedures for your scenario:
For a more general rolling upgrade, refer to Procedure 24.2.
For a specific rolling upgrade, refer to Procedure 24.3.
Before starting an upgrade for a node, stop the cluster stack on that node.
If the cluster resource manager on a node is active during the software update, this can lead to results such as fencing of active nodes.
The new features shipped with the latest product version will only be available after all cluster nodes have been upgraded to the latest product version. Mixed version clusters are only supported for a short time frame during the cluster rolling upgrade. Complete the cluster rolling upgrade within one week.
Log in as
root
on the node that you want to upgrade and stop the cluster stack:#
systemctl
stop pacemakerPerform an upgrade to the desired target version of SUSE Linux Enterprise Server and SUSE Linux Enterprise High Availability. To find the details for the individual upgrade processes, see Section 24.2.1, “Supported Upgrade Paths for SLE HA and SLE HA Geo”.
Restart the cluster stack on the upgraded node to make the node rejoin the cluster:
#
systemctl
start pacemakerTake the next node offline and repeat the procedure for that node.
Check the cluster status with
crm status
or with Hawk2.
The Hawk2
screen also shows a warning if different CRM versions are detected for your cluster nodes.Beside an in-place upgrade, many customers prefer a fresh installation even for moving to the next service pack. The following procedure shows a scenario where a two-node cluster with the nodes alice and bob is upgraded to the next service pack (SP):
Make a backup of your cluster configuration. A minimum set of files are shown in the following list:
/etc/corosync/corosync.conf /etc/corosync/authkey /etc/sysconfig/sbd /etc/modules-load.d/watchdog.conf /etc/hosts /etc/ntp.conf
Depending on your resources, you may also need the following files:
/etc/services /etc/passwd /etc/shadow /etc/groups /etc/drbd/* /etc/lvm/lvm.conf /etc/mdadm.conf /etc/mdadm.SID.conf
Start with node alice.
Put the node into standby node. That way, resources can move off the node:
#
crm
--wait node standby alice rebootWith the option
--wait
, the command returns only when the cluster finishes the transition and becomes idle. Thereboot
option has the effect that the node will be already out of standby mode when it is online again. Despite its name, thereboot
option works as long as the node goes offline and online.Stop the cluster services on node alice:
#
crm
cluster stopAt this point, alice does not have running resources anymore. Upgrade the node alice and reboot it afterward. Cluster services are assumed not to start on boot.
Copy your backup files from Step 1 to the original places.
Bring back node alice into cluster:
#
crm
cluster startCheck that resources are fine.
Repeat Step 2 for node bob.
24.3 Updating Software Packages on Cluster Nodes #
Before starting an update for a node, either stop the cluster stack on that node or put the node into maintenance mode, depending on whether the cluster stack is affected or not. See Step 1 for details.
If the cluster resource manager on a node is active during the software update, this can lead to results such as fencing of active nodes.
Before installing any package updates on a node, check the following:
Does the update affect any packages belonging to SUSE Linux Enterprise High Availability or the Geo clustering extension? If
yes
: Stop the cluster stack on the node before starting the software update:#
systemctl
stop pacemakerDoes the package update require a reboot? If
yes
: Stop the cluster stack on the node before starting the software update:#
systemctl
stop pacemakerIf none of the situations above apply, you do not need to stop the cluster stack. In that case, put the node into maintenance mode before starting the software update:
#
crm
node maintenance NODE_NAMEFor more details on maintenance mode, see Section 23.2, “Different Options for Maintenance Tasks”.
Install the package update using either YaST or Zypper.
After the update has been successfully installed:
Either start the cluster stack on the respective node (if you stopped it in Step 1):
#
systemctl
start pacemakeror remove the maintenance flag to bring the node back to normal mode:
#
crm
node ready NODE_NAME
Check the cluster status with
crm status
or with Hawk2.
24.4 For More Information #
For detailed information about any changes and new features of the product you are upgrading to, refer to its release notes. They are available from https://www.suse.com/releasenotes/.