|Index|Performing Maintenance Tasks on Cluster Nodes
SUSE Linux Enterprise High Availability 16.0

Performing Maintenance Tasks on Cluster Nodes

Publication Date: 17 Jul 2026

High Availability clusters require extra preparation steps before you can perform maintenance tasks, or you risk side effects such as resource failure, unsynchronized CIBs, or data loss. Use this workflow when making changes to the cluster nodes, such as hardware, software or firmware updates. This involves putting a node into standby mode and stopping the cluster services on just that node, then rebooting if required. To make changes to multiple nodes, repeat these steps on one node at a time.

Tip

If this isn't the workflow you need, see Section 7, “Other maintenance workflows” to choose the right one.

Important
Important: Maintenance window

Preparing a node for maintenance only takes a few minutes. In this scenario, most resources don't require downtime because the cluster moves them to other nodes, but some resources might stop if constraints prevent them from moving. The resources will still be highly available because only one node at a time is offline.

1 What are HA administrative modes?

 

While under High Availability cluster management, resources must not be started or stopped from outside the cluster (for example, with systemctl). This means that if you need to manually touch a resource, you must release it from cluster management first. Resources are also vulnerable if something goes wrong when you make changes to the cluster nodes or infrastructure. You must protect the cluster and its resources before performing maintenance tasks or you risk unwanted side effects such as resource failure, unsynchronized CIBs, or even data loss.

1.1 Basic steps

 

These are the basic steps for performing any maintenance tasks in a High Availability cluster:

  1. Determine the workflow you need, depending on your maintenance tasks.

  2. Apply one or more administrative modes, depending on the workflow.

  3. Perform your maintenance tasks on the cluster, node or individual resource.

  4. Put the cluster, node or resource back into “normal” operation.

1.2 Administrative modes

 

SUSE Linux Enterprise High Availability has the following modes for protecting the cluster during maintenance tasks. Depending on your task, you might need to use one of these modes or a combination of multiple modes.

Stopping the cluster services

Stopping the cluster services means stopping Pacemaker and Corosync. In this mode, the cluster is no longer running, either entirely or just on one node. Use this mode when you need to manage infrastructure, such as storage or networking, or when you need to change the configuration of the cluster services.

  • For the whole cluster: Stopping the cluster services on all nodes at once allows you to shut down a cluster while avoiding the mass migration of resources that would happen if you shut down each node one by one. When this mode is used on its own, all the resources stop. However, this doesn't guarantee a graceful shutdown. If stopping a resource fails or times out, the resource's node is fenced. Alternatively, you can avoid resource downtime by using this mode in combination with maintenance mode.

  • For one node: Stopping the cluster services on just one node allows the rest of the cluster to continue running while you perform maintenance tasks on the offline node. The node's resources either move to another node, or stop if constraints prevent them from moving. However, if stopping a resource fails or times out, the node is fenced. To avoid attempting to stop resources and the cluster services at the same time, we recommend using this mode in combination with standby mode.

Maintenance mode

In maintenance mode, the cluster stops managing the resources and won't start, stop or monitor them. This allows you to manually manage applications or services without interference from the cluster. However, resources aren't highly available while in this mode, so they won't fail over to another node if the current node shuts down or reboots.

  • For the whole cluster: Cluster maintenance mode puts all the resources into maintenance mode at once. If you stop or restart the cluster services while in this mode, most resources can continue to run even while the cluster is offline. This allows you to make cluster configuration changes with minimal resource downtime. However, any resources that depend on Corosync, such as DLM, must be stopped.

  • For one node: Node maintenance mode puts all the resources that are running on a specific node into maintenance mode at once. Don't stop or restart the cluster services while in this mode. Without proper caution, this can lead to undesirable effects, such as concurrency violations, where resources mistakenly run on multiple nodes instead of only one node.

  • For one resource: Resource maintenance mode puts a single resource into maintenance mode. You can also put groups and clones into resource maintenance mode.

Standby mode

When a node is in standby mode, it can't run resources. Any resources that were running on it either move to another node, or stop if constraints prevent them from moving. Standby mode gives the resources more time to move or stop successfully because they aren't limited by the shutdown timeout of the cluster services. Therefore, this mode is useful for gracefully evacuating a node before stopping the cluster services or rebooting the node.

Unmanaged mode

When a resource is in unmanaged mode, the cluster won't start or stop it. However, unlike maintenance mode, the cluster still monitors the resource and reports any failures. This can be useful for troubleshooting. In most other cases, maintenance mode is more appropriate.

1.3 For more information

 

For more information, see https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Administration/html/administrative.html.

2 Putting a node into standby mode

 

Use standby mode to gracefully evacuate a node before stopping the cluster services or rebooting the node. When you put a node into standby mode, its resources either move to another node, or stop if constraints prevent them from moving.

  • CRM Shell
  • Hawk

Perform this procedure on the node you want to put into standby mode.

  1. Log in to the node either as the root user or as a user with sudo privileges.

  2. Put the node into standby mode: 

    > sudo crm --wait node standby

    Use --wait to make sure the node's resources have really moved or stopped before the command returns.

    By default, the node will remain in standby mode after rebooting. Alternatively, you can set the node to come back online automatically with crm --wait node standby reboot.

  3. Check the status of the cluster: 

    > sudo crm status

    The node should show the status standby.

3 Stopping the cluster services for one node

 

Stopping the cluster services means stopping Pacemaker and Corosync. When used on one node, the node is no longer an active member of the cluster. The node's resources either move to another node, or stop if constraints prevent them from moving. However, if stopping a resource fails or times out, the node is fenced. To avoid attempting to stop resources and the cluster services at the same time, we recommend using this mode in combination with standby mode.

Warning
Warning: Don't stop the cluster services in single-node maintenance mode

Stopping the cluster services while in single-node maintenance mode can lead to undesirable effects. Resources running on this node might also start on another node. Caution is required to prevent potential concurrency violations.

Perform this procedure on the node you want to stop.

  1. Log in to the node either as the root user or as a user with sudo privileges.

  2. Check the status of the cluster: 

    > sudo crm status

    • If the node is in maintenance mode: Don't stop the cluster services, as this can lead to undesirable effects. Take the node out of maintenance mode first.

    • If the node is in standby mode: The resources have either moved to other nodes or stopped, so you can safely stop the cluster services.

    • If the node isn't in either mode: When you stop the cluster services, the resources attempt to move to other nodes. If they can't move, they stop. However, if any resources fail to stop before the cluster's shutdown timeout, the node will be fenced.

  3. Stop the cluster services on the node: 

    > sudo crm cluster stop

  4. Check the status of the cluster services: 

    > sudo crm cluster status

    Both Pacemaker and Corosync should be inactive.

4 Performing the maintenance tasks

 

Perform your maintenance tasks on the cluster node. This could be hardware, software or firmware updates, for example. In this workflow, you can safely shut down or reboot the node if required.

Warning
Warning: Only make changes to one node at a time

If you need to make changes to other nodes, wait until this node is an active member of the cluster again before repeating this workflow on the next node.

5 Starting the cluster services for one node

 

Starting the cluster services means starting Pacemaker and Corosync. The node becomes an active member of the cluster again.

Perform this procedure on the node you want to start.

  1. Log in to the node either as the root user or as a user with sudo privileges.

  2. If you rebooted the node, check whether the cluster services started automatically: 

    > sudo crm cluster status

    If the cluster services started automatically, both Pacemaker and Corosync will be active.

  3. If the cluster services didn't start automatically, start them manually: 

    > sudo crm cluster start

  4. Check the status of the cluster: 

    > sudo crm status

    The node should be Online. It might also have the status standby. If the node is in standby mode, it won't be able to run resources again until you take it out of standby mode.

6 Taking a node out of standby mode

 

Taking a node out of standby mode allows resources to run on it again. The resources that moved away from the node might return to it automatically, depending on their constraint scores and stickiness.

  • CRM Shell
  • Hawk

Perform this procedure on the node you want to take out of standby mode.

  1. Log in to the node either as the root user or as a user with sudo privileges.

  2. Check the status of the cluster: 

    > sudo crm status

    The node might have come back online automatically if you used the reboot option.

  3. If the node still has the status standby, bring it back online: 

    > sudo crm node online

  4. Check the status of the cluster again. The node should no longer have the status standby, and resource clones should be running on the node again. Other resources might return to the node automatically, depending on their constraint scores and stickiness.

7 Other maintenance workflows

 

If you need a different workflow, or to perform a specific task, check this list of other articles:

General workflows

These articles describe general workflows for different scenarios:

Performing maintenance tasks on cluster infrastructure

Make changes to the configuration of the cluster services, or to infrastructure used by the cluster services, such as storage or networking. This involves putting the whole cluster into maintenance mode and stopping the cluster services on all nodes at once, but not rebooting any nodes.

(This article) Performing maintenance tasks on cluster nodes

Make changes to the cluster nodes, such as hardware, software or firmware updates. This involves putting a node into standby mode and stopping the cluster services on just that node, then rebooting if required.

Performing maintenance tasks on cluster resources

Manually manage applications or services without interference from the cluster. This involves putting one or more resources into maintenance mode so you can temporarily manage them manually.

Specific tasks

These articles describe specific maintenance or administration tasks:

Changing the Configuration of SBD

This article describes different ways to update the configuration of SBD for node fencing. Most of these tasks require stopping the cluster services, and might also require cluster maintenance mode.