SUSE Linux Enterprise High Availability 16.0

Performing Maintenance Tasks on Cluster Infrastructure

Publication Date: 20 Jul 2026

High Availability clusters require extra preparation steps before you can perform maintenance tasks, or you risk side effects such as resource failure, unsynchronized CIBs, or data loss. Use this workflow when making changes to the configuration of the cluster services, or to infrastructure used by the cluster services, such as storage or networking. This involves putting the whole cluster into maintenance mode and stopping the cluster services on all nodes at once, but not rebooting any nodes.

If this isn't the workflow you need, see Section 9, “Other maintenance workflows” to choose the right one.

Important: Maintenance window

Preparing the cluster for maintenance only takes a few minutes, but the length of the maintenance window depends on the tasks you need to perform. In this scenario, most resources won't require downtime, but DLM and DLM-dependent resources will. The resources won't be highly available during the maintenance window.

Revision History: Performing Maintenance Tasks on Cluster Infrastructure

1 What are HA administrative modes? #

While under High Availability cluster management, resources must not be started or stopped from outside the cluster (for example, with systemctl). This means that if you need to manually touch a resource, you must release it from cluster management first. Resources are also vulnerable if something goes wrong when you make changes to the cluster nodes or infrastructure. You must protect the cluster and its resources before performing maintenance tasks or you risk unwanted side effects such as resource failure, unsynchronized CIBs, or even data loss.

1.1 Basic steps #

These are the basic steps for performing any maintenance tasks in a High Availability cluster:

Determine the workflow you need, depending on your maintenance tasks.
Apply one or more administrative modes, depending on the workflow.
Perform your maintenance tasks on the cluster, node or individual resource.
Put the cluster, node or resource back into “normal” operation.

1.2 Administrative modes #

SUSE Linux Enterprise High Availability has the following modes for protecting the cluster during maintenance tasks. Depending on your task, you might need to use one of these modes or a combination of multiple modes.

Stopping the cluster services

Stopping the cluster services means stopping Pacemaker and Corosync. In this mode, the cluster is no longer running, either entirely or just on one node. Use this mode when you need to manage infrastructure, such as storage or networking, or when you need to change the configuration of the cluster services.

For the whole cluster: Stopping the cluster services on all nodes at once allows you to shut down a cluster while avoiding the mass migration of resources that would happen if you shut down each node one by one. When this mode is used on its own, all the resources stop. However, this doesn't guarantee a graceful shutdown. If stopping a resource fails or times out, the resource's node is fenced. Alternatively, you can avoid resource downtime by using this mode in combination with maintenance mode.
For one node: Stopping the cluster services on just one node allows the rest of the cluster to continue running while you perform maintenance tasks on the offline node. The node's resources either move to another node, or stop if constraints prevent them from moving. However, if stopping a resource fails or times out, the node is fenced. To avoid attempting to stop resources and the cluster services at the same time, we recommend using this mode in combination with standby mode.

Maintenance mode

In maintenance mode, the cluster stops managing the resources and won't start, stop or monitor them. This allows you to manually manage applications or services without interference from the cluster. However, resources aren't highly available while in this mode, so they won't fail over to another node if the current node shuts down or reboots.

For the whole cluster: Cluster maintenance mode puts all the resources into maintenance mode at once. If you stop or restart the cluster services while in this mode, most resources can continue to run even while the cluster is offline. This allows you to make cluster configuration changes with minimal resource downtime. However, any resources that depend on Corosync, such as DLM, must be stopped.
For one node: Node maintenance mode puts all the resources that are running on a specific node into maintenance mode at once. Don't stop or restart the cluster services while in this mode. Without proper caution, this can lead to undesirable effects, such as concurrency violations, where resources mistakenly run on multiple nodes instead of only one node.
For one resource: Resource maintenance mode puts a single resource into maintenance mode. You can also put groups and clones into resource maintenance mode.

Standby mode

When a node is in standby mode, it can't run resources. Any resources that were running on it either move to another node, or stop if constraints prevent them from moving. Standby mode gives the resources more time to move or stop successfully because they aren't limited by the shutdown timeout of the cluster services. Therefore, this mode is useful for gracefully evacuating a node before stopping the cluster services or rebooting the node.

Unmanaged mode

When a resource is in unmanaged mode, the cluster won't start or stop it. However, unlike maintenance mode, the cluster still monitors the resource and reports any failures. This can be useful for troubleshooting. In most other cases, maintenance mode is more appropriate.

1.3 For more information #

For more information, see https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Administration/html/administrative.html.

2 Stopping DLM and DLM-dependent resources #

To put the cluster into maintenance mode and then stop the cluster services, you must first stop DLM and any resources that depend on it, such as GFS2 or lvmlockd.

DLM depends on Corosync. If Corosync stops but DLM is still running, the DLM instances on each node consider the other instances lost and request node fencing. Stopping the cluster services with crm cluster stop doesn't usually cause this problem because DLM also stops. However, if the cluster is in maintenance mode, DLM keeps running even when the cluster services are offline.

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Stop DLM and any DLM-dependent resources. If the resources are in a group, you can just stop the group. If the resources aren't in a group, stop the DLM-dependent resources first and then stop DLM. You can stop multiple resources at once:
```
> sudo crm --wait resource stop RESOURCE1 RESOURCE2
```
Use --wait to make sure the resources have really stopped before the command returns.
Check the status of the cluster:
```
> sudo crm status
```
DLM and any DLM-dependent resources should have the status Stopped.

3 Putting the cluster into maintenance mode #

Cluster maintenance mode puts all the resources into maintenance mode at once. The cluster stops managing the resources and won't start, stop or monitor them.

Warning: No cluster protection while in maintenance mode

Resources aren't highly available while in maintenance mode, so they won't fail over to another node if the current node shuts down or reboots.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Put the cluster into maintenance mode:
```
> sudo crm maintenance on
```
Check the status of the cluster:
```
> sudo crm status
```
The resources should all show the status maintenance.

4 Stopping the cluster services for the whole cluster #

Stopping the cluster services means stopping Pacemaker and Corosync. You can stop the whole cluster at once by using the --all option. In this mode, the cluster is no longer running and all the resources stop. Alternatively, you can avoid resource downtime by using this mode in combination with maintenance mode.

Warning: Graceful shutdown not guaranteed

When you stop the cluster services for the whole cluster without maintenance mode, all the resources stop. However, the resources must finish stopping before the cluster's shutdown timeout. If stopping a resource fails or times out, the resource's node is fenced.

If any resources take a long time to stop, consider stopping them before you stop the cluster services.

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Check the status of the cluster:
```
> sudo crm status
```
- If the resources are in maintenance mode, they will continue to run even while the cluster services are offline.
- If the resources aren't in maintenance mode, they will all stop when you stop the cluster services.
Stop the cluster services on all nodes at once:
```
> sudo crm cluster stop --all
```
Check the status of the cluster services:
```
> sudo crm cluster status
```
Both Pacemaker and Corosync should be inactive.

5 Performing the maintenance tasks #

Perform your maintenance tasks on the cluster services or infrastructure. This could be changes to the configuration of the cluster services, or to the underlying storage or networking that the cluster services rely on.

Warning: Don't shut down or reboot cluster nodes

In this workflow, don't shut down or reboot any cluster nodes. The resources aren't highly available while in maintenance mode, so they won't fail over to another node.

6 Starting the cluster services for the whole cluster #

Starting the cluster services means starting Pacemaker and Corosync. You can start the whole cluster at once by using the --all option.

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Start the cluster services on all nodes at once:
```
> sudo crm cluster start --all
```
Check the status of the cluster services:
```
> sudo crm cluster status
```
Both Corosync and Pacemaker should be active.
Check the status of the cluster:
```
> sudo crm status
```
The nodes will have the status UNCLEAN (offline), but will soon change to Online.

7 Taking the cluster out of maintenance mode #

Taking the cluster out of maintenance mode puts the resources back under cluster management and protection.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Check the status of the cluster:
```
> sudo crm status
```
If the nodes have the status UNCLEAN (Offline), wait until they change to Online.
If the nodes are Online, put the cluster back into normal operation:
```
> sudo crm maintenance off
```
Check the status of the cluster again. The resources should no longer show the status maintenance.

8 Starting DLM and DLM-dependent resources #

If you stopped DLM before putting the cluster into maintenance node, you can start it again when the cluster is out of maintenance mode.

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Check the status of the cluster to make sure the resources aren't in maintenance mode:
```
> sudo crm status
```
Start DLM and any DLM-dependent resources. If the resources are in a group, you can just start the group. If the resources aren't in a group, start DLM first and then start the DLM-dependent resources. You can start multiple resources at once:
```
> sudo crm resource start RESOURCE1 RESOURCE2
```
Check the status of the cluster again. DLM and any DLM-dependent resources should have the status Started.

9 Other maintenance workflows #

If you need a different workflow, or to perform a specific task, check this list of other articles:

General workflows

These articles describe general workflows for different scenarios:

(This article) Performing maintenance tasks on cluster infrastructure: Make changes to the configuration of the cluster services, or to infrastructure used by the cluster services, such as storage or networking. This involves putting the whole cluster into maintenance mode and stopping the cluster services on all nodes at once, but not rebooting any nodes.
Performing maintenance tasks on cluster nodes: Make changes to the cluster nodes, such as hardware, software or firmware updates. This involves putting a node into standby mode and stopping the cluster services on just that node, then rebooting if required.
Performing maintenance tasks on cluster resources: Manually manage applications or services without interference from the cluster. This involves putting one or more resources into maintenance mode so you can temporarily manage them manually.

Specific tasks

These articles describe specific maintenance or administration tasks:

Changing the Configuration of SBD: This article describes different ways to update the configuration of SBD for node fencing. Most of these tasks require stopping the cluster services, and might also require cluster maintenance mode.

10 Legal Notice #

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”.

For SUSE trademarks, see https://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.