Managing HA Cluster Resources

1 What are cluster resources? #

In a High Availability cluster, the applications and services that need to be highly available are called resources. Cluster resources can include Web sites, e-mail servers, databases, file systems, virtual machines or any other applications or services that need to be available to users at all times. You can start, stop, monitor and move resources as needed. You can also specify whether resources should run together on the same node, or start and stop in sequential order. If a node fails, the resources running on it fail over (move) to another node instead of being lost.

In SUSE Linux Enterprise High Availability, the cluster resource manager (CRM) is Pacemaker, which manages and coordinates all cluster services. Pacemaker uses resource agents (RAs) to start, stop and monitor resources. A resource agent abstracts the resource and presents its status to the cluster. This means that Pacemaker only interacts with the resource agent, not with the actual application or service. SUSE Linux Enterprise High Availability supports many different resource agents that are designed to manage specific types of resources.

1.1 Primitive resources #

Primitive resources represent the applications or services that need to be highly available. Configuring a primitive specifies the resource agent and any other information required for the cluster to manage a resource. To see the supported resource agent classes, run crm ra classes. To see the resource agents available within a class, run crm ra list CLASS.

Primitive resources can have the following settings:

Properties: Resource properties are the basic details required by the cluster: a unique name for the resource, and the resource agent that manages the application or service.
Parameters: Parameters are specific to each resource agent and determine the details and behavior of an application or service. To show all the parameters available for a resource agent, run crm ra info RESOURCE_AGENT.
Operations: Operations are actions that the cluster can perform on a resource, such as start, stop and monitor. To show the supported operations for a resource agent, run crm ra info RESOURCE_AGENT and scroll to the bottom of the output.
Meta attributes: Meta attributes change how the cluster treats a resource. To show all the meta attributes available for primitives, run crm_resource --list-options primitive.
Utilization attributes: If the cluster nodes have utilization attributes configured, you can add these attributes to cluster resources to help with load balancing. For more information, see https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/html/utilization.html.

1.2 Resource groups #

Cluster resources might depend on other resources, such as a Web server that requires an IP address and a file system. You can combine these resources into a resource group. Groups contain multiple primitives that need to be located together, started sequentially and stopped in the reverse order.

This diagram shows a box representing a node. The box contains four individual resources and a smaller box containing three more resources. The smaller box represents a resource group. A zoomed-in view of the resource group shows that the resources are an IP address, an XFS file system and an Apache Web server. The resources are numbered to show the start order in one direction and the reversed stop order in the other direction.

Figure 1: Resource group #

Groups have the following properties:

Contents: Groups can only contain primitive resources, not clones or other groups.
Starting and stopping: Resources start in the order they appear and stop in the reverse order.
Location: Resources in the group must all run on the same node.
Dependency: If one of the resources can't run anywhere, then the resources that appear after it in the group can't run anywhere either.
Meta attributes: The meta attributes target-role, priority, maintenance and is-managed can be added directly to the group. All other attributes are inherited from the group's resources.
Resource stickiness: Stickiness in groups is additive. Every active resource in the group contributes its stickiness value to the group's total.

1.3 Resource clones #

Resource clones can run simultaneously on multiple nodes. This is required for cluster file systems like GFS2, for example. You can clone a primitive or a group if the resource agent supports it. Clones can also have additional clone-specific meta attributes to change their behavior.

Clones can be one of the following types:

Anonymous clones

Anonymous clones are the simplest type. All instances of the clone behave identically. Therefore, only one instance of an anonymous clone can be active on each node.

Globally unique clones

Globally unique clones are distinct entities. An instance of the clone running on one node is not necessarily equivalent to another instance on another node.

Promotable clones

Promotable clones allow the instances to be in one of two operating modes: promoted or unpromoted (also known as primary and secondary, or active and passive). Promotable clones can be either anonymous or globally unique.

To make a clone promotable, the resource agent must support the promote and demote operations. The cluster determines which instance of the clone to promote using one of the following methods (or a combination of both):

If a resource agent supports promotable clones, it might automatically set promotion scores based on its own preference for which instance to promote.
You can manually set promotion rules or preferences by creating location or colocation constraints.

For more information, see https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/html/collective.html#determining-which-instance-is-promoted

1.4 For more information #

Primitive resources: https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/html/resources.html
Groups and clones: https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/html/collective.html
Resource operations: https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/html/operations.html
Utilization attributes: https://clusterlabs.org/projects/pacemaker/doc/3.0/Pacemaker_Explained/html/utilization.html

2 Showing cluster resources #

You can check the status and configuration of existing cluster resources with the CRM Shell (crmsh) or with the Hawk Web interface.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Show the status of the resources:
```
> sudo crm resource status
```
Show the configuration of the resources. The basic command shows the full CIB:
```
> sudo crm configure show
```
You can limit the output by using modifiers such as an object's name, type or tag. For more information about the available modifiers, run the following command:
```
> sudo crm configure help show
```

3 Starting cluster resources #

You can start cluster resources with the CRM Shell (crmsh) or with the Hawk Web interface.

Requirements #

The application or service that the cluster resource manages must be set up correctly. For example, if you have a resource managing an Apache server, the Apache configuration must be complete before you start the resource, or the start action will fail.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Check the current status of the resource:
```
> sudo crm resource status
```
Start the resource. If necessary, you can start multiple resources at once:
```
> sudo crm resource start RESOURCE1 RESOURCE2
```
Check the status again to make sure the resource started successfully:
```
> sudo crm resource status
```

4 Stopping cluster resources #

You can stop cluster resources with the CRM Shell (crmsh) or with the Hawk Web interface.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Check the current status of the resource:
```
> sudo crm resource status
```
Stop the resource. If necessary, you can stop multiple resources at once:
```
> sudo crm resource stop RESOURCE1 RESOURCE2
```
Check the status again to make sure the resource stopped successfully:
```
> sudo crm resource status
```

5 Editing primitive resources #

You can edit primitive resources with the CRM Shell (crmsh) or with the Hawk Web interface.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
To add a new monitor operation, use the following command:
```
> sudo crm configure monitor RESOURCE INTERVAL:TIMEOUT
```

To add a new parameter, meta attribute or utilization attribute, use one of the following commands:

> sudo crm resource param RESOURCE set PARAMETER VALUE
> sudo crm resource meta RESOURCE set ATTRIBUTE VALUE
> sudo crm resource utilization RESOURCE set ATTRIBUTE VALUE

To delete a parameter, meta attribute or utilization attribute, use one of the following commands:

> sudo crm resource param RESOURCE delete PARAMETER
> sudo crm resource meta RESOURCE delete ATTRIBUTE
> sudo crm resource utilization RESOURCE delete ATTRIBUTE

To change multiple attributes at once, use the following command:
```
> sudo crm configure edit RESOURCE
```
This command opens the CIB representation of the resource in a text editor. You can make multiple changes to the resource's configuration, then save and exit.
Don't use this command to rename a resource. The change won't apply to related objects.
To rename a resource, first stop the resource, then use the following command:
```
> sudo crm configure rename OLD_NAME NEW_NAME
```
This also changes the resource's name in related objects such as constraints, groups and clones.

For more information #

crm configure help edit
crm configure help monitor
crm configure help rename
crm resource help meta
crm resource help param
crm resource help util

6 Editing resource groups #

You can edit resource groups with the CRM Shell (crmsh) or with the Hawk Web interface.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
To remove a resource from a group, use the following command:
```
> sudo crm configure modgroup GROUP remove RESOURCE
```
To add a resource to a group, use the following command:
```
> sudo crm configure modgroup GROUP add RESOURCE
```
The resource is appended to the end of the group, meaning it starts last and stops first.
Alternatively, you can use before or after to add the resource to a specific position in the group's order:
```
> sudo crm configure modgroup GROUP add RESOURCE0 before RESOURCE1
```
```
> sudo crm configure modgroup GROUP add RESOURCE2 after RESOURCE1
```
To change the order of resources that are already in the group, use the following command:
```
> sudo crm configure edit GROUP
```
This command opens the CIB representation of the group in a text editor. You can make multiple changes to the group's configuration, then save and exit.
Don't use this command to rename a group. The change won't apply to related objects.
To rename a group, use the following command:
```
> sudo crm configure rename OLD_NAME NEW_NAME
```
This also changes the group's name in related objects such as constraints and clones.

For more information #

crm configure help edit
crm configure help modgroup
crm configure help rename

7 Cleaning up failed resources #

The cluster automatically restarts resources if they fail, but each failure increases the resource's fail count. If the fail count reaches the resource's migration threshold, the current node becomes ineligible to run the resource. The cluster either moves the resource to an eligible node or stops the resource if no eligible node is available.

By default, fail counts are not automatically reset. This procedure explains how to reset the fail count manually.

You can clean up failed resources with the CRM Shell (crmsh) or with the Hawk Web interface. If the resource is part of a group, the other resources in the group are also cleaned up.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Show the status of the cluster, including fail counts:
```
> sudo crm status failcounts
```
If a resource has a fail count, it appears in the Migration Summary section.
Clean up the resource on all nodes:
```
> sudo crm resource cleanup RESOURCE
```
Alternatively, you can clean up the resource on a specific node:
```
> sudo crm resource cleanup RESOURCE NODE
```

Tip: Automatically resetting fail counts

You can configure a cluster resource's fail count to reset automatically by adding the failure-timeout meta attribute to the resource.

For more information #

crm resource help cleanup

8 Deleting cluster resources #

You can delete a resource from the cluster configuration with the CRM Shell (crmsh) or with the Hawk Web interface.

Requirements #

If the resource has the status FAILED, you must clean it up before you can delete it.

CRM Shell
Hawk

You can perform this procedure on any node in the cluster.

Log in either as the root user or as a user with sudo privileges.
Check the current status of the resource:
```
> sudo crm resource status
```
If the resource is Started, you can stop it with the following command:
```
> sudo crm resource stop RESOURCE
```
Delete the resource. If necessary, you can delete multiple resources at once:
```
> sudo crm configure delete RESOURCE1 RESOURCE2
```

For more information #

crm configure help delete

9 Legal Notice #

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”.

For SUSE trademarks, see https://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.