The High Availability Extension supports the following scenarios:

Your cluster can contain up to 32 Linux servers. Using pacemaker_remote, the cluster can be extended to include additional Linux servers beyond this limit. Any server in the cluster can restart resources (applications, services, IP addresses, and file systems) from a failed server in the cluster.

The High Availability Extension ships with Corosync messaging and membership layer and Pacemaker Cluster Resource Manager. Using Pacemaker, administrators can continually monitor the health and status of their resources, and manage dependencies. They can automatically stop and start services based on highly configurable rules and policies. The High Availability Extension allows you to tailor a cluster to the specific applications and hardware infrastructure that fit your organization. Time-dependent configuration enables services to automatically migrate back to repaired nodes at specified times.

With the High Availability Extension you can dynamically assign and reassign server storage as needed. It supports Fibre Channel or iSCSI storage area networks (SANs). Shared disk systems are also supported, but they are not a requirement. SUSE Linux Enterprise High Availability Extension also comes with a cluster-aware file system (OCFS2) and the cluster Logical Volume Manager (cluster LVM2). For replication of your data, use DRBD* to mirror the data of a High Availability service from the active node of a cluster to its standby node. Furthermore, SUSE Linux Enterprise High Availability Extension also supports CTDB (Cluster Trivial Database), a technology for Samba clustering.

SUSE Linux Enterprise High Availability Extension supports the mixed clustering of both physical and virtual Linux servers. SUSE Linux Enterprise Server 15 SP1 ships with Xen, an open source virtualization hypervisor, and with KVM (Kernel-based Virtual Machine). KVM is a virtualization software for Linux which is based on hardware virtualization extensions. The cluster resource manager in the High Availability Extension can recognize, monitor, and manage services running within virtual servers and services running in physical servers. Guest systems can be managed as services by the cluster.

SUSE Linux Enterprise High Availability Extension has been extended to support different geographical scenarios. Support for geographically dispersed clusters (Geo clusters) is available with Geo Clustering for SUSE Linux Enterprise High Availability Extension.

The greater the geographical distance between individual cluster nodes, the more factors may potentially disturb the high availability of services the cluster provides. Network latency, limited bandwidth and access to storage are the main challenges for long-distance clusters.

SUSE Linux Enterprise High Availability Extension includes a huge number of resource agents to manage resources such as Apache, IPv4, IPv6 and many more. It also ships with resource agents for popular third party applications such as IBM WebSphere Application Server. For an overview of Open Cluster Framework (OCF) resource agents included with your product, use the crm ra command as described in Section 8.1.3, “Displaying Information about OCF Resource Agents”.

The High Availability Extension ships with a set of powerful tools. Use them for basic installation and setup of your cluster and for effective configuration and administration:

The High Availability Extension has a layered architecture. Figure 1.6, “Architecture” illustrates the different layers and their associated components.

Figure 1.6: Architecture #

The pacemakerd daemon launches and monitors all other related daemons. The daemon that coordinates all actions, pacemaker-controld, has an instance on each cluster node. Pacemaker centralizes all cluster decision-making by electing one of those instances as a master. Should the elected pacemaker-controld daemon fail, a new one is established.

Many actions performed in the cluster will cause a cluster-wide change. These actions can include things like adding or removing a cluster resource or changing resource constraints. It is important to understand what happens in the cluster when you perform such an action.

For example, suppose you want to add a cluster IP address resource. To do this, you can use the crm shell or the Web interface to modify the CIB. It is not required to perform the actions on the DC. You can use either tool on any node in the cluster and they will be relayed to the DC. The DC will then replicate the CIB change to all cluster nodes.

Based on the information in the CIB, the pacemaker-schedulerd then computes the ideal state of the cluster and how it should be achieved. It feeds a list of instructions to the DC. The DC sends commands via the messaging/infrastructure layer which are received by the pacemaker-controld peers on other nodes. Each of them uses its local resource agent executor (implemented as pacemaker-execd) to perform resource modifications. The pacemaker-execd is not cluster-aware and interacts directly with resource agents.

All peer nodes report the results of their operations back to the DC. After the DC concludes that all necessary operations are successfully performed in the cluster, the cluster will go back to the idle state and wait for further events. If any operation was not carried out as planned, the pacemaker-schedulerd is invoked again with the new information recorded in the CIB.

In some cases, it may be necessary to power off nodes to protect shared data or complete resource recovery. In a Pacemaker cluster, the implementation of node level fencing is STONITH. For this, Pacemaker comes with a fencing subsystem, pacemaker-fenced. STONITH devices have to be configured as cluster resources (that use specific fencing agents), because this allows to monitor the fencing devices. When clients detect a failure, they send a request to pacemaker-fenced, which then executes the fencing agent to bring down the node.

To successfully synchronize the files with Csync2, the following requirements must be met:

Before the first Csync2 run, you therefore need to make the following preparations:

For more information on the Csync2 options, run

csync2 -help

Note: Pushing Synchronization After Any Changes

Csync2 only pushes changes. It does not continuously synchronize files between the machines.

Each time you update files that need to be synchronized, you need to push the changes to the other machines: Run csync2  -xv on the machine where you did the changes. If you run the command on any of the other machines with unchanged files, nothing will happen.

This section applies to the following scenarios:

If your cluster is still based on an older product version than the ones listed above, first upgrade it to a version of SLES and SLE HA that can be used as a source for upgrading to the desired target version.

Note: Upgrading the CIB Syntax Version

Sometimes new features are only available with the latest CIB syntax version. When you upgrade to a new product version, your CIB syntax version will not be upgraded by default.

1. Check your version with:

   cibadmin -Q | grep validate-with

2. Upgrade to the latest CIB syntax version with:

   root # cibadmin --upgrade --force

Procedure 5.2: Upgrading from Product Version 12 to 15: Cluster Offline Upgrade #

Important: Installation from Scratch

If you decide to install the cluster nodes from scratch (instead of upgrading them), see Section 2.2, “Software Requirements” for the list of modules required for SUSE Linux Enterprise High Availability Extension 15 SP1. Find more information about modules, extensions and related products in the release notes for SUSE Linux Enterprise Server 15. They are available at https://www.suse.com/releasenotes/.

1. Before starting the offline upgrade to SUSE Linux Enterprise High Availability Extension 15, manually upgrade the CIB syntax in your current cluster as described in Note: Upgrading the CIB Syntax Version.

2. Log in to each cluster node and stop the cluster stack with:

   root # crm cluster stop

3. For each cluster node, perform an upgrade to the desired target version of SUSE Linux Enterprise Server and SUSE Linux Enterprise High Availability Extension—see Section 5.2.1, “Supported Upgrade Paths for SLE HA and SLE HA Geo”.

4. After the upgrade process has finished, log in to each node and boot it with the upgraded version of SUSE Linux Enterprise Server and SUSE Linux Enterprise High Availability Extension.

5. If you use Cluster LVM, you need to migrate from clvmd to lvmlockd. See the man page of lvmlockd, section changing a clvm VG to a lockd VG and Section 22.4, “Online Migration from Mirror LV to Cluster MD”.

6. Start the cluster stack with:

   root # crm cluster start

7. Check the cluster status with crm status or with Hawk2.

This section applies to the following scenarios:

Use one of the following procedures for your scenario:

Warning: Active Cluster Stack

Before starting an upgrade for a node, stop the cluster stack on that node.

If the cluster resource manager on a node is active during the software update, this can lead to unpredictable results like fencing of active nodes.

Important: Time Limit for Cluster Rolling Upgrade

The new features shipped with the latest product version will only be available after all cluster nodes have been upgraded to the latest product version. Mixed version clusters are only supported for a short time frame during the cluster rolling upgrade. Complete the cluster rolling upgrade within one week.

Once all the online nodes are running the upgraded version, it is not possible for any other nodes with the old version to (re-)join without having been upgraded.

Important: Time Limit for Rolling Upgrade

The new features shipped with the latest product version will only be available after all cluster nodes have been upgraded to the latest product version. Mixed version clusters are only supported for a short time frame during the rolling upgrade. Complete the rolling upgrade within one week.

The Hawk2 Status screen also shows a warning if different CRM versions are detected for your cluster nodes.

Beside an in-place upgrade, many customers prefer a fresh installation even for moving to the next service pack. The following procedure shows a scenario where a two-node cluster with the nodes alice and bob is upgraded to the next service pack (SP):

Global cluster options control how the cluster behaves when confronted with certain situations. They are grouped into sets and can be viewed and modified with the cluster management tools like Hawk2 and the crm shell.

The predefined values can usually be kept. However, to make key functions of your cluster work correctly, you need to adjust the following parameters after basic cluster setup:

This global option defines what to do when a cluster partition does not have quorum (no majority of nodes is part of the partition).

Allowed values are:

This global option defines whether to apply fencing, allowing STONITH devices to shoot failed nodes and nodes with resources that cannot be stopped. By default, this global option is set to true, because for normal cluster operation it is necessary to use STONITH devices. According to the default value, the cluster will refuse to start any resources if no STONITH resources have been defined.

If you need to disable fencing for any reasons, set stonith-enabled to false, but be aware that this has impact on the support status for your product. Furthermore, with stonith-enabled="false", resources like the Distributed Lock Manager (DLM) and all services depending on DLM (such as lvmlockd, GFS2, and OCFS2) will fail to start.

Important: No Support Without STONITH

A cluster without STONITH is not supported.

When using the bootstrap scripts, the Corosync configuration contains a quorum section with the following options:

quorum {
   # Enable and configure quorum subsystem (default: off)
   # see also corosync.conf.5 and votequorum.5
   provider: corosync_votequorum
   expected_votes: 2
   two_node: 1
}

As opposed to SUSE Linux Enterprise 11, the votequorum subsystem in SUSE Linux Enterprise 12 is powered by Corosync version 2.x. This means that the no-quorum-policy=ignore option must not be used.

By default, when two_node: 1 is set, the wait_for_all option is automatically enabled. If wait_for_all is not enbaled, the cluster should be started on both nodes in parallel. Otherwise the first node will perform a startup-fencing on the missing second node.

When not using a two-node cluster, we strongly recommend an odd number of nodes for your N-node cluster. With regards to quorum configuration, you have the following options:

If you adjust /etc/corosync/corosync.conf manually, use the following settings:

quorum {
   provider: corosync_votequorum 1
   expected_votes: N 2
   wait_for_all: 1 3
}

Before you can use a resource in the cluster, it must be set up. For example, to use an Apache server as a cluster resource, set up the Apache server first and complete the Apache configuration before starting the respective resource in your cluster.

If a resource has specific environment requirements, make sure they are present and identical on all cluster nodes. This kind of configuration is not managed by the High Availability Extension. You must do this yourself.

Note: Do Not Touch Services Managed by the Cluster

When managing a resource with the High Availability Extension, the same resource must not be started or stopped otherwise (outside of the cluster, for example manually or on boot or reboot). The High Availability Extension software is responsible for all service start or stop actions.

If you need to execute testing or maintenance tasks after the services are already running under cluster control, make sure to put the resources, nodes, or the whole cluster into maintenance mode before you touch any of them manually. For details, see Section 17.2, “Different Options for Maintenance Tasks”.

After having configured the resources in the cluster, use the cluster management tools to start, stop, clean up, remove or migrate any resources manually. For details how to do so with your preferred cluster management tool:

Important: Resource IDs and Node Names

Cluster resources and cluster nodes should be named differently. Otherwise Hawk2 will fail.

For each cluster resource you add, you need to define the standard that the resource agent conforms to. Resource agents abstract the services they provide and present an accurate status to the cluster, which allows the cluster to be non-committal about the resources it manages. The cluster relies on the resource agent to react appropriately when given a start, stop or monitor command.

Typically, resource agents come in the form of shell scripts. The High Availability Extension supports the following classes of resource agents:

The agents supplied with the High Availability Extension are written to OCF specifications.

The following types of resources can be created:

If you want to create lots of resources with similar configurations, defining a resource template is the easiest way. After having been defined, it can be referenced in primitives—or in certain types of constraints, as described in Section 6.5.3, “Resource Templates and Constraints”.

If a template is referenced in a primitive, the primitive will inherit all operations, instance attributes (parameters), meta attributes, and utilization attributes defined in the template. Additionally, you can define specific operations or attributes for your primitive. If any of these are defined in both the template and the primitive, the values defined in the primitive will take precedence over the ones defined in the template.

Learn how to define resource templates with your preferred cluster configuration tool:

Whereas primitives are the simplest kind of resources and therefore easy to configure, you will probably also need more advanced resource types for cluster configuration, such as groups, clones or promotable clone resources.

6.3.5.1 Groups #Edit source

Some cluster resources depend on other components or resources. They require that each component or resource starts in a specific order and runs together on the same server with resources it depends on. To simplify this configuration, you can use cluster resource groups.

Example 6.3: Resource Group for a Web Server #

An example of a resource group would be a Web server that requires an IP address and a file system. In this case, each component is a separate resource that is combined into a cluster resource group. The resource group would run on one or more servers. In case of a software or hardware malfunction, the group would fail over to another server in the cluster, similar to an individual cluster resource.

Figure 6.1: Group Resource #

Groups have the following properties:

Starting and Stopping

Resources are started in the order they appear in and stopped in the reverse order.

Dependency

If a resource in the group cannot run anywhere, then none of the resources located after that resource in the group is allowed to run.

Contents

Groups may only contain a collection of primitive cluster resources. Groups must contain at least one resource, otherwise the configuration is not valid. To refer to the child of a group resource, use the child’s ID instead of the group’s ID.

Constraints

Although it is possible to reference the group’s children in constraints, it is usually preferable to use the group’s name instead.

Stickiness

Stickiness is additive in groups. Every active member of the group will contribute its stickiness value to the group’s total. So if the default resource-stickiness is 100 and a group has seven members (five of which are active), the group as a whole will prefer its current location with a score of 500.

Resource Monitoring

To enable resource monitoring for a group, you must configure monitoring separately for each resource in the group that you want monitored.

Learn how to create groups with your preferred cluster management tool:

• Hawk2: Procedure 7.9, “Adding a Resource Group”

• crmsh: Section 8.3.10, “Configuring a Cluster Resource Group”

For each resource you add, you can define options. Options are used by the cluster to decide how your resource should behave—they tell the CRM how to treat a specific resource. Resource options can be set with the crm_resource --meta command or with Hawk2 as described in Procedure 7.5, “Adding a Primitive Resource”.

Table 6.1: Options for a Primitive Resource #

Option	Description	Default
priority	If not all resources can be active, the cluster will stop lower priority resources to keep higher priority ones active.	0
target-role	In what state should the cluster attempt to keep this resource? Allowed values: stopped, started, master.	started
is-managed	Is the cluster allowed to start and stop the resource? Allowed values: true, false. If the value is set to false, the status of the resource is still monitored and any failures are reported. This is different from setting a resource to maintenance="true".	true
maintenance	Can the resources be touched manually? Allowed values: true, false. If set to true, all resources become unmanaged: the cluster will stop monitoring them and thus be oblivious about their status. You can stop or restart cluster resources at will, without the cluster attempting to restart them.	false
resource-stickiness	How much does the resource prefer to stay where it is?	calculated
migration-threshold	How many failures should occur for this resource on a node before making the node ineligible to host this resource?	INFINITY (disabled)
multiple-active	What should the cluster do if it ever finds the resource active on more than one node? Allowed values: block (mark the resource as unmanaged), stop_only, stop_start.	stop_start
failure-timeout	How many seconds to wait before acting as if the failure had not occurred (and potentially allowing the resource back to the node on which it failed)?	0 (disabled)
allow-migrate	Allow resource migration for resources which support migrate_to/migrate_from actions.	false
remote-node	The name of the remote node this resource defines. This both enables the resource as a remote node and defines the unique name used to identify the remote node. If no other parameters are set, this value will also be assumed as the host name to connect to at remote-portport. Warning: Use Unique IDs This value must not overlap with any existing resource or node IDs.	none (disabled)
remote-port	Custom port for the guest connection to pacemaker_remote.	3121
remote-addr	The IP address or host name to connect to if the remote node’s name is not the host name of the guest.	remote-node (value used as host name)
remote-connect-timeout	How long before a pending guest connection will time out.	60s

The scripts of all resource classes can be given parameters which determine how they behave and which instance of a service they control. If your resource agent supports parameters, you can add them with the crm_resource command or with Hawk2 as described in Procedure 7.5, “Adding a Primitive Resource”. In the crm command line utility and in Hawk2, instance attributes are called params or Parameter, respectively. The list of instance attributes supported by an OCF script can be found by executing the following command as root:

root # crm ra info [class:[provider:]]resource_agent

or (without the optional parts):

root # crm ra info resource_agent

The output lists all the supported attributes, their purpose and default values.

For example, the command

root # crm ra info IPaddr

returns the following output:

Manages virtual IPv4 addresses (portable version) (ocf:heartbeat:IPaddr)
    
This script manages IP alias IP addresses
It can add an IP alias, or remove one.   
    
Parameters (* denotes required, [] the default):
    
ip* (string): IPv4 address
The IPv4 address to be configured in dotted quad notation, for example
"192.168.1.1".                                                        
    
nic (string, [eth0]): Network interface
The base network interface on which the IP address will be brought
online.                                                           
    
If left empty, the script will try and determine this from the    
routing table.                                                    
    
Do NOT specify an alias interface in the form eth0:1 or anything here;
rather, specify the base interface only.                              
    
cidr_netmask (string): Netmask
The netmask for the interface in CIDR format. (ie, 24), or in
dotted quad notation  255.255.255.0).                        
    
If unspecified, the script will also try to determine this from the
routing table.                                                     
    
broadcast (string): Broadcast address
Broadcast address associated with the IP. If left empty, the script will
determine this from the netmask.                                        
    
iflabel (string): Interface label
You can specify an additional label for your IP address here.
    
lvs_support (boolean, [false]): Enable support for LVS DR
Enable support for LVS Direct Routing configurations. In case a IP
address is stopped, only move it to the loopback device to allow the
local node to continue to service requests, but no longer advertise it
on the network.                                                       
    
local_stop_script (string): 
Script called when the IP is released
    
local_start_script (string): 
Script called when the IP is added
    
ARP_INTERVAL_MS (integer, [500]): milliseconds between gratuitous ARPs
milliseconds between ARPs                                         
    
ARP_REPEAT (integer, [10]): repeat count
How many gratuitous ARPs to send out when bringing up a new address
    
ARP_BACKGROUND (boolean, [yes]): run in background
run in background (no longer any reason to do this)
    
ARP_NETMASK (string, [ffffffffffff]): netmask for ARP
netmask for ARP - in nonstandard hexadecimal format.
    
Operations' defaults (advisory minimum):
    
start         timeout=90
stop          timeout=100
monitor_0     interval=5s timeout=20s

Note: Instance Attributes for Groups, Clones or Promotable Clones

Note that groups, clones and promotable clone resources do not have instance attributes. However, any instance attributes set will be inherited by the group's, clone's or promotable clone's children.

By default, the cluster will not ensure that your resources are still healthy. To instruct the cluster to do this, you need to add a monitor operation to the resource’s definition. Monitor operations can be added for all classes or resource agents. For more information, refer to Section 6.4, “Resource Monitoring”.

Timeouts values for resources can be influenced by the following parameters:

Note: Priority of Values

If a specific value is defined for a resource, it takes precedence over the global default. A specific value for a resource also takes precedence over a value that is defined in a resource template.

Getting timeout values right is very important. Setting them too low will result in a lot of (unnecessary) fencing operations for the following reasons:

You can adjust the global default for operations and set any specific timeout values with both crmsh and Hawk2. The best practice for determining and setting timeout values is as follows:

There are three different kinds of constraints available:

Important: Restrictions for Constraints and Certain Types of Resources

• Do not create colocation constraints for members of a resource group. Create a colocation constraint pointing to the resource group as a whole instead. All other types of constraints are safe to use for members of a resource group.

• Do not use any constraints on a resource that has a clone resource or a promotable clone resource applied to it. The constraints must apply to the clone or promotable clone resource, not to the child resource.

crm(live)configure# primitive vip1 ocf:heartbeat:IPaddr2 params ip=192.168.1.5
crm(live)configure# primitive vip2 ocf:heartbeat:IPaddr2 params ip=192.168.1.6
crm(live)configure# location loc-alice { vip1 vip2 } Mandatory: alice

<constraints>
     <rsc_colocation id="coloc-1" rsc="B" with-rsc="A" score="INFINITY"/>
     <rsc_colocation id="coloc-2" rsc="C" with-rsc="B" score="INFINITY"/>
     <rsc_colocation id="coloc-3" rsc="D" with-rsc="C" score="INFINITY"/>
</constraints>

<constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
     <resource_set id="colocated-set-example" sequential="true">
      <resource_ref id="A"/>
      <resource_ref id="B"/>
      <resource_ref id="C"/>
      <resource_ref id="D"/>
     </resource_set>
    </rsc_colocation>
</constraints>

<constraints>
     <rsc_order id="order-1" first="A" then="B" />
     <rsc_order id="order-2" first="B" then="C" />
     <rsc_order id="order-3" first="C" then="D" />
</constraints>

<constraints>
     <rsc_order id="order-1">
     <resource_set id="ordered-set-example" sequential="true">
     <resource_ref id="A"/>
     <resource_ref id="B"/>
     <resource_ref id="C"/>
     <resource_ref id="D"/>
     </resource_set>
     </rsc_order>
</constraints>

When defining constraints, you also need to deal with scores. Scores of all kinds are integral to how the cluster works. Practically everything from migrating a resource to deciding which resource to stop in a degraded cluster is achieved by manipulating scores in some way. Scores are calculated on a per-resource basis and any node with a negative score for a resource cannot run that resource. After calculating the scores for a resource, the cluster then chooses the node with the highest score.

INFINITY is currently defined as 1,000,000. Additions or subtractions with it stick to the following three basic rules:

When defining resource constraints, you specify a score for each constraint. The score indicates the value you are assigning to this resource constraint. Constraints with higher scores are applied before those with lower scores. By creating additional location constraints with different scores for a given resource, you can specify an order for the nodes that a resource will fail over to.

If you have defined a resource template (see Section 6.3.4, “Resource Templates”), it can be referenced in the following types of constraints:

However, colocation constraints must not contain more than one reference to a template. Resource sets must not contain a reference to a template.

Resource templates referenced in constraints stand for all primitives which are derived from that template. This means, the constraint applies to all primitive resources referencing the resource template. Referencing resource templates in constraints is an alternative to resource sets and can simplify the cluster configuration considerably. For details about resource sets, refer to Procedure 7.17, “Using a Resource Set for Constraints”.

A resource will be automatically restarted if it fails. If that cannot be achieved on the current node, or it fails N times on the current node, it will try to fail over to another node. Each time the resource fails, its failcount is raised. You can define several failures for resources (a migration-threshold), after which they will migrate to a new node. If you have more than two nodes in your cluster, the node a particular resource fails over to is chosen by the High Availability software.

However, you can specify the node a resource will fail over to by configuring one or several location constraints and a migration-threshold for that resource.

Learn how to specify failover nodes with your preferred cluster management tool:

There are two exceptions to the migration threshold concept, occurring when a resource either fails to start or fails to stop:

For details on using migration thresholds and resetting failcounts with your preferred cluster management tool:

A resource might fail back to its original node when that node is back online and in the cluster. To prevent a resource from failing back to the node that it was running on, or to specify a different node for the resource to fail back to, change its resource stickiness value. You can either specify resource stickiness when you are creating a resource or afterward.

Consider the following implications when specifying resource stickiness values:

Not all resources are equal. Some, such as Xen guests, require that the node hosting them meets their capacity requirements. If resources are placed such that their combined need exceed the provided capacity, the resources diminish in performance (or even fail).

To take this into account, the High Availability Extension allows you to specify the following parameters:

Learn how to configure these settings with your preferred cluster management tool:

A node is considered eligible for a resource if it has sufficient free capacity to satisfy the resource's requirements. The nature of the capacities is completely irrelevant for the High Availability Extension; it only makes sure that all capacity requirements of a resource are satisfied before moving a resource to a node.

To manually configure the resource's requirements and the capacity a node provides, use utilization attributes. You can name the utilization attributes according to your preferences and define as many name/value pairs as your configuration needs. However, the attribute's values must be integers.

If multiple resources with utilization attributes are grouped or have colocation constraints, the High Availability Extension takes that into account. If possible, the resources will be placed on a node that can fulfill all capacity requirements.

Note: Utilization Attributes for Groups

It is impossible to set utilization attributes directly for a resource group. However, to simplify the configuration for a group, you can add a utilization attribute with the total capacity needed to any of the resources in the group.

The High Availability Extension also provides means to detect and configure both node capacity and resource requirements automatically:

The NodeUtilization resource agent checks the capacity of a node (regarding CPU and RAM). To configure automatic detection, create a clone resource of the following class, provider, and type: ocf:pacemaker:NodeUtilization. One instance of the clone should be running on each node. After the instance has started, a utilization section will be added to the node's configuration in CIB.

For automatic detection of a resource's minimal requirements (regarding RAM and CPU) the Xen resource agent has been improved. Upon start of a Xen resource, it will reflect the consumption of RAM and CPU. Utilization attributes will automatically be added to the resource configuration.

Note: Different Resource Agents for Xen and libvirt

The ocf:heartbeat:Xen resource agent should not be used with libvirt, as libvirt expects to be able to modify the machine description file.

For libvirt, use the ocf:heartbeat:VirtualDomain resource agent.

Apart from detecting the minimal requirements, the High Availability Extension also allows to monitor the current utilization via the VirtualDomain resource agent. It detects CPU and RAM use of the virtual machine. To use this feature, configure a resource of the following class, provider and type: ocf:heartbeat:VirtualDomain. The following instance attributes are available: autoset_utilization_cpu and autoset_utilization_hv_memory. Both default to true. This updates the utilization values in the CIB during each monitoring cycle.

Independent of manually or automatically configuring capacity and requirements, the placement strategy must be specified with the placement-strategy property (in the global cluster options). The following values are available:

Note: Configuring Resource Priorities

The available placement strategies are best-effort—they do not yet use complex heuristic solvers to always reach optimum allocation results. Ensure that resource priorities are properly set so that your most important resources are scheduled first.

node alice utilization memory="4000"
node bob utilization memory="4000"
node  charlie utilization memory="4000"
primitive xenA ocf:heartbeat:Xen utilization hv_memory="3500" \
     params xmfile="/etc/xen/shared-vm/vm1"
     meta priority="10" 
primitive xenB ocf:heartbeat:Xen utilization hv_memory="2000" \
     params xmfile="/etc/xen/shared-vm/vm2"
     meta priority="1"
primitive xenC ocf:heartbeat:Xen utilization hv_memory="2000" \
     params xmfile="/etc/xen/shared-vm/vm3"
     meta priority="1"
primitive xenD ocf:heartbeat:Xen utilization hv_memory="1000" \
     params xmfile="/etc/xen/shared-vm/vm4"
     meta priority="5"
property placement-strategy="minimal"

Tags are a new feature that has been added to Pacemaker recently. Tags are a way to refer to multiple resources at once, without creating any colocation or ordering relationship between them. This can be useful for grouping conceptually related resources. For example, if you have several resources related to a database, create a tag called databases and add all resources related to the database to this tag. This allows you to stop or start them all with a single command.

Tags can also be used in constraints. For example, the following location constraint loc-db-prefer applies to the set of resources tagged with databases:

location loc-db-prefer databases 100: alice

Learn how to create tags with your preferred cluster management tool:

Monitoring of virtual machines can be done with the VM agent (which only checks if the guest shows up in the hypervisor), or by external scripts called from the VirtualDomain or Xen agent. Up to now, more fine-grained monitoring was only possible with a full setup of the High Availability stack within the virtual machines.

By providing support for monitoring plug-ins (formerly named Nagios plug-ins), the High Availability Extension now also allows you to monitor services on remote hosts. You can collect external statuses on the guests without modifying the guest image. For example, VM guests might run Web services or simple network resources that need to be accessible. With the Nagios resource agents, you can now monitor the Web service or the network resource on the guest. In case these services are not reachable anymore, the High Availability Extension will trigger a restart or migration of the respective guest.

If your guests depend on a service (for example, an NFS server to be used by the guest), the service can either be an ordinary resource, managed by the cluster, or an external service that is monitored with Nagios resources instead.

To configure the Nagios resources, the following packages must be installed on the host:

YaST or Zypper will resolve any dependencies on further packages, if required.

A typical use case is to configure the monitoring plug-ins as resources belonging to a resource container, which usually is a VM. The container will be restarted if any of its resources has failed. Refer to Example 6.10, “Configuring Resources for Monitoring Plug-ins” for a configuration example. Alternatively, Nagios resource agents can also be configured as ordinary resources to use them for monitoring hosts or services via the network.

primitive vm1 ocf:heartbeat:VirtualDomain \
    params hypervisor="qemu:///system" config="/etc/libvirt/qemu/vm1.xml" \
    op start interval="0" timeout="90" \
    op stop interval="0" timeout="90" \
    op monitor interval="10" timeout="30"
primitive vm1-sshd nagios:check_tcp \
    params hostname="vm1" port="22" \ 1
    op start interval="0" timeout="120" \ 2
    op monitor interval="10"
group g-vm1-and-services vm1 vm1-sshd \
    meta container="vm1" 3

group g-vm1-and-services vm1 vm1-sshd vm1-httpd \ 
     meta container="vm1" \ 
     params hostname="vm1"

With the pacemaker_remote service, High Availability clusters can be extended to virtual nodes or remote bare-metal machines. They do not need to run the cluster stack to become members of the cluster.

The High Availability Extension can now launch virtual environments (KVM and LXC), plus the resources that live within those virtual environments without requiring the virtual environments to run Pacemaker or Corosync.

For the use case of managing both virtual machines as cluster resources plus the resources that live within the VMs, you can now use the following setup:

As the remote nodes do not have the cluster stack installed, this has the following implications:

Find more information about the remote_pacemaker service, including multiple use cases with detailed setup instructions in Article “Pacemaker Remote Quick Start”.

Hawk2's top-level row shows the following entries:

Sometimes a cluster administrator needs to know the cluster configuration. Hawk2 can show the current configuration in crm shell syntax, as XML and as a graph. To view the cluster configuration in crm shell syntax, from the left navigation bar select Configuration › Edit Configuration and click Show. To show the configuration in raw XML instead, click XML. Click Graph for a graphical representation of the nodes and resources configured in the CIB. It also shows the relationships between resources.

The Hawk2 wizard is a convenient way of setting up simple resources like a virtual IP address or an SBD STONITH resource, for example. It is also useful for complex configurations that include multiple resources, like the resource configuration for a DRBD block device or an Apache Web server. The wizard guides you through the configuration steps and provides information about the parameters you need to enter.

Figure 7.2: Hawk2—Wizard for Apache Web Server #

To create the most basic type of resource, proceed as follows:

To create lots of resources with similar configurations, defining a resource template is the easiest way. After being defined, it can be referenced in primitives or in certain types of constraints. For detailed information about function and use of resource templates, refer to Section 6.5.3, “Resource Templates and Constraints”.

If you have created a resource, you can edit its configuration at any time by adjusting parameters, operations, or meta attributes as needed.

Important: No Support Without STONITH

• You must have a node fencing mechanism for your cluster.

• The global cluster options stonith-enabled and startup-fencing must be set to true. When you change them, you lose support.

By default, the global cluster option stonith-enabled is set to true. If no STONITH resources have been defined, the cluster will refuse to start any resources. Configure one or more STONITH resources to complete the STONITH setup. To add a STONITH resource for SBD, for libvirt (KVM/Xen) or for vCenter/ESX Server, the easiest way is to use the Hawk2 wizard (see Section 7.5.2, “Adding Resources with the Wizard”). While STONITH resources are configured similarly to other resources, their behavior is different in some respects. For details refer to Section 10.3, “STONITH Resources and Configuration”.

To complete your fencing configuration, add constraints. For more details, refer to Chapter 10, Fencing and STONITH.

Some cluster resources depend on other components or resources. They require that each component or resource starts in a specific order and runs on the same server. To simplify this configuration SUSE Linux Enterprise High Availability Extension supports the concept of groups.

Resource groups contain a set of resources that need to be located together, be started sequentially and stopped in the reverse order. For an example of a resource group and more information about groups and their properties, refer to Section 6.3.5.1, “Groups”.

Note: Empty Groups

Groups must contain at least one resource, otherwise the configuration is not valid. While creating a group, Hawk2 allows you to create more primitives and add them to the group. For details, see Section 7.7.1, “Editing Resources and Groups”.

Figure 7.6: Hawk2—Resource Group #

If you want certain resources to run simultaneously on multiple nodes in your cluster, configure these resources as clones. An example of a resource that can be configured as a clone is ocf:pacemaker:controld for cluster file systems like OCFS2. Any regular resources or resource groups can be cloned. Instances of cloned resources may behave identically. However, they may also be configured differently, depending on which node they are hosted on.

For an overview of the available types of resource clones, refer to Section 6.3.5.2, “Clones”.

Note: Child Resources for Clones

Clones can either contain a primitive or a group as child resources. In Hawk2, child resources cannot be created or modified while creating a clone. Before adding a clone, create child resources and configure them as desired. For details, refer to Section 7.5.3, “Adding Simple Resources” or Section 7.5.7, “Adding Cluster Resource Groups”.

Figure 7.7: Hawk2—Clone Resource #

Multi-state resources are a specialization of clones. They allow the instances to be in one of two operating modes (called active/passive, primary/secondary, or master/slave). Multi-state resources must contain exactly one group or one regular resource.

When configuring resource monitoring or constraints, multi-state resources have different requirements than simple resources. For details, see Pacemaker Explained, available from http://www.clusterlabs.org/pacemaker/doc/. Refer to section Multi-state - Resources That Have Multiple Modes.

Note: Child Resources for Multi-state Resources

Multi-state resources can either contain a primitive or a group as child resources. In Hawk2, child resources cannot be created or modified while creating a multi-state resource. Before adding a multi-state resource, create child resources and configure them as desired. For details, refer to Section 7.5.3, “Adding Simple Resources” or Section 7.5.7, “Adding Cluster Resource Groups”.

Figure 7.8: Hawk2—Multi-state Resource #

Tags are a way to refer to multiple resources at once, without creating any colocation or ordering relationship between them. You can use tags for grouping conceptually related resources. For example, if you have several resources related to a database, you can add all related resources to a tag named database.

All resources belonging to a tag can be started or stopped with a single command.

Figure 7.9: Hawk2—Tag #

The High Availability Extension does not only detect node failures, but also when an individual resource on a node has failed. If you want to ensure that a resource is running, configure resource monitoring for it. Usually, resources are only monitored by the cluster while they are running. However, to detect concurrency violations, also configure monitoring for resources which are stopped. For resource monitoring, specify a timeout and/or start delay value, and an interval. The interval tells the CRM how often it should check the resource status. You can also set particular parameters such as timeout for start or stop operations.

For the processes that take place if the resource monitor detects a failure, refer to Section 6.4, “Resource Monitoring”.

To view resource failures, switch to the Status screen in Hawk2 and select the resource you are interested in. In the Operations column click the arrow down icon and select Recent Events. The dialog that opens lists recent actions performed for the resource. Failures are displayed in red. To view the resource details, click the magnifier icon in the Operations column.

Figure 7.10: Hawk2—Resource Details #

A location constraint determines on which node a resource may be run, is preferably run, or may not be run. An example of a location constraint is to place all resources related to a certain database on the same node.

Figure 7.11: Hawk2—Location Constraint #

A colocational constraint tells the cluster which resources may or may not run together on a node. As a colocation constraint defines a dependency between resources, you need at least two resources to create a colocation constraint.

Figure 7.12: Hawk2—Colocation Constraint #

Order constraints define the order in which resources are started and stopped. As an order constraint defines a dependency between resources, you need at least two resources to create an order constraint.

Figure 7.13: Hawk2—Order Constraint #

As an alternative format for defining constraints, you can use Resource Sets. They have the same ordering semantics as Groups.

For more information on configuring constraints and detailed background information about the basic concepts of ordering and colocation, refer to the documentation available at http://www.clusterlabs.org/pacemaker/doc/:

A resource will be automatically restarted if it fails. If that cannot be achieved on the current node, or it fails N times on the current node, it will try to fail over to another node. You can define several failures for resources (a migration-threshold), after which they will migrate to a new node. If you have more than two nodes in your cluster, the node to which a particular resource fails over is chosen by the High Availability software.

You can specify a specific node to which a resource will fail over by proceeding as follows:

The process flow regarding migration thresholds and failcounts is demonstrated in Example 6.8, “Migration Threshold—Process Flow”.

Instead of letting the failcount for a resource expire automatically, you can also clean up failcounts for a resource manually at any time. Refer to Section 7.7.3, “Cleaning Up Resources” for details.

A resource may fail back to its original node when that node is back online and in the cluster. To prevent this or to specify a different node for the resource to fail back to, change the stickiness value of the resource. You can either specify the resource stickiness when creating it or afterward.

For the implications of different resource stickiness values, refer to Section 6.5.5, “Failback Nodes”.