High Availability is available as a module or extension for several products. For details, see https://documentation.suse.com/sles/html/SLES-all/article-modules.html#art-modules-high-availability.

SUSE® Linux Enterprise High Availability helps you ensure and manage the availability of your network resources. The following sections highlight some of the key features:

1.2.1 Wide range of clustering scenarios #

  

SUSE Linux Enterprise High Availability supports the following scenarios:

• Active/active configurations

• Active/passive configurations: N+1, N+M, N to 1, N to M

• Hybrid physical and virtual clusters, allowing virtual servers to be clustered with physical servers. This improves service availability and resource usage.

• Local clusters

• Metro clusters (“stretched” local clusters)

• Geo clusters (geographically dispersed clusters)

Important: No support for mixed architectures

All nodes belonging to a cluster should have the same processor platform: x86, IBM Z, or POWER. Clusters of mixed architectures are not supported.

Your cluster can contain up to 32 Linux servers. Using pacemaker_remote, the cluster can be extended to include additional Linux servers beyond this limit. Any server in the cluster can restart resources (applications, services, IP addresses, and file systems) from a failed server in the cluster.

1.2.4 Support for virtualized environments #

  

SUSE Linux Enterprise High Availability supports the mixed clustering of both physical and virtual Linux servers. SUSE Linux Enterprise Server 15 SP4 ships with Xen, an open source virtualization hypervisor, and with KVM (Kernel-based Virtual Machine). KVM is a virtualization software for Linux which is based on hardware virtualization extensions. The cluster resource manager in SUSE Linux Enterprise High Availability can recognize, monitor, and manage services running within virtual servers and services running in physical servers. Guest systems can be managed as services by the cluster.

Important: Live migration in High Availability clusters

Use caution when performing live migration of nodes in an active cluster. The cluster stack might not tolerate an operating system freeze caused by the live migration process, which could lead to the node being fenced.

We recommend either of the following actions to help avoid node fencing during live migration:

• Increase the Corosync token timeout and the SBD watchdog timeout, along with any other related settings. The appropriate values depend on your specific setup. For more information, see Section 13.5, “Calculation of timeouts”.

• Before performing live migration, stop the cluster services on either the node or the whole cluster. For more information, see Section 28.2, “Different options for maintenance tasks”.

You must thoroughly test this setup before attempting live migration in a production environment.

SUSE Linux Enterprise High Availability allows you to configure up to 32 Linux servers into a high-availability cluster (HA cluster). Resources can be dynamically switched or moved to any node in the cluster. Resources can be configured to automatically migrate if a node fails, or they can be moved manually to troubleshoot hardware or balance the workload.

SUSE Linux Enterprise High Availability provides high availability from commodity components. Lower costs are obtained through the consolidation of applications and operations onto a cluster. SUSE Linux Enterprise High Availability also allows you to centrally manage the complete cluster. You can adjust resources to meet changing workload requirements (thus, manually “load balance” the cluster). Allowing clusters of more than two nodes also provides savings by allowing several nodes to share a “hot spare”.

An equally important benefit is the potential reduction of unplanned service outages and planned outages for software and hardware maintenance and upgrades.

Reasons that you would want to implement a cluster include:

Shared disk fault tolerance can be obtained by implementing RAID on the shared disk subsystem.

The following scenario illustrates some benefits SUSE Linux Enterprise High Availability can provide.

Suppose you have configured a three-node cluster, with a Web server installed on each of the three nodes in the cluster. Each of the nodes in the cluster hosts two Web sites. All the data, graphics, and Web page content for each Web site are stored on a shared disk subsystem connected to each of the nodes in the cluster. The following figure depicts how this setup might look.

Figure 1.1: Three-server cluster #

  

During normal cluster operation, each node is in constant communication with the other nodes in the cluster and performs periodic polling of all registered resources to detect failure.

Suppose Web Server 1 experiences hardware or software problems and the users depending on Web Server 1 for Internet access, e-mail, and information lose their connections. The following figure shows how resources are moved when Web Server 1 fails.

Figure 1.2: Three-server cluster after one server fails #

  

Web Site A moves to Web Server 2 and Web Site B moves to Web Server 3. IP addresses and certificates also move to Web Server 2 and Web Server 3.

When you configured the cluster, you decided where the Web sites hosted on each Web server would go should a failure occur. In the previous example, you configured Web Site A to move to Web Server 2 and Web Site B to move to Web Server 3. This way, the workload formerly handled by Web Server 1 continues to be available and is evenly distributed between any surviving cluster members.

When Web Server 1 failed, the High Availability software did the following:

In this example, the failover process happened quickly and users regained access to Web site information within seconds, usually without needing to log in again.

Now suppose the problems with Web Server 1 are resolved, and Web Server 1 is returned to a normal operating state. Web Site A and Web Site B can either automatically fail back (move back) to Web Server 1, or they can stay where they are. This depends on how you configured the resources for them. Migrating the services back to Web Server 1 will incur some down-time. Therefore SUSE Linux Enterprise High Availability also allows you to defer the migration until a period when it will cause little or no service interruption. There are advantages and disadvantages to both alternatives.

SUSE Linux Enterprise High Availability also provides resource migration capabilities. You can move applications, Web sites, etc. to other servers in your cluster as required for system management.

For example, you could have manually moved Web Site A or Web Site B from Web Server 1 to either of the other servers in the cluster. Use cases for this are upgrading or performing scheduled maintenance on Web Server 1, or increasing performance or accessibility of the Web sites.

Cluster configurations with SUSE Linux Enterprise High Availability might or might not include a shared disk subsystem. The shared disk subsystem can be connected via high-speed Fibre Channel cards, cables, and switches, or it can be configured to use iSCSI. If a node fails, another designated node in the cluster automatically mounts the shared disk directories that were previously mounted on the failed node. This gives network users continuous access to the directories on the shared disk subsystem.

Important: Shared disk subsystem with LVM

When using a shared disk subsystem with LVM, that subsystem must be connected to all servers in the cluster from which it needs to be accessed.

Typical resources might include data, applications, and services. The following figures show how a typical Fibre Channel cluster configuration might look. The green lines depict connections to an Ethernet power switch. Such a device can be controlled over a network and can reboot a node when a ping request fails.

Figure 1.3: Typical Fibre Channel cluster configuration #

  

Although Fibre Channel provides the best performance, you can also configure your cluster to use iSCSI. iSCSI is an alternative to Fibre Channel that can be used to create a low-cost Storage Area Network (SAN). The following figure shows how a typical iSCSI cluster configuration might look.

Figure 1.4: Typical iSCSI cluster configuration #

  

Although most clusters include a shared disk subsystem, it is also possible to create a cluster without a shared disk subsystem. The following figure shows how a cluster without a shared disk subsystem might look.

Figure 1.5: Typical cluster configuration without shared storage #

  

This section provides a brief overview of SUSE Linux Enterprise High Availability architecture. It identifies and provides information on the architectural components, and describes how those components interoperate.

1.5.1 Architecture layers #

  

SUSE Linux Enterprise High Availability has a layered architecture. Figure 1.6, “Architecture” illustrates the different layers and their associated components.

Figure 1.6: Architecture #

  

1.5.1.1 Membership and messaging layer (Corosync) #

  

This component provides reliable messaging, membership, and quorum information about the cluster. This is handled by the Corosync cluster engine, a group communication system.

1.5.1.2 Cluster resource manager (Pacemaker) #

  

Pacemaker as cluster resource manager is the “brain” which reacts to events occurring in the cluster. It is implemented as pacemaker-controld, the cluster controller, which coordinates all actions. Events can be nodes that join or leave the cluster, failure of resources, or scheduled activities such as maintenance, for example.

Local resource manager

The local resource manager is located between the Pacemaker layer and the resources layer on each node. It is implemented as pacemaker-execd daemon. Through this daemon, Pacemaker can start, stop, and monitor resources.

Cluster Information Database (CIB)

On every node, Pacemaker maintains the cluster information database (CIB). It is an XML representation of the cluster configuration (including cluster options, nodes, resources, constraints and the relationship to each other). The CIB also reflects the current cluster status. Each cluster node contains a CIB replica, which is synchronized across the whole cluster. The pacemaker-based daemon takes care of reading and writing cluster configuration and status.

Designated Coordinator (DC)

The DC is elected from all nodes in the cluster. This happens if there is no DC yet or if the current DC leaves the cluster for any reason. The DC is the only entity in the cluster that can decide that a cluster-wide change needs to be performed, such as fencing a node or moving resources around. All other nodes get their configuration and resource allocation information from the current DC.

Policy Engine

The policy engine runs on every node, but the one on the DC is the active one. The engine is implemented as pacemaker-schedulerd daemon. When a cluster transition is needed, based on the current state and configuration, pacemaker-schedulerd calculates the expected next state of the cluster. It determines what actions need to be scheduled to achieve the next state.

1.5.1.3 Resources and resource agents #

  

In a High Availability cluster, the services that need to be highly available are called resources. Resource agents (RAs) are scripts that start, stop, and monitor cluster resources.

The following list specifies hardware requirements for a cluster based on SUSE® Linux Enterprise High Availability. These requirements represent the minimum hardware configuration. Additional hardware might be necessary, depending on how you intend to use your cluster.

All nodes that will be part of the cluster need at least the following modules and extensions:

Depending on the system roles you select during installation, the following software patterns are installed by default:

Note: Minimal installation

An installation via those system roles results in a minimal installation only. You might need to add more packages manually, if required.

For machines that originally had another system role assigned, you need to manually install the sles_ha or ha_geo patterns and any further packages that you need.

Some services require shared storage. If using an external NFS share, it must be reliably accessible from all cluster nodes via redundant communication paths.

To make data highly available, a shared disk system (Storage Area Network, or SAN) is recommended for your cluster. If a shared disk subsystem is used, ensure the following:

When using SBD as STONITH mechanism, additional requirements apply for the shared storage. For details, see Section 13.3, “Requirements and restrictions”.

For a supported and useful High Availability setup, consider the following recommendations:

For the manual installation of the packages for High Availability refer to Article “Installation and Setup Quick Start”. It leads you through the setup of a basic two-node cluster.

After you have installed and set up a two-node cluster, you can extend the cluster by cloning existing nodes with AutoYaST and adding the clones to the cluster.

AutoYaST uses profiles that contains installation and configuration data. A profile tells AutoYaST what to install and how to configure the installed system to get a ready-to-use system in the end. This profile can then be used for mass deployment in different ways (for example, to clone existing cluster nodes).

For detailed instructions on how to use AutoYaST in various scenarios, see the AutoYaST Guide for SUSE Linux Enterprise Server 15 SP4.

Important: Identical hardware

Procedure 3.1, “Cloning a cluster node with AutoYaST” assumes you are rolling out SUSE Linux Enterprise High Availability 15 SP4 to a set of machines with identical hardware configurations.

If you need to deploy cluster nodes on non-identical hardware, refer to the Deployment Guide for SUSE Linux Enterprise Server 15 SP4, chapter Automated Installation, section Rule-Based Autoinstallation.

After the clone has been successfully installed, execute the following steps to make the cloned node join the cluster:

The cloned node will now join the cluster because the /etc/corosync/corosync.conf file has been applied to the cloned node via Csync2. The CIB is automatically synchronized among the cluster nodes.

Several key terms used in the YaST cluster module and in this chapter are defined below.

Start YaST and select High Availability › Cluster. Alternatively, start the module from command line:

sudo yast2 cluster

The following list shows an overview of the available screens in the YaST cluster module. It also mentions whether the screen contains parameters that are required for successful cluster setup or whether its parameters are optional.

If you start the cluster module for the first time, it appears as a wizard, guiding you through all the steps necessary for basic setup. Otherwise, click the categories on the left panel to access the configuration options for each step.

Note: Settings in the YaST Cluster module

Some settings in the YaST cluster module apply only to the current node. Other settings may automatically be transferred to all nodes with Csync2. Find detailed information about this in the following sections.

For successful communication between the cluster nodes, define at least one communication channel. As transport protocol, either use multicast (UDP) or unicast (UDPU) as described in Procedure 4.1 or Procedure 4.2, respectively. If you want to define a second, redundant channel (Procedure 4.3), both communication channels must use the same protocol.

Note: Public clouds: use unicast

For deploying SUSE Linux Enterprise High Availability in public cloud platforms, use unicast as transport protocol. Multicast is generally not supported by the cloud platforms themselves.

All settings defined in the YaST Communication Channels screen are written to /etc/corosync/corosync.conf. Find example files for a multicast and a unicast setup in /usr/share/doc/packages/corosync/.

If you are using IPv4 addresses, node IDs are optional. If you are using IPv6 addresses, node IDs are required. Instead of specifying IDs manually for each node, the YaST cluster module contains an option to automatically generate a unique ID for every cluster node.

Figure 4.1: YaST Cluster—multicast configuration #

  

To use unicast instead of multicast for cluster communication, proceed as follows.

Figure 4.2: YaST Cluster—unicast configuration #

  

If network device bonding cannot be used for any reason, the second best choice is to define a redundant communication channel (a second ring) in Corosync. That way, two physically separate networks can be used for communication. If one network fails, the cluster nodes can still communicate via the other network.

The additional communication channel in Corosync will form a second token-passing ring. In /etc/corosync/corosync.conf, the first channel you configured is the primary ring and gets the ring number 0. The second ring (redundant channel) gets the ring number 1.

When having defined redundant communication channels in Corosync, use RRP to tell the cluster how to use these interfaces. With RRP, two physically separate networks are used for communication. If one network fails, the cluster nodes can still communicate via the other network.

RRP can have three modes:

Procedure 4.3: Defining a redundant communication channel #

  

Important: Redundant rings and /etc/hosts

If multiple rings are configured in Corosync, each node can have multiple IP addresses. This needs to be reflected in the /etc/hosts file of all nodes.

1. Start the YaST cluster module and switch to the Communication Channels category.

2. Activate Redundant Channel. The redundant channel must use the same protocol as the first communication channel you defined.

3. If you use multicast, enter the following parameters: the Bind Network Address to use, the Multicast Address and the Port for the redundant channel.

   If you use unicast, define the following parameters: the Bind Network Address to use, and the Port. Enter the IP addresses of all nodes that will be part of the cluster.

4. To tell Corosync how and when to use the different channels, select the rrp_mode to use:

   • If only one communication channel is defined, rrp_mode is automatically disabled (value none).

   • If set to active, Corosync uses both interfaces actively. However, this mode is deprecated.

   • If set to passive, Corosync sends messages alternatively over the available networks.

   When RRP is used, SUSE Linux Enterprise High Availability monitors the status of the current rings and automatically re-enables redundant rings after faults.

   Alternatively, check the ring status manually with corosync-cfgtool. View the available options with -h.

5. Confirm your changes.

To define the authentication settings for the cluster, you can use HMAC/SHA1 authentication. This requires a shared secret used to protect and authenticate messages. The authentication key (password) you specify will be used on all nodes in the cluster.

Figure 4.3: YaST Cluster—security #

  

To enable stateful packet inspection for iptables, configure and use the conntrack tools. This requires the following basic steps:

Figure 4.4: YaST Cluster—conntrackd #

  

After having configured the conntrack tools, you can use them for Linux Virtual Server (see Load balancing).

In the YaST cluster module define whether to start certain services on a node at boot time. You can also use the module to start and stop the services manually. To bring the cluster nodes online and start the cluster resource manager, Pacemaker must be running as a service.

Figure 4.5: YaST Cluster—services #

  

Instead of copying the resulting configuration files to all nodes manually, use the csync2 tool for replication across all nodes in the cluster.

This requires the following basic steps:

Csync2 helps you to keep track of configuration changes and to keep files synchronized across the cluster nodes:

Find detailed information about Csync2 at http://oss.linbit.com/csync2/ and http://oss.linbit.com/csync2/paper.pdf.

4.7.1 Configuring Csync2 with YaST #

  

Procedure 4.7: Configuring Csync2 with YaST #

  

1. Start the YaST cluster module and switch to the Csync2 category.

2. To specify the synchronization group, click Add in the Sync Host group and enter the local host names of all nodes in your cluster. For each node, you must use exactly the strings that are returned by the hostname command.

   

   Tip: Host name resolution

   If host name resolution does not work properly in your network, you can also specify a combination of host name and IP address for each cluster node. To do so, use the string HOSTNAME@IP such as alice@192.168.2.100, for example. Csync2 will then use the IP addresses when connecting.

3. Click Generate Pre-Shared-Keys to create a key file for the synchronization group. The key file is written to /etc/csync2/key_hagroup. After it has been created, it must be copied manually to all members of the cluster.

4. To populate the Sync File list with the files that usually need to be synchronized among all nodes, click Add Suggested Files.

5. To Edit, Add or Remove files from the list of files to be synchronized use the respective buttons. You must enter the absolute path name for each file.

6. Activate Csync2 by clicking Turn Csync2 ON. This will execute the following command to start Csync2 automatically at boot time:

   # systemctl enable csync2.socket

7. Click Finish. YaST writes the Csync2 configuration to /etc/csync2/csync2.cfg.

Figure 4.6: YaST Cluster—Csync2 #

  

4.7.2 Synchronizing changes with Csync2 #

  

Before running Csync2 for the first time, you need to make the following preparations:

Procedure 4.8: Preparing for initial synchronization with Csync2 #

  

1. Copy the file /etc/csync2/csync2.cfg manually to all nodes after you have configured it as described in Section 4.7.1, “Configuring Csync2 with YaST”.

2. Copy the file /etc/csync2/key_hagroup that you have generated on one node in Step 3 of Section 4.7.1 to all nodes in the cluster. It is needed for authentication by Csync2. However, do not regenerate the file on the other nodes—it needs to be the same file on all nodes.

3. Execute the following command on all nodes to start the service now:

   # systemctl start csync2.socket

Procedure 4.9: Synchronizing the configuration files with Csync2 #

  

1. To initially synchronize all files once, execute the following command on the machine that you want to copy the configuration from:

   # csync2 -xv

   This will synchronize all the files once by pushing them to the other nodes. If all files are synchronized successfully, Csync2 will finish with no errors.

   If one or several files that are to be synchronized have been modified on other nodes (not only on the current one), Csync2 reports a conflict. You will get an output similar to the one below:

   While syncing file /etc/corosync/corosync.conf:
   ERROR from peer hex-14: File is also marked dirty here!
   Finished with 1 errors.

2. If you are sure that the file version on the current node is the “best” one, you can resolve the conflict by forcing this file and resynchronizing:

   # csync2 -f /etc/corosync/corosync.conf
   # csync2 -x

For more information on the Csync2 options, run

# csync2 -help

Note: Pushing synchronization after any changes

Csync2 only pushes changes. It does not continuously synchronize files between the machines.

Each time you update files that need to be synchronized, you need to push the changes to the other machines: Run csync2  -xv on the machine where you did the changes. If you run the command on any of the other machines with unchanged files, nothing will happen.

Before starting the cluster, make sure passwordless SSH is configured between the nodes. If you did not already configure passwordless SSH before setting up the cluster, you can do so now by using the ssh stage of the bootstrap scripts:

After the initial cluster configuration is done, start the cluster services on all cluster nodes to bring the stack online:

After the basic configuration is done and the nodes are online, you can start to configure cluster resources. Use one of the cluster management tools like the crm shell (crmsh) or Hawk2. For more information, see Section 5.5, “Introduction to crmsh” or Section 5.4, “Introduction to Hawk2”.

In general, clusters fall into one of two categories:

Adding also different topologies, different use cases can be derived. The following use cases are the most common:

Whenever communication fails between one or more nodes and the rest of the cluster, a cluster partition occurs. The nodes can only communicate with other nodes in the same partition and are unaware of the separated nodes. A cluster partition is defined as having quorum (being “quorate”) if it has the majority of nodes (or votes). How this is achieved is done by quorum calculation. Quorum is a requirement for fencing.

Quorum is not calculated or determined by Pacemaker. Corosync can handle quorum for two-node clusters directly without changing the Pacemaker configuration.

How quorum is calculated is influenced by the following factors:

quorum {
   # Enable and configure quorum subsystem (default: off)
   # see also corosync.conf.5 and votequorum.5
   provider: corosync_votequorum
   expected_votes: 2
   two_node: 1
}

quorum {
   provider: corosync_votequorum 1
   expected_votes: N 2
   wait_for_all: 1 3
}

Global cluster options control how the cluster behaves when confronted with certain situations. They are grouped into sets and can be viewed and modified with the cluster management tools like Hawk2 and the crm shell.

The predefined values can usually be kept. However, to make key functions of your cluster work as expected, you might need to adjust the following parameters after basic cluster setup:

5.3.2 Global option stonith-enabled #

  

This global option defines whether to apply fencing, allowing STONITH devices to shoot failed nodes and nodes with resources that cannot be stopped. By default, this global option is set to true, because for normal cluster operation it is necessary to use STONITH devices. According to the default value, the cluster will refuse to start any resources if no STONITH resources have been defined.

If you need to disable fencing for any reasons, set stonith-enabled to false, but be aware that this has impact on the support status for your product. Furthermore, with stonith-enabled="false", resources like the Distributed Lock Manager (DLM) and all services depending on DLM (such as lvmlockd, GFS2, and OCFS2) will fail to start.

Important: No support without STONITH

A cluster without STONITH is not supported.

To configure and manage cluster resources, either use Hawk2, or the crm shell (crmsh) command line utility.

Hawk2's user-friendly Web interface allows you to monitor and administer your High Availability clusters from Linux or non-Linux machines alike. Hawk2 can be accessed from any machine inside or outside of the cluster by using a (graphical) Web browser.

5.4.3 Hawk2 overview: main elements #

  

After logging in to Hawk2, you will see a navigation bar on the left-hand side and a top-level row with several links on the right-hand side.

Note: Available functions in Hawk2

By default, users logged in as root or hacluster have full read-write access to all cluster configuration tasks. However, Access control lists (ACLs) can be used to define fine-grained access permissions.

If ACLs are enabled in the CRM, the available functions in Hawk2 depend on the user role and their assigned access permissions. The History Explorer in Hawk2 can only be executed by the user hacluster.

5.4.3.1 Left navigation bar #

  

Monitoring

• Status: Displays the current cluster status at a glance (similar to crm status on the crmsh). For details, see Section 11.1.1, “Monitoring a single cluster”. If your cluster includes guest nodes (nodes that run the pacemaker_remote daemon), they are displayed, too. The screen refreshes in near real-time: any status changes for nodes or resources are visible almost immediately.

• Dashboard: Allows you to monitor multiple clusters (also located on different sites, in case you have a Geo cluster setup). For details, see Section 11.1.2, “Monitoring multiple clusters”. If your cluster includes guest nodes (nodes that run the pacemaker_remote daemon), they are displayed, too. The screen refreshes in near real-time: any status changes for nodes or resources are visible almost immediately.

Troubleshooting

• History: Opens the History Explorer from which you can generate cluster reports. For details, see Section 11.3, “Viewing the cluster history”.

• Command Log: Lists the crmsh commands recently executed by Hawk2.

Configuration

• Add Resource: Opens the resource configuration screen. For details, see Chapter 6, Configuring cluster resources.

• Add Constraint: Opens the constraint configuration screen. For details, see Chapter 7, Configuring resource constraints.

• Wizards: Allows you to select from several wizards that guide you through the creation of resources for a certain workload, for example, a DRBD block device. For details, see Section 5.4.6, “Adding resources with the wizard”.

• Edit Configuration: Allows you to edit resources, constraints, node names and attributes, tags, alerts, and fencing topologies.

• Cluster Configuration: Allows you to modify global cluster options and resource and operation defaults. For details, see Section 5.4.4, “Configuring global cluster options”.

• Access Control › Roles: Opens a screen where you can create roles for access control lists (sets of rules describing access rights to the CIB). For details, see Procedure 15.2, “Adding a monitor role with Hawk2”.

• Access Control › Targets: Opens a screen where you can create targets (system users) for access control lists and assign roles to them. For details, see Procedure 15.3, “Assigning a role to a target with Hawk2”.

5.4.3.2 Top-level row #

  

Hawk2's top-level row shows the following entries:

• Batch: Click to switch to batch mode. This allows you to simulate and stage changes and to apply them as a single transaction. For details, see Section 5.4.7, “Using the batch mode”.

• USERNAME: Allows you to set preferences for Hawk2 (for example, the language for the Web interface, or whether to display a warning if STONITH is disabled).

• Help: Access the SUSE Linux Enterprise High Availability documentation, read the release notes or report a bug.

• Logout: Click to log out.

5.4.6 Adding resources with the wizard #

  

The Hawk2 wizard is a convenient way of setting up simple resources like a virtual IP address or an SBD STONITH resource, for example. It is also useful for complex configurations that include multiple resources, like the resource configuration for a DRBD block device or an Apache Web server. The wizard guides you through the configuration steps and provides information about the parameters you need to enter.

Procedure 5.4: Using the resource wizard #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. From the left navigation bar, select Configuration › Wizards.

3. Expand the individual categories by clicking the arrow down icon next to them and select the desired wizard.

4. Follow the instructions on the screen. After the last configuration step, Verify the values you have entered.

   Hawk2 shows which actions it is going to perform and what the configuration looks like. Depending on the configuration, you might be prompted for the root password before you can Apply the configuration.

Figure 5.2: Hawk2—wizard for Apache web server #

  

For more information, see Chapter 6, Configuring cluster resources.

5.4.7 Using the batch mode #

  

Hawk2 provides a Batch Mode, including a cluster simulator. It can be used for the following:

• Staging changes to the cluster and applying them as a single transaction, instead of having each change take effect immediately.

• Simulating changes and cluster events, for example, to explore potential failure scenarios.

For example, batch mode can be used when creating groups of resources that depend on each other. Using batch mode, you can avoid applying intermediate or incomplete configurations to the cluster.

While batch mode is enabled, you can add or edit resources and constraints or change the cluster configuration. It is also possible to simulate events in the cluster, including nodes going online or offline, resource operations and tickets being granted or revoked. See Procedure 5.6, “Injecting node, resource or ticket events” for details.

The cluster simulator runs automatically after every change and shows the expected outcome in the user interface. For example, this also means: If you stop a resource while in batch mode, the user interface shows the resource as stopped—while actually, the resource is still running.

Important: Wizards and changes to the live system

Some wizards include actions beyond mere cluster configuration. When using those wizards in batch mode, any changes that go beyond cluster configuration would be applied to the live system immediately.

Therefore wizards that require root permission cannot be executed in batch mode.

Procedure 5.5: Working with the batch mode #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. To activate the batch mode, select Batch from the top-level row.

   An additional bar appears below the top-level row. It indicates that batch mode is active and contains links to actions that you can execute in batch mode.

   

   Figure 5.3: Hawk2 batch mode activated #

     

3. While batch mode is active, perform any changes to your cluster, like adding or editing resources and constraints or editing the cluster configuration.

   The changes will be simulated and shown in all screens.

4. To view details of the changes you have made, select Show from the batch mode bar. The Batch Mode window opens.

   For any configuration changes it shows the difference between the live state and the simulated changes in crmsh syntax: Lines starting with a - character represent the current state whereas lines starting with + show the proposed state.

5. To inject events or view even more details, see Procedure 5.6. Otherwise Close the window.

6. Choose to either Discard or Apply the simulated changes and confirm your choice. This also deactivates batch mode and takes you back to normal mode.

When running in batch mode, Hawk2 also allows you to inject Node Events and Resource Events.

Node events

Let you change the state of a node. Available states are online, offline, and unclean.

Resource events

Let you change some properties of a resource. For example, you can set an operation (like start, stop, monitor), the node it applies to, and the expected result to be simulated.

Ticket events

Let you test the impact of granting and revoking tickets (used for Geo clusters).

Procedure 5.6: Injecting node, resource or ticket events #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. If batch mode is not active yet, click Batch at the top-level row to switch to batch mode.

3. In the batch mode bar, click Show to open the Batch Mode window.

4. To simulate a status change of a node:

   1. Click Inject › Node Event.

   2. Select the Node you want to manipulate and select its target State.

   3. Confirm your changes. Your event is added to the queue of events listed in the Batch Mode dialog.

5. To simulate a resource operation:

   1. Click Inject › Resource Event.

   2. Select the Resource you want to manipulate and select the Operation to simulate.

   3. If necessary, define an Interval.

   4. Select the Node on which to run the operation and the targeted Result. Your event is added to the queue of events listed in the Batch Mode dialog.

   5. Confirm your changes.

6. To simulate a ticket action:

   1. Click Inject › Ticket Event.

   2. Select the Ticket you want to manipulate and select the Action to simulate.

   3. Confirm your changes. Your event is added to the queue of events listed in the Batch Mode dialog.

7. The Batch Mode dialog (Figure 5.4) shows a new line per injected event. Any event listed here is simulated immediately and is reflected on the Status screen.

   If you have made any configuration changes, too, the difference between the live state and the simulated changes is shown below the injected events.

   

   Figure 5.4: Hawk2 batch mode—injected invents and configuration changes #

     

8. To remove an injected event, click the Remove icon next to it. Hawk2 updates the Status screen accordingly.

9. To view more details about the simulation run, click Simulator and choose one of the following:

   Summary

   Shows a detailed summary.

   CIB (in)/CIB (out)

   CIB (in) shows the initial CIB state. CIB (out) shows what the CIB would look like after the transition.

   Transition graph

   Shows a graphical representation of the transition.

   Transition

   Shows an XML representation of the transition.

10. If you have reviewed the simulated changes, close the Batch Mode window.

11. To leave the batch mode, either Apply or Discard the simulated changes.

To configure and manage cluster resources, either use the crm shell (crmsh) command line utility or Hawk2, a Web-based user interface.

This section introduces the command line tool crm. The crm command has several subcommands which manage resources, CIBs, nodes, resource agents, and others. It offers a thorough help system with embedded examples. All examples follow a naming convention described in Appendix B.

Events are logged to /var/log/crmsh/crmsh.log.

Note: User privileges

Sufficient privileges are necessary to manage a cluster. The crm command and its subcommands need to be run either as root user or as the CRM owner user (typically the user hacluster).

However, the user option allows you to run crm and its subcommands as a regular (unprivileged) user and to change its ID using sudo whenever necessary. For example, with the following command crm will use hacluster as the privileged user ID:

# crm options user hacluster

Note that you need to set up /etc/sudoers so that sudo does not ask for a password.

Tip: Interactive crm prompt

By using crm without arguments (or with only one sublevel as argument), the crm shell enters the interactive mode. This mode is indicated by the following prompt:

crm(live/HOSTNAME)

For readability reasons, we omit the host name in the interactive crm prompts in our documentation. We only include the host name if you need to run the interactive shell on a specific node, like alice for example:

crm(live/alice)

5.5.1 Getting help #

  

Help can be accessed in several ways:

• To output the usage of crm and its command line options:

  # crm --help

• To give a list of all available commands:

  # crm help

• To access other help sections, not only the command reference:

  # crm help topics

• To view the extensive help text of the configure subcommand:

  # crm configure help

• To print the syntax, its usage, and examples of the group subcommand of configure:

  # crm configure help group

  This is the same:

  # crm help configure group

Almost all output of the help subcommand (do not mix it up with the --help option) opens a text viewer. This text viewer allows you to scroll up or down and read the help text more comfortably. To leave the text viewer, press the Q key.

Tip: Use tab completion in Bash and interactive shell

The crmsh supports full tab completion in Bash directly, not only for the interactive shell. For example, typing crm help config→| will complete the word like in the interactive shell.

5.5.2 Executing crmsh's subcommands #

  

The crm command itself can be used in the following ways:

• Directly: Concatenate all subcommands to crm, press Enter and you see the output immediately. For example, enter crm help ra to get information about the ra subcommand (resource agents).

  It is possible to abbreviate subcommands as long as they are unique. For example, you can shorten status as st and crmsh will know what you mean.

  Another feature is to shorten parameters. Usually, you add parameters through the params keyword. You can leave out the params section if it is the first and only section. For example, this line:

  # crm primitive ipaddr IPaddr2 params ip=192.168.0.55

  is equivalent to this line:

  # crm primitive ipaddr IPaddr2 ip=192.168.0.55

• As crm shell script: Crm shell scripts contain subcommands of crm. For more information, see Section 5.5.4, “Using crmsh's shell scripts”.

• As crmsh cluster scripts:These are a collection of metadata, references to RPM packages, configuration files, and crmsh subcommands bundled under a single, yet descriptive name. They are managed through the crm script command.

  Do not confuse them with crmsh shell scripts: although both share some common objectives, the crm shell scripts only contain subcommands whereas cluster scripts incorporate much more than a simple enumeration of commands. For more information, see Section 5.5.5, “Using crmsh's cluster scripts”.

• Interactive as internal shell: Type crm to enter the internal shell. The prompt changes to crm(live). With help you can get an overview of the available subcommands. As the internal shell has different levels of subcommands, you can “enter” one by typing this subcommand and press Enter.

  For example, if you type resource you enter the resource management level. Your prompt changes to crm(live)resource#. To leave the internal shell, use the command quit. If you need to go one level back, use back, up, end, or cd.

  You can enter the level directly by typing crm and the respective subcommand(s) without any options and press Enter.

  The internal shell supports also tab completion for subcommands and resources. Type the beginning of a command, press →| and crm completes the respective object.

In addition to previously explained methods, crmsh also supports synchronous command execution. Use the -w option to activate it. If you have started crm without -w, you can enable it later with the user preference's wait set to yes (options wait yes). If this option is enabled, crm waits until the transition is finished. Whenever a transaction is started, dots are printed to indicate progress. Synchronous command execution is only applicable for commands like resource start.

Note: Differentiate between management and configuration subcommands

The crm tool has management capability (the subcommands resource and node) and can be used for configuration (cib, configure).

The following subsections give you an overview of some important aspects of the crm tool.

5.5.3 Displaying information about OCF resource agents #

  

As you need to deal with resource agents in your cluster configuration all the time, the crm tool contains the ra command. Use it to show information about resource agents and to manage them (for additional information, see also Section 6.2, “Supported resource agent classes”):

# crm ra
crm(live)ra# 

The command classes lists all classes and providers:

crm(live)ra# classes
 lsb
 ocf / heartbeat linbit lvm2 ocfs2 pacemaker
 service
 stonith
 systemd

To get an overview of all available resource agents for a class (and provider) use the list command:

crm(live)ra# list ocf
AoEtarget           AudibleAlarm        CTDB                ClusterMon
Delay               Dummy               EvmsSCC             Evmsd
Filesystem          HealthCPU           HealthSMART         ICP
IPaddr              IPaddr2             IPsrcaddr           IPv6addr
LVM                 LinuxSCSI           MailTo              ManageRAID
ManageVE            Pure-FTPd           Raid1               Route
SAPDatabase         SAPInstance         SendArp             ServeRAID
...

An overview of a resource agent can be viewed with info:

crm(live)ra# info ocf:linbit:drbd
This resource agent manages a DRBD* resource
as a master/slave resource. DRBD is a shared-nothing replicated storage
device. (ocf:linbit:drbd)

Master/Slave OCF Resource Agent for DRBD

Parameters (* denotes required, [] the default):

drbd_resource* (string): drbd resource name
    The name of the drbd resource from the drbd.conf file.

drbdconf (string, [/etc/drbd.conf]): Path to drbd.conf
    Full path to the drbd.conf file.

Operations' defaults (advisory minimum):

    start         timeout=240
    promote       timeout=90
    demote        timeout=90
    notify        timeout=90
    stop          timeout=100
    monitor_Slave_0 interval=20 timeout=20 start-delay=1m
    monitor_Master_0 interval=10 timeout=20 start-delay=1m

Leave the viewer by pressing Q.

Tip: Use crm directly

In the former example we used the internal shell of the crm command. However, you do not necessarily need to use it. You get the same results if you add the respective subcommands to crm. For example, you can list all the OCF resource agents by entering crm ra list ocf in your shell.

# A small example file with some crm subcommands
status
node list

# crm -f example.cli
# crm < example.cli

# crm script list

# crm script show mailto
mailto (Basic)
MailTo

 This is a resource agent for MailTo. It sends email to a sysadmin
whenever  a takeover occurs.

1. Notifies recipients by email in the event of resource takeover

  id (required)  (unique)
      Identifier for the cluster resource
  email (required)
      Email address
  subject
      Subject

# crm script verify mailto id=sysadmin email=tux@example.org
1. Ensure mail package is installed

        mailx

2. Configure cluster resources

        primitive sysadmin MailTo
                email="tux@example.org"
                op start timeout="10"
                op stop timeout="10"
                op monitor interval="10" timeout="10"

        clone c-sysadmin sysadmin

# crm script run mailto id=sysadmin email=tux@example.org
INFO: MailTo
INFO: Nodes: alice, bob
OK: Ensure mail package is installed
OK: Configure cluster resources

# crm status
[...]
 Clone Set: c-sysadmin [sysadmin]
     Started: [ alice bob ]

5.5.6 Using configuration templates #

  

Note: Deprecation notice

The use of configuration templates is deprecated and will be removed in the future. Configuration templates will be replaced by cluster scripts, see Section 5.5.5, “Using crmsh's cluster scripts”.

Configuration templates are ready-made cluster configurations for crmsh. Do not confuse them with the resource templates (as described in Section 6.8.2, “Creating resource templates with crmsh”). Those are templates for the cluster and not for the crm shell.

Configuration templates require minimum effort to be tailored to the particular user's needs. Whenever a template creates a configuration, warning messages give hints which can be edited later for further customization.

The following procedure shows how to create a simple yet functional Apache configuration:

1. Log in as root and start the crm interactive shell:

   # crm configure

2. Create a new configuration from a configuration template:

   1. Switch to the template subcommand:

      crm(live)configure# template

   2. List the available configuration templates:

      crm(live)configure template# list templates
      gfs2-base   filesystem  virtual-ip  apache   clvm     ocfs2    gfs2

   3. Decide which configuration template you need. As we need an Apache configuration, we select the apache template and name it g-intranet:

      crm(live)configure template# new g-intranet apache
      INFO: pulling in template apache
      INFO: pulling in template virtual-ip

3. Define your parameters:

   1. List the configuration you have created:

      crm(live)configure template# list
      g-intranet

   2. Display the minimum required changes that need to be filled out by you:

      crm(live)configure template# show
      ERROR: 23: required parameter ip not set
      ERROR: 61: required parameter id not set
      ERROR: 65: required parameter configfile not set

   3. Invoke your preferred text editor and fill out all lines that have been displayed as errors in Step 3.b:

      crm(live)configure template# edit

4. Show the configuration and check whether it is valid (bold text depends on the configuration you have entered in Step 3.c):

   crm(live)configure template# show
   primitive virtual-ip ocf:heartbeat:IPaddr \
       params ip="192.168.1.101"
   primitive apache apache \
       params configfile="/etc/apache2/httpd.conf"
       monitor apache 120s:60s
   group g-intranet \
       apache virtual-ip

5. Apply the configuration:

   crm(live)configure template# apply
   crm(live)configure# cd ..
   crm(live)configure# show

6. Submit your changes to the CIB:

   crm(live)configure# commit

It is possible to simplify the commands even more, if you know the details. The above procedure can be summarized with the following command on the shell:

# crm configure template \
   new g-intranet apache params \
   configfile="/etc/apache2/httpd.conf" ip="192.168.1.101"

If you are inside your internal crm shell, use the following command:

crm(live)configure template# new intranet apache params \
   configfile="/etc/apache2/httpd.conf" ip="192.168.1.101"

However, the previous command only creates its configuration from the configuration template. It does not apply nor commit it to the CIB.

# crm configure
crm(live)configure# show fence-bob
primitive fence-bob stonith:apcsmart \
        params hostlist="bob"
crm(live)configure# monitor fence-bob 120m:60s
crm(live)configure# show changed
primitive fence-bob stonith:apcsmart \
        params hostlist="bob" \
        op monitor interval="120m" timeout="60s"
crm(live)configure# ptest
crm(live)configure# commit

# crm configure graph dot config.svg svg

# crm corosync status
Printing ring status.
Local node ID 175704363
RING ID 0
        id      = 10.121.9.43
        status  = ring 0 active with no faults
Quorum information
------------------
Date:             Thu May  8 16:41:56 2014
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          175704363
Ring ID:          4032
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
 175704363          1 alice.example.com (local)
 175704619          1 bob.example.com

# crm corosync diff
--- bob
+++ alice
@@ -46,2 +46,2 @@
-       expected_votes: 2
-       two_node: 1
+       expected_votes: 1
+       two_node: 0

# crm configure show
primitive mydb mysql \
   params replication_user=admin ...

# crm resource secret mydb set passwd linux
INFO: syncing /var/lib/heartbeat/lrm/secrets/mydb/passwd to [your node list]

# crm resource secret mydb show passwd
linux

The following types of resources can be created:

For each cluster resource you add, you need to define the standard that the resource agent conforms to. Resource agents abstract the services they provide and present an accurate status to the cluster, which allows the cluster to be non-committal about the resources it manages. The cluster relies on the resource agent to react appropriately when given a start, stop or monitor command.

Typically, resource agents come in the form of shell scripts. SUSE Linux Enterprise High Availability supports the following classes of resource agents:

The agents supplied with SUSE Linux Enterprise High Availability are written to OCF specifications.

Timeouts values for resources can be influenced by the following parameters:

Note: Priority of values

If a specific value is defined for a resource, it takes precedence over the global default. A specific value for a resource also takes precedence over a value that is defined in a resource template.

Getting timeout values right is very important. Setting them too low will result in a lot of (unnecessary) fencing operations for the following reasons:

You can adjust the global default for operations and set any specific timeout values with both crmsh and Hawk2. The best practice for determining and setting timeout values is as follows:

Before you can use a resource in the cluster, it must be set up. For example, to use an Apache server as a cluster resource, set up the Apache server first and complete the Apache configuration before starting the respective resource in your cluster.

If a resource has specific environment requirements, make sure they are present and identical on all cluster nodes. This kind of configuration is not managed by SUSE Linux Enterprise High Availability. You must do this yourself.

You can create primitive resources using either Hawk2 or crmsh.

Note: Do not touch services managed by the cluster

When managing a resource with SUSE Linux Enterprise High Availability, the same resource must not be started or stopped otherwise (outside of the cluster, for example manually or on boot or reboot). The High Availability software is responsible for all service start or stop actions.

If you need to execute testing or maintenance tasks after the services are already running under cluster control, make sure to put the resources, nodes, or the whole cluster into maintenance mode before you touch any of them manually. For details, see Section 28.2, “Different options for maintenance tasks”.

Important: Resource IDs and node names

Cluster resources and cluster nodes should be named differently. Otherwise Hawk2 will fail.

Some cluster resources depend on other components or resources. They require that each component or resource starts in a specific order and runs together on the same server with resources it depends on. To simplify this configuration, you can use cluster resource groups.

You can create resource groups using either Hawk2 or crmsh.

Figure 6.2: Group resource #

  

Groups have the following properties:

6.5.1 Creating resource groups with Hawk2 #

  

Note: Empty groups

Groups must contain at least one resource, otherwise the configuration is not valid. While creating a group, Hawk2 allows you to create more primitives and add them to the group.

Procedure 6.4: Adding a resource group with Hawk2 #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. From the left navigation bar, select Configuration › Add Resource › Group.

3. Enter a unique Group ID.

4. To define the group members, select one or multiple entries in the list of Children. Re-sort group members by dragging and dropping them into the order you want by using the “handle” icon on the right.

5. If needed, modify or add Meta Attributes.

6. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Figure 6.3: Hawk2—resource group #

  

You may want certain resources to run simultaneously on multiple nodes in your cluster. To do this you must configure a resource as a clone. Examples of resources that might be configured as clones include cluster file systems like OCFS2. You can clone any resource provided. This is supported by the resource's Resource Agent. Clone resources may even be configured differently depending on which nodes they are hosted.

There are three types of resource clones:

Clones must contain exactly one group or one regular resource.

When configuring resource monitoring or constraints, clones have different requirements than simple resources. For details, see Pacemaker Explained, available from http://www.clusterlabs.org/pacemaker/doc/.

You can create clone resources using either Hawk2 or crmsh.

6.6.1 Creating clone resources with Hawk2 #

  

Note: Child resources for clones

Clones can either contain a primitive or a group as child resources. In Hawk2, child resources cannot be created or modified while creating a clone. Before adding a clone, create child resources and configure them as desired.

Procedure 6.6: Adding a clone resource with Hawk2 #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. From the left navigation bar, select Configuration › Add Resource › Clone.

3. Enter a unique Clone ID.

4. From the Child Resource list, select the primitive or group to use as a sub-resource for the clone.

5. If needed, modify or add Meta Attributes.

6. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Figure 6.4: Hawk2—clone resource #

  

Promotable clones (formerly known as multi-state resources) are a specialization of clones. They allow the instances to be in one of two operating modes (primary or secondary). Promotable clones must contain exactly one group or one regular resource.

When configuring resource monitoring or constraints, promotable clones have different requirements than simple resources. For details, see Pacemaker Explained, available from http://www.clusterlabs.org/pacemaker/doc/.

You can create promotable clones using either Hawk2 or crmsh.

6.7.1 Creating promotable clones with Hawk2 #

  

Note: Child resources for promotable clones

Promotable clones can either contain a primitive or a group as child resources. In Hawk2, child resources cannot be created or modified while creating a promotable clone. Before adding a promotable clone, create child resources and configure them as desired. See Section 6.4.1, “Creating primitive resources with Hawk2” or Section 6.5.1, “Creating resource groups with Hawk2”.

Procedure 6.8: Adding a promotable clone with Hawk2 #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. From the left navigation bar, select Configuration › Add Resource › Multi-state.

3. Enter a unique Multi-state ID.

4. From the Child Resource list, select the primitive or group to use as a sub-resource for the multi-state resource.

5. If needed, modify or add Meta Attributes.

6. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

If you want to create lots of resources with similar configurations, defining a resource template is the easiest way. After being defined, it can be referenced in primitives, or in certain types of constraints as described in Section 7.3, “Resource templates and constraints”.

If a template is referenced in a primitive, the primitive will inherit all operations, instance attributes (parameters), meta attributes, and utilization attributes defined in the template. Additionally, you can define specific operations or attributes for your primitive. If any of these are defined in both the template and the primitive, the values defined in the primitive will take precedence over the ones defined in the template.

You can create resource templates using either Hawk2 or crmsh.

# crm configure rsc_template
 usage: rsc_template <name> [<class>:[<provider>:]]<type>
        [params <param>=<value> [<param>=<value>...]]
        [meta <attribute>=<value> [<attribute>=<value>...]]
        [utilization <attribute>=<value> [<attribute>=<value>...]]
        [operations id_spec
            [op op_type [<attribute>=<value>...] ...]]

crm(live)configure# rsc_template BigVM ocf:heartbeat:Xen \
   params allow_mem_management="true" \
   op monitor timeout=60s interval=15s \
   op stop timeout=10m \
   op start timeout=10m

crm(live)configure# primitive MyVM1 @BigVM \
   params xmfile="/etc/xen/shared-vm/MyVM1" name="MyVM1"

crm(live)configure# primitive MyVM1 Xen \
   params xmfile="/etc/xen/shared-vm/MyVM1" name="MyVM1" \
   params allow_mem_management="true" \
   op monitor timeout=60s interval=15s \
   op stop timeout=10m \
   op start timeout=10m

crm(live)configure# primitive MyVM2 @BigVM \
   params xmfile="/etc/xen/shared-vm/MyVM2" name="MyVM2" \
   op monitor timeout=120s interval=30s

Important: No Support Without STONITH

• You must have a node fencing mechanism for your cluster.

• The global cluster options stonith-enabled and startup-fencing must be set to true. When you change them, you lose support.

By default, the global cluster option stonith-enabled is set to true. If no STONITH resources have been defined, the cluster will refuse to start any resources. Configure one or more STONITH resources to complete the STONITH setup. While STONITH resources are configured similarly to other resources, their behavior is different in some respects. For details refer to Section 12.3, “STONITH resources and configuration”.

You can create STONITH resources using either Hawk2 or crmsh.

If you want to ensure that a resource is running, you must configure resource monitoring for it. You can configure resource monitoring using either Hawk2 or crmsh.

If the resource monitor detects a failure, the following takes place:

If you do not configure resource monitoring, resource failures after a successful start will not be communicated, and the cluster will always show the resource as healthy.

Usually, resources are only monitored by the cluster while they are running. However, to detect concurrency violations, also configure monitoring for resources which are stopped. For resource monitoring, specify a timeout and/or start delay value, and an interval. The interval tells the CRM how often it should check the resource status. You can also set particular parameters such as timeout for start or stop operations.

For more information about monitor operation parameters, see Section 6.14, “Resource operations”.

6.10.1 Configuring resource monitoring with Hawk2 #

  

Procedure 6.13: Adding and modifying an operation #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. Add a resource as described in Procedure 6.2, “Adding a primitive resource with Hawk2” or select an existing primitive to edit.

   Hawk2 automatically shows the most important Operations (start, stop, monitor) and proposes default values.

   To see the attributes belonging to each proposed value, hover the mouse pointer over the respective value.

   

3. To change the suggested timeout values for the start or stop operation:

   1. Click the pen icon next to the operation.

   2. In the dialog that opens, enter a different value for the timeout parameter, for example 10, and confirm your change.

4. To change the suggested interval value for the monitor operation:

   1. Click the pen icon next to the operation.

   2. In the dialog that opens, enter a different value for the monitoring interval.

   3. To configure resource monitoring in the case that the resource is stopped:

      1. Select the role entry from the empty drop-down box below.

      2. From the role drop-down box, select Stopped.

      3. Click Apply to confirm your changes and to close the dialog for the operation.

5. Confirm your changes in the resource configuration screen. A message at the top of the screen shows if the action has been successful.

To view resource failures, switch to the Status screen in Hawk2 and select the resource you are interested in. In the Operations column click the arrow down icon and select Recent Events. The dialog that opens lists recent actions performed for the resource. Failures are displayed in red. To view the resource details, click the magnifier icon in the Operations column.

Figure 6.6: Hawk2—resource details #

  

crm(live)configure# primitive apache apache \
  params ... \
  op monitor interval=60s timeout=30s

crm(live)configure# primitive apache apache \
   params ...
 crm(live)configure# monitor apache 60s:30s

Parts or all of the configuration can be loaded from a local file or a network URL. Three different methods can be defined:

To load the new configuration from the file mycluster-config.txt use the following syntax:

# crm configure load push mycluster-config.txt

For each resource you add, you can define options. Options are used by the cluster to decide how your resource should behave; they tell the CRM how to treat a specific resource. Resource options can be set with the crm_resource --meta command or with Hawk2.

The following resource options are available:

The scripts of all resource classes can be given parameters which determine how they behave and which instance of a service they control. If your resource agent supports parameters, you can add them with the crm_resource command or with Hawk2. In the crm command line utility and in Hawk2, instance attributes are called params or Parameter, respectively. The list of instance attributes supported by an OCF script can be found by executing the following command as root:

# crm ra info [class:[provider:]]resource_agent

or (without the optional parts):

# crm ra info resource_agent

The output lists all the supported attributes, their purpose and default values.

Note: Instance attributes for groups, clones or promotable clones

Note that groups, clones and promotable clone resources do not have instance attributes. However, any instance attributes set will be inherited by the group's, clone's or promotable clone's children.

By default, the cluster will not ensure that your resources are still healthy. To instruct the cluster to do this, you need to add a monitor operation to the resource's definition. Monitor operations can be added for all classes or resource agents.

Monitor operations can have the following properties:

There are three different kinds of constraints available:

Important: Restrictions for constraints and certain types of resources

• Do not create colocation constraints for members of a resource group. Create a colocation constraint pointing to the resource group as a whole instead. All other types of constraints are safe to use for members of a resource group.

• Do not use any constraints on a resource that has a clone resource or a promotable clone resource applied to it. The constraints must apply to the clone or promotable clone resource, not to the child resource.

When defining constraints, you also need to deal with scores. Scores of all kinds are integral to how the cluster works. Practically everything from migrating a resource to deciding which resource to stop in a degraded cluster is achieved by manipulating scores in some way. Scores are calculated on a per-resource basis and any node with a negative score for a resource cannot run that resource. After calculating the scores for a resource, the cluster then chooses the node with the highest score.

INFINITY is currently defined as 1,000,000. Additions or subtractions with it stick to the following three basic rules:

When defining resource constraints, you specify a score for each constraint. The score indicates the value you are assigning to this resource constraint. Constraints with higher scores are applied before those with lower scores. By creating additional location constraints with different scores for a given resource, you can specify an order for the nodes that a resource will fail over to.

If you have defined a resource template (see Section 6.8, “Creating resource templates”), it can be referenced in the following types of constraints:

However, colocation constraints must not contain more than one reference to a template. Resource sets must not contain a reference to a template.

Resource templates referenced in constraints stand for all primitives which are derived from that template. This means, the constraint applies to all primitive resources referencing the resource template. Referencing resource templates in constraints is an alternative to resource sets and can simplify the cluster configuration considerably. For details about resource sets, refer to Section 7.7, “Using resource sets to define constraints”.

A location constraint determines on which node a resource may be run, is preferably run, or may not be run. An example of a location constraint is to place all resources related to a certain database on the same node. This type of constraint may be added multiple times for each resource. All location constraints are evaluated for a given resource.

You can add location constraints using either Hawk2 or crmsh.

7.4.1 Adding location constraints with Hawk2 #

  

Procedure 7.1: Adding a location constraint #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. From the left navigation bar, select Configuration › Add Constraint › Location.

3. Enter a unique Constraint ID.

4. From the list of Resources select the resource or resources for which to define the constraint.

5. Enter a Score. The score indicates the value you are assigning to this resource constraint. Positive values indicate the resource can run on the Node you specify in the next step. Negative values mean it should not run on that node. Constraints with higher scores are applied before those with lower scores.

   Some often-used values can also be set via the drop-down box:

   • To force the resources to run on the node, click the arrow icon and select Always. This sets the score to INFINITY.

   • If you never want the resources to run on the node, click the arrow icon and select Never. This sets the score to -INFINITY, meaning that the resources must not run on the node.

   • To set the score to 0, click the arrow icon and select Advisory. This disables the constraint. This is useful when you want to set resource discovery but do not want to constrain the resources.

6. Select a Node.

7. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Figure 7.1: Hawk2—location constraint #

  

crm(live)configure# location loc-fs1 fs1 100: alice

crm(live)configure# primitive ping ping \
    params name=ping dampen=5s multiplier=100 host_list="r1 r2"
crm(live)configure# clone cl-ping ping meta interleave=true
crm(live)configure# location loc-node_pref internal_www \
    rule 50: #uname eq alice \
    rule ping: defined ping

crm(live)configure# primitive vip1 IPaddr2 params ip=192.168.1.5
crm(live)configure# primitive vip2 IPaddr2 params ip=192.168.1.6
crm(live)configure# location loc-alice { vip1 vip2 } inf: alice

crm(live)configure# location loc-alice /vip.*/ inf: alice

A colocation constraint tells the cluster which resources may or may not run together on a node. As a colocation constraint defines a dependency between resources, you need at least two resources to create a colocation constraint.

You can add colocation constraints using either Hawk2 or crmsh.

7.5.1 Adding colocation constraints with Hawk2 #

  

Procedure 7.2: Adding colocation constraints #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. From the left navigation bar, select Configuration › Add Constraint › Colocation.

3. Enter a unique Constraint ID.

4. Enter a Score. The score determines the location relationship between the resources. Positive values indicate that the resources should run on the same node. Negative values indicate that the resources should not run on the same node. The score will be combined with other factors to decide where to put the resource.

   Some often-used values can also be set via the drop-down box:

   • To force the resources to run on the same node, click the arrow icon and select Always. This sets the score to INFINITY.

   • If you never want the resources to run on the same node, click the arrow icon and select Never. This sets the score to -INFINITY, meaning that the resources must not run on the same node.

5. To define the resources for the constraint:

   1. From the drop-down box in the Resources category, select a resource (or a template).

      The resource is added and a new empty drop-down box appears beneath.

   2. Repeat this step to add more resources.

      As the topmost resource depends on the next resource and so on, the cluster will first decide where to put the last resource, then place the depending ones based on that decision. If the constraint cannot be satisfied, the cluster may not allow the dependent resource to run.

   3. To swap the order of resources within the colocation constraint, click the arrow up icon next to a resource to swap it with the entry above.

6. If needed, specify further parameters for each resource (such as Started, Stopped, Promote, Demote): Click the empty drop-down box next to the resource and select the desired entry.

7. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Figure 7.2: Hawk2—colocation constraint #

  

crm(live)configure# colocation coloc-2resource inf: resource1 resource2

Order constraints can be used to start or stop a service right before or after a different resource meets a special condition, such as being started, stopped, or promoted to primary. For example, you cannot mount a file system before the device is available to a system. As an order constraint defines a dependency between resources, you need at least two resources to create an order constraint.

You can add order constraints using either Hawk2 or crmsh.

7.6.1 Adding order constraints with Hawk2 #

  

Procedure 7.3: Adding an order constraint #

  

1. Log in to Hawk2:

   https://HAWKSERVER:7630/

2. From the left navigation bar, select Configuration › Add Constraint › Order.

3. Enter a unique Constraint ID.

4. Enter a Score. If the score is greater than zero, the order constraint is mandatory, otherwise it is optional.

   Some often-used values can also be set via the drop-down box:

   • To make the order constraint mandatory, click the arrow icon and select Mandatory.

   • To make the order constraint a suggestion only, click the arrow icon and select Optional.

   • Serialize: To ensure that no two stop/start actions occur concurrently for the resources, click the arrow icon and select Serialize. This makes sure that one resource must complete starting before the other can be started. A typical use case are resources that put a high load on the host during start-up.

5. For order constraints, you can usually keep the option Symmetrical enabled. This specifies that resources are stopped in reverse order.

6. To define the resources for the constraint:

   1. From the drop-down box in the Resources category, select a resource (or a template).

      The resource is added and a new empty drop-down box appears beneath.

   2. Repeat this step to add more resources.

      The topmost resource will start first, then the second, etc. Usually the resources will be stopped in reverse order.

   3. To swap the order of resources within the order constraint, click the arrow up icon next to a resource to swap it with the entry above.

7. If needed, specify further parameters for each resource (like Started, Stopped, Promote, Demote): Click the empty drop-down box next to the resource and select the desired entry.

8. Confirm your changes to finish the configuration. A message at the top of the screen shows if the action has been successful.

Figure 7.3: Hawk2—order constraint #

  

crm(live)configure# order res1_before_res2 Mandatory: resource1 resource2