Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Linux Enterprise High Availability Extension 15 SP2

7 Configuring and Managing Cluster Resources with Hawk2 Edit source

To configure and manage cluster resources, either use Hawk2, or the crm shell (crmsh) command line utility. If you upgrade from an earlier version of SUSE® Linux Enterprise High Availability Extension where Hawk was installed, the package will be replaced with the current version, Hawk2.

Hawk2's user-friendly Web interface allows you to monitor and administer your High Availability clusters from Linux or non-Linux machines alike. Hawk2 can be accessed from any machine inside or outside of the cluster by using a (graphical) Web browser.

7.1 Hawk2 Requirements Edit source

Before users can log in to Hawk2, the following requirements need to be fulfilled:

hawk2 Package

The hawk2 package must be installed on all cluster nodes you want to connect to with Hawk2.

Web Browser

On the machine from which to access a cluster node using Hawk2, you need a (graphical) Web browser (with JavaScript and cookies enabled) to establish the connection.

Hawk2 Service

To use Hawk2, the respective Web service must be started on the node that you want to connect to via the Web interface. See Procedure 7.1, “Starting Hawk2 Services”.

If you have set up your cluster with the scripts from the ha-cluster-bootstrap package, the Hawk2 service is already enabled.

Username, Group and Password on Each Cluster Node

Hawk2 users must be members of the haclient group. The installation creates a Linux user named hacluster, who is added to the haclient group.

When using the ha-cluster-init script for setup, a default password is set for the hacluster user. Before starting Hawk2, change it to a secure password. If you did not use the ha-cluster-init script, either set a password for the hacluster first or create a new user which is a member of the haclient group. Do this on every node you will connect to with Hawk2.

Procedure 7.1: Starting Hawk2 Services
  1. On the node you want to connect to, open a shell and log in as root.

  2. Check the status of the service by entering

    root # systemctl status hawk
  3. If the service is not running, start it with

    root # systemctl start hawk

    If you want Hawk2 to start automatically at boot time, execute the following command:

    root # systemctl enable hawk

7.2 Logging In Edit source

The Hawk2 Web interface uses the HTTPS protocol and port 7630.

Instead of logging in to an individual cluster node with Hawk2, you can configure a floating, virtual IP address (IPaddr or IPaddr2) as a cluster resource. It does not need any special configuration. It allows clients to connect to the Hawk service no matter which physical node the service is running on.

When setting up the cluster with the ha-cluster-bootstrap scripts, you will be asked whether to configure a virtual IP for cluster administration.

Procedure 7.2: Logging In to the Hawk2 Web Interface
  1. On any machine, start a Web browser and enter the following URL:

    https://HAWKSERVER:7630/

    Replace HAWKSERVER with the IP address or host name of any cluster node running the Hawk Web service. If a virtual IP address has been configured for cluster administration with Hawk2, replace HAWKSERVER with the virtual IP address.

    Note
    Note: Certificate Warning

    If a certificate warning appears when you try to access the URL for the first time, a self-signed certificate is in use. Self-signed certificates are not considered trustworthy by default.

    To verify the certificate, ask your cluster operator for the certificate details.

    To proceed anyway, you can add an exception in the browser to bypass the warning.

    For information on how to replace the self-signed certificate with a certificate signed by an official Certificate Authority, refer to Replacing the Self-Signed Certificate.

  2. On the Hawk2 login screen, enter the Username and Password of the hacluster user (or of any other user that is a member of the haclient group).

  3. Click Log In.

7.3 Hawk2 Overview: Main Elements Edit source

After logging in to Hawk2, you will see a navigation bar on the left-hand side and a top-level row with several links on the right-hand side.

Note
Note: Available Functions in Hawk2

By default, users logged in as root or hacluster have full read-write access to all cluster configuration tasks. However, Access Control Lists (ACLs) can be used to define fine-grained access permissions.

If ACLs are enabled in the CRM, the available functions in Hawk2 depend on the user role and their assigned access permissions. The History Explorer in Hawk2 can only be executed by the user hacluster.

7.3.1 Left Navigation Bar Edit source

Monitoring
  • Status: Displays the current cluster status at a glance (similar to crm status on the crmsh). For details, see Section 7.8.1, “Monitoring a Single Cluster”. If your cluster includes guest nodes (nodes that run the pacemaker_remote daemon), they are displayed, too. The screen refreshes in near real-time: any status changes for nodes or resources are visible almost immediately.

  • Dashboard: Allows you to monitor multiple clusters (also located on different sites, in case you have a Geo cluster setup). For details, see Section 7.8.2, “Monitoring Multiple Clusters”. If your cluster includes guest nodes (nodes that run the pacemaker_remote daemon), they are displayed, too. The screen refreshes in near real-time: any status changes for nodes or resources are visible almost immediately.

Troubleshooting
Configuration

7.3.2 Top-Level Row Edit source

Hawk2's top-level row shows the following entries:

  • Batch: Click to switch to batch mode. This allows you to simulate and stage changes and to apply them as a single transaction. For details, see Section 7.9, “Using the Batch Mode”.

  • USERNAME: Allows you to set preferences for Hawk2 (for example, the language for the Web interface, or whether to display a warning if STONITH is disabled).

  • Help: Access the SUSE Linux Enterprise High Availability Extension documentation, read the release notes or report a bug.

  • Logout: Click to log out.

7.4 Configuring Global Cluster Options Edit source

Global cluster options control how the cluster behaves when confronted with certain situations. They are grouped into sets and can be viewed and modified with cluster management tools like Hawk2 and crmsh. The predefined values can usually be kept. However, to ensure the key functions of your cluster work correctly, you need to adjust the following parameters after basic cluster setup:

Procedure 7.3: Modifying Global Cluster Options
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Cluster Configuration.

    The Cluster Configuration screen opens. It displays the global cluster options and their current values.

    To display a short description of the parameter on the right-hand side of the screen, hover your mouse over a parameter.

    Hawk2—Cluster Configuration
    Figure 7.1: Hawk2—Cluster Configuration
  3. Check the values for no-quorum-policy and stonith-enabled and adjust them, if necessary:

    1. Set no-quorum-policy to the appropriate value. See Section 6.2.2, “Global Option no-quorum-policy for more details.

    2. If you need to disable fencing for any reason, set stonith-enabled to no. By default, it is set to true, because using STONITH devices is necessary for normal cluster operation. According to the default value, the cluster will refuse to start any resources if no STONITH resources have been configured.

      Important
      Important: No Support Without STONITH
      • You must have a node fencing mechanism for your cluster.

      • The global cluster options stonith-enabled and startup-fencing must be set to true. When you change them, you lose support.

    3. To remove a parameter from the cluster configuration, click the Minus icon next to the parameter. If a parameter is deleted, the cluster will behave as if that parameter had the default value.

    4. To add a new parameter to the cluster configuration, choose one from the drop-down box.

  4. If you need to change Resource Defaults or Operation Defaults, proceed as follows:

    1. To adjust a value, either select a different value from the drop-down box or edit the value directly.

    2. To add a new resource default or operation default, choose one from the empty drop-down box and enter a value. If there are default values, Hawk2 proposes them automatically.

    3. To remove a parameter, click the Minus icon next to it. If no values are specified for Resource Defaults and Operation Defaults, the cluster uses the default values that are documented in Section 6.3.6, “Resource Options (Meta Attributes)” and Section 6.3.8, “Resource Operations”.

  5. Confirm your changes.

7.5 Configuring Cluster Resources Edit source

A cluster administrator needs to create cluster resources for every resource or application that runs on the servers in your cluster. Cluster resources can include Web sites, mail servers, databases, file systems, virtual machines, and any other server-based applications or services you want to make available to users at all times.

For an overview of the resource types you can create, refer to Section 6.3.3, “Types of Resources”. After you have specified the resource basics (ID, class, provider, and type), Hawk2 shows the following categories:

Parameters (Instance Attributes)

Determines which instance of a service the resource controls. For more information, refer to Section 6.3.7, “Instance Attributes (Parameters)”.

When creating a resource, Hawk2 automatically shows any required parameters. Edit them to get a valid resource configuration.

Operations

Needed for resource monitoring. For more information, refer to Section 6.3.8, “Resource Operations”.

When creating a resource, Hawk2 displays the most important resource operations (monitor, start, and stop).

Meta Attributes

Tells the CRM how to treat a specific resource. For more information, refer to Section 6.3.6, “Resource Options (Meta Attributes)”.

When creating a resource, Hawk2 automatically lists the important meta attributes for that resource (for example, the target-role attribute that defines the initial state of a resource. By default, it is set to Stopped, so the resource will not start immediately).

Utilization

Tells the CRM what capacity a certain resource requires from a node. For more information, refer to Section 7.6.8, “Configuring Placement of Resources Based on Load Impact”.

You can adjust the entries and values in those categories either during resource creation or later.

7.5.1 Showing the Current Cluster Configuration (CIB) Edit source

Sometimes a cluster administrator needs to know the cluster configuration. Hawk2 can show the current configuration in crm shell syntax, as XML and as a graph. To view the cluster configuration in crm shell syntax, from the left navigation bar select Configuration › Edit Configuration and click Show. To show the configuration in raw XML instead, click XML. Click Graph for a graphical representation of the nodes and resources configured in the CIB. It also shows the relationships between resources.

7.5.2 Adding Resources with the Wizard Edit source

The Hawk2 wizard is a convenient way of setting up simple resources like a virtual IP address or an SBD STONITH resource, for example. It is also useful for complex configurations that include multiple resources, like the resource configuration for a DRBD block device or an Apache Web server. The wizard guides you through the configuration steps and provides information about the parameters you need to enter.

Procedure 7.4: Using the Resource Wizard
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Wizards.

  3. Expand the individual categories by clicking the arrow down icon next to them and select the desired wizard.

  4. Follow the instructions on the screen. After the last configuration step, Verify the values you have entered.

    Hawk2 shows which actions it is going to perform and what the configuration looks like. Depending on the configuration, you might be prompted for the root password before you can Apply the configuration.

Hawk2—Wizard for Apache Web Server
Figure 7.2: Hawk2—Wizard for Apache Web Server

7.5.3 Adding Simple Resources Edit source

To create the most basic type of resource, proceed as follows:

Procedure 7.5: Adding a Primitive Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Resource › Primitive.

  3. Enter a unique Resource ID.

  4. In case a resource template exists on which you want to base the resource configuration, select the respective Template. For details about configuring templates, see Procedure 7.6, “Adding a Resource Template”.

  5. Select the resource agent Class you want to use: lsb, ocf, service, stonith, or systemd. For more information, see Section 6.3.2, “Supported Resource Agent Classes”.

  6. If you selected ocf as class, specify the Provider of your OCF resource agent. The OCF specification allows multiple vendors to supply the same resource agent.

  7. From the Type list, select the resource agent you want to use (for example, IPaddr or Filesystem). A short description for this resource agent is displayed.

    With that, you have specified the resource basics.

    Note
    Note

    The selection you get in the Type list depends on the Class (and for OCF resources also on the Provider) you have chosen.

    Hawk2—Primitive Resource
    Figure 7.3: Hawk2—Primitive Resource
  8. To keep the Parameters, Operations, and Meta Attributes as suggested by Hawk2, click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

    To adjust the parameters, operations, or meta attributes, refer to Section 7.5.5, “Modifying Resources”. To configure Utilization attributes for the resource, see Procedure 7.21, “Configuring the Capacity a Resource Requires”.

7.5.4 Adding Resource Templates Edit source

To create lots of resources with similar configurations, defining a resource template is the easiest way. After being defined, it can be referenced in primitives or in certain types of constraints. For detailed information about function and use of resource templates, refer to Section 6.5.3, “Resource Templates and Constraints”.

Procedure 7.6: Adding a Resource Template

Resource templates are configured like primitive resources.

  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Resource › Template.

  3. Enter a unique Resource ID.

  4. Follow the instructions in Procedure 7.5, “Adding a Primitive Resource”, starting from Step 5.

7.5.5 Modifying Resources Edit source

If you have created a resource, you can edit its configuration at any time by adjusting parameters, operations, or meta attributes as needed.

Procedure 7.7: Modifying Parameters, Operations, or Meta Attributes for a Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. On the Hawk2 Status screen, go to the Resources list.

  3. In the Operations column, click the arrow down icon next to the resource or group you want to modify and select Edit.

    The resource configuration screen opens.

    Hawk2—Editing A Primitive Resource
    Figure 7.4: Hawk2—Editing A Primitive Resource
  4. To add a new parameter, operation, or meta attribute, select an entry from the empty drop-down box.

  5. To edit any values in the Operations category, click the Edit icon of the respective entry, enter a different value for the operation, and click Apply.

  6. When you are finished, click the Apply button in the resource configuration screen to confirm your changes to the parameters, operations, or meta attributes.

    A message at the top of the screen shows if the action has been successful.

7.5.6 Adding STONITH Resources Edit source

Important
Important: No Support Without STONITH
  • You must have a node fencing mechanism for your cluster.

  • The global cluster options stonith-enabled and startup-fencing must be set to true. When you change them, you lose support.

By default, the global cluster option stonith-enabled is set to true. If no STONITH resources have been defined, the cluster will refuse to start any resources. Configure one or more STONITH resources to complete the STONITH setup. To add a STONITH resource for SBD, for libvirt (KVM/Xen) or for vCenter/ESX Server, the easiest way is to use the Hawk2 wizard (see Section 7.5.2, “Adding Resources with the Wizard”). While STONITH resources are configured similarly to other resources, their behavior is different in some respects. For details refer to Section 10.3, “STONITH Resources and Configuration”.

Procedure 7.8: Adding a STONITH Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Resource › Primitive.

  3. Enter a unique Resource ID.

  4. From the Class list, select the resource agent class stonith.

  5. From the Type list, select the STONITH plug-in to control your STONITH device. A short description for this plug-in is displayed.

  6. Hawk2 automatically shows the required Parameters for the resource. Enter values for each parameter.

  7. Hawk2 displays the most important resource Operations and proposes default values. If you do not modify any settings here, Hawk2 adds the proposed operations and their default values when you confirm.

  8. If there is no reason to change them, keep the default Meta Attributes settings.

    Hawk2—STONITH Resource
    Figure 7.5: Hawk2—STONITH Resource
  9. Confirm your changes to create the STONITH resource.

    A message at the top of the screen shows if the action has been successful.

To complete your fencing configuration, add constraints. For more details, refer to Chapter 10, Fencing and STONITH.

7.5.7 Adding Cluster Resource Groups Edit source

Some cluster resources depend on other components or resources. They require that each component or resource starts in a specific order and runs on the same server. To simplify this configuration SUSE Linux Enterprise High Availability Extension supports the concept of groups.

Resource groups contain a set of resources that need to be located together, be started sequentially and stopped in the reverse order. For an example of a resource group and more information about groups and their properties, refer to Section 6.3.5.1, “Groups”.

Note
Note: Empty Groups

Groups must contain at least one resource, otherwise the configuration is not valid. While creating a group, Hawk2 allows you to create more primitives and add them to the group. For details, see Section 7.7.1, “Editing Resources and Groups”.

Procedure 7.9: Adding a Resource Group
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Resource › Group.

  3. Enter a unique Group ID.

  4. To define the group members, select one or multiple entries in the list of Children. Re-sort group members by dragging and dropping them into the order you want by using the handle icon on the right.

  5. If needed, modify or add Meta Attributes.

  6. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Hawk2—Resource Group
Figure 7.6: Hawk2—Resource Group

7.5.8 Adding Clone Resources Edit source

If you want certain resources to run simultaneously on multiple nodes in your cluster, configure these resources as clones. An example of a resource that can be configured as a clone is ocf:pacemaker:controld for cluster file systems like OCFS2. Any regular resources or resource groups can be cloned. Instances of cloned resources may behave identically. However, they may also be configured differently, depending on which node they are hosted on.

For an overview of the available types of resource clones, refer to Section 6.3.5.2, “Clones”.

Note
Note: Child Resources for Clones

Clones can either contain a primitive or a group as child resources. In Hawk2, child resources cannot be created or modified while creating a clone. Before adding a clone, create child resources and configure them as desired. For details, refer to Section 7.5.3, “Adding Simple Resources” or Section 7.5.7, “Adding Cluster Resource Groups”.

Procedure 7.10: Adding a Clone Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Resource › Clone.

  3. Enter a unique Clone ID.

  4. From the Child Resource list, select the primitive or group to use as a sub-resource for the clone.

  5. If needed, modify or add Meta Attributes.

  6. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Hawk2—Clone Resource
Figure 7.7: Hawk2—Clone Resource

7.5.9 Adding Multi-state Resources Edit source

Multi-state resources are a specialization of clones. They allow the instances to be in one of two operating modes (called active/passive, primary/secondary, or master/slave). Multi-state resources must contain exactly one group or one regular resource.

When configuring resource monitoring or constraints, multi-state resources have different requirements than simple resources. For details, see Pacemaker Explained, available from http://www.clusterlabs.org/pacemaker/doc/. Refer to section Multi-state - Resources That Have Multiple Modes.

Note
Note: Child Resources for Multi-state Resources

Multi-state resources can either contain a primitive or a group as child resources. In Hawk2, child resources cannot be created or modified while creating a multi-state resource. Before adding a multi-state resource, create child resources and configure them as desired. For details, refer to Section 7.5.3, “Adding Simple Resources” or Section 7.5.7, “Adding Cluster Resource Groups”.

Procedure 7.11: Adding a Multi-state Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Resource › Multi-state.

  3. Enter a unique Multi-state ID.

  4. From the Child Resource list, select the primitive or group to use as a sub-resource for the multi-state resource.

  5. If needed, modify or add Meta Attributes.

  6. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Hawk2—Multi-state Resource
Figure 7.8: Hawk2—Multi-state Resource

7.5.10 Grouping Resources by Using Tags Edit source

Tags are a way to refer to multiple resources at once, without creating any colocation or ordering relationship between them. You can use tags for grouping conceptually related resources. For example, if you have several resources related to a database, you can add all related resources to a tag named database.

All resources belonging to a tag can be started or stopped with a single command.

Procedure 7.12: Adding a Tag
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Resource › Tag.

  3. Enter a unique Tag ID.

  4. From the Objects list, select the resources you want to refer to with the tag.

  5. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Hawk2—Tag
Figure 7.9: Hawk2—Tag

7.5.11 Configuring Resource Monitoring Edit source

The High Availability Extension does not only detect node failures, but also when an individual resource on a node has failed. If you want to ensure that a resource is running, configure resource monitoring for it. Usually, resources are only monitored by the cluster while they are running. However, to detect concurrency violations, also configure monitoring for resources which are stopped. For resource monitoring, specify a timeout and/or start delay value, and an interval. The interval tells the CRM how often it should check the resource status. You can also set particular parameters such as timeout for start or stop operations.

Procedure 7.13: Adding and Modifying an Operation
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. Add a resource as described in Procedure 7.5, “Adding a Primitive Resource” or select an existing primitive to edit.

    Hawk2 automatically shows the most important Operations (start, stop, monitor) and proposes default values.

    To see the attributes belonging to each proposed value, hover the mouse pointer over the respective value.

  3. To change the suggested timeout values for the start or stop operation:

    1. Click the pen icon next to the operation.

    2. In the dialog that opens, enter a different value for the timeout parameter, for example 10, and confirm your change.

  4. To change the suggested interval value for the monitor operation:

    1. Click the pen icon next to the operation.

    2. In the dialog that opens, enter a different value for the monitoring interval.

    3. To configure resource monitoring in the case that the resource is stopped:

      1. Select the role entry from the empty drop-down box below.

      2. From the role drop-down box, select Stopped.

      3. Click Apply to confirm your changes and to close the dialog for the operation.

  5. Confirm your changes in the resource configuration screen. A message at the top of the screen shows if the action has been successful.

For the processes that take place if the resource monitor detects a failure, refer to Section 6.4, “Resource Monitoring”.

To view resource failures, switch to the Status screen in Hawk2 and select the resource you are interested in. In the Operations column click the arrow down icon and select Recent Events. The dialog that opens lists recent actions performed for the resource. Failures are displayed in red. To view the resource details, click the magnifier icon in the Operations column.

Hawk2—Resource Details
Figure 7.10: Hawk2—Resource Details

7.6 Configuring Constraints Edit source

After you have configured all resources, specify how the cluster should handle them correctly. Resource constraints let you specify on which cluster nodes resources can run, in which order to load resources, and what other resources a specific resource depends on.

For an overview of available types of constraints, refer to Section 6.5.1, “Types of Constraints”. When defining constraints, you also need to specify scores. For more information on scores and their implications in the cluster, see Section 6.5.2, “Scores and Infinity”.

7.6.1 Adding Location Constraints Edit source

A location constraint determines on which node a resource may be run, is preferably run, or may not be run. An example of a location constraint is to place all resources related to a certain database on the same node.

Procedure 7.14: Adding a Location Constraint

  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Constraint › Location.

  3. Enter a unique Constraint ID.

  4. From the list of Resources select the resource or resources for which to define the constraint.

  5. Enter a Score. The score indicates the value you are assigning to this resource constraint. Positive values indicate the resource can run on the Node you specify in the next step. Negative values mean it should not run on that node. Constraints with higher scores are applied before those with lower scores.

    Some often-used values can also be set via the drop-down box:

    • To force the resources to run on the node, click the arrow icon and select Always. This sets the score to INFINITY.

    • If you never want the resources to run on the node, click the arrow icon and select Never. This sets the score to -INFINITY, meaning that the resources must not run on the node.

    • To set the score to 0, click the arrow icon and select Advisory. This disables the constraint. This is useful when you want to set resource discovery but do not want to constrain the resources.

  6. Select a Node.

  7. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Hawk2—Location Constraint
Figure 7.11: Hawk2—Location Constraint

7.6.2 Adding Colocation Constraints Edit source

A colocational constraint tells the cluster which resources may or may not run together on a node. As a colocation constraint defines a dependency between resources, you need at least two resources to create a colocation constraint.

Procedure 7.15: Adding a Colocation Constraint
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Constraint › Colocation.

  3. Enter a unique Constraint ID.

  4. Enter a Score. The score determines the location relationship between the resources. Positive values indicate that the resources should run on the same node. Negative values indicate that the resources should not run on the same node. The score will be combined with other factors to decide where to put the resource.

    Some often-used values can also be set via the drop-down box:

    • To force the resources to run on the same node, click the arrow icon and select Always. This sets the score to INFINITY.

    • If you never want the resources to run on the same node, click the arrow icon and select Never. This sets the score to -INFINITY, meaning that the resources must not run on the same node.

  5. To define the resources for the constraint:

    1. From the drop-down box in the Resources category, select a resource (or a template).

      The resource is added and a new empty drop-down box appears beneath.

    2. Repeat this step to add more resources.

      As the topmost resource depends on the next resource and so on, the cluster will first decide where to put the last resource, then place the depending ones based on that decision. If the constraint cannot be satisfied, the cluster may not allow the dependent resource to run.

    3. To swap the order of resources within the colocation constraint, click the arrow up icon next to a resource to swap it with the entry above.

  6. If needed, specify further parameters for each resource (such as Started, Stopped, Master, Slave, Promote, Demote): Click the empty drop-down box next to the resource and select the desired entry.

  7. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Hawk2—Colocation Constraint
Figure 7.12: Hawk2—Colocation Constraint

7.6.3 Adding Order Constraints Edit source

Order constraints define the order in which resources are started and stopped. As an order constraint defines a dependency between resources, you need at least two resources to create an order constraint.

Procedure 7.16: Adding an Order Constraint
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Add Constraint › Order.

  3. Enter a unique Constraint ID.

  4. Enter a Score. If the score is greater than zero, the order constraint is mandatory, otherwise it is optional.

    Some often-used values can also be set via the drop-down box:

    • If you want to make the order constraint mandatory, click the arrow icon and select Mandatory.

    • If you want the order constraint to be a suggestion only, click the arrow icon and select Optional.

    • Serialize: To ensure that no two stop/start actions occur concurrently for the resources, click the arrow icon and select Serialize. This makes sure that one resource must complete starting before the other can be started. A typical use case are resources that put a high load on the host during start-up.

  5. For order constraints, you can usually keep the option Symmetrical enabled. This specifies that resources are stopped in reverse order.

  6. To define the resources for the constraint:

    1. From the drop-down box in the Resources category, select a resource (or a template).

      The resource is added and a new empty drop-down box appears beneath.

    2. Repeat this step to add more resources.

      The topmost resource will start first, then the second, etc. Usually the resources will be stopped in reverse order.

    3. To swap the order of resources within the order constraint, click the arrow up icon next to a resource to swap it with the entry above.

  7. If needed, specify further parameters for each resource (like Started, Stopped, Master, Slave, Promote, Demote): Click the empty drop-down box next to the resource and select the desired entry.

  8. Confirm your changes to finish the configuration. A message at the top of the screen shows if the action has been successful.

Hawk2—Order Constraint
Figure 7.13: Hawk2—Order Constraint

7.6.4 Using Resource Sets for Constraints Edit source

As an alternative format for defining constraints, you can use Resource Sets. They have the same ordering semantics as Groups.

Procedure 7.17: Using a Resource Set for Constraints
  1. To use a resource set within a location constraint:

    1. Proceed as outlined in Procedure 7.14, “Adding a Location Constraint”, apart from Step 4: Instead of selecting a single resource, select multiple resources by pressing Ctrl or Shift and mouse click. This creates a resource set within the location constraint.

    2. To remove a resource from the location constraint, press Ctrl and click the resource again to deselect it.

  2. To use a resource set within a colocation or order constraint:

    1. Proceed as described in Procedure 7.15, “Adding a Colocation Constraint” or Procedure 7.16, “Adding an Order Constraint”, apart from the step where you define the resources for the constraint (Step 5.a or Step 6.a):

    2. Add multiple resources.

    3. To create a resource set, click the chain icon next to a resource to link it to the resource above. A resource set is visualized by a frame around the resources belonging to a set.

    4. You can combine multiple resources in a resource set or create multiple resource sets.

      Hawk2—Two Resource Sets in a Colocation Constraint
      Figure 7.14: Hawk2—Two Resource Sets in a Colocation Constraint
    5. To unlink a resource from the resource above, click the scissors icon next to the resource.

  3. Confirm your changes to finish the constraint configuration.

7.6.5 For More Information Edit source

For more information on configuring constraints and detailed background information about the basic concepts of ordering and colocation, refer to the documentation available at http://www.clusterlabs.org/pacemaker/doc/:

  • Pacemaker Explained, chapter Resource Constraints

  • Colocation Explained

  • Ordering Explained

7.6.6 Specifying Resource Failover Nodes Edit source

A resource will be automatically restarted if it fails. If that cannot be achieved on the current node, or it fails N times on the current node, it will try to fail over to another node. You can define several failures for resources (a migration-threshold), after which they will migrate to a new node. If you have more than two nodes in your cluster, the node to which a particular resource fails over is chosen by the High Availability software.

You can specify a specific node to which a resource will fail over by proceeding as follows:

Procedure 7.18: Specifying a Failover Node
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. Configure a location constraint for the resource as described in Procedure 7.14, “Adding a Location Constraint”.

  3. Add the migration-threshold meta attribute to the resource as described in Procedure 7.7: Modifying Parameters, Operations, or Meta Attributes for a Resource, Step 4 and enter a value for the migration-threshold. The value should be positive and less than INFINITY.

  4. If you want to automatically expire the failcount for a resource, add the failure-timeout meta attribute to the resource as described in Procedure 7.5: Adding a Primitive Resource, Step 4 and enter a Value for the failure-timeout.

  5. If you want to specify additional failover nodes with preferences for a resource, create additional location constraints.

The process flow regarding migration thresholds and failcounts is demonstrated in Example 6.8, “Migration Threshold—Process Flow”.

Instead of letting the failcount for a resource expire automatically, you can also clean up failcounts for a resource manually at any time. Refer to Section 7.7.3, “Cleaning Up Resources” for details.

7.6.7 Specifying Resource Failback Nodes (Resource Stickiness) Edit source

A resource may fail back to its original node when that node is back online and in the cluster. To prevent this or to specify a different node for the resource to fail back to, change the stickiness value of the resource. You can either specify the resource stickiness when creating it or afterward.

For the implications of different resource stickiness values, refer to Section 6.5.5, “Failback Nodes”.

Procedure 7.19: Specifying Resource Stickiness
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. Add the resource-stickiness meta attribute to the resource as described in Procedure 7.7: Modifying Parameters, Operations, or Meta Attributes for a Resource, Step 4.

  3. Specify a value between -INFINITY and INFINITY for resource-stickiness.

7.6.8 Configuring Placement of Resources Based on Load Impact Edit source

Not all resources are equal. Some, such as Xen guests, require that the node hosting them meets their capacity requirements. If resources are placed so that their combined needs exceed the provided capacity, the performance of the resources diminishes or they fail.

To take this into account, the High Availability Extension allows you to specify the following parameters:

  1. The capacity a certain node provides.

  2. The capacity a certain resource requires.

  3. An overall strategy for placement of resources.

For more details and a configuration example, refer to Section 6.5.6, “Placing Resources Based on Their Load Impact”.

Utilization attributes are used to configure both the resource's requirements and the capacity a node provides. You first need to configure a node's capacity before you can configure the capacity a resource requires.

Procedure 7.20: Configuring the Capacity a Node Provides
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Monitoring › Status.

  3. On the Nodes tab, select the node whose capacity you want to configure.

  4. In the Operations column, click the arrow down icon and select Edit.

    The Edit Node screen opens.

  5. Below Utilization, enter a name for a utilization attribute into the empty drop-down box.

    The name can be arbitrary (for example, RAM_in_GB).

  6. Click the Add icon to add the attribute.

  7. In the empty text box next to the attribute, enter an attribute value. The value must be an integer.

  8. Add as many utilization attributes as you need and add values for all of them.

  9. Confirm your changes. A message at the top of the screen shows if the action has been successful.

Procedure 7.21: Configuring the Capacity a Resource Requires

Configure the capacity a certain resource requires from a node either when creating a primitive resource or when editing an existing primitive resource.

Before you can add utilization attributes to a resource, you need to have set utilization attributes for your cluster nodes as described in Procedure 7.20.

  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. To add a utilization attribute to an existing resource: Go to Manage › Status and open the resource configuration dialog as described in Section 7.7.1, “Editing Resources and Groups”.

    If you create a new resource: Go to Configuration › Add Resource and proceed as described in Section 7.5.3, “Adding Simple Resources”.

  3. In the resource configuration dialog, go to the Utilization category.

  4. From the empty drop-down box, select one of the utilization attributes that you have configured for the nodes in Procedure 7.20.

  5. In the empty text box next to the attribute, enter an attribute value. The value must be an integer.

  6. Add as many utilization attributes as you need and add values for all of them.

  7. Confirm your changes. A message at the top of the screen shows if the action has been successful.

After you have configured the capacities your nodes provide and the capacities your resources require, set the placement strategy in the global cluster options. Otherwise the capacity configurations have no effect. Several strategies are available to schedule the load: for example, you can concentrate it on as few nodes as possible, or balance it evenly over all available nodes. For more information, refer to Section 6.5.6, “Placing Resources Based on Their Load Impact”.

Procedure 7.22: Setting the Placement Strategy
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Cluster Configuration to open the respective screen. It shows global cluster options and resource and operation defaults.

  3. From the empty drop-down box in the upper part of the screen, select placement-strategy.

    By default, its value is set to Default, which means that utilization attributes and values are not considered.

  4. Depending on your requirements, set Placement Strategy to the appropriate value.

  5. Confirm your changes.

7.7 Managing Cluster Resources Edit source

In addition to configuring your cluster resources, Hawk2 allows you to manage existing resources from the Status screen. For a general overview of the screen refer to Section 7.8.1, “Monitoring a Single Cluster”.

7.7.1 Editing Resources and Groups Edit source

In case you need to edit existing resources, go to the Status screen. In the Operations column, click the arrow down icon next to the resource or group you want to modify and select Edit.

The editing screen appears. If you edit a primitive resource, the following operations are available:

Operations for Primitives
  • Copying the resource.

  • Renaming the resource (changing its ID).

  • Deleting the resource.

If you edit a group, the following operations are available:

Operations for Groups
  • Creating a new primitive which will be added to this group.

  • Renaming the group (changing its ID).

  • Re-sort group members by dragging and dropping them into the order you want using the handle icon on the right.

7.7.2 Starting Resources Edit source

Before you start a cluster resource, make sure it is set up correctly. For example, if you use an Apache server as a cluster resource, set up the Apache server first. Complete the Apache configuration before starting the respective resource in your cluster.

Note
Note: Do Not Touch Services Managed by the Cluster

When managing a resource via the High Availability Extension, the resource must not be started or stopped otherwise (outside of the cluster, for example manually or on boot or reboot). The High Availability Extension software is responsible for all service start or stop actions.

However, if you want to check if the service is configured properly, start it manually, but make sure that it is stopped again before the High Availability Extension takes over.

For interventions in resources that are currently managed by the cluster, set the resource to maintenance mode first. For details, see Procedure 17.5, “Putting a Resource into Maintenance Mode with Hawk2”.

When creating a resource with Hawk2, you can set its initial state with the target-role meta attribute. If you set its value to stopped, the resource does not start automatically after being created.

Procedure 7.23: Starting A New Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.

  3. Select the resource to start. In its Operations column click the Start icon. To continue, confirm the message that appears.

    When the resource has started, Hawk2 changes the resource's Status to green and shows on which node it is running.

7.7.3 Cleaning Up Resources Edit source

A resource will be automatically restarted if it fails, but each failure increases the resource's failcount.

If a migration-threshold has been set for the resource, the node will no longer run the resource when the number of failures reaches the migration threshold.

A resource's failcount can either be reset automatically (by setting a failure-timeout option for the resource) or it can be reset manually as described below.

Procedure 7.24: Cleaning Up A Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Status. The list of Resources also shows the Status.

  3. Go to the resource to clean up. In the Operations column click the arrow down button and select Cleanup. To continue, confirm the message that appears.

    This executes the command crm resource cleanup and cleans up the resource on all nodes.

7.7.4 Removing Cluster Resources Edit source

If you need to remove a resource from the cluster, follow the procedure below to avoid configuration errors:

Procedure 7.25: Removing a Cluster Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. Clean up the resource on all nodes as described in Procedure 7.24, “Cleaning Up A Resource”.

  3. Stop the resource:

    1. From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.

    2. In the Operations column click the Stop button next to the resource.

    3. To continue, confirm the message that appears.

      The Status column will reflect the change when the resource is stopped.

  4. Delete the resource:

    1. From the left navigation bar, select Configuration › Edit Configuration.

    2. In the list of Resources, go to the respective resource. From the Operations column click the Delete icon next to the resource.

    3. To continue, confirm the message that appears.

7.7.5 Migrating Cluster Resources Edit source

As mentioned in Section 7.6.6, “Specifying Resource Failover Nodes”, the cluster will fail over (migrate) resources automatically in case of software or hardware failures—according to certain parameters you can define (for example, migration threshold or resource stickiness). You can also manually migrate a resource to another node in the cluster. Or you decide to move the resource away from the current node and let the cluster decide where to put it.

Procedure 7.26: Manually Migrating a Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.

  3. In the list of Resources, select the respective resource.

  4. In the Operations column click the arrow down button and select Migrate.

  5. In the window that opens you have the following choices:

    • Away from current node: This creates a location constraint with a -INFINITY score for the current node.

    • Alternatively, you can move the resource to another node. This creates a location constraint with an INFINITY score for the destination node.

  6. Confirm your choice.

To allow a resource to move back again, proceed as follows:

Procedure 7.27: Unmigrating a Resource
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.

  3. In the list of Resources, go to the respective resource.

  4. In the Operations column click the arrow down button and select Clear. To continue, confirm the message that appears.

    Hawk2 uses the crm_resource  --clear command. The resource can move back to its original location or it may stay where it is (depending on resource stickiness).

For more information, see Pacemaker Explained, available from http://www.clusterlabs.org/pacemaker/doc/. Refer to section Resource Migration.

7.8 Monitoring Clusters Edit source

Hawk2 has different screens for monitoring single clusters and multiple clusters: the Status and the Dashboard screen.

7.8.1 Monitoring a Single Cluster Edit source

To monitor a single cluster, use the Status screen. After you have logged in to Hawk2, the Status screen is displayed by default. An icon in the upper right corner shows the cluster status at a glance. For further details, have a look at the following categories:

Errors

If errors have occurred, they are shown at the top of the page.

Resources

Shows the configured resources including their Status, Name (ID), Location (node on which they are running), and resource agent Type. From the Operations column, you can start or stop a resource, trigger several actions, or view details. Actions that can be triggered include setting the resource to maintenance mode (or removing maintenance mode), migrating it to a different node, cleaning up the resource, showing any recent events, or editing the resource.

Nodes

Shows the nodes belonging to the cluster site you are logged in to, including the nodes' Status and Name. In the Maintenance and Standby columns, you can set or remove the maintenance or standby flag for a node. The Operations column allows you to view recent events for the node or further details: for example, if a utilization, standby or maintenance attribute is set for the respective node.

Tickets

Only shown if tickets have been configured (for use with Geo clustering).

Hawk2—Cluster Status
Figure 7.15: Hawk2—Cluster Status

7.8.2 Monitoring Multiple Clusters Edit source

To monitor multiple clusters, use the Hawk2 Dashboard. The cluster information displayed in the Dashboard screen is stored on the server side. It is synchronized between the cluster nodes (if passwordless SSH access between the cluster nodes has been configured). For details, see Section D.2, “Configuring a Passwordless SSH Account”. However, the machine running Hawk2 does not even need to be part of any cluster for that purpose—it can be a separate, unrelated system.

In addition to the general Hawk2 Requirements, the following prerequisites need to be fulfilled to monitor multiple clusters with Hawk2:

Prerequisites
  • All clusters to be monitored from Hawk2's Dashboard must be running SUSE Linux Enterprise High Availability Extension 15 SP2.

  • If you did not replace the self-signed certificate for Hawk2 on every cluster node with your own certificate (or a certificate signed by an official Certificate Authority) yet, do the following: Log in to Hawk2 on every node in every cluster at least once. Verify the certificate (or add an exception in the browser to bypass the warning). Otherwise Hawk2 cannot connect to the cluster.

Procedure 7.28: Monitoring Multiple Clusters with the Dashboard
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Monitoring › Dashboard.

    Hawk2 shows an overview of the resources and nodes on the current cluster site. In addition, it shows any Tickets that have been configured for use with a Geo cluster. If you need information about the icons used in this view, click Legend. To search for a resource ID, enter the name (ID) into the Search text box. To only show specific nodes, click the filter icon and select a filtering option.

    Hawk2 Dashboard with One Cluster Site (amsterdam)
    Figure 7.16: Hawk2 Dashboard with One Cluster Site (amsterdam)
  3. To add dashboards for multiple clusters:

    1. Click Add Cluster.

    2. Enter the Cluster name with which to identify the cluster in the Dashboard. For example, berlin.

    3. Enter the fully qualified host name of one of the nodes in the second cluster. For example, charlie.

    4. Click Add. Hawk2 will display a second tab for the newly added cluster site with an overview of its nodes and resources.

      Note
      Note: Connection Error

      If instead you are prompted to log in to this node by entering a password, you probably did not connect to this node yet and have not replaced the self-signed certificate. In that case, even after entering the password, the connection will fail with the following message: Error connecting to server. Retrying every 5 seconds... '.

      To proceed, see Replacing the Self-Signed Certificate.

  4. To view more details for a cluster site or to manage it, switch to the site's tab and click the chain icon.

    Hawk2 opens the Status view for this site in a new browser window or tab. From there, you can administer this part of the Geo cluster.

  5. To remove a cluster from the dashboard, click the x icon on the right-hand side of the cluster's details.

7.9 Using the Batch Mode Edit source

Hawk2 provides a Batch Mode, including a cluster simulator. It can be used for the following:

  • Staging changes to the cluster and applying them as a single transaction, instead of having each change take effect immediately.

  • Simulating changes and cluster events, for example, to explore potential failure scenarios.

For example, batch mode can be used when creating groups of resources that depend on each other. Using batch mode, you can avoid applying intermediate or incomplete configurations to the cluster.

While batch mode is enabled, you can add or edit resources and constraints or change the cluster configuration. It is also possible to simulate events in the cluster, including nodes going online or offline, resource operations and tickets being granted or revoked. See Procedure 7.30, “Injecting Node, Resource or Ticket Events” for details.

The cluster simulator runs automatically after every change and shows the expected outcome in the user interface. For example, this also means: If you stop a resource while in batch mode, the user interface shows the resource as stopped—while actually, the resource is still running.

Important
Important: Wizards and Changes to the Live System

Some wizards include actions beyond mere cluster configuration. When using those wizards in batch mode, any changes that go beyond cluster configuration would be applied to the live system immediately.

Therefore wizards that require root permission cannot be executed in batch mode.

Procedure 7.29: Working with the Batch Mode
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. To activate the batch mode, select Batch from the top-level row.

    An additional bar appears below the top-level row. It indicates that batch mode is active and contains links to actions that you can execute in batch mode.

    Hawk2 Batch Mode Activated
    Figure 7.17: Hawk2 Batch Mode Activated
  3. While batch mode is active, perform any changes to your cluster, like adding or editing resources and constraints or editing the cluster configuration.

    The changes will be simulated and shown in all screens.

  4. To view details of the changes you have made, select Show from the batch mode bar. The Batch Mode window opens.

    For any configuration changes it shows the difference between the live state and the simulated changes in crmsh syntax: Lines starting with a - character represent the current state whereas lines starting with + show the proposed state.

  5. To inject events or view even more details, see Procedure 7.30. Otherwise Close the window.

  6. Choose to either Discard or Apply the simulated changes and confirm your choice. This also deactivates batch mode and takes you back to normal mode.

When running in batch mode, Hawk2 also allows you to inject Node Events and Resource Events.

Node Events

Let you change the state of a node. Available states are online, offline, and unclean.

Resource Events

Let you change some properties of a resource. For example, you can set an operation (like start, stop, monitor), the node it applies to, and the expected result to be simulated.

Ticket Events

Let you test the impact of granting and revoking tickets (used for Geo clusters).

Procedure 7.30: Injecting Node, Resource or Ticket Events
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. If batch mode is not active yet, click Batch at the top-level row to switch to batch mode.

  3. In the batch mode bar, click Show to open the Batch Mode window.

  4. To simulate a status change of a node:

    1. Click Inject › Node Event.

    2. Select the Node you want to manipulate and select its target State.

    3. Confirm your changes. Your event is added to the queue of events listed in the Batch Mode dialog.

  5. To simulate a resource operation:

    1. Click Inject › Resource Event.

    2. Select the Resource you want to manipulate and select the Operation to simulate.

    3. If necessary, define an Interval.

    4. Select the Node on which to run the operation and the targeted Result. Your event is added to the queue of events listed in the Batch Mode dialog.

    5. Confirm your changes.

  6. To simulate a ticket action:

    1. Click Inject › Ticket Event.

    2. Select the Ticket you want to manipulate and select the Action to simulate.

    3. Confirm your changes. Your event is added to the queue of events listed in the Batch Mode dialog.

  7. The Batch Mode dialog (Figure 7.18) shows a new line per injected event. Any event listed here is simulated immediately and is reflected on the Status screen.

    If you have made any configuration changes, too, the difference between the live state and the simulated changes is shown below the injected events.

    Hawk2 Batch Mode—Injected Invents and Configuration Changes
    Figure 7.18: Hawk2 Batch Mode—Injected Invents and Configuration Changes
  8. To remove an injected event, click the Remove icon next to it. Hawk2 updates the Status screen accordingly.

  9. To view more details about the simulation run, click Simulator and choose one of the following:

    Summary

    Shows a detailed summary.

    CIB (in)/CIB (out)

    CIB (in) shows the initial CIB state. CIB (out) shows what the CIB would look like after the transition.

    Transition Graph

    Shows a graphical representation of the transition.

    Transition

    Shows an XML representation of the transition.

  10. If you have reviewed the simulated changes, close the Batch Mode window.

  11. To leave the batch mode, either Apply or Discard the simulated changes.

7.10 Viewing the Cluster History Edit source

Hawk2 provides the following possibilities to view past events on the cluster (on different levels and in varying detail):

7.10.1 Viewing Recent Events of Nodes or Resources Edit source

  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Monitoring › Status. It lists Resources and Nodes.

  3. To view recent events of a resource:

    1. Click Resources and select the respective resource.

    2. In the Operations column for the resource, click the arrow down button and select Recent events.

      Hawk2 opens a new window and displays a table view of the latest events.

  4. To view recent events of a node:

    1. Click Nodes and select the respective node.

    2. In the Operations column for the node, select Recent events.

      Hawk2 opens a new window and displays a table view of the latest events.

7.10.2 Using the History Explorer for Cluster Reports Edit source

From the left navigation bar, select Troubleshooting › History to access the History Explorer. The History Explorer allows you to create detailed cluster reports and view transition information. It provides the following options:

Generate

Create a cluster report for a certain time. Hawk2 calls the crm report command to generate the report.

Upload

Allows you to upload crm report archives that have either been created with the crm shell directly or even on a different cluster.

After reports have been generated or uploaded, they are shown below Reports. From the list of reports, you can show a report's details, download or delete the report.

Hawk2—History Explorer Main View
Figure 7.19: Hawk2—History Explorer Main View
Procedure 7.31: Generating or Uploading a Cluster Report
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Troubleshooting › History.

    The History Explorer screen opens in the Generate view. By default, the suggested time frame for a report is the last hour.

  3. To create a cluster report:

    1. To immediately start a report, click Generate.

    2. To modify the time frame for the report, click anywhere on the suggested time frame and select another option from the drop-down box. You can also enter a Custom start date, end date and hour, respectively. To start the report, click Generate.

      After the report has finished, it is shown below Reports.

  4. To upload a cluster report, the crm report archive must be located on a file system that you can access with Hawk2. Proceed as follows:

    1. Switch to the Upload tab.

    2. Browse for the cluster report archive and click Upload.

      After the report is uploaded, it is shown below Reports.

  5. To download or delete a report, click the respective icon next to the report in the Operations column.

  6. To view Report Details in History Explorer, click the report's name or select Show from the Operations column.

  7. Return to the list of reports by clicking the Reports button.

Report Details in History Explorer
  • Name of the report.

  • Start time of the report.

  • End time of the report.

  • Number of transitions plus time line of all transitions in the cluster that are covered by the report. To learn how to view more details for a transition, see Section 7.10.3.

  • Node events.

  • Resource events.

7.10.3 Viewing Transition Details in the History Explorer Edit source

For each transition, the cluster saves a copy of the state which it provides as input to pacemaker-schedulerd. The path to this archive is logged. All pe-* files are generated on the Designated Coordinator (DC). As the DC can change in a cluster, there may be pe-* files from several nodes. Any pe-* files are saved snapshots of the CIB, used as input of calculations by pacemaker-schedulerd.

In Hawk2, you can display the name of each pe-* file plus the time and node on which it was created. In addition, the History Explorer can visualize the following details, based on the respective pe-* file:

Transition Details in the History Explorer
Details

Shows snippets of logging data that belongs to the transition. Displays the output of the following command (including the resource agents' log messages):

crm history transition peinput
Configuration

Shows the cluster configuration at the time that the pe-* file was created.

Diff

Shows the differences of configuration and status between the selected pe-* file and the following one.

Log

Shows snippets of logging data that belongs to the transition. Displays the output of the following command:

crm history transition log peinput

This includes details from the following daemons: pacemaker-schedulerd, pacemaker-controld, and pacemaker-execd.

Graph

Shows a graphical representation of the transition. If you click Graph, the calculation is simulated (exactly as done by pacemaker-schedulerd) and a graphical visualization is generated.

Procedure 7.32: Viewing Transition Details
  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Troubleshooting › History.

    If reports have already been generated or uploaded, they are shown in the list of Reports. Otherwise generate or upload a report as described in Procedure 7.31.

  3. Click the report's name or select Show from the Operations column to open the Report Details in History Explorer.

  4. To access the transition details, you need to select a transition point in the transition time line that is shown below. Use the Previous and Next icons and the Zoom In and Zoom Out icons to find the transition that you are interested in.

  5. To display the name of a pe-input* file plus the time and node on which it was created, hover the mouse pointer over a transition point in the time line.

  6. To view the Transition Details in the History Explorer, click the transition point for which you want to know more.

  7. To show Details, Configuration, Diff, Logs or Graph, click the respective buttons to show the content described in Transition Details in the History Explorer.

  8. To return to the list of reports, click the Reports button.

7.11 Verifying Cluster Health Edit source

Hawk2 provides a wizard which checks and detects issues with your cluster. After the analysis is complete, Hawk2 creates a cluster report with further details. To verify cluster health and generate the report, Hawk2 requires passwordless SSH access between the nodes. Otherwise it can only collect data from the current node. If you have set up your cluster with the bootstrap scripts, provided by the ha-cluster-bootstrap package, passwordless SSH access is already configured. In case you need to configure it manually, see Section D.2, “Configuring a Passwordless SSH Account”.

  1. Log in to Hawk2:

    https://HAWKSERVER:7630/
  2. From the left navigation bar, select Configuration › Wizards.

  3. Expand the Basic category.

  4. Select the Verify health and configuration wizard.

  5. Confirm with Verify.

  6. Enter the root password for your cluster and click Apply. Hawk2 will generate the report.