|Index|SUSE Linux Enterprise High Availability Documentation|Administration Guide|Configuration and administration|Managing cluster resources
Applies to SUSE Linux Enterprise High Availability 15 SP7

12 Managing cluster resources

  

After configuring the resources in the cluster, use the cluster management tools to start, stop, clean up, remove or migrate the resources. This chapter describes how to use Hawk2 or crmsh for resource management tasks.

12.1 Showing cluster resources

  

12.1.1 Showing cluster resources with Hawk2

  

You can use Hawk2 on any machine with a Web browser and network access to the nodes.

  1. In a Web browser, go to https://HAWKSERVER:7630/ and log in.

    The Status page shows the cluster resources and their statuses. If the resource is a group or a clone, you can click the plus icon to show the child resources.

  2. From the left navigation bar, select Configuration › Edit Configuration.

    On this page, you can click different tabs to view more details about cluster objects such as resources, constraints or nodes. You can also limit the results by using the Search bar.

  3. To view the full CIB, click Show.

    Alternatively, you can click XML to view the XML representation of the CIB.

12.1.2 Showing cluster resources with crmsh

  

When administering a cluster the command crm configure show lists the current CIB objects like cluster configuration, global options, primitives, and others: 

# crm configure show
node 178326192: alice
node 178326448: bob
primitive admin_addr IPaddr2 \
       params ip=192.168.2.1 \
       op monitor interval=10 timeout=20
primitive stonith-sbd stonith:external/sbd \
       params pcmk_delay_max=30
property cib-bootstrap-options: \
       have-watchdog=true \
       dc-version=1.1.15-17.1-e174ec8 \
       cluster-infrastructure=corosync \
       cluster-name=hacluster \
       stonith-enabled=true \
       placement-strategy=balanced \
       standby-mode=true
rsc_defaults rsc-options: \
       resource-stickiness=1 \
       migration-threshold=3
op_defaults op-options: \
       timeout=600 \
       record-pending=true

If you have lots of resources, the output of show is too verbose. To restrict the output, use the name of the resource. For example, to list the properties of the primitive admin_addr only, append the resource name to show: 

# crm configure show admin_addr
primitive admin_addr IPaddr2 \
       params ip=192.168.2.1 \
       op monitor interval=10 timeout=20

However, in some cases, you want to limit the output of specific resources even more. This can be achieved with filters. Filters limit the output to specific components. For example, to list the nodes only, use type:node: 

# crm configure show type:node
node 178326192: alice
node 178326448: bob

If you are also interested in primitives, use the or operator:

# crm configure show type:node or type:primitive
node 178326192: alice
node 178326448: bob
primitive admin_addr IPaddr2 \
       params ip=192.168.2.1 \
       op monitor interval=10 timeout=20
primitive stonith-sbd stonith:external/sbd \
       params pcmk_delay_max=30

Furthermore, to search for an object that starts with a certain string, use this notation:

# crm configure show type:primitive and 'admin*'
primitive admin_addr IPaddr2 \
       params ip=192.168.2.1 \
       op monitor interval=10 timeout=20

To list all available types, enter crm configure show type: and press the →| key. The Bash completion gives you a list of all types.

12.2 Editing cluster resources

  

You can edit resources using either Hawk2 or crmsh.

12.2.1 Editing resources with Hawk2

  

If you have created a resource, you can edit its configuration at any time by adjusting parameters, operations or meta attributes as needed.

Procedure 12.1: Modifying a resource
  

  1. Log in to Hawk2: 

    https://HAWKSERVER:7630/

  2. On the Hawk2 Status screen, go to the Resources list.

  3. In the Operations column, click the arrow down icon next to the resource you want to modify and select Edit.

    The resource configuration screen opens.

    In the Edit Primitive screen, the Resource ID, Class, Provider, and Type fields cannot be changed for an existing primitive. You can edit the parameters, operations, meta attributes and utilization settings.
    Figure 12.1: Hawk2—editing a primitive resource
      

  4. At the top of the configuration screen, you can select operations to perform:

    • Copying the resource

    • Renaming the resource (changing its ID)

    • Deleting the resource

  5. To add a new parameter, operation or meta attribute, select an entry from the empty drop-down box.

  6. To edit any values in the Operations category, click the Edit icon of the respective entry, enter a different value for the operation, and click Apply.

  7. When you are finished, click the Apply button in the resource configuration screen to confirm your changes to the parameters, operations or meta attributes.

    A message at the top of the screen shows if the action has been successful.

12.2.2 Editing resources with crmsh

  
Procedure 12.2: Editing resources with crmsh
  

  1. Log in either as the root user or as a user with sudo privileges.

  2. To add a new monitor operation, use the following command: 

    # crm configure monitor RESOURCE INTERVAL:TIMEOUT

  3. To add a new parameter, meta attribute or utilization attribute, use one of the following commands: 

    # crm resource param RESOURCE set PARAMETER VALUE
# crm resource meta RESOURCE set ATTRIBUTE VALUE
# crm resource utilization RESOURCE set ATTRIBUTE VALUE

  4. To delete a parameter, meta attribute or utilization attribute, use one of the following commands: 

    # crm resource param RESOURCE delete PARAMETER
# crm resource meta RESOURCE delete ATTRIBUTE
# crm resource utilization RESOURCE delete ATTRIBUTE

  5. To change multiple attributes at once, use the following command: 

    # crm configure edit RESOURCE

    This command opens the CIB representation of the resource in a text editor. You can make multiple changes to the resource's configuration, then save and exit.

    Important

    Don't use this command to rename a resource. The change won't apply to related objects.

  6. To rename a resource, first stop the resource, then use the following command: 

    # crm configure rename OLD_NAME NEW_NAME

    This also changes the resource's name in related objects such as constraints, groups and clones.

12.3 Editing resource groups

  

You can edit groups using either Hawk2 or crmsh.

12.3.1 Editing groups with Hawk2

  

If you have created a group, you can edit its configuration at any time by adjusting parameters, operations or meta attributes as needed.

Procedure 12.3: Modifying a group
  

  1. Log in to Hawk2: 

    https://HAWKSERVER:7630/

  2. On the Hawk2 Status screen, go to the Resources list.

  3. In the Operations column, click the arrow down icon next to the group you want to modify and select Edit.

    The group configuration screen opens.

  4. At the top of the configuration screen, you can select operations to perform:

    • Creating a new primitive to be added to this group

    • Renaming the group (changing its ID)

    • Dragging and dropping group members into a new order

  5. To add a new parameter, operation or meta attribute, select an entry from the empty drop-down box.

  6. To edit any values in the Operations category, click the Edit icon of the respective entry, enter a different value for the operation, and click Apply.

  7. When you are finished, click the Apply button in the resource configuration screen to confirm your changes to the parameters, operations or meta attributes.

    A message at the top of the screen shows if the action has been successful.

12.3.2 Editing groups with crmsh

  
Procedure 12.4: Editing groups with crmsh
  

  1. Log in either as the root user or as a user with sudo privileges.

  2. To remove a resource from a group, use the following command: 

    # crm configure modgroup GROUP remove RESOURCE

  3. To add a resource to a group, use the following command: 

    # crm configure modgroup GROUP add RESOURCE

    The resource is appended to the end of the group, meaning it starts last and stops first.

    Alternatively, you can use before or after to add the resource to a specific position in the group's order: 

    # crm configure modgroup GROUP add RESOURCE0 before RESOURCE1
    # crm configure modgroup GROUP add RESOURCE2 after RESOURCE1

  4. To change the order of resources that are already in the group, use the following command: 

    # crm configure edit GROUP

    This command opens the CIB representation of the group in a text editor. You can make multiple changes to the group's configuration, then save and exit.

    Important

    Don't use this command to rename a group. The change won't apply to related objects.

  5. To rename a group, use the following command: 

    # crm configure rename OLD_NAME NEW_NAME

    This also changes the group's name in related objects such as constraints and clones.

12.4 Starting cluster resources

  

Before you start a cluster resource, make sure it is set up correctly. For example, if you use an Apache server as a cluster resource, set up the Apache server first. Complete the Apache configuration before starting the respective resource in your cluster.

Note
Note: Do not touch services managed by the cluster

When managing a resource via the High Availability software, the resource must not be started or stopped otherwise (outside the cluster, for example manually or on boot or reboot). The High Availability software is responsible for all service start or stop actions.

However, if you want to check if the service is configured properly, start it manually, but make sure that it is stopped again before the High Availability software takes over.

For interventions in resources that are currently managed by the cluster, set the resource to maintenance mode first. For details, see Procedure 32.5, “Putting a resource into maintenance mode with Hawk2”.

You can start a cluster resource using either Hawk2 or crmsh.

12.4.1 Starting cluster resources with Hawk2

  

When creating a resource with Hawk2, you can set its initial state with the target-role meta attribute. If you set its value to stopped, the resource does not start automatically after being created.

Procedure 12.5: Starting a new resource
  

  1. In a Web browser, go to https://HAWKSERVER:7630/ and log in.

  2. From the left navigation bar, select Monitoring › Status.

    The Resources tab shows the Status of the cluster resources. A stopped resource has a red Status icon and an empty Location column.

  3. Find the resource you want to start. In its Operations column, click the Start icon.

  4. Click OK to continue.

    The resource's Status icon changes to green, and the Location column shows which node the resource is running on.

When the resource has started, Hawk2 changes the resource's Status to green and shows on which node it is running.

12.4.2 Starting cluster resources with crmsh

  

To start a new cluster resource, you need the respective identifier.

Procedure 12.6: Starting a cluster resource with crmsh
  

  1. Log in either as the root user or as a user with sudo privileges.

  2. Check the current status of the resource: 

    # crm resource status

  3. Start the resource. If necessary, you can start multiple resources at once: 

    # crm resource start RESOURCE1 RESOURCE2

  4. Check the status again to make sure the resource started successfully: 

    # crm resource status

12.5 Stopping cluster resources

  

12.5.1 Stopping cluster resources with Hawk2

  
Procedure 12.7: Stopping cluster resources with Hawk2
  

  1. In a Web browser, go to https://HAWKSERVER:7630/ and log in.

  2. From the left navigation bar, select Monitoring › Status.

    The Resources tab shows the Status of the cluster resources. A running resource has a green Status icon.

  3. Find the resource you want to stop. In its Operations column, click the Stop icon.

  4. Click OK to continue.

    The resource's Status icon changes to red.

12.5.2 Stopping cluster resources with crmsh

  

To stop one or more existing cluster resources, you need the respective identifiers.

Procedure 12.8: Stopping cluster resources with crmsh
  

  1. Log in either as the root user or as a user with sudo privileges.

  2. Check the current status of the resource: 

    # crm resource status

  3. Stop the resource. If necessary, you can stop multiple resources at once: 

    # crm resource stop RESOURCE1 RESOURCE2

  4. Check the status again to make sure the resource stopped successfully: 

    # crm resource status

12.6 Cleaning up cluster resources

  

A resource is automatically restarted if it fails, but each failure increases the resource's fail count.

If a migration-threshold has been set for the resource, the node can no longer run the resource when the number of failures reaches the migration threshold.

By default, fail counts are not automatically reset. You can configure a fail count to be reset automatically by setting a failure-timeout option for the resource, or you can manually reset the fail count using either Hawk2 or crmsh.

12.6.1 Cleaning up cluster resources with Hawk2

  
Procedure 12.9: Cleaning up a resource
  

  1. Log in to Hawk2: 

    https://HAWKSERVER:7630/

  2. From the left navigation bar, select Status. The list of Resources also shows the Status.

  3. Go to the resource to clean up. In the Operations column, click the arrow down button and select Cleanup. To continue, confirm the message that appears.

    This executes the command crm resource cleanup and cleans up the resource on all nodes.

12.6.2 Cleaning up cluster resources with crmsh

  
Procedure 12.10: Cleaning up a resource with crmsh
  

  1. Open a shell and log in as user root.

  2. Get a list of all your resources: 

    # crm resource status
Full List of Resources
   * admin-ip      (ocf:heartbeat:IPaddr2):    Started
   * stonith-sbd   (stonith:external/sbd):     Started
   * Resource Group: dlm-clvm:
     * dlm:        (ocf:pacemaker:controld)    Started
     * clvm:       (ocf:heartbeat:lvmlockd)    Started

  3. Show the fail count of a resource: 

    # crm resource failcount RESOURCE show NODE

    For example, to show the fail count of the resource dlm on node alice: 

    # crm resource failcount dlm show alice
scope=status name=fail-count-dlm value=2

  4. Clean up the resource: 

    # crm resource cleanup RESOURCE

    This command cleans up the resource on all nodes. If the resource is part of a group, crmsh also cleans up the other resources in the group.

12.7 Removing cluster resources

  

To remove a resource from the cluster, follow either the Hawk2 or crmsh procedure below to avoid configuration errors.

12.7.1 Removing cluster resources with Hawk2

  
Procedure 12.11: Removing a cluster resource
  

  1. Log in to Hawk2: 

    https://HAWKSERVER:7630/

  2. Clean up the resource on all nodes as described in Procedure 12.9, “Cleaning up a resource”.

  3. Stop the resource:

    1. From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.

    2. In the Operations column, click the Stop button next to the resource.

    3. To continue, confirm the message that appears.

      The Status column reflects the change when the resource is stopped.

  4. Delete the resource:

    1. From the left navigation bar, select Configuration › Edit Configuration.

    2. In the list of Resources, go to the respective resource. From the Operations column, click the Delete icon next to the resource.

    3. To continue, confirm the message that appears.

12.7.2 Removing cluster resources with crmsh

  
Procedure 12.12: Removing a cluster resource with crmsh
  

  1. Log in as root and start the crm interactive shell: 

    # crm

  2. Get a list of your resources: 

    crm(live)# resource status
Full List of Resources:
  * admin-ip     (ocf:heartbeat:IPaddr2):     Started
  * stonith-sbd  (stonith:external/sbd):      Started
  * nfsserver    (ocf:heartbeat:nfsserver):   Started

  3. Stop the resource you want to remove: 

    crm(live)# resource stop RESOURCE

  4. Delete the resource: 

    crm(live)# configure delete RESOURCE

12.8 Migrating cluster resources

  

The cluster fails over (migrates) resources automatically during software or hardware failures, according to certain parameters you can define (for example, migration threshold or resource stickiness). You can also manually migrate a resource to another node in the cluster, or move the resource away from the current node and let the cluster decide where to put it.

You can migrate a cluster resource using either Hawk2 or crmsh.

12.8.1 Migrating cluster resources with Hawk2

  
Procedure 12.13: Manually migrating a resource
  

  1. Log in to Hawk2: 

    https://HAWKSERVER:7630/

  2. From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.

  3. In the list of Resources, select the respective resource.

  4. In the Operations column, click the arrow down button and select Migrate.

  5. In the window that opens, you have the following choices:

    • Away from current node: This creates a location constraint with a -INFINITY score for the current node.

    • Alternatively, you can move the resource to another node. This creates a location constraint with an INFINITY score for the destination node.

  6. Confirm your choice.

To allow a resource to move back again, proceed as follows:

Procedure 12.14: Unmigrating a resource
  

  1. Log in to Hawk2: 

    https://HAWKSERVER:7630/

  2. From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.

  3. In the list of Resources, go to the respective resource.

  4. In the Operations column, click the arrow down button and select Clear. To continue, confirm the message that appears.

    Hawk2 uses the crm_resource  --clear command. The resource can move back to its original location or it may stay where it is (depending on resource stickiness).

For more information, see Pacemaker Explained, available from https://clusterlabs.org/projects/pacemaker/doc/. Refer to section Resource Migration.

12.8.2 Migrating cluster resources with crmsh

  
Procedure 12.15: Manually migrating a resource with crmsh
  

  1. Log in either as the root user or as a user with sudo privileges.

  2. Check which node the resource is currently running on: 

    # crm status

  3. Move the resource to another node: 

    # crm resource move RESOURCE NODE

    Alternatively, you can move the resource away from the current node without specifying a specific node to move to: 

    # crm resource move RESOURCE force

  4. Check the status again to make sure the resource is now running on the other node: 

    # crm status

To allow a resource to move back again, proceed as follows:

Procedure 12.16: Unmigrating a resource with crmsh
  

  1. Log in either as the root user or as a user with sudo privileges.

  2. Show the existing location constraints: 

    # crm configure show type:location

    Relocation constraints are named either cli-prefer-RESOURCE or cli-ban-RESOURCE-on-NODE.

  3. Remove any relocation constraints from a specific resource: 

    # crm resource clear RESOURCE

  4. Check the status to see which node the resource is now running on: 

    # crm status

12.9 Grouping resources by using tags

  

Tags are a way to refer to multiple resources at once, without creating any colocation or ordering relationship between them. This can be useful for grouping conceptually related resources. For example, if you have several resources related to a database, create a tag called databases and add all resources related to the database to this tag. This allows you to stop or start them all with a single command.

Tags can also be used in constraints. For example, the following location constraint loc-db-prefer applies to the set of resources tagged with databases: 

location loc-db-prefer databases 100: alice

You can create tags using either Hawk2 or crmsh.

12.9.1 Grouping resources by using tags with Hawk2

  
Procedure 12.17: Adding a tag
  

  1. Log in to Hawk2: 

    https://HAWKSERVER:7630/

  2. From the left navigation bar, select Configuration › Add Resource › Tag.

  3. Enter a unique Tag ID.

  4. From the Objects list, select the resources you want to refer to with the tag.

  5. Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

The Create Tag screen includes a field to add a Tag ID and a list of Objects (resources). Activate the check boxes to add the tag to the selected resources.
Figure 12.2: Hawk2—tag
  

12.9.2 Grouping resources by using tags with crmsh

  

For example, if you have several resources related to a database, create a tag called databases and add all resources related to the database to this tag: 

# crm configure tag databases: db1 db2 db3

This allows you to start them all with a single command: 

# crm resource start databases

Similarly, you can stop them all too: 

# crm resource stop databases