Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE OpenStack Cloud Crowbar 8

9 Customizing Crowbar

9.1 Skip Unready Nodes

In large deployments with many nodes, there are always some nodes that are in a fail or unknown state. New barclamps cannot be applied to them and values cannot be updated in some barclamps that are already deployed. This happens because Crowbar will refuse to apply a barclamp to a list of nodes if they are not all in ready state.

To avoid having to manually take out nodes that are not ready, there is a feature called skip unready nodes. Instead of refusing to apply the barclamp, it will skip the nodes that it finds in any other state than ready.

Enabling the Feature

In /opt/dell/crowbar_framework/config/crowbar.yml, set the option skip_unready_nodes to true.

default: &default
skip_unready_nodes:
  enabled: false <<< change to true

Roles Affected

All Barclamp roles are affected. The default config file includes all the roles that have been tested and found to be working. Adding roles to the default list is not supported for the skip_unready_nodes feature. Removing default roles is supported.

The list of currently supported roles to skip:

- bmc-nat-client
- ceilometer-agent
- deployer-client
- dns-client
- ipmi
- logging-client
- nova-compute-ironic
- nova-compute-kvm
- nova-compute-qemu
- nova-compute-vmware
- nova-compute-xen
- ntp-client
- provisioner-base
- suse-manager-client
- swift-storage
- updater

Determining Which Nodes Were Skipped

Skipped nodes are logged to the Crowbar log (/var/log/crowbar/production.log) where you can search for the text skipped until next chef run. This will print the log lines where nodes were skipped, the name of the node, and the barclamp which was being applied.

tux > grep "skipped until next chef run" /var/log/crowbar/production.log
Important
Important

After enabling/disabling the skip_unready_nodes feature or adding/removing roles, the Crowbar framework service must be restarted (systemctl restart crowbar) in order to use the updated settings.

9.2 Skip Unchanged Nodes

When a barclamp is applied, all nodes will run chef-client. Sometimes it is not necessary to run chef for each node as attributes or roles may have not changed for some of the nodes.

The skip unchanged nodes feature allows nodes to be skipped if their attributes or roles have not changed. This can help speed up applying a barclamp, especially in large deployments and with roles that are usually applied to a lot of nodes, such as Nova.

Enabling the Feature

In /opt/dell/crowbar_framework/config/crowbar.yml, set the option skip_unchanged_nodes to true.

default: &default
skip_unchanged_nodes:
  enabled: false <<< change to true

9.3 Controlling Chef Restarts Manually

When a service configuration has changed, Chef forces this service to restart. Sometimes it is useful to have manual control of these restarts. This feature supports avoiding automatic restart of services and including them in a pending restart list. Disabling restarts is enabled/disabled by barclamp and cannot be done on a service level. In other words, enabling this feature on the Cinder barclamp will disable automatic restarts for all Cinder-* services.

Two steps are necessary to activate this feature:

  1. Enable disallow_restart in the Crowbar configuration file

  2. Set the disable_restart flag for a specific barclamp using crowbar-client or API

Enabling the Feature

  1. In /opt/dell/crowbar_framework/config/crowbar.yml, set the option disallow_restart to true.

    default: &default
    disallow_restart:
      enabled: false <<< change to true
  2. The disable_restart flag can be set with the Crowbar client or with the API.

    • Set Flag with Crowbar Client

      The crowbar client options for this feature are accessed with the crowbarctl services command, and only work for OpenStack services. The options are:

      disable_restart

      The parameters are the barclamp name and the flag value (true to disable automatic restart and false to enable automatic restart). The command is crowbarctl services disable_restart BARCLAMP <true|false>. For example, to disable restart of the Keystone barclamp, enter the command crowbarctl services disable_restart keystone true.

      restart_flags

      Used to check the disable_restart flag for each barclamp

      crowbarctl services restart_flags
      {
        "nova": true,
        "keystone": true,
        "database": true
      }
      list_restarts

      Displays the list of pending restarts. The pending restart flag indicates that a service tried to restart due to the Chef run but it did not due to the automatic restart being disabled. It also indicates that the service might have a new configuration and it will not be applied until it is manually restarted.

      In the following example, the pacemaker_service attribute indicates whether this service is managed by Pacemaker (usually in an HA environment) or it is a standalone service managed by systemd (usually in non-HA environments). More on Pacemaker at Section 12.2, “Deploying Pacemaker (Optional, HA Setup Only)”. The service to restart will be apache2 not managed by Pacemaker.

      crowbarctl services list_restarts
      {
        "NODE_IP": {
          "alias": "controller1",
          "keystone": {
            "apache2": {
              "pacemaker_service": false,
              "timestamp": "2017-11-22 11:17:49 UTC"
            }
          }
        }
      }
      clear_restart

      Removes the flag on a specific node for a specific service. It can be executed when the service has restarted manually. The command is crowbarctl services clear_restart NODE SERVICE. For example, crowbarctl services clear_restart NODE_IP apache2

    • Using the API to List, Get Status, Set and Clear Flags

      • /restart_management/configuration

        GET

        Lists the barclamps and the status of service reboots disallowed

        POST (parameters: disallow_restarts, barclamp)

        Sets the disallow_restart flag for a barclamp

      • /restart_management/restarts

        GET

        Lists all of the services needing restarts

        POST (parameters: node, service)

        Clears the restart flag for the given service in the given node

9.4 Prevent Automatic Restart

Sometimes a change in a proposal requires services to be restarted on all implicated nodes. This could be a problem in a production environment where interrupting a service completely is not an option, and where manual restart control is needed. With the service disable_restart feature in crowbarctl, you can set a flag for certain proposals to prevent the automatic restart.

Enabling the Feature

  1. In /opt/dell/crowbar_framework/config/crowbar.yml, set the option disallow_restart to true.

    default: &default
    disallow_restart:
      enabled: false <<< change to true
  2. Restart Crowbar

  3. Enable the disable_restart flag in the affected cookbook.

    crowbarctl services disable_restart COOKBOOK true

    For example, to disable Keystone and RabbitMQ restarts:

    crowbarctl services disable_restart keystone true
    crowbarctl services disable_restart rabbitmq true
  4. To check the proposals that have disable_restart enabled:

    crowbarctl services restart_flags
    {
      "rabbitmq": true,
      "keystone": true,
    }

Check Pending Restarts

When a proposal is applied with the disable_restart flag enabled, the implicated nodes will not restart. They will be listed as pending restarts. To check this list, run the command crowbarctl services list_restarts.

In the following example, disable_restart is enabled for RabbitMQ. After applying rabbitmq, list_restarts will show the affected nodes, the proposal, and the services to restart.

crowbarctl services list_restarts
{
   "d52-54-77-77-01-01.vo5.cloud.suse.de": {
    "alias": "controller1",
    "rabbitmq": {
      "rabbitmq-server": {
        "pacemaker_service": false,
        "timestamp": "2018-03-07 15:30:30 UTC"
      }
    }
  }
}

After a manual restart of the service, the flags should be cleaned. The following command will clean the flag for a specific node, for a specific proposal or all proposals, and for a specific service.

crowbarctl services clear_restart NODE [COOKBOOK [SERVICE]]

For example, to clean the RabbitMQ flag from the controller1 node, run the command:

crowbarctl service clear_restart controller1 rabbitmq

To clean the controller1 node, run the command:

crowbarctl service clear_restart controller1