Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / SUSE Linux Enterprise High Availability Documentation / Administration Guide / Configuration and administration / QDevice and QNetd
Applies to SUSE Linux Enterprise High Availability 15 SP5

14 QDevice and QNetd

  

QDevice and QNetd participate in quorum decisions. With assistance from the arbitrator corosync-qnetd, corosync-qdevice provides a configurable number of votes, allowing a cluster to sustain more node failures than the standard quorum rules allow. We recommend deploying corosync-qnetd and corosync-qdevice for clusters with an even number of nodes, and especially for two-node clusters.

14.1 Conceptual overview

  

In comparison to calculating quora among cluster nodes, the QDevice-and-QNetd approach has the following benefits:

  • It provides better sustainability in case of node failures.

  • You can write your own heuristics scripts to affect votes. This is especially useful for complex setups.

  • It enables you to configure a QNetd server to provide votes for multiple clusters.

  • It allows using diskless SBD for two-node clusters.

  • It helps with quorum decisions for clusters with an even number of nodes under split-brain situations, especially for two-node clusters.

A setup with QDevice/QNetd consists of the following components and mechanisms:

QDevice/QNetd components and mechanisms
  
QNetd (corosync-qnetd)

A systemd service (a daemon, the “QNetd server”) which is not part of the cluster. The systemd service provides a vote to the corosync-qdevice daemon.

To improve security, corosync-qnetd can work with TLS for client certificate checking.

QDevice (corosync-qdevice)

A systemd service (a daemon) on each cluster node running together with Corosync. This is the client of corosync-qnetd. Its primary use is to allow a cluster to sustain more node failures than standard quorum rules allow.

QDevice is designed to work with different arbitrators. However, currently, only QNetd is supported.

Algorithms

QDevice supports different algorithms, which determine the behavior of how votes are assigned. Currently, the following exist:

  • FFSplit (“fifty-fifty split”) is the default. It is used for clusters with an even number of nodes. If the cluster splits into two similar partitions, this algorithm provides one vote to one of the partitions, based on the results of heuristics checks and other factors.

  • LMS (“last man standing”) allows the only remaining node that can see the QNetd server to get the votes. So this algorithm is useful when a cluster with only one active node should remain quorate.

Heuristics

QDevice supports a set of commands (“heuristics”). The commands are executed locally on start-up of cluster services, cluster membership change, successful connection to corosync-qnetd, or, optionally, at regular times. The heuristics can be set with the quorum.device.heuristics key (in the corosync.conf file) or with the --qdevice-heuristics-mode option. Both know the values off (default), sync, and on. The difference between sync and on is that you can additionally execute the above commands regularly.

Only if all commands are executed successfully are the heuristics considered to have passed; otherwise, they failed. The heuristics' result is sent to corosync-qnetd where it is used in calculations to determine which partition should be quorate.

Tiebreaker

This is used as a fallback if the cluster partitions are completely equal even with the same heuristics results. It can be configured to be the lowest, the highest, or a specific node ID.

14.2 Requirements and prerequisites

  

Before setting up QDevice and QNetd, you need to prepare the environment as follows:

  • In addition to the cluster nodes, you have a separate machine that will become the QNetd server. See Section 14.3, “Setting up the QNetd server”.

  • A different physical network than the one that Corosync uses. It is recommended for QDevice to reach the QNetd server. Ideally, the QNetd server should be in a separate rack than the main cluster, or at least on a separate PSU and not in the same network segment as the Corosync ring or rings.

14.3 Setting up the QNetd server

  

The QNetd server runs outside the cluster and can support multiple clusters (as long as each cluster has a unique name). As such, you cannot move resources to this server.

The QNetd server is almost “state free”. Usually, you do not need to change anything in the configuration file /etc/sysconfig/corosync-qnetd. By default, the corosync-qnetd service runs the daemon as user coroqnetd in the group coroqnetd. This avoids running the daemon as root.

To create a QNetd server, proceed as follows:

  1. On the machine that will become the QNetd server, install SUSE Linux Enterprise Server 15 SP5.

  2. Enable the SUSE Linux Enterprise High Availability using the command listed in SUSEConnect --list-extensions.

  3. Install the corosync-qnetd package: 

    # zypper install corosync-qnetd

    You do not need to manually start the corosync-qnetd service. It starts automatically when you configure QDevice on the cluster.

Your QNetd server is ready to accept connections from a QDevice client (corosync-qdevice). Further configuration is handled by crmsh when you connect QDevice clients.

14.4 Connecting QDevice clients to the QNetd server

  

After you set up your QNetd server, you can set up and run the clients. You can connect the clients to the QNetd server during the installation of your cluster, or you can add them later. This procedure documents how to add them later.

Important

To complete the QNetd configuration, the clients must have root SSH access to the QNetd server.

  1. On all nodes, install the corosync-qdevice package: 

    # zypper install corosync-qdevice

  2. On one of the nodes, run the following command to configure QDevice: 

    # crm cluster init qdevice
Do you want to configure QDevice (y/n)? y
HOST or IP of the QNetd server to be used []QNETD_SERVER
TCP PORT of QNetd server [5403]
QNetd decision ALGORITHM (ffsplit/lms) [ffsplit]
QNetd TIE_BREAKER (lowest/highest/valid node id) [lowest]
Whether using TLS on QDevice/QNetd (on/off/required) [on]
Heuristics COMMAND to run with absolute path; For multiple commands, use ";" to separate []

    Confirm with y that you want to configure QDevice, then enter the host name or IP address of the QNetd server. For the remaining fields, you can accept the default values or change them if required.

The script configures QDevice on the nodes and completes the QNetd configuration on the QNetd server, including generating CA and server certificates, starting the QNetd daemon, and updating the cluster's quorum configuration.

Important
Important: SBD_WATCHDOG_TIMEOUT for diskless SBD and QDevice

If you use QDevice with diskless SBD, the SBD_WATCHDOG_TIMEOUT value must be greater than QDevice's sync_timeout value, or SBD will time out and fail to start.

The default value for sync_timeout is 30 seconds. Therefore, in the file /etc/sysconfig/sbd, make sure that SBD_WATCHDOG_TIMEOUT is set to a greater value, such as 35.

14.5 Setting up a QDevice with heuristics

  

If you need additional control over how votes are determined, use heuristics. Heuristics are a set of commands that are executed in parallel.

For this purpose, the command crm cluster init qdevice provides the option --qdevice-heuristics. You can pass one or more commands (separated by semicolons) with absolute paths.

For example, if your own command for heuristic checks is located at /usr/sbin/my-script.sh you can run it on one of your cluster nodes as follows: 

# crm cluster init qdevice --qnetd-hostname=charlie \
     --qdevice-heuristics=/usr/sbin/my-script.sh \
     --qdevice-heuristics-mode=on

The command or commands can be written in any language such as Shell, Python, or Ruby. If they succeed, they return 0 (zero), otherwise they return an error code.

You can also pass a set of commands. Only when all commands finish successfully (return code is zero), have the heuristics passed.

The --qdevice-heuristics-mode=on option lets the heuristics commands run regularly.

14.6 Checking and showing quorum status

  

You can query the quorum status on one of your cluster nodes as shown in Example 14.1, “Status of QDevice”. It shows the status of your QDevice nodes.

Example 14.1: Status of QDevice
  
# corosync-quorumtool1
 Quorum information
------------------
Date:             ...
Quorum provider:  corosync_votequorum
Nodes:            2 2
Node ID:          3232235777 3
Ring ID:          3232235777/8
Quorate:          Yes 4

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
 3232235777         1    A,V,NMW 192.168.1.1 (local) 5
 3232235778         1    A,V,NMW 192.168.1.2 5
         0          1            Qdevice

1

As an alternative with an identical result, you can also use the crm corosync status quorum command.

2

The number of nodes we are expecting. In this example, it is a two-node cluster.

3

As the node ID is not explicitly specified in corosync.conf this ID is a 32-bit integer representation of the IP address. In this example, the value 3232235777 stands for the IP address 192.168.1.1.

4

The quorum status. In this case, the cluster has quorum.

5

The status for each cluster node means:

A (alive) or NA (not alive)

Shows the connectivity status between QDevice and Corosync. If there is a heartbeat between QDevice and Corosync, it is shown as alive (A).

V (vote) or NV (non vote)

Shows if the quorum device has given a vote (letter V) to the node. A letter V means that both nodes can communicate with each other. In a split-brain situation, one node would be set to V and the other node would be set to NV.

MW (master wins) or NMW(not master wins)

Shows if the quorum device master_wins flag is set. By default, the flag is not set, so you see NMW (not master wins). See the man page votequorum_qdevice_master_wins(3) for more information.

NR (not registered)

Shows that the cluster is not using a quorum device.

If you query the status of the QNetd server, you get a similar output to that shown in Example 14.2, “Status of QNetd server”:

Example 14.2: Status of QNetd server
  
# corosync-qnetd-tool -lv1
Cluster "hacluster": 2
    Algorithm:          Fifty-Fifty split 3
    Tie-breaker:        Node with lowest node ID
    Node ID 3232235777: 4
        Client address:         ::ffff:192.168.1.1:54732
        HB interval:            8000ms
        Configured node list:   3232235777, 3232235778
        Ring ID:                aa10ab0.8
        Membership node list:   3232235777, 3232235778
        Heuristics:             Undefined (membership: Undefined, regular: Undefined)
        TLS active:             Yes (client certificate verified)
        Vote:                   ACK (ACK)
    Node ID 3232235778:
        Client address:         ::ffff:192.168.1.2:43016
        HB interval:            8000ms
        Configured node list:   3232235777, 3232235778
        Ring ID:                aa10ab0.8
        Membership node list:   3232235777, 3232235778
        Heuristics:             Undefined (membership: Undefined, regular: Undefined)
        TLS active:             Yes (client certificate verified)
        Vote:                   No change (ACK)

1

As an alternative with an identical result, you can also use the crm corosync status qnetd command on one of the cluster nodes.

2

The name of your cluster as set in the configuration file /etc/corosync/corosync.conf in the totem.cluster_name section.

3

The algorithm currently used. In this example, it is FFSplit.

4

This is the entry for the node with the IP address 192.168.1.1. TLS is active.

14.7 For more information

  

For additional information about QDevice and QNetd, see the man pages of corosync-qdevice(8) and corosync-qnetd(8).