SAP HANA System Replication Scale-Up - Performance Optimized Scenario #

For the SAPHanaSR-angi package configuration as decribed in this document, we limit the support to scale-up (single-box to single-box) system replication with the following configurations and parameters:

For the HA/DR provider hook scripts susHanaSR.py and susTkOver.py, the following requirements apply:

For the HA/DR provider hook script susChkSrv.py, the following requirements apply:

See also manual pages SAPHanaSR(7), susHanaSR.py(7), susTkOver.py(7) and susChkSrv.py(7) for more details and requirements.

Important

Without a valid fencing method, the complete cluster is unsupported and will not work properly.

If you need to implement a different scenario, we strongly recommend to define a Proof of Concept (PoC) with SUSE. This PoC will focus on testing the existing solution in your scenario. Most of the above mentioned limitations are set because careful testing is needed.

This document describes how to set up the cluster to control the database instances and system replication for the scenario SAP HANA scale-up performance optimized.

The document focuses on the steps to integrate an already installed and working SAP HANA with System Replication. This guide can be used with SUSE Linux Enterprise Server for SAP applications 16.0 or newer.

The described example setup builds an SAP HANA HA cluster in two data centers in Walldorf (WDF) and in Rot (ROT), installed on two SLES for SAP 16.0 systems.

Figure 2: SAP HANA System Replication Scale-Up in the Cluster - performance optimized #

 

This guide focuses on the manual setup of the cluster to explain the details and to give you the possibility to create your own automation.

The seven main setup steps are:

Planning the installation is essential for a successful SAP HANA cluster setup.

Before you start, you need the following:

4.1 Minimum lab requirements and prerequisites #

 

Note

The minimum lab requirements mentioned here are by no means SAP sizing information. These data are provided only to rebuild the described cluster in a lab for test purposes. Even for tests the requirements can increase, depending on your test scenario. For productive systems ask your hardware vendor or use the official SAP sizing tools and services.

Note

Refer to SAP HANA TDI documentation for allowed storage configuration and file systems.

Requirements with one SAP system replication instance per site (1 : 1) - without a majority maker (2-node cluster):

• 2 VMs with each 32GB RAM, 50GB disk space for the system

• 1 shared disk for SBD with 10 MB disk space

• 2 data disks (one per site) with a capacity of each 96GB for SAP HANA

• 1 additional IP address for takeover

• 1 optional IP address for the read-enabled setup

Requirements with one SAP system replication instance per site (1 : 1) - with a majority maker (3-node cluster):

• 2 VMs with each 32GB RAM, 50GB disk space for the system

• 1 VM with 2GB RAM, 50GB disk space for the system

• 2 data disks (one per site) with a capacity of each 96GB for SAP HANA

• 1 additional IP address for takeover

• 1 optional IP address for the read-enabled setup

This section contains information you should consider during the installation of the operating system.

For the scope of this document, first SUSE Linux Enterprise Server for SAP applications is installed and configured. Then the SAP HANA database including the system replication is set up. Finally the automation with the cluster is set up and configured.

suse01:~ # zypper in --type pattern ha_sles sles_sap_HADB

Even though this document focuses on the integration of an installed SAP HANA with system replication already set up into the Linux cluster, this chapter summarizes the test environment. Always use the official documentation from SAP to install SAP HANA and to set up the system replication.

This guide shows SAP HANA and saphostagent with native systemd integration.

# systemctl list-unit-files "sap*"
UNIT FILE                        STATE           PRESET
saphostagent.service             enabled         disabled
saptune.service                  enabled         enabled

# systemctl list-unit-files "SAP*"
UNIT FILE                        STATE           PRESET
SAPHA1_10.service                enabled         disabled
SAP.slice                        static          -

# systemd-cgls -u SAP.slice
Unit SAP.slice (/SAP.slice):
├─saphostagent.service
│ ├─2630 /usr/sap/hostctrl/exe/saphostexec pf=/usr/sap/hostctrl/exe/host_profile -systemd
│ ├─2671 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D
│ └─3591 /usr/sap/hostctrl/exe/saposcol -l -w60 pf=/usr/sap/hostctrl/exe/host_profile
└─SAPHA1_10.service
  ├─ 1257 hdbcompileserver
  ├─ 1274 hdbpreprocessor
  ├─ 1353 hdbindexserver -port 31003
  ├─ 1356 hdbxsengine -port 31007
  ├─ 2077 hdbwebdispatcher
  ├─ 2300 hdbrsutil --start --port 31003 --volume 3 --volumesuffix mnt00001/hdb00003.00003 --identifier 1644426276
  ├─28462 /usr/sap/HA1/HDB10/exe/sapstartsrv pf=/usr/sap/HA1/SYS/profile/HA1_HDB10_suse01
  ├─31314 sapstart pf=/usr/sap/HA1/SYS/profile/HA1_HDB10_suse01
  ├─31372 /usr/sap/HA1/HDB10/suse01/trace/hdb.sapHA1_HDB10 -d -nw -f /usr/sap/HA1/HDB10/suse01/daemon.ini pf=/usr/sap/HA1/SYS/profile/HA1_HDB10_suse01
  ├─31479 hdbnameserver
  └─32201 hdbrsutil --start --port 31001 --volume 1 --volumesuffix mnt00001/hdb00001 --identifier 1644426203

For more information read the section Setting Up System Replication of the SAP HANA Administration Guide.

Procedure

7.1 Backing up the primary database #

 

Back up the primary database as described in the SAP HANA Administration Guide, section SAP HANA Database Backup and Recovery. We provide an example with SQL commands. You need to adapt these backup commands to match your backup infrastructure.

Example 1: Simple backup for the system database and all tenants with one single backup call #

 

As user <sid>adm enter the following command:

~> hdbsql -i 10 -u SYSTEM -d SYSTEMDB \
   "BACKUP DATA FOR FULL SYSTEM USING FILE ('backup')"

You will get a command output similar to the following:

0 rows affected (overall time 15.352069 sec; server time 15.347745 sec)

Example 2: Simple backup for a single container (non MDC) database #

 

Enter the following command as user <sid>adm:

~> hdbsql -i <instanceNumber> -u <dbuser> \
   "BACKUP DATA USING FILE ('backup')"

Important

Without a valid backup, you cannot bring SAP HANA into a system replication configuration.

7.2 Enabling the primary node #

 

As Linux user <sid>adm, enable the system replication at the primary node. You need to define a site name (like WDF). This site name must be unique for all SAP HANA databases which are connected via system replication. This means the secondary must have a different site name. The site names must not be changed later when the cluster has been activated.

Note

Do not use strings like "primary" and "secondary" as site names.

7.2.1 Enable primary node #

 

Enable the primary using the -sr_enable option.

suse01:~> hdbnsutil -sr_enable --name=WDF
checking local nameserver:
checking for active nameserver ...
nameserver is running, proceeding ...
configuring ini files ...
successfully enabled system as primary site ...
done.

7.2.2 Check system replication configuration on primary node #

 

Check the primary using the command hdbnsutil -sr_stateConfiguration.

suse01:~> hdbnsutil -sr_stateConfiguration --sapcontrol=1
SAPCONTROL-OK: <begin>
mode=primary
site id=1
site name=WDF
SAPCONTROL-OK: <end>
done.

The mode has changed from “none” to “primary”. The site now has a site name and a site ID.

suse02:~> HDB stop

~> cd /usr/sap/<SID>/SYS/global/security/rsecssfs
~> rsync -va {,<node1-siteB>:}$PWD/data/SSFS_<SID>.DAT
~> rsync -va {,<node1-siteB>:}$PWD/key/SSFS_<SID>.KEY

...
suse02:~> hdbnsutil -sr_register --name=ROT \
     --remoteHost=suse01 --remoteInstance=10 \
     --replicationMode=sync --operationMode=logreplay
adding site ...
checking for inactive nameserver ...
nameserver suse02:30001 not responding.
collecting information ...
updating local ini files ...
done.

suse02:~> HDB start
...
suse02:~> hdbnsutil -sr_stateConfiguration --sapcontrol=1
SAPCONTROL-OK: <begin>
mode=sync
site id=2
site name=ROT
active primary site=1
primary masters=suse01
SAPCONTROL-OK: <end>
done.

suse01:~> cdpy; python3 systemReplicationStatus.py --sapcontrol=1
...
site/2/SITE_NAME=ROT
site/2/SOURCE_SITE_ID=1
site/2/REPLICATION_MODE=SYNC
site/2/REPLICATION_STATUS=ACTIVE
overall_replication_status=ACTIVE
site/1/REPLICATION_MODE=PRIMARY
site/1/SITE_NAME=WDF
local_site_id=1
...

This step is mandatory to inform the cluster immediately if the secondary gets out of sync. The hook is called by SAP HANA using the HA/DR provider interface in point-of-time when the secondary gets out of sync. This is typically the case when the first commit pending is released. The hook is called by SAP HANA again when the system replication is back. This HA/DR provider method is srConnectionChanged(), the related SUSE hook script is susHanaSR.py. The hook script susHanaSR.py shall be used.

Another hook is called by SAP HANA before an SR takeover is processed. This method can be used to block a manual takeover during normal cluster operation. This HA/DR provider method is preTakeover(), the related SUSE hook script is susTkOver.py.

A third hook is called by SAP HANA when a service status changes. This method can be used to speed up the takeover in case the indexserver process fails. This HA/DR provider method is srServiceStateChanged(), the related SUSE hook script is susChkSrv.py.

Procedure

This will implement three SAP HANA HA/DR provider hook scripts. The hook script susHanaSR.py does not need any configuration parameters. The configuration for susTkOver.py normally does not need to be adapted. The configuration shown for susChkSrv.py is a good starting point. Any tuning should be aligned with the SAP experts.

Note

All hook scripts should be used directly from the SAPHanaSR-angi package. If the scripts are moved or copied, regular SUSE package updates will not work.

SAP HANA must be stopped to change the global.ini and allow SAP HANA to integrate the HA/DR hook scripts during start. Alternatively, SAPHanaSR-manageProvider might be used for adapting the global.ini. See manual page SAPHanaSR-manageProvider(8) for details.

~> sapcontrol -nr <instanceNumber> -function StopSystem

[ha_dr_provider_sushanasr]
provider = susHanaSR
path = /usr/share/SAPHanaSR-angi/
execution_order = 1

[trace]
ha_dr_sushanasr = info

~> sapcontrol -nr <instanceNumber> -function StopSystem

[ha_dr_provider_sustkover]
provider = susTkOver
path = /usr/share/SAPHanaSR-angi/
execution_order = 2

[trace]
ha_dr_sustkover = info
...

~> sapcontrol -nr <instanceNumber> -function StopSystem

[ha_dr_provider_suschksrv]
provider = susChkSrv
path = /usr/share/SAPHanaSR-angi/
execution_order = 3
action_on_lost=fence

[trace]
ha_dr_suschksrv = info
...

[system_replication]
operation_mode = logreplay

# SAPHanaSR-ScaleUp entries for writing srHook cluster attribute and SAPHanaSR-hookHelper
<sid>adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_<sid>_*
<sid>adm ALL=(ALL) NOPASSWD: /usr/bin/SAPHanaSR-hookHelper --sid=<SID> *

# SAPHanaSR-ScaleUp entries for writing srHook cluster attribute
Cmnd_Alias SOK_SITEA    = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_<siteA> -v SOK   -t crm_config -s SAPHanaSR
Cmnd_Alias SFAIL_SITEA  = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_<siteA> -v SFAIL -t crm_config -s SAPHanaSR
Cmnd_Alias SOK_SITEB    = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_<siteB> -v SOK   -t crm_config -s SAPHanaSR
Cmnd_Alias SFAIL_SITEB  = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_<siteB> -v SFAIL -t crm_config -s SAPHanaSR
Cmnd_Alias HOOK_HELPER  = /usr/bin/SAPHanaSR-hookHelper --sid=<SID> --case=checkTakeover

<sid>adm ALL=(ALL) NOPASSWD: SOK_SITEA, SFAIL_SITEA, SOK_SITEB, SFAIL_SITEB, HOOK_HELPER

~> HDB start

~> cdtrace
~> grep HADR.*load.*susHanaSR nameserver_*.trc
~> grep susHanaSR.init nameserver_*.trc

~> cdtrace
~> grep HADR.*load.*susTkOver nameserver_*.trc
~> grep susTkOver.init nameserver_*.trc

~> cdtrace
~> grep HADR.*load.*susChkSrv nameserver_*.trc
~> grep susChkSrv.init nameserver_*.trc
~> egrep '(LOST:|STOP:|START:|DOWN:|init|load|fail)' nameserver_suschksrv.trc

~> cdtrace
~> grep susHanaSR.srConnection.*CRM nameserver_*.trc
~> grep susHanaSR.srConnection.*fallback nameserver_*.trc

# grep "sudo.*SAPHanaSR-hookHelper"  /var/log/messages

~> cdtrace
~> grep susTkOver.preTakeover.*permit nameserver_*.trc
~> grep susTkOver.preTakeover.*failed.*50277 nameserver_*.trc

This chapter describes the configuration of the cluster software SUSE Linux Enterprise High Availability, which is part of SUSE Linux Enterprise Server for SAP applications, and the SAP HANA database integration.

9.1 Configuring the basic cluster #

 

The first step is to set up the basic cluster framework. For convenience, use the crmsh command crm cluster init .

9.1.1 Setting up watchdog for SBD fencing #

 

If you use the storage-based fencing (SBD) mechanism (diskless or disk-based), you must also configure a watchdog. The watchdog is needed to reset a node if the system cannot longer access the SBD (diskless or disk-based). It is mandatory to configure the Linux system for loading a watchdog driver. It is strongly recommended to use a watchdog with hardware assistance (as is available on most modern systems), such as hpwdt, iTCO_wdt, or others. As fallback, you can use the softdog module.

Important: Setup for Watchdog

Access to the watchdog timer: No other software must access the watchdog timer; it can only be accessed by one process at any time. Some hardware vendors ship systems management software that use the watchdog for system resets. Such software must be disabled if the watchdog is to be used by SBD.

Determine the right watchdog module. Alternatively, you can find a list of installed drivers with your kernel version.

# ls -l /lib/modules/$(uname -r)/kernel/drivers/watchdog

Check if any watchdog module is already loaded.

# lsmod | egrep "(wd|dog|i6|iT|ibm)"

If you get a result, the system has already a loaded watchdog. If the watchdog does not match your watchdog device, you need to unload the module.

To safely unload the module, check first if an application is using the watchdog device.

# lsof /dev/watchdog
# rmmod <wrong_module>

Enable your watchdog module and make it persistent. For the example below, softdog has been used. However, softdog has some restrictions and should not be used as first option.

# echo softdog > /etc/modules-load.d/watchdog.conf
# systemctl restart systemd-modules-load

Check if the watchdog module is loaded correctly.

# lsmod | grep dog
# ls -l /dev/watchdog
# systool -vc watchdog

Testing the watchdog can be done with a simple action. Make sure to stop your SAP HANA first, because the watchdog will force an unclean reset or shutdown of your system.

Important

Triggering the watchdog without continuously updating the watchdog resets/switches off the system. This is the intended mechanism. The following commands will force your system to be reset/switched off.

The following action will activate the watchdog without feeding it and should cause a reset/switch off:

# sync; cat /dev/watchdog & while date; do sleep 10; done

After your test was successful, you must implement the watchdog on all cluster members.

9.1.2 Setting up the initial cluster using crm cluster init #

 

For more detailed information about setting up a cluster, refer to the sections Setting Up the First Node and Adding the Second Node of the Installation and Setup Quick Start for SUSE Linux Enterprise High Availability 16 0 at https://documentation.suse.com/sle-ha/16.0/html/HA-installing-two-node-cluster/.

This setup uses knet for corosync communication. Refer to the https://documentation.suse.com/sle-ha/16.0/ on detailed explanations of the term knet.

Create an initial setup, using the crm cluster init command, and follow the dialogs. Do this only on the first cluster node. Answer "no" to "Do you wish to configure a virtual IP address" and to "Do you want to configure QDevice".

suse01:~ # crm cluster init -s /dev/disk/by-id/SBDA

This command configures the basic cluster framework including:

• SSH keys

• csync2 to transfer configuration files

• SBD (one in this guide, better three for production)

• corosync (at least one ring, two for production)

Important

As requested by crm cluster init, change the password of the user hacluster.

9.1.3 Checking and adapting the corosync and SBD configuration #

 

9.1.3.1 Checking the corosync configuration #

 

Check the following blocks in the file /etc/corosync/corosync.conf. The important parts are knet and the correct ring/IP configuration.

See also the example at the end of this document and refer to the manual pages corosync.conf(5), votequorum(5) and corosync_overview(8) for details on parameters and features.

totem {
    ...
    transport: knet
    ...
}

nodelist {
    node {
            ring0_addr: 192.168.1.11
            name: suse01
            nodeid: 1
    }

    node {
            ring0_addr: 192.168.1.12
            name: suse02
            nodeid: 2
    }
}
    ...

9.1.3.2 Adapting the SBD configuration #

 

You can skip this section if you do not have any SBD devices, but be sure to implement another supported fencing mechanism.

See the manual pages sbd(8) and stonith_sbd(7) for details.

Table 3: SBD Options in File /etc/sysconfig/SBD #

 

Parameter	Description
SBD_WATCHDOG_DEV	Define the watchdog device. It is mandatory to use a watchdog. SBD does not work reliable without watchdog. Refer to the SLES manual and SUSE TID 7016880 for setting up a watchdog.
SBD_WATCHDOG_TIMEOUT	This parameter is used with diskless SBD. It defines the timeout, in seconds, the watchdog will wait before panicking the node if noone tickles it. The value of fencing-watchdog-timeout in CIB should be more than twice the value of SBD_WATCHDOG_TIMEOUT.
SBD_STARTMODE	Start mode. If set to clean, sbd will only start if the node was previously shut down cleanly or if the slot is empty.
SBD_DELAY_START	Whether to delay after starting sbd on boot. Should be no for SBD_STSRTMODE clean.
SBD_PACEMAKER	Check Pacemaker quorum and node health.

In the following, replace /dev/disk/by-id/SBDA by your real sbd device names. As an example, the SBD_WATCHDOG_TIMEOUT is set to 15s.

# egrep -v "(^#|^$)" /etc/sysconfig/sbd
SBD_PACEMAKER="yes"
SBD_STARTMODE="clean"
SBD_DELAY_START="no"
SBD_WATCHDOG_DEV="/dev/watchdog"
SBD_WATCHDOG_TIMEOUT="15"
SBD_TIMEOUT_ACTION="flush,reboot"
SBD_MOVE_TO_ROOT_CGROUP="auto"
SBD_OPTS=""
SBD_DEVICE="/dev/disk/by-id/SBDA"

Important

Also read the SUSE product documentation about calculation of timeouts for more details: https://documentation.suse.com/sle-ha/16.0/html/HA-sbd-configuring-diskbased/index.html#ha-sbd-setting-up-watchdog

9.1.3.3 Verifying the SBD device #

 

You can skip this section if you do not have any SBD devices, but make sure to implement a supported fencing mechanism.

It is a good practice to check if the SBD device can be accessed from both nodes and does contain valid records. Check this for all devices configured in /etc/sysconfig/sbd. You can do so, for example, by calling cs_show_sbd_devices.

suse01:~ # cs_show_sbd_devices
==Dumping header on disk /dev/disk/by-id/SBDA
Header version     : 2.1
UUID               : c2d5b54e-aa7e-421d-8f08-ed929499cd84
Number of slots    : 255
Sector size        : 512
Timeout (watchdog) : 15
Timeout (allocate) : 2
Timeout (loop)     : 1
Timeout (msgwait)  : 30
==Header on disk /dev/disk/by-id/SBDA is dumped
0       suse01 clear

Important

The timeout values in our example are only start values. It is a requirement that they are tuned to your environment. Refer to the TIDs 7011346 and 7023689 for more information.

For more information on SBD configuration parameters, consult the respective sections of the SUSE Linux Enterprise High Availability Administration Guide and the TIDs 7016880 and 7008216.

9.1.4 Configuring the cluster on the second node #

 

The second node of the two nodes cluster can be integrated by starting the command crm cluster join. This command asks for the IP address or name of the first cluster node. With this command, all needed configuration files are copied over. As a result, the cluster is started on both nodes.

# crm cluster join -c <host1>

Press RETURN to acknowledge the IP address.

9.1.5 Checking the cluster for the first time #

 

Now it is time to check and optionally start the cluster for the first time on both nodes.

suse01:~ # systemctl status pacemaker
suse01:~ # systemctl status sbd
suse02:~ # systemctl status pacemaker
suse02:~ # systemctl status sbd
suse01:~ # crm cluster stop --all
suse01:~ # crm cluster start --all

Check the cluster status with crm_mon. We use the option "-r" to also see resources, which are configured but stopped.

# crm_mon -r1

The command will show the "empty" cluster and will print something similar to the screen output below. The most interesting pieces of information for now are that there are two nodes in the status "online", the message "partition with quorum", and a running SBD resource.

Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: suse01 (version 3.0.2+20260429.40d19b75-160099.1.8-3.0.2+20260429.40d19b75) - partition with quorum
  * Last updated: Fri Jul 24 15:05:26 2026 on suse01
  * Last change:  Tue Jul 14 13:36:53 2026 by hacluster via crmd on suse01
  * 2 nodes configured
  * 1 resource instance configured

Node List:
  * Online: [ suse01 suse02  ]

Full List of Resources:
  * fencing-sbd	(stonith:fence_sbd):	 Started suse01

suse01:~ # vi crm-fileXX
suse01:~ # crm configure load update crm-fileXX

suse01:~ # vi crm-bs.txt
# enter the following to crm-bs.txt
property cib-bootstrap-options: \
    fencing-enabled="true" \
    fencing-action="reboot" \
    fencing-timeout="150" \
    priority-fencing-delay="30"
rsc_defaults build-resource-defaults: \
    resource-stickiness="1000" \
    migration-threshold="5000"
op_defaults op-options: \
    timeout="600" \
    record-pending=true

suse01:~ # crm configure load update crm-bs.txt

suse01:~ # vi crm-sbd.txt
# enter the following to crm-sbd.txt
primitive fencing-sbd stonith:fence_sbd \
        params pcmk_delay_max=15

suse01:~ # crm configure load update crm-sbd.txt

suse01:~ # vi crm-saphanatop.txt
# enter the following to crm-saphanatop.txt
primitive rsc_SAPHanaTop_HA1_HDB10 ocf:suse:SAPHanaTopology \
    op start interval=0 timeout=600 \
    op stop interval=0 timeout=300 \
    op monitor interval=50 timeout=600 \
    params SID=HA1 InstanceNumber=10
clone cln_SAPHanaTop_HA1_HDB10 rsc_SAPHanaTop_HA1_HDB10 \
    meta clone-node-max=1 interleave=true

suse01:~ # crm configure load update crm-saphanatop.txt

suse01:~ # vi crm-saphanafil.txt
# enter the following to crm-saphanafil.txt
primitive rsc_SAPHanaFil_HA1_HDB10 ocf:suse:SAPHanaFilesystem \
    op start interval=0 timeout=10 \
    op stop interval=0 timeout=20 \
    op monitor interval=120 timeout=120 \
    params SID=HA1 InstanceNumber=10 ON_FAIL_ACTION=fence
clone cln_SAPHanaFil_HA1_HDB10 rsc_SAPHanaFil_HA1_HDB10 \
    meta clone-node-max=1 interleave=true

suse01:~ # crm configure load update crm-saphanafil.txt

suse01:~ # vi crm-saphanacon.txt
# enter the following to crm-saphanacon.txt
primitive rsc_SAPHanaCon_HA1_HDB10 ocf:suse:SAPHanaController \
    op start interval=0 timeout=3600 \
    op stop interval=0 timeout=600 \
    op promote interval=0 timeout=900 \
    op demote interval=0 timeout=320 \
    op monitor interval=60 role=Promoted timeout=700 \
    op monitor interval=61 role=Unpromoted timeout=700 \
    params SID=HA1 InstanceNumber=10 PREFER_SITE_TAKEOVER=true \
        DUPLICATE_PRIMARY_TIMEOUT=7200 AUTOMATED_REGISTER=false \
        ON_FAIL_ACTION=fence \
    meta priority=100
clone mst_SAPHanaCon_HA1_HDB10 rsc_SAPHanaCon_HA1_HDB10 \
    meta clone-node-max=1 promotable=true interleave=true maintenance=true

suse01:~ # crm configure load update crm-saphanacon.txt

# vi crm-vip.txt
# enter the following to crm-vip.txt
primitive rsc_ip_HA1_HDB10 ocf:heartbeat:IPaddr2 \
    op monitor interval=10 timeout=20 \
    params ip=192.168.1.20

suse01:~ # crm configure load update crm-vip.txt

suse01:~ # vi crm-cs.txt
# enter the following to crm-cs.txt
location loc_saphana_ip_HA1_HDB10 rsc_ip_HA1_HDB10 \
    rule 2000: master-rsc_SAPHanaCon_HA1_HDB10 eq 150
order ord_saphana_HA1_HDB10 Optional: cln_SAPHanaTop_HA1_HDB10 \
    mst_SAPHanaCon_HA1_HDB10

suse01:~ # crm configure load update crm-cs.txt

suse01:~ # crm resource refresh mst_SAPHanaCon_HA1_HDB10
suse01:~ # cs_wait_for_idle -s 5
suse01:~ # crm resource maintenance mst_SAPHanaCon_HA1_HDB10 off

suse01:~ # vi crm-re.txt
# enter the following to crm-re.txt
primitive rsc_ip_HA1_HDB10_readenabled ocf:heartbeat:IPaddr2 \
    op monitor interval=10 timeout=20 \
    params ip=192.168.1.21
location loc_saphana_ip_readenabled_HA1_HDB10 rsc_ip_HA1_HDB10_readenabled \
    rule 2000: master-rsc_SAPHanaCon_HA1_HDB10 eq 100

suse01:~ # crm configure load update crm-re.txt

The chapter "Tests" refers to our two-node lab cluster suse12. The two nodes are named suse01 and suse02. SAP HANA has SID HA1 and instance number 10.

The lists of tests will be further enhanced in one of the next updates of this document.

As with any cluster testing is crucial. Make sure that all test cases derived from customer expectations are implemented and passed fully. Otherwise the project is likely to fail in production.

The test prerequisite, if not described differently, is always that both nodes are booted, normal members of the cluster and the HANA RDBMS is running. The system replication is in sync (SOK).

In the following test descriptions, we assume resource agents and HA/DR hooks are configured for the progressive variant. See manual page SAPHanaSR-angi-scenarios(7) for details. The SAPHanaController uses PREFER_SITE_TAKEOVER=true, AUTOMATED_REGISTER=true and ON_FAIL_ACTION=fence. The susChkSrv uses action_on_lost=fence. The SBD uses SBD_STARTMODE=clean and SBD_DELAY_START=0. The pacemaker service is disabled.

Note

The following tests are designed to be run in sequence and depend on the exit state of the preceding tests.

The chapter "Administration" refers to our two-node lab cluster suse12. The two nodes are named suse01 and suse02. SAP HANA has the SID HA1 and instance number 10.

11.1 Dos and don’ts #

 

In your project, you should:

• Define fencing before adding other resources to the cluster.

• Tune the timeouts of operations of SAPHanaController, SAPHanaTopology and SAPHanaFilesystem.

• Use SUSE Trento to check your cluster.

• Do intensive testing.

• Always check HANA and cluster status before doing anything and after having done something, see manual page SAPHanaSR-showStatus(8).

• Always wait for pending cluster actions to finish before doing something, see manual page cs_wait_for_idle(8).

• Set up a test cluster for testing configuration changes and administrative procedure before applying them on the production cluster.

In your project, avoid:

• Rapidly changing/changing back a cluster configuration, such as setting nodes to standby and online again or stopping/starting the multi-state resource.

• Creating a cluster without proper time synchronization or unstable name resolutions for hosts, users and groups.

• Using site names other than the ones already known by the cluster when manually re-registering a site.

• Adding location rules for the clone, multi-state or IP resource. Only location rules mentioned in this setup guide are allowed. For public cloud refer to the cloud specific documentation.

• Using SAP tools for attempting start/stop/takeover actions on a database while the cluster is in charge of managing that database. Same for unregistering/disabling system replication.

Important

As "migrating" or "moving" resources in crm-shell, HAWK or other tools would add client-prefer location rules, support is limited to maintenance procedures described in this document. See Section 10, “Testing the cluster” and Section 11.3, “Maintenance” for proven procedures.

11.2 Monitoring and tools #

 

You can use the High Availability Web Console (HAWK), SAP HANA Cockpit, SAP HANA Studio and different command line tools for cluster status requests.

11.2.1 HAWK – cluster status and more #

 

You can use a Web browser to check the cluster status.

Figure 3: Cluster Status in HAWK #

 

If you set up the cluster using crm cluster init and you have installed all packages as described above, your system will provide a very useful Web interface. You can use this graphical Web interface to get an overview of the complete cluster status, perform administrative tasks or configure resources and cluster bootstrap parameters. Read the product manuals for a complete documentation of this user interface. For the SAP HANA system replication scale-up performance optimized scenario the use of HAWK should follow the guidance given in this guide.

11.2.2 SAP HANA Cockpit #

 

Database-specific administration and checks can be done with SAP HANA Cockpit. Before trying start/stop/takeover for the database, make sure the cluster is not in charge of managing the respective resource. See also Section 11.3, “Maintenance”.

Figure 4: SAP HANA Cockpit – database directory #

 

11.2.3 Cluster command line tools #

 

A simple overview can be obtained by calling crm_mon. Using option -r shows also stopped but already configured resources. Option -1 tells crm_mon to output the status once instead of periodically.

besler3:~ # crm_mon -1r
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: besler4 (version 3.0.2+20260601.c75e25851c-160099.1.5-3.0.2+20260601.c75e25851c) - partition with quorum
  * Last updated: Mon Aug  3 16:49:12 2026 on besler3
  * Last change:  Mon Aug  3 16:48:56 2026 by root via crm_attribute on besler3
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ besler3 besler4 ]

Full List of Resources:
  * fencing-sbd (stonith:fence_sbd):     Started besler3
  * Clone Set: cln_SAPHanaTop_HA1_HDB00 [rsc_SAPHanaTop_HA1_HDB00]:
    * Started: [ besler3 besler4 ]
  * Clone Set: cln_SAPHanaFil_HA1_HDB00 [rsc_SAPHanaFil_HA1_HDB00]:
    * Started: [ besler3 besler4 ]
  * Clone Set: mst_SAPHanaCon_HA1_HDB00 [rsc_SAPHanaCon_HA1_HDB00] (promotable):
    * Promoted: [ besler3 ]
    * Unpromoted: [ besler4 ]
  * rsc_ip_HA1_HDB00    (ocf:heartbeat:IPaddr2):         Started besler3

See the manual page crm_mon(8) for details.

11.2.4 SAPHanaSR command line tools #

 

To show SAPHanaController and SAPHanaTopology resource agent internal values, you can call the program SAPHanaSR-showAttr. The internal values, the attribute location and their parameter names may change. The command SAPHanaSR-showAttr will always fetch the values from the correct attribute location. Do not use cluster commands like crm_attribute to fetch the values directly from the cluster. SAPHanaSR-showAttr should not be used for automated system monitoring.

besler3:~ # SAPHanaSR-showAttr
Global cib-update dcid prim sec sid topology
---------------------------------------------
global 0.165.0    2    S1   S2  HA1 ScaleUp

Resource                 maintenance promotable
------------------------------------------------
mst_SAPHanaCon_HA1_HDB00 false       true
cln_SAPHanaTop_HA1_HDB00

Site lpt        lss mns     opMode    srHook srMode  srPoll srr
----------------------------------------------------------------
S2   30         4   besler4 logreplay SOK    syncmem SOK    S
S1   1785768660 4   besler3 logreplay PRIM   syncmem PRIM   P

Host    clone_state roles                        score site srah version     vhost
-------------------------------------------------------------------------------------
besler3 PROMOTED    master1:master:worker:master 150   S1   -    2.00.089.00 besler3
besler4 DEMOTED     master1:master:worker:master 100   S2   -    2.00.089.00 besler4

SAPHanaSR-showAttr also supports other output formats such as script. The script format is intended to allow running filters. Based on SAPHanaSR-showAttr output format script you can define effective queries:

# SAPHanaSR-showAttr --format=script --experimental_attributes host:vhost
0 Host/suse01/vhost="suse01"
0 Host/suse02/vhost="suse02"

See the manual page SAPHanaSR-showAttr(8) and SAPHanaSR-showStatus(8) for details.

SAPHanaSR-replay-archive can help to analyze the SAPHanaSR attribute values from crm_report archives. This allows post mortem analysis.

In our example, the administrator killed the primary SAP HANA instance using the command HDB kill-9. This happened around 9:10 pm.

suse01:~ # crm_report -f 17:00
suse01: INFO: The report is saved in ./crm_report-Mon-03-Aug-2026.tar.gz
suse01: INFO: Report timespan: 2026-08-03 17:00:00 - 2026-08-03 17:03:29
suse01: INFO: Including nodes: suse01 suse02
suse01: INFO: Thank you for taking time to create this report

suse01:~ # SAPHanaSR-replay-archive --format=script --experimental_attributes host:score site:lss --dir=crm_report-Mon-03-Aug-2026 --sid=HA1

2026-08-03T15:01:20 Site/S2/srHook="SOK"
2026-08-03T15:01:20 Site/S2/lss="4"
2026-08-03T15:01:20 Host/suse02/score="100"
2026-08-03T15:02:04 Site/S2/srHook="SFAIL"
2026-08-03T15:02:04 Site/S2/lss="4"
2026-08-03T15:02:04 Host/suse02/score="100"
2026-08-03T15:02:11 Site/S2/srHook="SFAIL"
2026-08-03T15:02:11 Site/S2/lss="4"
2026-08-03T15:02:11 Host/suse02/score="-INFINITY"
...
2026-08-03T15:02:14 Site/S2/srHook="SFAIL"
2026-08-03T15:02:14 Site/S2/lss="1"
2026-08-03T15:02:14 Host/suse02/score="-1"
2026-08-03T15:03:09 Site/S2/srHook="SOK"
2026-08-03T15:03:09 Site/S2/lss="4"
2026-08-03T15:03:09 Host/suse02/score="0"
2026-08-03T15:03:11 Site/S2/srHook="SOK"
2026-08-03T15:03:11 Site/S2/lss="4"
2026-08-03T15:03:11 Host/suse02/score="100"

In the above example the attributes indicate that at the beginning suse02 was running secondary (lss "4", score "100").

At 15:02:04 (UTC) suddenly the system replication is maked as srHook="SFAIL".

At 15:02:11 (UTC) The secondary score is degraded: score="-INFINITY".

At 15:02:14 (UTC) The secondary is also marked as in error (lss="1"). The cluster initiated a restart for the failed secondary.

At 15:03:09 (UTC) The seconary is marked as back (lss="4") and the system replication is up again (srHook="SOK").

At 15:03:10 (UTC) The seconary score is set back to "100".

11.2.5 SAP HANA LandscapeHostConfiguration #

 

To check the status of an HANA database and to find out if the cluster should react, you can use the script landscapeHostConfiguration.py to be called as Linux user <sid>adm.

suse01:~> cdpy; python3 landscapeHostConfiguration.py
| Host    | Host   | ... NameServer   | NameServer  | IndexServer | IndexServer |
|         | Active | ... Config Role  | Actual Role | Config Role | Actual Role |
| ------  | ------ | ... ------------ | ----------- | ----------- | ----------- |
| suse01 | yes    | ... master 1     | master      | worker      | master      |

overall host status: ok

Following the SAP HA guideline, the SAPHanaController resource agent interprets the return codes in the following way:

Table 5: Interpretation of Return Codes #

 

Return Code	Description
4	SAP HANA database is up and OK. The cluster interprets this as a correctly running database.
3	SAP HANA database is up and in status info. The cluster interprets this as a correctly running database.
2	SAP HANA database is up and in status warning. The cluster interprets this as a correctly running database.
1	SAP HANA database is down. If the database should be up and is not down by intention, this could trigger a takeover.
0	Internal Script Error - Database status unknown. Cluster action depends on RA parameter ON_FAIL_ACTION.

The chapter "Examples" refers to our two-node lab cluster besler34. The two nodes are named besler3 and besler4. SAP HANA has the SID HA1 and instance number 00.

INFO: Loading "default" profile from /etc/crm/profiles.yml
INFO: Loading "knet-default" profile from /etc/crm/profiles.yml
INFO: A public key is added to authorized_keys for user root: 2048 SHA256:BpU1NYKSI1pLWQVkZCotf1jdiThQlMrCd77Pz2XTGf0 root@glishorn01 (RSA)
INFO: The user 'hacluster' will have the login shell configuration changed to /bin/bash
Continue (y/n)? y
INFO: A new ssh keypair is generated for user hacluster.
INFO: A public key is added to authorized_keys for user hacluster: 3072 SHA256:sOI39ogMRs3WOJqFzFO12mTy+gAf9jS7Bu7Y334W5g4 Cluster internal on besler3 (RSA)
INFO: Added firewalld service high-availability on besler3
Address for ring0 [192.168.2.62]
INFO: Configuring disk-based SBD
INFO: Initializing SBD device /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0
INFO: Update SBD_DEVICE in /etc/sysconfig/sbd: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0
INFO: Update SBD_WATCHDOG_DEV in /etc/sysconfig/sbd: /dev/watchdog0
INFO: Enable sbd.service on node besler3
INFO: Configure Qdevice/Qnetd:
  QDevice participates in quorum decisions. With the assistance of
  a third-party arbitrator Qnetd, it provides votes so that a cluster
  is able to sustain more node failures than standard quorum rules
  allow. It is recommended for clusters with an even number of nodes
  and highly recommended for 2 node clusters.

Do you want to configure QDevice (y/n)? n
INFO: Hawk cluster interface is now running. To see cluster status, open:
INFO:   https://besler:37630/
INFO: Log in with username 'hacluster', password 'linux'
WARNING: You should change the hacluster password to something more secure!
INFO: Starting and enable pacemaker.service on besler3
INFO: BEGIN Waiting for cluster
............
INFO: END Waiting for cluster
INFO: Loading initial cluster configuration
INFO: Configuring fence agent stonith:fence_sbd
INFO: Update SBD_DELAY_START in /etc/sysconfig/sbd: 41
INFO: Adjusting stonith-timeout to 71
WARNING: "stonith-timeout" in crm_config is set to 71, it was 60s
INFO: Setting stonith-enabled to true
WARNING: "stonith-enabled" in crm_config is set to true, it was false
INFO: Createing runtime drop-in file /run/systemd/system/sbd.service.d/sbd_delay_start_disabled.conf to unset SBD_DELAY_START
WARNING: "stonith-timeout" is deprecated, please consider using "fencing-timeout"
WARNING: "stonith-enabled" is deprecated, please consider using "fencing-enabled"
INFO: Configure Administration IP Address:
  Optionally configure an administration virtual IP
  address. The purpose of this IP address is to
  provide a single IP that can be used to interact
  with the cluster, rather than using the IP address
  of any specific cluster node.

Do you wish to configure a virtual IP address (y/n)? n
INFO: Done (log saved to /var/log/crmsh/crmsh.log on besler3)

besler4:~ # crm cluster join -c besler3
INFO: The user 'hacluster' will have the login shell configuration changed to /bin/bash
Continue (y/n)? y
INFO: A new ssh keypair is generated for user hacluster.
INFO: A public key is added to authorized_keys for user hacluster: 3072 SHA256:XU8AHmSS8Zr1UT5FSBuoOoDP7ljFFHdyjHZY7w55kXs Cluster internal on besler4 (RSA)
INFO: Added firewalld service high-availability on besler4
INFO: Merging known_hosts
INFO: BEGIN Probing for new partitions
INFO: END Probing for new partitions
INFO: BEGIN Retrieving all configuration files from besler3
INFO: END Retrieving all configuration files from besler3
Address for ring0 [192.168.2.64]

INFO: Sync file /etc/corosync/corosync.conf to besler3
INFO: Got SBD configuration
INFO: Enable sbd.service on node besler4
INFO: Hawk cluster interface is now running. To see cluster status, open:
INFO:   https://192.168.2.64:7630/
INFO: Log in with username 'hacluster', password 'linux'
WARNING: You should change the hacluster password to something more secure!
INFO: Starting and enable pacemaker.service on besler4
INFO: BEGIN Waiting for cluster
..
INFO: END Waiting for cluster
INFO: Setting pcmk_delay_max parameter to 30s in resource fencing-sbd
INFO: Update SBD_DELAY_START in /etc/sysconfig/sbd: 71
INFO: Sync file /etc/sysconfig/sbd to besler3
WARNING: "stonith-timeout" is deprecated, please consider using "fencing-timeout"
WARNING: "stonith-enabled" is deprecated, please consider using "fencing-enabled"
WARNING: "priority" in rsc_defaults is set to 1, it was 0
WARNING: "priority-fencing-delay" in crm_config is set to 60, it was 0
INFO: Sync file /etc/corosync/corosync.conf to besler3
INFO: BEGIN Reloading cluster configuration
INFO: END Reloading cluster configuration
INFO: Done (log saved to /var/log/crmsh/crmsh.log on besler4)

# Generated by crmsh
# For more details please see corosync.conf.5 man page
totem {
        version: 2
        cluster_name: besler34
        transport: knet
        token: 5000
        join: 60
        max_messages: 20
        token_retransmits_before_loss_const: 10
        crypto_hash: sha256
        crypto_cipher: aes256
}

quorum {
        provider: corosync_votequorum
        two_node: 1
}

logging {
        to_logfile: yes
        logfile: /var/log/cluster/corosync.log
        to_syslog: yes
        timestamp: hires
}

nodelist {
        node {
                ring0_addr: 192.168.2.62
                name: besler3
                nodeid: 1
        }

        node {
                ring0_addr: 192.168.2.64
                name: besler4
                nodeid: 2
        }

}

node 1: besler3 \
        attributes hana_ha1_vhost=besler3 hana_ha1_site=S1
node 2: besler4 \
        attributes hana_ha1_vhost=besler4 hana_ha1_site=S2
#
primitive fencing-sbd stonith:fence_sbd \
        params pcmk_delay_max=15
#
primitive rsc_SAPHanaCon_HA1_HDB00 ocf:suse:SAPHanaController \
        op start interval=0 timeout=3600 \
        op stop interval=0 timeout=600 \
        op promote interval=0 timeout=900 \
        op demote interval=0 timeout=320 \
        op monitor interval=60 role=Promoted timeout=700 \
        op monitor interval=61 role=Unpromoted timeout=700 \
        params SID=HA1 InstanceNumber=00 PREFER_SITE_TAKEOVER=true \
        DUPLICATE_PRIMARY_TIMEOUT=7200 AUTOMATED_REGISTER=false \
        ON_FAIL_ACTION=fence \
        meta priority=100
#
primitive rsc_SAPHanaFil_HA1_HDB00 ocf:suse:SAPHanaFilesystem \
        op start interval=0 timeout=10 \
        op stop interval=0 timeout=20 \
        op monitor interval=120 timeout=120 \
        params SID=HA1 InstanceNumber=00 ON_FAIL_ACTION=fence
#
primitive rsc_SAPHanaTop_HA1_HDB00 ocf:suse:SAPHanaTopology \
        op start interval=0 timeout=600 \
        op stop interval=0 timeout=300 \
        op monitor interval=50 timeout=600 \
        params SID=HA1 InstanceNumber=00
#
primitive rsc_ip_HA1_HDB00 IPaddr2 \
        op monitor interval=10 timeout=20 \
        params ip=192.168.2.66
#
clone cln_SAPHanaFil_HA1_HDB00 rsc_SAPHanaFil_HA1_HDB00 \
        meta clone-node-max=1 interleave=true
#
clone cln_SAPHanaTop_HA1_HDB00 rsc_SAPHanaTop_HA1_HDB00 \
        meta clone-node-max=1 interleave=true
#
clone mst_SAPHanaCon_HA1_HDB00 rsc_SAPHanaCon_HA1_HDB00 \
        meta clone-node-max=1 promotable=true interleave=true
#
location loc_ip_with_HA1_HDB00 rsc_ip_HA1_HDB00 \
        rule 2000: master-rsc_SAPHanaCon_HA1_HDB00 eq 150
#
order ord_saphana_HA1_HDB00 Optional: cln_SAPHanaTop_HA1_HDB00 \
        mst_SAPHanaCon_HA1_HDB00
#
property SAPHanaSR: \
        hana_ha1_site_srHook_S2=SOK \
        hana_ha1_glob_topology=ScaleUp \
        hana_ha1_site_lss_S2=4 \
        hana_ha1_site_lss_S1=4 \
        hana_ha1_site_srr_S2=S \
        hana_ha1_site_srr_S1=P \
        hana_ha1_site_srMode_S2=syncmem \
        hana_ha1_site_srMode_S1=syncmem \
        hana_ha1_site_mns_S1=besler3 \
        hana_ha1_site_mns_S2=besler4 \
        hana_ha1_site_lpt_S2=30 \
        hana_ha1_glob_sec=S2 \
        hana_ha1_site_srHook_S1=PRIM \
        hana_ha1_site_lpt_S1=1785762330 \
        hana_ha1_site_opMode_S2=logreplay \
        hana_ha1_site_opMode_S1=logreplay \
        hana_ha1_glob_prim=S1 \
        hana_ha1_site_srPoll_S1=PRIM \
        hana_ha1_site_srPoll_S2=SOK
property cib-bootstrap-options: \
        dc-version="3.0.2+20260601.c75e25851c-160099.1.5-3.0.2+20260601.c75e25851c" \
        cluster-infrastructure=corosync \
        have-watchdog=true \
        cluster-name=hacluster \
        priority-fencing-delay=30 \
        fencing-enabled=true \
        fencing-action=reboot \
        fencing-timeout=150 \
        last-lrm-refresh=1785761575
rsc_defaults build-resource-defaults: \
        resource-stickiness=1000 \
        migration-threshold=5000 \
        priority=1
op_defaults op-options: \
        timeout=600 \
        record-pending=true

For more detailed information, have a look at the documents listed below.

Copyright © 2006-2026 SUSE LLC and contributors. All rights reserved.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled "GNU Free Documentation License".

SUSE, the SUSE logo and YaST are registered trademarks of SUSE LLC in the United States and other countries. For SUSE trademarks, see https://www.suse.com/company/legal/.

Linux is a registered trademark of Linus Torvalds. All other names or trademarks mentioned in this document may be trademarks or registered trademarks of their respective owners.

Documents published as part of the SUSE Best Practices series have been contributed voluntarily by SUSE employees and third parties. They are meant to serve as examples of how particular actions can be performed. They have been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. SUSE cannot verify that actions described in these documents do what is claimed or whether actions described have unintended consequences. SUSE LLC, its affiliates, the authors, and the translators may not be held liable for possible errors or the consequences thereof.

Below we draw your attention to the license under which the articles are published.

Copyright © 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work’s title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’s license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties—​for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See https://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

Copyright (c) YEAR YOUR NAME.
   Permission is granted to copy, distribute and/or modify this document
   under the terms of the GNU Free Documentation License, Version 1.2
   or any later version published by the Free Software Foundation;
   with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
   A copy of the license is included in the section entitled “GNU
   Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “ with…​Texts.” line with this:

with the Invariant Sections being LIST THEIR TITLES, with the
   Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.

Copyright © 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work’s title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’s license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties—​for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See https://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

Copyright (c) YEAR YOUR NAME.
   Permission is granted to copy, distribute and/or modify this document
   under the terms of the GNU Free Documentation License, Version 1.2
   or any later version published by the Free Software Foundation;
   with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
   A copy of the license is included in the section entitled “GNU
   Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “ with…​Texts.” line with this:

with the Invariant Sections being LIST THEIR TITLES, with the
   Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.