SAP HANA System Replication Scale-Out - Performance Optimized Scenario #

This document describes how to set up automation of an SAP HANA scale-out system replication cluster installed on two sites based on SUSE Linux Enterprise Server for SAP Applications 15 SP3. This concept can also be used with SUSE Linux Enterprise Server for SAP Applications 15 SP1 or newer.

To give a better overview the installation and setup is subdivided into seven steps.

Figure 1: [Planning] [OsSetup] [SAPHanaInst] [SAPHanaHsr] [Integration] [Cluster] [Testing] #

 

After the setup process you will have a SUSE Linux Enterprise Server for SAP Applications cluster controlling two 'swarms' of SAP HANA scale-out in a system replication configuration. The architecture is named the 'performance optimized scenario'.

Figure 2: Cluster with SAP HANA SR - performance optimized #

 

With SAPHanaSR-ScaleOut, various HANA scale-out configurations are supported. Details on requirements and supported scenarios are given below. In this guide, we describe two scenarios where SAP HANA is configured for host auto-failover. That means there are standby nodes which are managed by SAP HANA. This setup is addressing the needs for scale-out systems as they are used for SAP Business Warehouse (BW) systems.

For SAP Enterprise Resource Planning (ERP) systems, another setup guide exists. That setup guide describes how to manage a specific HANA scale-out configuration without standby-nodes. The multi-target configuration is shown there as well. You will find background information on that setup in a series of blog articles on {#towardszerodowntime}, for example https://www.suse.com/c/sap-hana-scale-out-system-replication-for-large-erp-systems/ .

Note

For upgrading an existing SAP HANA scale-out system replication cluster from SAPHanaSR-ScaleOut version 0.160, consult manual page SAPHanaSR-manageAttr(8) and the blog article https://www.suse.com/c/sap-hana-scale-out-multi-target-upgrade/ .

Figure 3: Planning [OsSetup] [SAPHanaInst] [SAPHanaHsr] [Integration] [Cluster] [Testing] #

 

Planning the installation is essential for a successful SAP HANA cluster setup.

What you need before you start:

3.1 Minimum lab requirements and prerequisites #

 

This section defines some minimum requirements to install SAP HANA scale-out.

Note

The minimum lab requirements mentioned here are no SAP sizing information. These data are provided only to rebuild the described cluster in a lab for test purposes. Even for such tests the requirements can increase depending on your test scenario. For productive systems ask your hardware vendor or use the official SAP sizing tools and services.

Note

Refer to SAP HANA TDI documentation for allowed storage configuration and file systems.

Figure 4: Simplified NFS share structure of a 2+1:2+1 SAP HANA system replication #

 

Requirements with 3 SAP HANA instances per site (2+1 : 2+1) - with a majority maker:

• 6 VMs with each 32 GB RAM, 50 GB disk space

• 1 VM with 2 GB RAM, 50 GB disk space

• 1 shared disk for SBD with 10 MB disk space

• 2 NFS pools (one per site) with a capacity of each 96 GB

• 1 additional IP address for takeover

Figure 5: Simplified NFS share structure of a 3+2:3+2 SAP HANA system replication #

 

Requirements with 5 SAP HANA instances per site (3+2 : 3+2) - with a majority maker:

• 10 VMs with each 32 GB RAM, 50 GB disk space

• 1 VM with 2 GB RAM, 50 GB disk space

• 1 shared disk for SBD with 10 MB disk space

• 2 NFS pools (one per site) with a capacity of each 132 GB

• 1 additional IP address for takeover

The SBD based fencing needs up to 3 shared block devices.

Figure 6: Additionally to the NFS shares: All cluster nodes need to have access to the SBD block devices #

 

3.3 Scale-out scenario and resource agents #

 

To automate the failover, the High Availability Extension built into SUSE Linux Enterprise Server for SAP Applications is used. Two resource agents have been created to handle the scenario.

The first is the SAPHanaController resource agent (RA), which checks and manages the SAP HANA database instances. This RA is configured as a multi-state resource.

The master assumes responsibility for the active master name server of the SAP HANA database running in primary mode. All other instances are represented by the slave mode.

Figure 7: Cluster resource agents and multi-state status mapping #

 

The second resource agent is SAPHanaTopology. This RA has been created to make configuring the cluster as simple as possible. It runs on all nodes (except the majority maker) of a SUSE Linux Enterprise High Availability Extension 15 cluster. It gathers information about the statuses and configurations of the SAP HANA system replication. It is designed as a normal (stateless) clone resource.

With the current version of resource agents, SAP HANA system replication for scale-out is supported in the following scenarios or use cases:

Performance optimized, single container (A ⇒ B)

This scenario and setup is described in this document. In the performance optimized scenario an SAP HANA RDBMS on site "A" is synchronizing with an SAP HANA RDBMS on a second site "B". As the SAP HANA RDBMS on the second site is configured to preload the tables the takeover time is typically very short. See also the requirements section below for details.

Performance optimized, multi-tenancy also named MDC (%A ⇒ %B)

Multi-tenancy is available for all of the supported scenarios and use cases in this document. This scenario is supported since SAP HANA 1.0 SPS12, it is the default installation type for SAP HANA 2.0. The setup and configuration from a cluster point of view is the same for multi-tenancy and single containers. The one caveat is, that the tenants are managed all together by the Linux cluster. See also the requirements section below.

Multi-Tier Replication (A ⇒ B → C)

A Multi-Tier system replication has an additional target, which must be connected to the secondary (chain topology). This is a special case of the Multi-Target replication. Because of the mandatory chain topology, the RA feature AUTOMATED_REGISTER=true is not possible with pure Multi-Tier replication. See also the requirements section below.

Multi-Target Replication (A ⇐ B → C)

A Multi-Target system replication has an additional target, which is connected to either the secondary (chain topology) or to the primary (star topology). Multi-Target replication is possible since SAP HANA 2.0 SPS04. See also the requirements section below.

3.5 Important prerequisites #

 

Read the SAP Notes and papers first.

The SAPHanaSR-ScaleOut resource agent software package supports scale-out (multiple-box to multiple-box) system replication with the following configurations and parameters:

• The cluster must include a valid STONITH method. SBD disk-based or diskless is recommended STONITH method.

• Both sites are either in the same network segment (layer 2) to allow an easy takeover of an IP Address, or you need a technique like overlay IP addresses in virtual private clouds.

• Technical users and groups, such as <sid>adm are defined locally in the Linux system.

• Name resolution of the cluster nodes and the virtual IP address should be done locally on all cluster nodes to not depend on DNS services (as it can fail, too).

• Time synchronization between the cluster nodes using reliable time services like NTP.

• Both SAP HANA sites have the same SAP Identifier (SID) and instance number.

• The SAP HANA scale-out system must have only one active master name server per site. It should have up to three master name server candidates (SAP HANA nodes with a configured role 'master<N>').

• For SAP HANA databases without additional master name server candidate, the package SAPHanaSR-ScaleOut version 0.180 or newer is needed.

• The SAP HANA scale-out system must have only one failover group.

• There is maximum one additional SAP HANA system replication connected from outside the Linux cluster. Thus two sites are managed by the Linux cluster, one site outside is recognized. For SAP HANA multi-tier and multi-target system replication, the package SAPHanaSR-ScaleOut version 0.180 or newer is needed.

• Only one SAP HANA SID is installed. Thus the performance optimized setup is supported. But the cost optimized and MCOS scenarios are currently not supported.

• The saphostagent must be running. saphostagent is needed to translate between the system node names and SAP host names used during the installation of SAP HANA.

  • For SystemV style, the sapinit script needs to be active.

  • For systemd style, the services saphostagent and SAP<SID>_<INO> can stay enabled. The systemd enabled saphostagent and instance´s sapstartsrv is supported from SAPHanaSR-ScaleOut 0.181 onwards. Refer to the OS documentation for the systemd version. SAP HANA comes with native systemd integration as default starting with version 2.0 SPS07. Refer to SAP documentation for the SAP HANA version.

  • Combining systemd style hostagent with SystemV style instance is allowed. However, all nodes in one Linux cluster have to use the same style.

• All SAP HANA instances controlled by the cluster must not be activated via sapinit auto-start.

• The replication mode should be either 'sync' or 'syncmem'. But 'async' is not supported.

• SAP HANA 2.0 SPS05 rev.059 and later provides Python 3 as well as the HA/DR provider hook method srConnectionChanged() with needed parameters for SAPHanaSrMultiTarget.py.

• SAP HANA 2.0 SPS05 or later provides the HA/DR provider hook method srServiceStateChanged() with needed parameters for susChkSrv.py.

• SAP HANA 2.0 SPS06 or later provides the HA/DR provider hook method preTakeover() with multi-target aware parameters and separate return code for Linux HA clusters.

• No other HA/DR provider hook script should be configured for the above mentioned methods. Hook scripts for other methods, provided in SAPHanaSR-ScaleOut, can be used in parallel, if not documented contradictingly.

• The Linux cluster needs to be up and running to allow HA/DR provider events being written into CIB attributes. The current HANA SR status might differ from CIB srHook attribute after Linux cluster maintenance.

• The user <sid>adm needs execution permission as user root for the command SAPHanaSR-hookHelper.

• The Linux cluster can be either freshly installed as described in this guide, or it can be upgraded as described in respective documentation. Not allowed is mixing old and new cluster attributes or hook scripts within one Linux cluster.

Find more details in the REQUIREMENTS section of manual pages SAPHanaSR-ScaleOut(7), ocf_suse_SAPHanaController(7), SAPHanaSrMultiTarget.py(7), susTkOver.py(7) and SAPHanaSR-manageAttr(8).

Important

Automated registration of a failed primary after takeover is possible. But as a good starting configuration for projects, it is recommended to switch off the automated registration of a failed primary, therefore the AUTOMATED_REGISTER="false" is the default.

In this case, you need to register a failed primary after a takeover manually. Use SAP tools like SAP HANA Cockpit or hdbnsutil. Make sure to use always the exact site names as already known to the cluster.

• For optimal automation, AUTOMATED_REGISTER="true" is recommended.

• Automated start of SAP HANA instances during system boot must be switched off.

• You need at least SAPHanaSR-ScaleOut version 0.180, SUSE Linux Enterprise Server for SAP Applications 15 SP1 and SAP HANA 2.0 SPS 5 for all mentioned setups.

Important

You must implement a valid STONITH method. Without a valid STONITH method, the complete cluster is unsupported and will not work properly.

In this setup guide, NFS is used as storage for the SAP HANA database. This has been chosen for simplicity. However, any storage supported by SAP HANA and the SAP HANA storage API can be used. Refer to the SAP HANA TDI documentation for supported storage and follow the respective storage vendor’s configuration instructions.

This setup guide focuses on the performance optimized setup as it is the only supported scale-out scenario at the point of writing this guide.

If you need to implement a different scenario, it is strongly recommended to define a Proof-of-Concept (PoC) with SUSE. This PoC will focus on testing the existing solution in your scenario. The limitation of most of the above items is mostly due to testing limits.

Figure 8: [Planning] OsSetup [SAPHanaInst] [SAPHanaHsr] [Integration] [Cluster] [Testing] #

 

This section includes information you should consider during the installation of the operating system.

In this document, first SUSE Linux Enterprise Server for SAP Applications is installed and configured. Then the SAP HANA database including the system replication is set up. Finally the automation with the cluster is set up and configured.

# zypper remove SAPHanaSR

Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 3 packages are going to be REMOVED:
  patterns-sap-hana SAPHanaSR yast2-sap-ha

The following pattern is going to be REMOVED:
  sap-hana

3 packages to remove.
After the operation, 494.2 KiB will be freed.
Continue? [y/n/...? shows all options] (y): y
(1/3) Removing patterns-sap-hana-15.3-6.8.2.x86_64 ..............................[done]
(2/3) Removing yast2-sap-ha-1.0.0-2.5.12.noarch .................................[done]
(3/3) Removing SAPHanaSR-0.161.21-1.1.noarch ....................................[done]

# zypper in SAPHanaSR-ScaleOut SAPHanaSR-ScaleOut-doc

Refreshing service 'Advanced_Systems_Management_Module_15_x86_64'.
Refreshing service 'SUSE_Linux_Enterprise_Server_for_SAP_Applications_15_SP3_x86_64'.
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 2 NEW packages are going to be installed:
  SAPHanaSR-ScaleOut SAPHanaSR-ScaleOut-doc

2 new packages to install.
Overall download size: 539.1 KiB. Already cached: 0 B. After the operation, additional 763.1 KiB will be used.
Continue? [y/n/...? shows all options] (y): y
Retrieving package SAPHanaSR-ScaleOut-0.180.1-1.1.noarch                 (1/2),  48.7 KiB (211.8 KiB unpacked)
Retrieving: SAPHanaSR-ScaleOut-0.180.1-1.1.noarch.rpm ....................................[done]
Retrieving package SAPHanaSR-ScaleOut-doc-0.180.1-1.1.noarch             (2/2), 490.4 KiB (551.3 KiB unpacked)
Retrieving: SAPHanaSR-ScaleOut-doc-0.180.1-1.1.noarch.rpm ................................[done (48.0 KiB/s)]
Checking for file conflicts: .............................................................[done]
(1/2) Installing: SAPHanaSR-ScaleOut-0.180.1-1.1.noarch ..................................[done]
(2/2) Installing: SAPHanaSR-ScaleOut-doc-0.180.1-1.1.noarch ..............................[done]

# zypper in --type pattern ha_sles

# zypper patch

# zypper update

4.2 Configuring SUSE Linux Enterprise Server for SAP Applications to run SAP HANA #

 

4.2.1 Tuning or modifying the operating system #

 

Operating system tuning are described in SAP note 1275776 and 2684254. The SAP note 1275776 explains three ways to implementing the settings.

Example 4: Using saptune (preferred) #

 

# saptune solution apply HANA

4.2.2 Enabling SSH access via public key (optional) #

 

Public key authentication provides SSH users access to their servers without entering their passwords. SSH keys are also more secure than passwords, because the private key used to secure the connection is never shared. Private keys can also be encrypted. Their encrypted contents cannot easily be read. For the document at hand, a very simple but useful setup is used. This setup is based on only one ssh-key pair which enables SSH access to all cluster nodes.

Note

Follow your company security policy to set up access to the systems.

Example 5: SSH key creation and exchange #

 

As user root create an SSH key on one node.

# ssh-keygen -t rsa

The ssh-key generation asks for missing parameters.

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:ip/8kdTbYZNuuEUAdsaYOAErkwnkAPBR7d2SQIpIZCU root@<host1>
The key's randomart image is:
+---[RSA 2048]----+
|XEooo+ooo+o      |
|=+.= o=.o+.      |
|..B o. + o.      |
|   (°<           |
|  /  )           |
|   --            |
|   B. . o o =    |
|     o . . +     |
|      +.. .      |
+----[SHA256]-----+

After the ssh-keygen is set up, you will have two new files under /root/.ssh/ .

# ls /root/.ssh/
id_rsa  id_rsa.pub

To allow password-free access for the user root between nodes in the cluster copy id_rsa.pub to authorized_keys and set the required permissions.

# cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
# chmod 600 /root/.ssh/authorized_keys

Collect the public host keys from all other node. For the document at hand, the ssh-keyscan command is used.

# ssh-keyscan

The SSH host key is automatically collected and stored in the file /root/.ssh/known_host during the first SSH connection. To avoid to confirm the first login with "yes", which accepts the host key, collect and store them beforehand.

# ssh-keyscan -t ecdsa-sha2-nistp256 <host1>,<host1 ip> >>.ssh/known_hosts
# ssh-keyscan -t ecdsa-sha2-nistp256 <host2>,<host2 ip> >>.ssh/known_hosts
# ssh-keyscan -t ecdsa-sha2-nistp256 <host3>,<host3 ip> >>.ssh/known_hosts
...

After collecting all host keys push the entire directory /root/.ssh/ from the first node to all further cluster members.

# rsync -ay /root/.ssh/ <host2>:/root/.ssh/
# rsync -ay /root/.ssh/ <host3>:/root/.ssh/
# rsync -ay /root/.ssh/ <host4>:/root/.ssh/
...

4.2.3 Setting up disk layout for SAP HANA #

 

An SAP certified storage system with a validated storage API is generally recommended. This is a prerequisite of a stable and reliable scale-out installation.

• /hana/shared/<SID>

• /hana/data/<SID>

• /hana/log/<SID>

Create the mount directories on all SAP HANA nodes.

# mkdir -p /hana/shared/<SID>
# mkdir -p /hana/data/<SID>
# mkdir -p /hana/log/<SID>
# mkdir -p /usr/sap

The SAP HANA installation needs a special storage setup. The NFS setup used for this guide must be reboot-persistent. You can achieve this with entries in the /etc/fstab.

Note

NFS version 4 is required in the setup at hand.

Example 6: Create permanent mount entries for all NFS pools #

 

Create /etc/fstab entries for the three NFS pools.

<nfs1>  /hana/data/<SID>      nfs4  defaults  0 0
<nfs2>  /hana/shared/<SID>    nfs4  defaults  0 0
<nfs3>  /hana/log/<SID>       nfs4  defaults  0 0

In the sample environment those lines are as follows:

/exports/HA1_WDF1/shared  /hana/data/HA1      nfs4  defaults  0 0
/exports/HA1_WDF1/data  /hana/shared/HA1    nfs4  defaults  0 0
/exports/HA1_WDF1/log  /hana/log/HA1        nfs4  defaults  0 0

Mount all NFS shares.

# mount -a

Create other directories (optional).

# mkdir -p /sapsoftware

File systems

/hana/shared/<SID>

The mount directory is used for shared files between all hosts in an SAP HANA system. This directory needs to be accessible to each of the servers in the SAP HANA cluster.

/hana/log/<SID>

The default path to the log directory depends on the system ID of the SAP HANA host.

/hana/data/<SID>

The default path to the data directory depends on the system ID of the SAP HANA host.

/usr/sap

This is the path to the local SAP system instance directories. It is possible to join this location with the Linux installation.

/sapsoftware

(optional) Space for copying the SAP install software media. This NFS pool is mounted on both sites and contains the SAP HANA installation media and installation parameter files.

Set up host name resolution for all machines.

You can either use a DNS server or modify the /etc/hosts on all nodes.

With maintaining the /etc/hosts file, you minimize the impact of a failing DNS service. Replace the IP address and the host name in the following commands.

# vi /etc/hosts

Insert the following lines to /etc/hosts. Change the IP address and host name to match your environment.

192.168.201.151 suse01
192.168.201.152 suse02
...

Enable NTP service on all nodes.

Simply enable an ntp service on all node in the cluster to have proper time synchronization.

# yast2 ntp-client

Figure 9: [Planning] [OsSetup] SAPHanaInst [SAPHanaHsr] [Integration] [Cluster] [Testing] #

 

The infrastructure is set up. Now install the SAP HANA database on both sites. This chapter summarizes the test environment. In a cluster a machine is also called a node. Always use the official documentation from SAP to install SAP HANA and to set up the system replication.

This guide shows SAP HANA and saphostagent with native systemd integration. An example for legacy SystemV is outlined in the appendix Section 10.3, “Example for checking legacy SystemV integration”.

In the example at hand, to make it easier to follow the documentation, the machines (or nodes) are named suse01, …​ suseXX. The nodes with odd numbers (suse01, suse03, suse05, …​) will be part of site "A" (WDF1) and the nodes with even numbers (suse02, suse04, suse06, …​) will be part of site "B"(ROT1) .

The following users are automatically created during the SAP HANA installation:

# systemctl list-unit-files | grep sap
saphostagent.service enabled
sapinit.service generated
saprouter.service disabled
saptune.service enabled

# systemctl list-unit-files | grep SAP
SAPHA1_10.service enabled

# systemd-cgls -u SAP.slice
Unit SAP.slice (/SAP.slice):
├─saphostagent.service
│ ├─2630 /usr/sap/hostctrl/exe/saphostexec pf=/usr/sap/hostctrl/exe/host_profile -systemd
│ ├─2671 /usr/sap/hostctrl/exe/sapstartsrv pf=/usr/sap/hostctrl/exe/host_profile -D
│ └─3591 /usr/sap/hostctrl/exe/saposcol -l -w60 pf=/usr/sap/hostctrl/exe/host_profile
└─SAPHA1_10.service
  ├─ 1257 hdbcompileserver
  ├─ 1274 hdbpreprocessor
  ├─ 1353 hdbindexserver -port 31003
  ├─ 1356 hdbxsengine -port 31007
  ├─ 2077 hdbwebdispatcher
  ├─ 2300 hdbrsutil --start --port 31003 --volume 3 --volumesuffix mnt00001/hdb00003.00003 --identifier 1644426276
  ├─28462 /usr/sap/HA1/HDB10/exe/sapstartsrv pf=/usr/sap/HA1/SYS/profile/HA1_HDB10_suse01
  ├─31314 sapstart pf=/usr/sap/HA1/SYS/profile/HA1_HDB10_suse01
  ├─31372 /usr/sap/HA1/HDB10/suse01/trace/hdb.sapHA1_HDB10 -d -nw -f /usr/sap/HA1/HDB00/suse21/daemon.ini pf=/usr/sap/HA1/SYS/profile/HA1_HDB10_suse02
  ├─31479 hdbnameserver
  └─32201 hdbrsutil --start --port 31001 --volume 1 --volumesuffix mnt00001/hdb00001 --identifier 1644426203

~> HDBSettings.sh landscapeHostConfiguration.py

| Host   | Host   |... NameServer  | NameServer  | IndexServer | IndexServer
|        | Active |... Config Role | Actual Role | Config Role | Actual Role
| ------ | ------ |... ----------- | ----------- | ----------- | -----------
| suse01 | yes    |... master 1    | master      | worker      | master
| suse03 | yes    |... master 2    | slave       | worker      | slave
| suse05 | yes    |... master 3    | slave       | worker      | slave
...

overall host status: ok

~> sapcontrol -nr <instanceNumber> -function GetSystemInstanceList
25.07.2022 17:25:16
GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
suse01, 00, 50013, 50014, 0.3, HDB|HDB_WORKER, GREEN
suse03, 00, 50013, 50014, 0.3, HDB|HDB_WORKER, GREEN
suse05, 00, 50013, 50014, 0.3, HDB|HDB_WORKER, GREEN
...

Figure 10: [Planning] [OsSetup] [SAPHanaInst] SAPHanaHsr [Integration] [Cluster] [Testing] #

 

This section describes the setup of the system replication (HSR) after SAP HANA has been installed properly.

For more information read the Section Setting Up System Replication of the SAP HANA Administration Guide.

6.1 Backing up the primary database #

 

First back up the primary database as described in the SAP HANA Administration Guide, Section SAP HANA Database Backup and Recovery.

Below find examples to back up SAP HANA with SQL Commands:

Example 11: Simple backup for the system database and all tenants with one singe backup call #

 

As user <sid>adm enter the following command:

~> hdbsql -i <instanceNumber> -u SYSTEM -d SYSTEMDB \
   "BACKUP DATA FOR FULL SYSTEM USING FILE ('backup')"

You get the following command output (or similar):

0 rows affected (overall time 15.352069 sec; server time 15.347745 sec)

Example 12: Simple backup for a single container (non-MDC) database #

 

Enter the following command as user <sid>adm:

~> hdbsql -i <instanceNumber> -u <dbuser> \
   "BACKUP DATA USING FILE ('backup')"

Important

Without a valid backup, you cannot bring SAP HANA into a system replication configuration.

~> hdbnsutil -sr_enable --name=WDF1

nameserver is active, proceeding ...
successfully enabled system as system replication source site
done.

~> hdbnsutil -sr_stateConfiguration

checking for active or inactive nameserver ...
System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

mode: primary
site id: 1
site name: WDF1
done.

~> sapcontrol -nr <instanceNumber> -function StopSystem

cd /usr/sap/<SID>/SYS/global/security/rsecssfs
rsync -va {,<node1-siteB>:}$PWD/data/SSFS_<SID>.DAT
rsync -va {,<node1-siteB>:}$PWD/key/SSFS_<SID>.KEY

~> hdbnsutil -sr_register --name=<site2> \
     --remoteHost=<node1-siteA> --remoteInstance=<instanceNumber> \
     --replicationMode=sync --operationMode=logreplay

adding site ...
checking for inactive nameserver ...
nameserver suse02:30001 not responding.
collecting information ...
updating local ini files ...
done.

~> sapcontrol -nr <instanceNumber> -function StartSystem

~> hdbnsutil -sr_stateConfiguration

System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~
mode: sync
site id: 2
site name: ROT1
active primary site: 1

primary masters: suse01 suse03 suse05
done.

~> HDBSettings.sh systemReplicationStatus.py

| Database | Host   | .. Site Name | Secondary | .. Secondary | .. Replication
|          |        | ..           | Host      | .. Site Name | .. Status
| -------- | ------ | .. --------- | --------- | .. --------- | .. ------
| SYSTEMDB | suse01 | .. WDF1      | suse02    | .. ROT1      | .. ACTIVE
| HA1      | suse01 | .. WDF1      | suse02    | .. ROT1      | .. ACTIVE
| HA1      | suse01 | .. WDF1      | suse02    | .. ROT1      | .. ACTIVE
| HA1      | suse03 | .. WDF1      | suse04    | .. ROT1      | .. ACTIVE
| HA1      | suse05 | .. WDF1      | suse06    | .. ROT1      | .. ACTIVE
...

status system replication site "2": ACTIVE
overall system replication status: ACTIVE

Local System Replication State
~~~~~~~~~~

mode: PRIMARY
site id: 1
site name: WDF1

Figure 11: [Planning] [OsSetup] [SAPHanaInst] [SAPHanaHsr] Integration [Cluster] [Testing] #

 

This chapter describes what to change on the SAP HANA configuration for the BW style scale-out scenario.

This step is mandatory to inform the cluster immediately if the secondary gets out of sync. The hook is called by SAP HANA using the HA/DR provider interface in point-of-time when the secondary gets out of sync. This is typically the case when the first commit pending is released. The hook is called by SAP HANA again when the system replication is back. This HA/DR provider method is srConnectionChanged(), the related SUSE hook script is SAPHanaSrMultiTarget.py.

# su - <sid>adm
~> sapcontrol -nr <instanceNumber> -function StopSystem
~> sapcontrol -nr <instanceNumber> -function WaitforStopped 300 20
~> sapcontrol -nr <instanceNumber> -function GetSystemInstanceList

[ha_dr_provider_saphanasrmultitarget]
provider = SAPHanaSrMultiTarget
path = /usr/share/SAPHanaSR-ScaleOut/
execution_order = 1

[trace]
ha_dr_saphanasrmultitarget = info

[ha_dr_provider_sustkover]
provider = susTkOver
path = /usr/share/SAPHanaSR-ScaleOut/
execution_order = 2
sustkover_timeout = 30

[trace]
ha_dr_sustkover = info

[system_replication]
operation_mode = logreplay

# SAPHanaSR-ScaleOut needs for srHook
<sid>adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_*
<sid>adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_<sid>_gsh *
<sid>adm ALL=(ALL) NOPASSWD: /usr/sbin/SAPHanaSR-hookHelper --sid=<sid> *

# SAPHanaSR-ScaleOut needs for srHook
ha1adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_ha1_site_srHook_*
ha1adm ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_ha1_gsh *
ha1adm ALL=(ALL) NOPASSWD: /usr/sbin/SAPHanaSR-hookHelper --sid=ha1 *

# SAPHanaSR-ScaleOut needs for /usr/share/SAPHanaSR-ScaleOut/SAPHanaSrMultiTarget.py
Cmnd_Alias GSH_QUERY      = /usr/sbin/crm_attribute -n hana_<sid>_gsh -G
Cmnd_Alias GSH_UPDATE     = /usr/sbin/crm_attribute -n hana_<sid>_gsh -v 2.2 -l reboot
# be compatible with non-multi-target mode
Cmnd_Alias SOK_GLOB       = /usr/sbin/crm_attribute -n hana_<sid>_glob_srHook -v SOK -t crm_config -s SAPHanaSR
Cmnd_Alias SFAIL_GLOB     = /usr/sbin/crm_attribute -n hana_<sid>_glob_srHook -v SOK -t crm_config -s SAPHanaSR
# be compatible with multi-target mode
Cmnd_Alias SOK_GLOB_MTS   = /usr/sbin/crm_attribute -n hana_<sid>_glob_mts -v SOK -t crm_config -s SAPHanaSR
Cmnd_Alias SFAIL_GLOB_MTS = /usr/sbin/crm_attribute -n hana_<sid>_glob_mts -v SOK -t crm_config -s SAPHanaSR
Cmnd_Alias SOK_SITEA      = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_Site1 -v SOK   -t crm_config -s SAPHanaSR
Cmnd_Alias SFAIL_SITEA    = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_Site1 -v SFAIL -t crm_config -s SAPHanaSR
Cmnd_Alias SOK_SITEB      = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_Site2 -v SOK   -t crm_config -s SAPHanaSR
Cmnd_Alias SFAIL_SITEB    = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_Site2 -v SFAIL -t crm_config -s SAPHanaSR
<sid>adm ALL=(ALL) NOPASSWD: GSH_QUERY, GSH_UPDATE, SOK_GLOB, SFAIL_GLOB, SOK_GLOB_MTS, SFAIL_GLOB_MTS, SOK_SITEA, SFAIL_SITEA, SOK_SITEB, SFAIL_SITEB
<sid>adm ALL=(ALL) NOPASSWD: /usr/sbin/SAPHanaSR-hookHelper --sid=<sid> --case=checkTakeover

~> sapcontrol -nr <instanceNumber> -function StartSystem

11.06.2021 18:30:16
StartSystem
OK

~> sapcontrol -nr <instanceNumber> -function WaitforStarted 300 20
~> sapcontrol -nr <instanceNumber> -function GetSystemInstanceList

~> cdtrace
~> grep HADR.*load.*SAPHanaSrMultiTarget nameserver_*.trc | tail -3
~> grep SAPHanaSr.*init nameserver_*.trc | tail -3
~> grep -A5 "init.called" nameserver_saphanasr_multitarget_hook.trc

~> cdtrace
~> grep HADR.*load.*susTkOver nameserver_*.trc | tail -3
~> grep susTkOver.init nameserver_*.trc | tail -3

Figure 12: [Planning] [OsSetup] [SAPHanaInst] [SAPHanaHsr] [Integration] Cluster [Testing] #

 

This chapter describes the configuration of the SUSE Linux Enterprise High Availability Extension cluster. The SUSE Linux Enterprise High Availability Extension is part of SUSE Linux Enterprise Server for SAP Applications. Further, the integration of SAP HANA System Replication with the SUSE Linux Enterprise High Availability Extension cluster is explained. The integration is done by using the SAPHanaSR-ScaleOut package which is also part of SUSE Linux Enterprise Server for SAP Applications.

# zypper in -t pattern ha_sles

# zypper in SAPHanaSR-ScaleOut

# zypper patch

8.2 Configuring the basic cluster #

 

After having installed the cluster packages, the next step is to set up the basic cluster framework. For convenience, use YaST or the ha-cluster-init script.

Important

It is strongly recommended to add a second corosync ring, implement unicast (UCAST) communication and adjust the timeout values to your environment.

Prerequisites

• Name resolution

• Time synchronization

• Redundant network for cluster intercommunication

• STONITH method

8.2.1 Seting up watchdog for "Storage-based Fencing" #

 

It is recommended to use SBD as central STONITH device, as done in the example at hand. Each node constantly monitors connectivity to the storage device, and terminates itself in case the partition becomes unreachable. Whenever SBD is used, a correctly working watchdog is crucial. Modern systems support a hardware watchdog that needs to be "tickled" or "fed" by a software component. The software component (usually a daemon) regularly writes a service pulse to the watchdog. If the daemon stops feeding the watchdog, the hardware will enforce a system restart. This protects against failures of the SBD process itself, such as dying, or getting stuck on an I/O error.

Example 26: Set up for Watchdog #

 

Important

Access to the Watchdog Timer: No other software must access the watchdog timer. Some hardware vendors ship systems management software that uses the watchdog for system resets (for example, HP ASR daemon). Disable such software, if watchdog is used by SBD.

Determine the right watchdog module. Alternatively, you can find a list of installed drivers with your kernel version.

# ls -l /lib/modules/$(uname -r)/kernel/drivers/watchdog

Check if any watchdog module is already loaded.

# lsmod | egrep "(wd|dog|i6|iT|ibm)"

If you get a result, the system has already a loaded watchdog. If the watchdog does not match your watchdog device, you need to unload the module.

To safely unload the module, check first if an application is using the watchdog device.

# lsof /dev/watchdog
# rmmod <wrong_module>

Enable your watchdog module and make it persistent. For the example below, softdog has been used which has some restrictions and should not be used as first option.

# echo softdog > /etc/modules-load.d/watchdog.conf
# systemctl restart systemd-modules-load

Check if the watchdog module is loaded correctly.

# lsmod | grep dog

Testing the watchdog can be done with a simple action. Ensure to switch of your SAP HANA first because watchdog will force an unclean reset / shutdown of your system.

In case of a hardware watchdog a desired action is predefined after the timeout of the watchdog has reached. If your watchdog module is loaded and not controlled by any other application, do the following:

Important

Triggering the watchdog without continuously updating the watchdog resets/switches off the system. This is the intended mechanism. The following commands will force your system to be reset/switched off.

# touch /dev/watchdog

In case the softdog module is used the following action can be performed:

# echo 1 > /dev/watchdog

After your test was successful you can implement the watchdog on all cluster members. The example below applies to the softdog module. Replace <wrong_module> by the module name queried before.

# for i in suse{02,03,04,05,06,-mm}; do
    ssh -T $i <<EOSSH
        hostname
        rmmod <wrong_module>
        echo softdog > /etc/modules-load.d/watchdog.conf
        systemctl restart systemd-modules-load
        lsmod |grep -e dog
EOSSH
done

8.2.2 Setting up the initial cluster using ha-cluster-init #

 

For more detailed information about ha-cluster-* tools, see section Overview of the Bootstrap Scripts of the Installation and Setup Quick Start Guide for SUSE Linux Enterprise High Availability Extension at https://documentation.suse.com/sle-ha/15-SP1/single-html/SLE-HA-install-quick/#sec-ha-inst-quick-bootstrap.

Create an initial setup by using ha-cluster-init command. Follow the dialog steps.

Note

This is only to be done on the first cluster node. If you are using SBD as STONITH mechanism, you need to first load the watchdog kernel module matching your setup. In the example at hand the softdog kernel module is used.

The command ha_cluster-init configures the basic cluster framework including:

• SSH keys

• csync2 to transfer configuration files

• SBD (at least one device)

• corosync (at least one ring)

• HAWK Web interface

# ha-cluster-init -u -s <sbd-device>

As requested by ha-cluster-init, change the password of the user hacluster on all cluster nodes.

Note

Do not forget to change the password of the user hacluster.

8.2.3 Cluster configuration for all other cluster nodes #

 

The other nodes of the cluster could be integrated by starting the command ha-cluster-join. This command asks for the IP address or name of the first cluster node. Than all needed configuration files are copied over. As a result the cluster is started on all nodes. Do not forget the majority maker.

If you are using SBD as STONITH method, you need to activate the softdog kernel module matching your systems. In the example at hand the softdog kernel module is used.

# ha-cluster-join -c <host1>

8.2.4 Checking the cluster for the first time #

 

Now it is time to check and optionally start the cluster for the first time on all nodes.

# crm cluster run "crm cluster start"

Note

All nodes should be started in parallel. Otherwise unseen nodes might get fenced.

Check whether all cluster nodes have registered at the SBD device(s). See manual page cs_show_sbd_devices(8) for details.

# cs_show_sbd_devices

Check the cluster status with crm_mon. Use the option -r to also see resources which are configured but stopped.

# crm_mon -r

The command will show the "empty" cluster and will print something like the screen output below. The most interesting information in this output is that there are two nodes in the status "online" and the message "partition with quorum".

Stack: corosync
Current DC: suse05 (version 1.1.16-4.8-77ea74d) - partition with quorum
Last updated: Mon Jun 11 16:55:04 2021
Last change: Mon Jun 11 16:53:58 2021 by root via crm_attribute on suse02

7 nodes configured
1 resource configured

Online: [ suse-mm suse01 suse02 suse03 suse04 suse05 suse06 ]

Full list of resources:

stonith-sbd     (stonith:external/sbd): Started suse-mm

# vi crm-file<XX>
# crm configure load update crm-file<XX>

# vi crm-bs.txt

property cib-bootstrap-options: \
    have-watchdog=true \
    cluster-infrastructure=corosync \
    cluster-name=hacluster \
    placement-strategy=balanced \
    no-quorum-policy=freeze \
    stonith-enabled=true \
    concurrent-fencing=true \
    stonith-action=reboot \
    stonith-timeout=150
rsc_defaults rsc-options: \
    resource-stickiness=1000 \
    migration-threshold=50
op_defaults op-options: \
    timeout=600 \
    record-pending=true

# crm configure load update crm-bs.txt

# vi crm-fencing.txt

primitive stonith-sbd stonith:external/sbd \
	params pcmk_action_limit=-1 pcmk_delay_max=1

# crm configure load update crm-fencing.txt

# crm maintenance on

suse01:~ # vi crm-saphanatop.txt

primitive rsc_SAPHanaTop_<SID>_HDB<instanceNumber> ocf:suse:SAPHanaTopology \
    op monitor interval="10" timeout="600" \
    op start interval="0" timeout="600" \
    op stop interval="0" timeout="300" \
    params SID="<SID>" InstanceNumber="<instanceNumber>"

clone cln_SAPHanaTop_<SID>_HDB<instanceNumber> rsc_SAPHanaTop_<SID>_HDB<instanceNumber> \
    meta clone-node-max="1" interleave="true"

primitive rsc_SAPHanaTop_HA1_HDB00 ocf:suse:SAPHanaTopology \
    op monitor interval="10" timeout="600" \
    op start interval="0" timeout="600" \
    op stop interval="0" timeout="300" \
    params SID="HA1" InstanceNumber="00"

clone cln_SAPHanaTop_HA1_HDB00 rsc_SAPHanaTop_HA1_HDB00 \
    meta clone-node-max="1" interleave="true"

# crm configure load update crm-saphanatop.txt

# vi crm-saphanacon.txt

primitive rsc_SAPHanaCon_<SID>_HDB<instanceNumber> ocf:suse:SAPHanaController \
    op start interval="0" timeout="3600" \
    op stop interval="0" timeout="3600" \
    op promote interval="0" timeout="3600" \
    op demote interval="0" timeout="320" \
    op monitor interval="60" role="Master" timeout="700" \
    op monitor interval="61" role="Slave" timeout="700" \
    params SID="<SID>" InstanceNumber="<instanceNumber>" \
        PREFER_SITE_TAKEOVER="true" \
        DUPLICATE_PRIMARY_TIMEOUT="7200" AUTOMATED_REGISTER="false" \
        HANA_CALL_TIMEOUT="120"

ms msl_SAPHanaCon_<SID>_HDB<instanceNumber> \
    rsc_SAPHanaCon_<SID>_HDB<instanceNumber> \
    meta master-node-max="1" master-max="1" clone-node-max="1" interleave="true" \
    maintenance="true"

primitive rsc_SAPHanaCon_HA1_HDB00 ocf:suse:SAPHanaController \
    op start interval="0" timeout="3600" \
    op stop interval="0" timeout="3600" \
    op promote interval="0" timeout="900" \
    op demote interval="0" timeout="320" \
    op monitor interval="60" role="Master" timeout="700" \
    op monitor interval="61" role="Slave" timeout="700" \
    params SID="HA1" InstanceNumber="00" PREFER_SITE_TAKEOVER="true" \
        DUPLICATE_PRIMARY_TIMEOUT="7200" AUTOMATED_REGISTER="false" \
        HANA_CALL_TIMEOUT="120"

ms msl_SAPHanaCon_HA1_HDB00 rsc_SAPHanaCon_HA1_HDB00 \
    meta master-node-max="1" master-max="1" clone-node-max="1" interleave="true" \
    maintenance="true"

# crm configure load update crm-saphanacon.txt

# vi crm-vip.txt

primitive rsc_ip_<SID>_HDB<instanceNumber> ocf:heartbeat:IPaddr2 \
    op monitor interval="10s" timeout="20s" \
    params ip="<IP>"

primitive rsc_ip_HA1_HDB00 ocf:heartbeat:IPaddr2 \
    op monitor interval="10s" timeout="20s" \
    params ip="192.168.201.109"

# crm configure load update crm-vip.txt

# vi crm-cs.txt

colocation col_saphana_ip_<SID>_HDB<instanceNumber> 2000: rsc_ip_<SID>_HDB<instanceNumber>:Started \
    msl_SAPHanaCon_<SID>_HDB<instanceNumber>:Master

order ord_SAPHana_<SID>_HDB<instanceNumber> Optional: cln_SAPHanaTop_<SID>_HDB<instanceNumber> \
    msl_SAPHanaCon_<SID>_HDB<instanceNumber>

location SAPHanaCon_not_on_majority_maker msl_SAPHanaCon_<SID>_HDB<instanceNumber> \
    -inf: <majority maker>
location SAPHanaTop_not_on_majority_maker cln_SAPHanaTop_<SID>_HDB<instanceNumber> \
    -inf: <majority maker>

colocation col_saphana_ip_HA1_HDB00 2000: rsc_ip_HA1_HDB00:Started \
    msl_SAPHanaCon_HA1_HDB00:Master

order ord_SAPHana_HA1_HDB00 Optional: cln_SAPHanaTop_HA1_HDB00 \
    msl_SAPHanaCon_HA1_HDB00

location SAPHanaCon_not_on_majority_maker msl_SAPHanaCon_HA1_HDB00 -inf: suse-mm
location SAPHanaTop_not_on_majority_maker cln_SAPHanaTop_HA1_HDB00 -inf: suse-mm

# configure load update crm-cs.txt

# crm_attribute -G  -n hana_<sid>_glob_srHook

scope=crm_config  name=hana_<sid>_glob_srHook value=SFAIL

suse01:ha1adm> /usr/sap/hostctrl/exe/saphostctrl -function ListInstances
 Inst Info : HA1 - 00 - suse01 - 749, patch 418, changelist 1816226

# crm maintenance off
# cs_wait_for_idle -s 5
# crm resource refresh msl_SAPHanaCon_HA1_HDB00
# cs_wait_for_idle -s 5
# crm resource maintenance msl_SAPHanaCon_HA1_HDB00 off

Figure 13: [Planning] [OsSetup] [SAPHanaInst] [SAPHanaHsr] [Integration] [Cluster] Testing #

 

Testing is one of the most important project tasks for implementing clusters. Proper testing is crucial. Make sure that all test cases derived from project or customer expectations are defined and passed completely. Without testing the project is likely to fail in production use.

The test prerequisite, if not described differently, is always that all cluster nodes are booted, are already normal members of the cluster and the SAP HANA RDBMS is running. The system replication is in state SYNC represented by 'srHook SOK' for the secondary site. The cluster is idle, no actions are pending, no migration constraints left over, no fail-counts left over.

The expected results are given for SAPHanaController parameter AUTOMATED_REGISTER="false" and SBD parameter SBD_START_MODE="always". These are installation defaults.

In the current version of this setup guide a plain list of test cases is provided. A more detailed description of the test cases is planned for future versions. These details will be provided either in an updated version of this guide or the test cases will be extracted to a separate test plan document.

For more detailed information, have a look at the documents listed below.

Copyright © 2006–2024 SUSE LLC and contributors. All rights reserved.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled "GNU Free Documentation License".

SUSE, the SUSE logo and YaST are registered trademarks of SUSE LLC in the United States and other countries. For SUSE trademarks, see https://www.suse.com/company/legal/.

Linux is a registered trademark of Linus Torvalds. All other names or trademarks mentioned in this document may be trademarks or registered trademarks of their respective owners.

Documents published as part of the SUSE Best Practices series have been contributed voluntarily by SUSE employees and third parties. They are meant to serve as examples of how particular actions can be performed. They have been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. SUSE cannot verify that actions described in these documents do what is claimed or whether actions described have unintended consequences. SUSE LLC, its affiliates, the authors, and the translators may not be held liable for possible errors or the consequences thereof.

Below we draw your attention to the license under which the articles are published.

Copyright © 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work’s title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’s license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties—​for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

Copyright (c) YEAR YOUR NAME.
   Permission is granted to copy, distribute and/or modify this document
   under the terms of the GNU Free Documentation License, Version 1.2
   or any later version published by the Free Software Foundation;
   with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
   A copy of the license is included in the section entitled “GNU
   Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “ with…​Texts.” line with this:

with the Invariant Sections being LIST THEIR TITLES, with the
   Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.