SAP HANA High Availability Cluster for the AWS Cloud #

With the SAPHanaSR resource agent software package, we limit the support to scale-up (single-box to single-box) system replication with the following configurations and parameters:

You need at least SAPHanaSR version 0.152 and in best SUSE Linux Enterprise Server for SAP Applications 12 SP4 or newer. SAP HANA 1.0 is supported since SPS09 (095) for all mentioned setups. SAP HANA 2.0 is supported with all known SPS versions.

Important

Without a valid STONITH method, the complete cluster is unsupported and will not work properly.

If you need to implement a different scenario, we strongly recommend to define a Proof of Concept (PoC) with SUSE. This PoC will focus on testing the existing solution in your scenario. Most of the above mentioned limitations exist because careful testing is needed.

Besides SAP HANA, you need SAP Host Agent to be installed on your system.

This document describes how to set up the cluster to control SAP HANA in System Replication Scenarios. The document focuses on the steps to integrate an already installed and working SAP HANA with System Replication.

The described example setup builds an SAP HANA HA cluster in two Availability Zones in one AWS Region. Availability Zone 1 is "A" and Availability Zone 2 is "B", installed on two SUSE Linux Enterprise Server for SAP Applications 12 SP5 systems.

Figure 7: Cluster with SAP HANA SR - performance optimized #

 

This guide focuses on the manual setup of the cluster to explain the details and to give you the possibility to create your own automation.

The seven main setup steps are:

Planning the installation is essential for a successful SAP HANA cluster setup.

What you need before you start:

Note

The preferred method to deploy SAP HANA Scale-Up clusters in AWS is to use the AWS Launch Wizard for SAP. However, if you are installing SAP HANA Scale-Up manually, refer to the AWS SAP HANA Guides for detailed installation instructions, including recommended storage configuration and file systems.

4.3 Creating AWS EC2 Instance #

 

Create two EC2 instances to build up your SUSE Linux Enterprise High Availability Extension cluster.

The EC2 instances must be located in two different Availability Zones to make them independent of each other, and it is recommended to be one of the certified SAP HANA instances as per the SAP HANA Certified Hardware Directory.

There are two options for which Amazon Machine Image (AMI) to use:

• Use the AWS Marketplace AMI "SUSE Linux Enterprise Server for SAP Applications 12 SP5" which already includes the required SUSE subscription and all High Availability components for this solution.

• Use a "SUSE Linux Enterprise Server for SAP" AMI. Search for "suse-sles-sap-12-sp5-byos" in the list of AMIs. There are several BYOS (Bring Your Own Subscription) AMIs available. Use these AMIs if you have a valid SUSE subscription. Register your system with the Subscription Management Tool (SMT) from SUSE, SUSE Manager or directly with the SUSE Customer Center.

Launch all EC2 instances into the Availability Zones (AZ) specific subnets. The subnets need to be able to communicate with each other.

Note

It is not possible to migrate from standard "SUSE Linux Enterprise Server" to "SUSE Linux Enterprise Server for SAP Applications" in AWS. Therefore, use a "SLES for SAP" AMI which includes the SUSE Linux Enterprise High Availability Extension.

4.4 Tagging the EC2 Instances #

 

The AWS EC2 STONITH agents use AWS resource tags to identify the EC2 instances.

Tag the two EC2 instances through the console or the AWS Command Line Interface (CLI) with arbitrarily chosen tags like pacemaker and the host name as it will be shown in the command uname. Use the same tag (like pacemaker) and the individual host names for both instances.

To add a tag to an EC2 instance, refer to the AWS Documentation: * Tagging your Amazon EC2 resources: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html

See an example screenshot after the EC2 instance has been tagged. A tag with the key pacemaker and the host name has been created. The host name in this example is suse-node52.

Figure 8: Tag EC2 instance #

 

Make sure that both EC2 instances part of the cluster are tagged.

Note

Use only ASCII characters in any AWS tag assigned to cluster managed resources.

4.4.1 Disabling Source/Destination Check for Cluster Instances #

 

The source/destination check needs to be disabled. This can be done through scripts using the AWS CLI or by using the AWS console.

The following command needs to be executed one time for both EC2 instances that are part of the cluster:

Example 1: Disabling Source/Destination Check using AWS CLI #

 

# aws ec2 modify-instance-attribute --instance-id EC2-instance --no-source-dest-check

Replace the variable EC2-instance with the EC2 instance IDs of the two cluster AWS EC2 instances.

The system on which this command gets executed needs temporarily a role with the following policy:

Example 2: IAM Policy required to change Source/Destination Check #

 

{
   "Version": "2012-10-17",
   "Statement": [
   {
      "Sid": "Stmt1424870324000",
      "Effect": "Allow",
      "Action": [ "ec2:ModifyInstanceAttribute" ],
      "Resource": [
      "arn:aws:ec2:region-name:account-id:instance/instance-a",
      "arn:aws:ec2:region-name:account-id:instance/instance-b"
      ]
   }
   ]
}

Replace the following individual parameters with the appropriate values:

• region-name : the name of the AWS region

• account-id : The number of the AWS account in which the policy is used

• instance-a and instance-b : The two EC2 instance ids participating in the cluster

The source/destination check can be also disabled from the AWS console. It requires the following action in the console on both EC2 instances (see below).

Figure 9: Disable Source/Destination Check at Console #

 

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "EC2:DescribeInstances",
                "EC2:DescribeVolumes"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "cloudwatch:GetMetricStatistics",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::aws-sap-data-provider/config.properties"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1424870324000",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1424870324001",
            "Effect": "Allow",
            "Action": [
                "ec2:RebootInstances",
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": [
                "arn:aws:ec2:region-name:account-id:instance/instance-a",
                "arn:aws:ec2:region-name:account-id:instance/instance-b"

            ]
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:ReplaceRoute",
            "Resource": "arn:aws:ec2:region-name:account-id:route-table/rtb-XYZ"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "ec2:DescribeRouteTables",
            "Resource": "*"
        }
    ]
}

4.6 Adding Overlay IP Addresses to Routing Tables #

 

Manually add the Overlay IP address as a routing entry to the VPC routing tables which are assigned to the subnets. The Overlay IP address is the virtual service IP address of the SAP HANA cluster. The Overlay IP address needs to be outside of the CIDR range of the VPC.

To add the Overlay IP address, do the following:

• Use the AWS console and search for “VPC”.

• Select the correct VPC ID.

• Click “Route Tables” in the left column.

• Select the route table used by the subnets from one of your SAP EC2 instances and their application servers.

• Click the tabulator “Routes”.

• Click “Edit”.

• Scroll to the end of the list and click “Add another route”.

• Add the Overlay IP address of the SAP HANA database. Use as filter /32 (example: 192.168.10.1/32). Add the Elastic Network Interface (ENI) name to one of your existing instance. The resource agent will modify this later automatically.

• Save your changes by clicking “Save”.

Note

The VPC routing table containing the routing entry needs to be inherited to all subnets in the VPC which have consumers or clients of the service. Add more routing tables if required. Check the AWS VPC documentation at http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Introduction.html for more details on routing table inheritance.

This section contains information you should consider during the installation of the operating system.

For the scope of this document, first SUSE Linux Enterprise Server for SAP Applications is configured. Then the SAP HANA database including the system replication is set up. Finally the automation with the cluster is set up and configured.

5.1 Setting System Host Name #

 

The EC2 instances will have host names which are automatically generated, and these automatically generated host names must be changed. Select host names which comply with SAP requirements, see SAP Note 611361.

To change the host name you need to edit /etc/cloud/cloud.cfg and change the option preserve_hostname to true for host names to persist:

Example 6: Option changed in cloud.cfg file #

 

preserve_hostname: true

Note

To learn how to change the default host name for an EC2 instance running SUSE Linux Enterprise, refer to the AWS' public documentation at https://aws.amazon.com/premiumsupport/knowledge-center/linux-static-hostname-suse/.

5.1.1 Configuring System Logging #

 

SUSE recommends to use rsyslogd for logging in the SUSE cluster. Despite of this being the default configuration on newer AMIs, some AWS AMIs may still be using syslogd logging.

Perform the following commands as root on all cluster nodes:

Example 7: SUSE Linux Enterprise Server 12 rsyslog Installation #

 

suse01:~> zypper install rsyslog

Depending on the installed packages, a conflict may be shown, like in the below example:

suse01:~ # zypper install rsyslog
Refreshing service 'SMT-http_smt-ec2_susecloud_net'.
Refreshing service 'cloud_update'.
Loading repository data...
Reading installed packages...
Resolving package dependencies...
Problem: syslog-ng-3.6.4-11.1.x86_64 conflicts with namespace:otherproviders(syslog)
provided by rsyslog-8.24.0-3.16.1.x86_64
 Solution 1: deinstallation of syslog-ng-3.6.4-11.1.x86_64
 Solution 2: do not install rsyslog-8.24.0-3.16.1.x86_64
Choose from above solutions by number or cancel [1/2/c] (c):

Select "Solution 1: deinstallation of syslog-ng", and then reboot both nodes.

Additionally, some cluster components require ha_logd to properly log events, thus it needs to be set to start at boot:

Example 8: Enabling logd to start automatically #

 

suse01:~> systemctl enable --now logd

[default]
region = region-name
[profile cluster]
region = region-name
output = text

# aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: region-name
Default output format [None]:

# aws configure --profile cluster
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: region-name
Default output format [None]: text

export HTTP_PROXY=http://a.b.c.d:n
export HTTPS_PROXY=http://a.b.c.d:m
export NO_PROXY=169.254.169.254

export HTTP_PROXY=http://username:password@a.b.c.d:n
export HTTPS_PROXY=http://username:password@a.b.c.d:m
export NO_PROXY=169.254.169.254

IPADDR_1="XX.XX.XX.XX/XX"
LABEL_1="1"

ip address add XX.XX.XX.XX/XX dev eth0

# zypper info cloud-netconfig-ec2

CLOUD_NETCONFIG_MANAGE='no'

Even though this document focuses on the integration of an installed SAP HANA with system replication already set up into the pacemaker cluster, this chapter summarizes the test environment. Always use the official documentation from SAP to install SAP HANA and to set up the system replication.

As Linux user <sid>adm, use the command line tool HDB to get an overview of the running HANA processes. The output of HDB info should be similar to the output shown below:

suse02:~> HDB info
USER           PID  ...  COMMAND
ha1adm         6561 ...  -csh
ha1adm         6635 ...    \_ /bin/sh /usr/sap/HA1/HDB10/HDB info
ha1adm         6658 ...        \_ ps fx -U ha1 -o user,pid,ppid,pcpu,vsz,rss,args
ha1adm         5442 ...  sapstart pf=/hana/shared/HA1/profile/HA1_HDB10_suse02
ha1adm         5456 ...   \_ /usr/sap/HA1/HDB10/suse02/trace/hdb.sapHA1_HDB10 -d
-nw -f /usr/sap/HA1/HDB10/suse
ha1adm         5482 ...       \_ hdbnameserver
ha1adm         5551 ...       \_ hdbpreprocessor
ha1adm         5554 ...       \_ hdbcompileserver
ha1adm         5583 ...       \_ hdbindexserver
ha1adm         5586 ...       \_ hdbstatisticsserver
ha1adm         5589 ...       \_ hdbxsengine
ha1adm         5944 ...       \_ sapwebdisp_hdb
pf=/usr/sap/HA1/HDB10/suse02/wdisp/sapwebdisp.pfl -f /usr/sap/SL
ha1adm         5363 ...  /usr/sap/HA1/HDB10/exe/sapstartsrv
pf=/hana/shared/HA1/profile/HA1_HDB10_suse02 -D -u s

For more information, read the section Setting Up System Replication of the SAP HANA Administration Guide.

Procedure

7.1 Backing Up the Primary Database #

 

Back up the primary database as described in the SAP HANA Administration Guide, section SAP HANA Database Backup and Recovery. We provide an example with SQL commands. You need to adapt these backup commands to match your backup infrastructure.

Example 17: Simple backup for the system database and all tenants with one single backup call #

 

As user <sidadm> enter the following command:

hdbsql -u SYSTEM -d SYSTEMDB \
   "BACKUP DATA FOR FULL SYSTEM USING FILE ('backup')"

You will get the following command output (or similar):

0 rows affected (overall time 15.352069 sec; server time 15.347745 sec)

Example 18: Simple backup for a single container (non MDC) database #

 

Enter the following command as user <sidadm>:

hdbsql -i <instanceNumber> -u <dbuser> \
   "BACKUP DATA USING FILE ('backup')"

Important

Without a valid backup, you cannot bring SAP HANA into a system replication configuration.

7.2 Enabling the Primary Node #

 

As Linux user <sid>adm, enable the system replication at the primary node. You need to define a site name (like WDF). This site name must be unique for all SAP HANA databases which are connected via system replication. This means the secondary must have a different site name.

Note

Do not use strings like "primary" and "secondary" as site names.

Example 19: Enable the Primary #

 

Enable the primary using the -sr_enable option.

suse01:~> hdbnsutil -sr_enable --name=WDF
checking local nameserver:
checking for active nameserver ...
nameserver is running, proceeding ...
configuring ini files ...
successfully enabled system as primary site ...
done.

Example 20: Check SR Configuration on the Primary #

 

Check the primary using the command hdbnsutil -sr_stateConfiguration.

suse01:~> hdbnsutil -sr_stateConfiguration --sapcontrol=1
SAPCONTROL-OK: <begin>
mode=primary
site id=1
site name=WDF
SAPCONTROL-OK: <end>
done.

The mode has changed from “none” to “primary” and the site now has a site name and a site ID.

suse02:~> HDB stop

cd /usr/sap/<SID>/SYS/global/security/rsecssfs
rsync -va {,<node1-siteB>:}$PWD/data/SSFS_<SID>.DAT
rsync -va {,<node1-siteB>:}$PWD/key/SSFS_<SID>.KEY

...
suse02:~> hdbnsutil -sr_register --name=ROT \
     --remoteHost=suse01 --remoteInstance=10 \
     --replicationMode=sync --operationMode=logreplay
adding site ...
checking for inactive nameserver ...
nameserver suse02:30001 not responding.
collecting information ...
updating local ini files ...
done.

suse02:~> HDB start
...
suse02:~> hdbnsutil -sr_stateConfiguration --sapcontrol=1
SAPCONTROL-OK: <begin>
mode=sync
site id=2
site name=ROT
active primary site=1
primary masters=suse01
SAPCONTROL-OK: <end>
done.

suse01:~> HDBSettings.sh systemReplicationStatus.py --sapcontrol=1
...
site/2/SITE_NAME=ROT1
site/2/SOURCE_SITE_ID=1
site/2/REPLICATION_MODE=SYNC
site/2/REPLICATION_STATUS=ACTIVE
site/1/REPLICATION_MODE=PRIMARY
site/1/SITE_NAME=WDF1
local_site_id=1
...

This step is mandatory to inform the cluster immediately if the secondary gets out of sync. The hook is called by SAP HANA using the HA/DR provider interface at that point of time when the secondary gets out of sync. This is typically the case when the first commit pending is released. The hook is called by SAP HANA again when the system replication is back.

Procedure

sapcontrol -nr <instanceNumber> -function StopSystem

[ha_dr_provider_SAPHanaSR]
provider = SAPHanaSR
path = /usr/share/SAPHanaSR
execution_order = 1

[trace]
ha_dr_saphanasr = info

[system_replication]
operation_mode = logreplay

# SAPHanaSR-ScaleUp entries for writing srHook cluster attribute
<sidadm> ALL=(ALL) NOPASSWD: /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_*

# SAPHanaSR-ScaleUp entries for writing srHook cluster attribute
Cmnd_Alias SOK_SITEA   = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_<siteA> -v SOK   -t crm_config -s SAPHanaSR
Cmnd_Alias SFAIL_SITEA = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_<siteA> -v SFAIL -t crm_config -s SAPHanaSR
Cmnd_Alias SOK_SITEB   = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_<siteB> -v SOK   -t crm_config -s SAPHanaSR
Cmnd_Alias SFAIL_SITEB = /usr/sbin/crm_attribute -n hana_<sid>_site_srHook_<siteB> -v SFAIL -t crm_config -s SAPHanaSR
<sidadm> ALL=(ALL) NOPASSWD: SOK_SITEA, SFAIL_SITEA, SOK_SITEB, SFAIL_SITEB

This chapter describes the configuration of the cluster software SUSE Linux Enterprise High Availability Extension, which is part of SUSE Linux Enterprise Server for SAP Applications, and SAP HANA Database Integration.

suse01:~> zypper update

9.2 Configuring the Basic Cluster #

 

The first step is to set up the basic cluster framework.

9.2.1 Configuring Corosync #

 

By default, the cluster service (pacemaker) is disabled and not set to start during boot. Thus at this point the cluster should not be running. However, if you previously configured pacemaker and it is running, proceed with a "stop" by using the following command:

Example 31: Stopping the cluster #

 

suse01:~ # systemctl stop pacemaker

The cluster service (pacemaker) status can be checked with:

Example 32: Checking cluster status #

 

suse01:~ # systemctl status pacemaker

9.2.2 Creating Keys #

 

On Node 1, generate a corosync secret key used to encrypt all cluster communication:

Example 33: Generating corosync security keys #

 

suse01:~# corosync-keygen

A new key file will be created on /etc/corosync/authkey, and this file needs to be copied to the same location on Node 2. After generating and transferring the key file to the second node, verify that permissions and ownerships on both nodes are the same:

Example 34: Checking permissions and ownership for corosync key file #

 

suse01:~ # ls -l /etc/corosync/authkey
-r-------- 1 root root 128 Oct 23 10:51 /etc/corosync/authkey

9.2.3 Creating the Corosync Configuration File #

 

The corosync configuration will leverage both IP addresses associated to each cluster node. The two IP configurations will use the second IP if the primary IP addresses for the two node cluster are no longer able to communicate with each other.

All cluster nodes are required to have a local configuration file "/etc/corosync/corosync.conf" where the relevant information is being located in the two sections describing interface and nodelist. The other entries can be configured as needed for a specific implementation.

AWS requires a specific corosync configuration, which can be structured as the example below.

Note

When using the following configuration as an example for the file /etc/corosync/corosync.conf, replace the IP addresses from the file below.

Example 35: Sample corosyc.conf file #

 

# Read the corosync.conf.5 manual page
totem {
   version: 2
   rrp_mode: passive
   token: 30000
   consensus: 36000
   token_retransmits_before_loss_const: 6
   secauth: on
   crypto_hash: sha1
   crypto_cipher: aes256
   clear_node_high_bit: yes
   interface {
      ringnumber: 0
      bindnetaddr: ip-local-node
      mcastport: 5405
      ttl: 1
   }
   transport: udpu
}
logging {
   fileline: off
   to_logfile: yes
   to_syslog: yes
   logfile: /var/log/cluster/corosync.log
   debug: off
   timestamp: on
   logger_subsys {
      subsys: QUORUM
      debug: off
   }
}
nodelist {
   node {
     ring0_addr: ip-node-1-a
     # redundant ring
     ring1_addr: ip-node-1-b
     nodeid: 1
   }
   node {
     ring0_addr: ip-node-2-a
     # redundant ring
     ring1_addr: ip-node-2-b
     nodeid: 2
   }
}
quorum {
# Enable and configure quorum subsystem (default: off)
# see also corosync.conf.5 and votequorum.5
   provider: corosync_votequorum
   expected_votes: 2
   two_node: 1
}

Replace the variables ip-node-1-a, ip-node-1-b, ip-node-2-a, ip-node-2-b and ip-local-node from the above sample file.

• ip-local-node: Use the IP address of the node where the file is being configured. This IP will be different between cluster nodes.

• ip-node-1-a: Primary IP address of cluster node node-1

• ip-node-1-b: Secondary IP address of cluster node node-1

• ip-node-2-a: Primary IP address of cluster node node-2

• ip-node-2-b: Secondary IP address of cluster node node-2

The chosen settings for crypto_cipher and crypto_hash are suitable for clusters in AWS. They may be modified according to SUSE’s documentation if strong encryption of cluster communication is desired.

Note

Remember to change the password of the user hacluster.

9.2.4 Checking the Cluster for the First Time #

 

Now it is time to check and start the cluster for the first time on both nodes.

Example 36: Starting the cluster on both cluster nodes #

 

suse01:~ # systemctl status pacemaker
suse02:~ # systemctl status pacemaker
suse01:~ # systemctl start pacemaker
suse02:~ # systemctl start pacemaker

Check the cluster status with crm_mon. We use the option -r to also see resources which may be configured but stopped. But at this stage crm_mon is expected to display no services.

Example 37: Checking cluster status using crm_mon #

 

# crm_mon -r

The command will show the "empty" cluster and will print something like the computer output shown below. The most interesting information for now is that there are two nodes in the status "online", and the message "partition with quorum".

Example 38: Cluster status after first start #

 

Stack: corosync
Current DC: prihana (version 1.1.19+20181105.ccd6b5b10-3.19.1-1.1.19+20181105.ccd6b5b10) - partition with quorum
Last updated: Mon Sep 28 18:36:16 2020
Last change: Mon Sep 28 18:36:09 2020 by root via crm_attribute on suse01

2 nodes configured

2 nodes configured
0 resources configured

Online: [ suse01 suse02 ]

No resources

Corosync’s redundant ring configuration can be checked with the following command:

Example 39: Corosync redundant ring status #

 

corosync-cfgtool -s

This will display a result like the following one for a cluster node with redundant corosync rings and IP addresses 172.16.100.179 and 172.16.100.138:

Printing ring status.
Local node ID 1
RING ID 0
	id	= 172.16.100.179
	status	= ring 0 active with no faults
RING ID 1
	id	= 172.16.100.138
	status	= ring 1 active with no faults

Note

It is not recommended to automatically rejoin a node to a cluster after a system crash with a reboot. A full inspection and a root cause analysis of the crash is highly recommended before rejoining the cluster.

9.3 Configuring Cluster Properties and Resources #

 

This section describes how to configure constraints, resources, bootstrap and STONITH using the crm configure shell command as described in section Configuring and Managing Cluster Resources (Command Line) of the SUSE Linux Enterprise High Availability Extension documentation.

Use the command crm to add the objects to CRM. Copy the following examples to a local file, edit the file and then load the configuration to the CIB:

suse01:~ # vi crm-fileXX
suse01:~ # crm configure load update crm-fileXX

9.3.1 Cluster Bootstrap and More #

 

The first example defines the cluster bootstrap options, the resource and operation defaults.

suse01:~ # vi crm-bs.txt
# enter the following to the file crm-bs.txt
property $id="cib-bootstrap-options" \
   stonith-enabled="true" \
   stonith-action="off" \
   stonith-timeout="600s"
rsc_defaults $id="rsc-options" \
   resource-stickiness="1000" \
   migration-threshold="5000"
op_defaults $id="op-options" \
   timeout="600"

Note

In some older SUSE versions, the parameter stonith-action may require a change to stonith-action="poweroff".

The setting off forces the EC2 STONITH agent to shut down the EC2 instance in case of fencing operation. This is desirable to avoid split brain scenarios on the AWS platform.

Now, add the configuration to the cluster:

suse01:~ # crm configure load update crm-bs.txt

9.3.2 STONITH Device #

 

The next configuration part defines an AWS EC2 STONITH resource.

suse01::~ # vi aws-stonith.txt
# enter the following to the file aws-stonith.txt
primitive res_AWS_STONITH stonith:external/ec2 \
    op start interval=0 timeout=180 \
    op stop interval=0 timeout=180 \
    op monitor interval=120 timeout=60 \
    meta target-role=Started \
    params tag=pacemaker profile=cluster pcmk_delay_max=15

The "tag=pacemaker" entry needs to match the tag chosen for the EC2 instances. The value for this tag contains the host name returned by the uname -n command. The name of the profile ("cluster" in this example) needs to match the previously configured profile in the AWS CLI.

Name this file for example aws-stonith.txt and add it to the configuration. The following command needs to be issued as root user:

suse01:~ # crm configure load update aws-stonith.txt

A working STONITH method is mandatory to run a supported SUSE cluster on AWS.

Note

Make sure to execute the STONITH tests as outlined in section Troubleshooting of this document to verify STONITH on both nodes.

Note

The STONITH agent currently requires the pacemaker tag on the instance to be in lowercase format. i.e a hostname of 'NODE1', should have the pacemaker tag on the instance set to 'node1'. This will be updated in the future.

9.3.3 Configuring the Overlay IP address #

 

This step requires the Overlay IP address and the resource IDs of the AWS VPC Route Table(s). Create a file with the following content:

suse01:~ # vi aws-move-ip.txt
# enter the following to the file aws-move-ip.txt
primitive res_AWS_IP ocf:suse:aws-vpc-move-ip \
   params ip=overlay-ip-address routing_table=rtb-table interface=eth0 profile=cluster \
   op start interval=0 timeout=180 \
   op stop interval=0 timeout=180 \
   op monitor interval=60 timeout=60

Replace the following individual parameters with the appropriate values:

• overlay-ip-address : the Overlay IP address used

• rtb-table : The AWS VPC Route Table(s) resource ids - if using more than one VPC Route Table use comma (,) as a separator (see below).

• interface : The Linux' network interface identificator

• profile : The name of the profile (cluster in this example) needs to match the previously configured profile in the AWS CLI.

Load this file into the cluster configuration by issuing the following command as superuser:

suse01:~ # crm configure load update aws-move-ip.txt

Optionally, it is possible to specify multiple routing tables in the primitive configuration separated by a comma (,), as shown in the following example:

suse01:~ # vi aws-move-ip.txt
# enter the following to the file aws-move-ip.txt
primitive res_AWS_IP ocf:suse:aws-vpc-move-ip \
   params ip=overlay-ip-address routing_table=rtb-table-1,rtb-table-2,rtb-table-N interface=eth0 profile=cluster \
   op start interval=0 timeout=180 \
   op stop interval=0 timeout=180 \
   op monitor interval=60 timeout=60

Note

Make sure to execute the IP tests as outlined in section Troubleshooting of this document to verify them on both nodes. Checking the configuration for potential problems at current point in time will increase the chances to launch the cluster successfully.

9.3.4 SAPHanaTopology #

 

Next, define the group of resources needed, before the HANA instances can be started. Prepare the changes in a text file, for example crm-saphanatop.txt, and load it with the command:

crm configure load update crm-saphanatop.txt

# vi crm-saphanatop.txt
# enter the following to crm-saphanatop.txt
primitive rsc_SAPHanaTopology_HA1_HDB10 ocf:suse:SAPHanaTopology \
        op monitor interval="10" timeout="600" \
        op start interval="0" timeout="600" \
        op stop interval="0" timeout="300" \
        params SID="HA1" InstanceNumber="10"
clone cln_SAPHanaTopology_HA1_HDB10 rsc_SAPHanaTopology_HA1_HDB10 \
        meta clone-node-max="1" interleave="true"

Additional information about all parameters can be found with the command:

man ocf_suse_SAPHanaTopology

Again, add the configuration to the cluster.

suse01:~ # crm configure load update crm-saphanatop.txt

The most important parameters here are SID and InstanceNumber, which are quite self-explaining in the SAP context. Beside these parameters, the timeout values or the operations (start, monitor, stop) are typical tuneables.

9.3.5 SAPHana #

 

Next, define the group of resources needed, before the HANA instances can be started. Edit the changes in a text file, for example crm-saphana.txt, and load it with the command:

crm configure load update crm-saphana.txt

Table 2: Typical Resource Agent parameter settings for different scenarios #

 

Parameter	Performance Optimized	Cost Optimized	Multi-Tier
PREFER_SITE_TAKEOVER	true	false	false / true
AUTOMATED_REGISTER	false / true	false / true	false
DUPLICATE_PRIMARY_TIMEOUT	7200	7200	7200

Table 3: Description of important Resource Agent parameters #

 

Parameter	Description
PREFER_SITE_TAKEOVER	Defines whether RA should prefer to takeover to the secondary instance instead of restarting the failed primary locally.
AUTOMATED_REGISTER	Defines whether a former primary should be automatically registered to be secondary of the new primary. With this parameter you can adapt the level of system replication automation. If set to false, the former primary must be manually registered. The cluster will not start this SAP HANA RDBMS until it is registered to avoid double primary up situations.
DUPLICATE_PRIMARY_TIMEOUT	Time difference needed between two primary time stamps if a dual-primary situation occurs. If the time difference is less than the time gap, than the cluster hold one or both instances in a "WAITING" status. This is to give an administrator the chance to react on a fail-over. If the complete node of the former primary crashed, the former primary will be registered after the time difference is passed. If "only" the SAP HANA RDBMS has crashed, then the former primary will be registered immediately. After this registration to the new primary all data will be overwritten by the system replication.

Additional information about all parameters can be found with the command:

man ocf_suse_SAPHana

# vi crm-saphana.txt
# enter the following to crm-saphana.txt
primitive rsc_SAPHana_HA1_HDB10 ocf:suse:SAPHana \
        op start interval="0" timeout="3600" \
        op stop interval="0" timeout="3600" \
        op promote interval="0" timeout="3600" \
        op monitor interval="60" role="Master" timeout="700" \
        op monitor interval="61" role="Slave" timeout="700" \
        params SID="HA1" InstanceNumber="10" PREFER_SITE_TAKEOVER="true" \
        DUPLICATE_PRIMARY_TIMEOUT="7200" AUTOMATED_REGISTER="false"
ms msl_SAPHana_HA1_HDB10 rsc_SAPHana_HA1_HDB10 \
        meta clone-max="2" clone-node-max="1" interleave="true"

Add the configuration to the cluster.

suse01:~ # crm configure load update crm-saphana.txt

The most important parameters here are again SID and InstanceNumber. Beside these parameters, the timeout values for the operations (start, promote, monitors, stop) are typical tuneables.

9.3.6 Constraints #

 

Two constraints are organizing the correct placement of the virtual IP address for the client database access and the start order between the two resource agents SAPHana and SAPHanaTopology.

The AWS IP agent needs to operate on the same node as the SAP HANA Master database. A constraint forces it to be on the same node.

# vi crm-cs.txt
# enter the following to crm-cs.txt

colocation col_saphana_ip_HA1_HDB10 2000: res_AWS_IP:Started \
    msl_SAPHana_HA1_HDB10:Master
order ord_SAPHana_HA1_HDB10 Optional: cln_SAPHanaTopology_HA1_HDB10 \
    msl_SAPHana_HA1_HDB10

Add this file to the configuration. The following command needs to be issued as superuser. It uses the file name crm-cs.txt:

suse01:~ # crm configure load update crm-cs.txt

9.3.7 Active/Active Read-Enabled Scenario #

 

This step is optional. If you have an active/active SAP HANA system replication with a read-enabled secondary, it is possible to integrate the needed second Overlay IP address into the cluster. This is done by adding a second Overlay IP address resource and a location constraint binding the address to the secondary site.

# vi crm-re.txt
# enter the following to crm-re.txt
primitive res_AWS_IP_readenabled ocf:suse:aws-vpc-move-ip \
   params ip=readenabled-overlay-ip-address routing_table=rtb-table interface=eth0 profile=cluster \
   op start interval=0 timeout=180 \
   op stop interval=0 timeout=180 \
   op monitor interval=60 timeout=60
colocation col_saphana_ip_HA1_HDB10_readenabled 2000: \
    res_AWS_IP_readenabled:Started msl_SAPHana_HA1_HDB10:Slave

9.3.8 Cluster Status After Configuration #

 

Now that the cluster has been configured, the basic it should have two online nodes, and six resources. If you configured a second Overlay IP for the read enabled replica, then the cluster will display seven resources.

The cluster status can be checked with crm status command:

suse01:~ # crm status
Stack: corosync
Current DC: prihana (version 1.1.19+20181105.ccd6b5b10-3.19.1-1.1.19+20181105.ccd6b5b10) - partition with quorum
Last updated: Tue Sep 29 16:15:51 2020
Last change: Tue Sep 29 16:15:05 2020 by root via crm_attribute on prihana

2 nodes configured
6 resources configured

Online: [ suse01 suse02 ]

Full list of resources:

 res_AWS_STONITH	(stonith:external/ec2):	Started suse01
 res_AWS_IP	(ocf::suse:aws-vpc-move-ip):	Started suse01
 Clone Set: cln_SAPHanaTopology_HDB_HDB00 [rsc_SAPHanaTopology_HDB_HDB00]
     Started: [ suse01 suse02 ]
 Master/Slave Set: msl_SAPHana_HDB_HDB00 [rsc_SAPHana_HDB_HDB00]
     Masters: [ suse01 ]
     Slaves: [ suse02 ]

The above example shows that the Overlay IP resource (res_AWS_IP) is "Started" on node suse01, along with SAPHanaTopology resource (cln_SAPHanaTopology_HA1_HDB10) running on both cluster nodes, and Master/Slave SAPHana (msl_SAPHana_HA1_HDB10), which in the above example is Master (Primary) on node suse01, and Secondary on node suse02.

The lists of tests will be enhanced with future updates of this document.

As with any cluster testing is crucial. Make sure that all test cases derived from customer expectations are implemented and fully passed. Otherwise the project is likely to fail in production.

The test prerequisite, if not described differently, is always that both nodes are booted, normal members of the cluster, and the HANA RDBMS is running. The system replication is in sync (SOK).

10.1 Test Cases for Semi Automation #

 

In the following test descriptions we assume the following values:

PREFER_SITE_TAKEOVER="true" and AUTOMATED_REGISTER="false"

Note

The following tests are designed to run in a sequence. They depend on the exit state of the proceeding tests.

10.1.1 Test: Stop Primary Database on Availability Zone A (Node 1) #

 

Example 40: Test STOP_PRIMARY_SITE_A #

 

Component: #

 

• Primary Database

Description: #

 

• The primary HANA database is stopped during normal cluster operation.

Test Procedure: #

 

1. Stop the primary HANA database gracefully as <sid>adm.

   suse01# HDB stop

Recovery Procedure: #

 

1. Manually register the old primary (on node 1) with the new primary after takeover (on node 2) as <sid>adm.

   suse01# hdbnsutil -sr_register --remoteHost=suse02 --remoteInstance=10 \
             --replicationMode=sync --operationMode=logreplay \
             --name=WDF

2. Restart the HANA database (now secondary) on node 1 as root.

   suse01# crm resource refresh rsc_SAPHana_HA1_HDB10 suse01

Expected: #

 

1. The cluster detects the stopped primary HANA database (on node 1) and marks the resource failed.

2. The cluster promotes the secondary HANA database (on node 2) to take over as primary.

3. The cluster migrates the IP address to the new primary (on node 2).

4. After some time the cluster shows the sync_state of the stopped primary (on node 1) as SFAIL.

5. Because of AUTOMATED_REGISTER="false" the cluster does not restart the failed HANA database or register it against the new primary.

6. After the manual register and resource refresh, the system replication pair is marked as in sync (SOK).

7. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.1.2 Test: Stop Primary Database on Availability Zone B (Node 2) #

 

Example 41: Test STOP_PRIMARY_DB_SITE_B #

 

Component:

Primary Database

Description:

The primary HANA database is stopped during normal cluster operation.

Test Procedure: #

 

1. Stop the database gracefully as <sid>adm.

   suse02# HDB stop

Recovery Procedure: #

 

1. Manually register the old primary (on node 2) with the new primary after takeover (on node 1) as <sid>adm.

   suse02# hdbnsutil -sr_register --remoteHost=suse01 --remoteInstance=10 \
                  --replicationMode=sync --operationMode=logreplay \
                  --name=ROT

2. Restart the HANA database (now secondary) on node 1 as root.

   suse02# crm resource refresh rsc_SAPHana_HA1_HDB10 suse02

Expected: #

 

1. The cluster detects the stopped primary HANA database (on node 2) and marks the resource failed.

2. The cluster promotes the secondary HANA database (on node 1) to take over as primary.

3. The cluster migrates the IP address to the new primary (on node 1).

4. After some time, the cluster shows the sync_state of the stopped primary (on node 2) as SFAIL.

5. Because of AUTOMATED_REGISTER="false" the cluster does not restart the failed HANA database or register it against the new primary.

6. After the manual register and resource refresh, the system replication pair is marked as in sync (SOK).

7. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.1.3 Test: Crash Primary Database on Availability Zone A (Node 1) #

 

Example 42: Test CRASH_PRIMARY_DB_SITE_A #

 

Component:

Primary Database

Description:

Simulate a complete breakdown of the primary database system.

Test Procedure: #

 

1. Kill the primary database system using signals as <sid>adm.

   suse01# HDB kill-9

Recovery Procedure: #

 

1. Manually register the old primary (on node 1) with the new primary after takeover (on node 2) as <sid>adm.

   suse01# hdbnsutil -sr_register --remoteHost=suse02 --remoteInstance=10 \
                  --replicationMode=sync  --operationMode=logreplay \
                  --name=WDF

2. Restart the HANA database (now secondary) on node 1 as root.

   suse01# crm resource refresh rsc_SAPHana_HA1_HDB10 suse01

Expected: #

 

1. The cluster detects the stopped primary HANA database (on node 1) and marks the resource failed.

2. The cluster promotes the secondary HANA database (on node 2) to take over as primary.

3. The cluster migrates the IP address to the new primary (on node 2).

4. After some time, the cluster shows the sync_state of the stopped primary (on node 1) as SFAIL.

5. Because of AUTOMATED_REGISTER="false" the cluster does not restart the failed HANA database or register it against the new primary.

6. After the manual register and resource refresh, the system replication pair is marked as in sync (SOK).

7. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.1.4 Test: Crash Primary Database on Availability Zone B (Node 2) #

 

Example 43: Test CRASH_PRIMARY_DB_SITE_B #

 

Component:

Primary Database

Description:

Simulate a complete breakdown of the primary database system.

Test Procedure: #

 

1. Kill the primary database system using signals as <sid>adm.

   suse02# HDB kill-9

Recovery Procedure: #

 

1. Manually register the old primary (on node 2) with the new primary after takeover (on node 1) as <sid>adm.

   suse02# hdbnsutil -sr_register --remoteHost=suse01 --remoteInstance=10 \
                  --replicationMode=sync  --operationMode=logreplay \
                  --name=ROT

2. Restart the HANA database (now secondary) on node 1 as root.

   suse02# crm resource refresh rsc_SAPHana_HA1_HDB10 suse02

Expected: #

 

1. The cluster detects the stopped primary HANA database (on node 2) and marks the resource failed.

2. The cluster promotes the secondary HANA database (on node 1) to take over as primary.

3. The cluster migrates the IP address to the new primary (on node 1).

4. After some time, the cluster shows the sync_state of the stopped primary (on node 2) as SFAIL.

5. Because of AUTOMATED_REGISTER="false" the cluster does not restart the failed HANA database or register it against the new primary.

6. After the manual register and resource refresh, the system replication pair is marked as in sync (SOK).

7. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.1.5 Test: Crash Primary Node on Availability Zone A (Node 1) #

 

Example 44: Test CRASH_PRIMARY_NODE_SITE_A #

 

Component:

Cluster node of primary site

Description:

Simulate a crash of the primary site node running the primary HANA database.

Test Procedure: #

 

1. Crash the primary node by sending a 'fast-reboot' system request.

   suse01# echo 'b' > /proc/sysrq-trigger

Recovery Procedure: #

 

1. AWS infrastructure has stopped the fenced instance. Restart it with AWS console or AWS CLI tools. Execute the following command after the instance has booted.

2. Start the cluster framework.

   suse01# systemctl start pacemaker

3. Manually register the old primary (on node 1) with the new primary after takeover (on node 2) as <sid>adm.

   suse01# hdbnsutil -sr_register --remoteHost=suse02 --remoteInstance=10 \
                  --replicationMode=sync  --operationMode=logreplay \
                  --name=WDF

4. Restart the HANA database (now secondary) on node 1 as root.

   suse01# crm resource refresh rsc_SAPHana_HA1_HDB10 suse01

Expected: #

 

1. The cluster detects the failed node (node 1) and declares it UNCLEAN and sets the secondary node (node 2) to status "partition with quorum".

2. The cluster fences the failed node (node 1).

3. The cluster declares the failed node (node 1) OFFLINE.

4. The cluster promotes the secondary HANA database (on node 2) to take over as primary.

5. The cluster migrates the IP address to the new primary (on node 2).

6. After some time, the cluster shows the sync_state of the stopped primary (on node 2) as SFAIL.

7. Because of AUTOMATED_REGISTER="false" the cluster does not restart the failed HANA database or register it against the new primary.

8. After the manual register and resource refresh, the system replication pair is marked as in sync (SOK).

9. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.1.6 Test: Crash Primary Node on Availability Zone B (Node 2) #

 

Example 45: Test CRASH_PRIMARY_NODE_SITE_B #

 

Component:

Cluster node of secondary site

Description:

Simulate a crash of the secondary site node running the primary HANA database.

Test Procedure: #

 

1. Crash the secondary node by sending a 'fast-reboot' system request.

   suse02# echo 'b' > /proc/sysrq-trigger

Recovery Procedure: #

 

1. AWS infrastructure has stopped the fenced instance. Restart it with AWS console or AWS CLI tools. Execute the following command after the instance has booted.

2. Start the cluster Framework

   suse02# systemctl start pacemaker

3. Manually register the old primary (on node 2) with the new primary after takeover (on node 1) as <sid>adm.

   suse02# hdbnsutil -sr_register --remoteHost=suse01 --remoteInstance=10 \
                  --replicationMode=sync  --operationMode=logreplay \
                  --name=ROT

4. Restart the HANA database (now secondary) on node 2 as root.

   suse02# crm resource refresh rsc_SAPHana_HA1_HDB10 suse02

Expected: #

 

1. The cluster detects the failed secondary node (node 2) and declares it UNCLEAN and sets the primary node (node 1) to status "partition with quorum".

2. The cluster fences the failed secondary node (node 2).

3. The cluster declares the failed secondary node (node 2) OFFLINE.

4. The cluster promotes the secondary HANA database (on node 1) to take over as primary.

5. The cluster migrates the IP address to the new primary (on node 1).

6. After some time, the cluster shows the sync_state of the stopped secondary (on node 2) as SFAIL.

7. Because of AUTOMATED_REGISTER="false" the cluster does not restart the failed HANA database or register it against the new primary.

8. After the manual register and resource refresh, the system replication pair is marked as in sync (SOK).

9. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.1.7 Test: Stop Secondary Database on Availability Zone B (Node 2) #

 

Example 46: Test STOP_SECONDARY_DB_SITE_B #

 

Component:

Secondary HANA database

Description:

The secondary HANA database is stopped during normal cluster operation.

Test Procedure: #

 

1. Stop the secondary HANA database gracefully as <sid>adm.

   suse02# HDB stop

Recovery Procedure: #

 

1. Refresh the failed resource status of the secondary HANA database (on node 2) as root.

   suse02# crm resource refresh rsc_SAPHana_HA1_HDB10 suse02

Expected: #

 

1. The cluster detects the stopped secondary database (on node 2) and marks the resource failed.

2. The cluster detects the broken system replication and marks it as failed (SFAIL).

3. The cluster restarts the secondary HANA database on the same node (node 2).

4. The cluster detects that the system replication is in sync again and marks it as ok (SOK).

5. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.1.8 Test: Crash Secondary Database on Availability Zone B (Node 2) #

 

Example 47: Test CRASH_SECONDARY_DB_SITE_B #

 

Component:

Secondary HANA database

Description:

Simulate a complete breakdown of the secondary database system.

Test Procedure: #

 

1. Kill the secondary database system using signals as <sid>adm.

   suse02# HDB kill-9

Recovery Procedure: #

 

1. Clean up the failed resource status of the secondary HANA database (on node 2) as root.

   suse02# crm resource refresh rsc_SAPHana_HA1_HDB10 suse02

Expected: #

 

1. The cluster detects the stopped secondary database (on node 2) and marks the resource failed.

2. The cluster detects the broken system replication and marks it as failed (SFAIL).

3. The cluster restarts the secondary HANA database on the same node (node 2).

4. The cluster detects that the system replication is in sync again and marks it as ok (SOK).

5. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.1.9 Test: Crash Secondary Node on Availability Zone B (Node2) #

 

Example 48: Test CRASH_SECONDARY_NODE_SITE_B #

 

Component:

Cluster node of secondary site

Description:

Simulate a crash of the secondary site node running the secondary HANA database.

Test Procedure: #

 

1. Crash the secondary node by sending a 'fast-reboot' system request.

   suse02# echo 'b' > /proc/sysrq-trigger

Recovery Procedure: #

 

1. AWS infrastructure has stopped the fenced instance. Restart it with AWS console or AWS CLI tools. Execute the following command after the instance has booted.

2. Start the cluster framework.

   suse02# systemctl start pacemaker

Expected: #

 

1. The cluster detects the failed secondary node (node 2) and declares it UNCLEAN and sets the primary node (node 1) to status "partition with quorum".

2. The cluster fences the failed secondary node (node 2).

3. The cluster declares the failed secondary node (node 2) OFFLINE.

4. After some time, the cluster shows the sync_state of the stopped secondary (on node 2) as SFAIL.

5. When the fenced node (node 2) rejoins the cluster, the former secondary HANA database is started automatically.

6. The cluster detects that the system replication is in sync again and marks it as ok (SOK).

10.3 Test Cases for Full Automation #

 

In the following test descriptions we assume PREFER_SITE_TAKEOVER="true"' and `AUTOMATED_REGISTER="true".

Note

The following tests are designed to run in a sequence. They depend on the exit state of the proceeding tests.

10.3.1 Test: Stop Primary Database on Site A #

 

Example 49: Test STOP_PRIMARY_DB_SITE_A #

 

Component: #

 

• Primary Database

Description: #

 

• The primary HANA database is stopped during normal cluster operation.

Test Procedure: #

 

• Stop the primary HANA database gracefully as <sid>adm.

suse01# HDB stop

Recovery Procedure: #

 

1. Not needed, everything is automated

2. Refresh the cluster resources on node 1 as root.

suse01# crm resource refresh rsc_SAPHana_HA1_HDB10 suse01

Expected: #

 

1. The cluster detects the stopped primary HANA database (on node 1) and marks the resource failed.

2. The cluster promotes the secondary HANA database (on node 2) to take over as primary.

3. The cluster migrates the IP address to the new primary (on node 2).

4. After some time, the cluster shows the sync_state of the stopped primary (on node 1) as SFAIL.

5. Because of AUTOMATED_REGISTER="true" the cluster does restart the failed HANA database and register it against the new primary.

6. After the automated register and resource refresh, the system replication pair is marked as in sync (SOK).

7. The cluster "failed actions" are cleaned up after following the recovery procedure.

10.3.2 Test: Crash the Primary Node on Site B (Node 2) #

 

Example 50: Test CRASH_PRIMARY_NODE_SITE_B #

 

Component: #

 

• Cluster node of site B

Description: #

 

• Simulate a crash of the site B node running the primary HANA database.

Test Procedure: #

 

• Crash the secondary node by sending a 'fast-reboot' system request.

suse02# echo 'b' > /proc/sysrq-trigger

Recovery Procedure: #

 

• Start the cluster framework.

suse02# systemctl start pacemaker

• Refresh the cluster resources on node 2 as root.

suse02# crm resource refresh rsc_SAPHana_HA1_HDB10 suse02

Expected: #

 

1. The cluster detects the failed primary node (node 2) and declares it UNCLEAN and sets the primary node (node 2) to status "partition with quorum".

2. The cluster fences the failed primary node (node 2).

3. The cluster declares the failed primary node (node 2) OFFLINE.

4. The cluster promotes the secondary HANA database (on node 1) to take over as primary.

5. The cluster migrates the IP address to the new primary (on node 1).

6. After some time, the cluster shows the sync_state of the stopped secondary (on node 2) as SFAIL.

7. When the fenced node (node 2) rejoins the cluster, the former primary becomes a secondary.

8. Because of AUTOMATED_REGISTER="true" the cluster does restart the failed HANA database and register it against the new primary.

9. The cluster detects that the system replication is in sync again and marks it as ok (SOK).

For more detailed information, have a look at the documents listed below.

Copyright © 2006–2024 SUSE LLC and contributors. All rights reserved.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled "GNU Free Documentation License".

SUSE, the SUSE logo and YaST are registered trademarks of SUSE LLC in the United States and other countries. For SUSE trademarks, see https://www.suse.com/company/legal/.

Linux is a registered trademark of Linus Torvalds. All other names or trademarks mentioned in this document may be trademarks or registered trademarks of their respective owners.

Documents published as part of the SUSE Best Practices series have been contributed voluntarily by SUSE employees and third parties. They are meant to serve as examples of how particular actions can be performed. They have been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. SUSE cannot verify that actions described in these documents do what is claimed or whether actions described have unintended consequences. SUSE LLC, its affiliates, the authors, and the translators may not be held liable for possible errors or the consequences thereof.

Below we draw your attention to the license under which the articles are published.

Copyright © 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work’s title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’s license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties—​for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

Copyright (c) YEAR YOUR NAME.
   Permission is granted to copy, distribute and/or modify this document
   under the terms of the GNU Free Documentation License, Version 1.2
   or any later version published by the Free Software Foundation;
   with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
   A copy of the license is included in the section entitled “GNU
   Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “ with…​Texts.” line with this:

with the Invariant Sections being LIST THEIR TITLES, with the
   Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.