SAP NetWeaver High Availability Cluster 7.40 for the AWS Cloud #

The EC2 instances will have host names which are automatically generated. Select host names which comply with SAP requirements, as detailed in SAP note 611361.

The cluster agents need to be able to identify the EC2 instances in the correct way. This happens through instance tags.

Tag the two EC2 instances through the console or the AWS Command Line Interface (CLI) with arbitrarily chosen tags like cluster and the host name as it will be shown in the command uname . Use the same tag (like cluster) and the individual host names for both instances. The AWS documentation (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) explains how to tag EC2 instances.

Note

Refrain from using non-ASCII characters in any tag assigned to cluster-managed resources.

Important

This section does not cover a discussion of SAP related ports in security groups, it only lists the ports which need to be available for the SUSE cluster only.

The following ports and protocols need to be configured to allow the two cluster nodes to communicate with each other:

We assume that there are no restrictions for outbound network communication.

The SUSE Linux Enterprise Server agents use the AWS Command Line Interface (CLI). This AWS CLI profile needs to be created for the root account root on both instances. The SUSE resources require a profile which creates output in text format. The name of the profile is arbitrary. The name chosen in this example is cluster. The region of the instance needs to be added as well. Replace the string region-name with your target region in the following example.

One way to create such a profile is to create a file /root/.aws/config with the following content:

[default]
region = region-name
[profile cluster]
region = region-name
output = text

The other way is to use the aws configure CLI command in the following way:

# aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: region-name
Default output format [None]:

# aws configure --profile cluster
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]: region-name
Default output format [None]: text

This command sequence generates a default profile and a cluster profile.

Configuring HTTP Proxies is not needed if the system has transparent access to the Internet. The resource agents execute the AWS CLI (Command Line Interface) commands. These commands send HTTP/HTTPS requests to an access point in the Internet. These access point are usually directly reachable.

However, systems which do not offer transparent Internet access will need to provide an HTTP/HTTPS proxy. The configuration of the proxy access is described in full detail in the AWS documentation.

Add the following environment variables to the root user’s .bashrc file:

export HTTP_PROXY=http://a.b.c.d:n
export HTTPS_PROXY=http://a.b.c.d:m

If authentication is required, add the following environment variables instead of the ones above:

export HTTP_PROXY=http://username:password@a.b.c.d:n
export HTTPS_PROXY=http://username:password@a.b.c.d:m

The AWS Data Provider for SAP needs to reach the instance meta data service directly. Add the following environment variable to the root user’s .bashrc file:

export NO_PROXY=169.254.169.254

The cluster configuration will require two IP addresses for each cluster instance. Adding a second IP address on the instance will allow the SUSE cluster to implement a two ring corosync configuration. The two ring corosync configuration will allow the cluster nodes to communicate with each other using the secondary IP address in the event that there is an issue communicating with each other over the primary IP address.

Please refer to AWS documentation to understand how to assign a secondary IP address: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html#assignIP-existing

After the secondary IP address is associated to the cluster instance in AWS, you will need to add the secondary IP address to the cluster instance. Update the file /etc/sysconfig/network/ifcfg-eth0. Replace XX.XX.XX.XX with the new secondary IP address and replace 'XX' with the two digit subnet mask.

IPADDR_1=‘XX.XX.XX.XX/XX'
LABEL_1="1"

The system will read the file and add the secondary IP address after the cluster instance is rebooted. Additionally, executing the command below as root will add the IP address to the cluster instance network stack with out rebooting.

ip address add XX.XX.XX.XX/XX dev eth0

Replace XX.XX.XX.XX with the new secondary IP address and replace 'XX' with the two digit subnet mask.

The source/destination check can be disabled through scripts using the AWS command line interface (AWS-CLI). The following command needs to be executed one time for both EC2 instances, which are supposed to receive traffic from the Overlay IP address:

# aws ec2 modify-instance-attribute --profile cluster --instance-id EC2-instance --no-source-dest-check

The system on which this command gets executed needs temporarily a role with the following policy:

{
   "Version": "2012-10-17",
   "Statement": [
   {
      "Sid": "Stmt1424870324000",
      "Effect": "Allow",
      "Action": [ "ec2:ModifyInstanceAttribute"],
      "Resource": [
      "arn:aws:ec2:region-name:account-id:instance/instance-a",
      "arn:aws:ec2:region-name:account-id:instance/instance-b"
      ]
   }
   ]
}

Replace the individual parameter for the region, the account identifier and the two identifiers for the EC2 instances with appropriate values.

The source/destination check can be disabled as well from the AWS console. It takes the execution of the following drop-down box in the console for both EC2 instances (see below).

Figure 1: Disable Source/Destination Check at Console #

 

SUSE Linux Enterprise Server 12 SP3 is the first version which ships the cloud-netconfig package. This package will remove any secondary IP address which is managed by the cluster agents from the eth0 interface. This can cause service interruptions for users of the HA service. Perform the following task on all cluster nodes:

Check whether the package cloud-netconfig-ec2 is installed with the command

# zypper info cloud-netconfig-ec2

Update the file /etc/sysconfig/network/ifcfg-eth0 if this package is installed. Change the following line to a „no“ setting or add the line if the package is not yet installed:

CLOUD_NETCONFIG_MANAGE='no'

The SAP ASCS and ESR will run the SUSE Linux Enterprise Server Pacemaker software and the agents. This software needs several AWS IAM privileges to operate the cluster. Create a new role for every ASCS/ESR cluster and associate this role to the two instances. Attach the policies detailed below to this role.

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "EC2:DescribeInstances",
                "EC2:DescribeVolumes"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "cloudwatch:GetMetricStatistics",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::aws-data-provider/config.properties"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1424870324000",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceAttribute",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1424870324001",
            "Effect": "Allow",
            "Action": [
                "ec2:ModifyInstanceAttribute",
                "ec2:RebootInstances",
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": [
                "arn:aws:ec2:region-name:aws-account:instance/i-node1",
                "arn:aws:ec2:region-name:aws-account:instance/i-node2"
            ]
        }
    ]
}

The Overlay IP agent changes a routing entry in an AWS routing table. Create a policy with a name like Manage-Overlay-IP-Policy and attach it to the role of the cluster instances:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:ReplaceRoute",
            "Resource": "arn:aws:ec2:region-name:account-id:route-table/rtb-XYZ"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "ec2:DescribeRouteTables",
            "Resource": "*"
        }
    ]
}

This policy allows the agent to update the routing tables which get used. Replace the following variables with the appropriate names:

Optionally, you can install the Route 53 agent in the cluster. The following policy is needed only when the Route 53 agent is used. Create a policy with the name Route53-Update and attach it to the role of the two cluster nodes:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1471878724000",
            "Effect": "Allow",
            "Action": "route53:GetChange",
            "Resource": "arn:aws:route53:::change/*"
        },
        {
            "Sid": "Stmt1471878724001",
            "Effect": "Allow",
            "Action": "route53:ChangeResourceRecordSets",
            "Resource": "arn:aws:route53:::hostedzone/hosted zone ID/full name"
        },
        {
            "Sid": "Stmt1471878724002",
            "Effect": "Allow",
            "Action": [
                "route53:ListResourceRecordSets",
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": "arn:aws:route53:::hostedzone/hosted zone ID"
        }
    ]
}

This policy is specific to the hosted zone and the resource record set. Replace the variables hosted zone ID with the AWS ID of the hosted zone. Replace full name with the name of the entry. An individual Route 53 policy needs to be created for every cluster.

Add the service IP address of the ASCS service (node hacert01). Use as filter /32 (example: 192.168.10.1/32). Add the Elastic Network Interface (ENI) name of your instance which is initially serving as ASCS. Save your changes by clicking “Save”.

This is the service IP address with the name sapha1as.

Add the service IP address of the ERS service (node hacert02). Use as filter /32 (example: 192.168.10.2/32). Add the Elastic Network Interface (ENI) name of your instance which is initially serving as ERS. Save your changes by clicking “Save”.

This is the IP address with the name sapha1er.

The two cluster instances need the Overlay IP address to be configured as secondary IP address on their standard interface eth0. This can be achieved by the command:

# ip address add OVERLAY-IP dev eth0

Execute this command with root privileges on both instances. Add the ASCS IP address on the ASCS node hacert01. Add the Enqueue Replication address on the ERS node hacert02.

A perfect enhancement of the five node scenario described in this document is to implement an SAP HANA system replication (SR) automation.

Figure 3: One cluster for central services, one for SAP HANA SR #

 

The following Databases are supported in combination with this scenario:

Create the following sub directories on both cluster nodes as root:

# mkdir -p /usr/sap/HA1/ASCS00 /usr/sap/HA1/ERS10

The file systems for the ASCS and ERS instances need to be shared and assigned to the cluster nodes hacert01 and hacert02. Create an EFS file system.

During the SAP installation we need the filesystems /usr/sap/HA1/ASCS00 to be mounted on hacert01 and /usr/sap/HA1/ERS10 to be mounted on hacert02.

hacert01: efs-name:/ASCS00 /usr/sap/HA1/ASCS00
hacert02: efs-name:/ERS10 /usr/sap/HA1/ERS10

Replace the variable efs-name with the appropriate DNS name of the EFS file system.

Note

hacert01 and hacert02 operate in different availability zones. They need mount points named "efs-name" which are individual to the availability zone. Use the DNS name provided by AWS. The DNS name will point to the files system mount point local to a given Availability Zone. During the SAP installation we need /usr/sap/HA1/ASCS00 to be mounted on hacert01 and /usr/sap/HA1/ERS10 to be mounted on hacert02.

On hacert01

# su - ha1adm
# sapcontrol -nr 00 -function Stop
# sapcontrol -nr 00 -function StopService

On hacert02

# su - ha1adm
# sapcontrol -nr 10 -function Stop
# sapcontrol -nr 10 -function StopService

Ensure that the file /usr/sap/sapservices holds both entries (ASCS+ERS) on both cluster nodes. Modify the file by copying the missing entry from both hosts across. Alternatively add the missing command string with sapstartsrv.

Example steps for ASCS profile on the ERS host:

# cd /usr/sap/hostctrl/exe
# export LD_LIBRARY_PATH=.
# ./sapstartsrv pf=/usr/sap/HA1/SYS/profile/HA1_ASCS00_sapha1as -reg

This allows the sapstartsrv clients to start the service like:

As user ha1adm

# sapcontrol -nr 10 -function StartService HA1

The /usr/sap/sapservices looks like (typically one line per instance):

#!/bin/sh
LD_LIBRARY_PATH=/usr/sap/HA1/ASCS00/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/HA1/ASCS00/exe/sapstartsrv pf=/usr/sap/HA1/SYS/profile/HA1_ASCS00_sapha1as -D -u ha1adm
LD_LIBRARY_PATH=/usr/sap/HA1/ERS10/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/HA1/ERS10/exe/sapstartsrv pf=/usr/sap/HA1/ERS10/profile/HA1_ERS10_sapha1er -D -u ha1adm

Install the package sap-suse-cluster-connector version >3.0.0 from our repositories:

# zypper install sap-suse-cluster-connector

Note

The package sap-suse-cluster-connector with version 3.0.x implements the SUSE SAP API version 3. New features like SAP Rolling Kernel Switch (RKS) and migration of ASCS are only supported with this new version.

For the ERS and ASCS instance edit the instance profile files HA1_ASCS00_sapha1as and HA1_ERS10_sapha1er in the profile directory /usr/sap/HA1/SYS/profile/.

You need to tell the sapstartsrv service to load the HA script connector library and to use the sap-suse-cluster-connector.

service/halib = $(DIR_EXECUTABLE)/saphascriptco.so
service/halib_cluster_connector = /usr/bin/sap_suse_cluster_connector

Add the user ha1adm to the unix user group haclient.

# usermod -a -G haclient ha1adm

For the ASCS, change the start command from Restart_Program_xx to Start_Program_xx for the enqueue server (enserver). This change tells the SAP start framework not to self-restart the enqueue process. Such a restart would lead in loss of the locks.

Start_Program_01 = local $(_EN) pf=$(_PF)

Optionally you could limit the number of restarts of services (in the case of ASCS this limits the restart of the message server).

For the ERS change instance the start command from Restart_Program_xx to Start_Program_xx for the enqueue replication server (enrepserver).

Start_Program_00 = local $(_ER) pf=$(_PFL) NR=$(SCSID)

On hacert01

# su - ha1adm
# sapcontrol -nr 00 -function StartService HA1
# sapcontrol -nr 00 -function Start

On hacert02

# su - ha1adm
# sapcontrol -nr 10 -function StartService HA1
# sapcontrol -nr 10 -function Start

SUSE recommends to use rsyslogd for logging with the SUSE cluster. This is a default configuration. Some AWS AMIs however use syslogd logging. Perform the following commands as root on all cluster nodes:

# zypper install rsyslog

Use option 1 (deinstallation of competing software, syslogd). Reboot both nodes.

The next step is to start the cluster with the command on both nodes:

# systemctl start pacemaker

The configuration can be checked with the command:

# corosync-cfgtool -s

It will create a result like the following one for a cluster node with the IP address 10.0.0.111:

Printing ring status.
Local node ID 1
RING ID 0
id = 10.0.0.111
status = ring 0 active with no faults

The cluster in question has been using ring 0, the node had the ID 1.

Stack: corosync
Current DC: hacert01 (version 1.1.15-19.15-e174ec8) - partition with quorum
Last updated: Wed Dec  6 16:02:42 2017
Last change: Wed Dec  6 15:44:45 2017 by hacluster via crmd on hacert01

2 nodes configured
0 resources configured

Online: [ hacert01 hacert02 ]

Full list of resources:

To avoid that the cluster starts partially defined resources we set the cluster to the maintenance mode. This deactivates all monitor actions.

As user root

# crm configure property maintenance-mode="true"

Execute the following commands on one of the two cluster nodes:

# vi crm-bs.txt

Enter the following information to the file crm-bs.txt:

property cib-bootstrap-options: \
    stonith-enabled="true" \
    stonith-action="off" \
    stonith-timeout="600s"
rsc_defaults rsc-options: \
	resource-stickiness=1 \
	migration-threshold=3
op_defaults op-options: \
	timeout=600 \
	record-pending=true

The setting off forces the agents to shut down the instance. This is desirable to avoid split brain scenarios on AWS.

Add the configuration to the cluster:

# crm configure load update crm-bs.txt

Create a file with the following content:

primitive res_AWS_STONITH stonith:external/ec2 \
op start interval=0 timeout=180 \
op stop interval=0 timeout=180 \
op monitor interval=300 timeout=60 \
params tag=pacemaker profile=cluster

The EC2 tag pacemaker entry needs to match the tag chosen for the EC2 instances. The value for this tag will contain the host name. The name of the profile (cluster in this example) will need to match the previously configured AWS profile.

Name this file for example aws-stonith.txt and add this file to the configuration. The following command needs to be issued as root. It uses the file name aws-stonith.txt:

# crm configure load update aws-stonith.txt

First we configure the resources for the file system, IP address and the SAP instance. Of course you need to adapt the parameters to your environment.

Create a file with your editor of choice with a name aws-ascs.txt. Add the ASCS primitive and the ASCS group to it. Do not forget to save your changes.

primitive rsc_fs_HA1_ASCS00 Filesystem \
    params  device="efs-name:/ASCS00" directory="/usr/sap/HA1/ASCS00" \
            fstype="nfs4" \
            options="rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2" \
    op start timeout=60s interval=0 \
    op stop timeout=60s interval=0 \
    op monitor interval=200s timeout=40s
primitive rsc_ip_HA1_ASCS00 ocf:suse:aws-vpc-move-ip \
    params  address=192.168.201.116 routing_table=rtb-table \
            interface=eth0 profile=cluster \
    op start interval=0 timeout=180 \
    op stop interval=0 timeout=180 \
    op monitor interval=60 timeout=60
primitive rsc_sap_HA1_ASCS00 SAPInstance \
    operations $id=rsc_sap_HA1_ASCS00-operations \
    op monitor interval=120 timeout=60 on-fail=restart \
    params InstanceName=HA1_ASCS00_sapha1as \
        START_PROFILE="/sapmnt/HA1/profile/HA1_ASCS00_sapha1as" \
        AUTOMATIC_RECOVER=false \
    meta resource-stickiness=5000 failure-timeout=60 \
        migration-threshold=1 priority=10

Replace the variable efs-name with the name of your EFS server.

Replace the variable rtb-table with the identifier of the appropriate AWS routing table for the subnets. The name of the AWS CLI profile (cluster in this example) will need to match the previously configured AWS profile.

group grp_HA1_ASCS00 \
  rsc_ip_HA1_ASCS00 rsc_fs_HA1_ASCS00 rsc_sap_HA1_ASCS00 \
	meta resource-stickiness=3000

Create a txt file aws_ascs.txt with your preferred text editor, enter both examples (primitives and group) to that file and load the configuration to the cluster manager configuration.

# crm configure load update aws_ascs.txt

Name this file for example aws-route53.txt and add this file to the configuration. The following command needs to be issued as root. It uses the file name aws-route53.txt:

Enter the following primitive before or after the existing primitives in the editor:

primitive rsc_r53_HA1_ASCS00 ocf:heartbeat:aws-vpc-route53 \
   params hostedzoneid=route-53-name ttl=10 fullname=name-full. profile=cluster \
   op start interval=0 timeout=180 \
   op stop interval=0 timeout=180 \
   op monitor interval=300 timeout=180

Replace the variable route-53-name with the name of the associated private hosted Route 53 zone.

Replace the variable name-full. will the fully qualified host name with matches the private hosted Route 53 zone.

The agent uses a time-to-live (ttl) of 10 seconds in this example. Change this parameter if needed.

Insert the rsc_r53_HA1_ASCS00 after the rsc_ip_HA1_ASCS00. This will force the group to update then Route 53 as second item after the Overlay IP address.

group grp_HA1_ASCS00 \
  rsc_ip_HA1_ASCS00 rsc_r53_HA1_ASCS00 \
  rsc_fs_HA1_ASCS00 rsc_sap_HA1_ASCS00 \
	meta resource-stickiness=3000

# crm configure load update aws-route53.txt

Note

Version 1.0.2 of the Route 53 agent will not work if the EC2 metadata contains a string like "local-ipv4" in the userdata section!

Second we configure the resources for the file system, IP address and the SAP instance. Of course you need to adapt the parameters to your environment.

Replace efs-name with the name of your EFS server.

The specific parameter IS_ERS=true should only be set for the ERS instance.

primitive rsc_fs_HA1_ERS10 Filesystem \
  params device="efs-name:/ERS10" directory="/usr/sap/HA1/ERS10" fstype=nfs4 \
  options="rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2" \
  op start timeout=60s interval=0 \
  op stop timeout=60s interval=0 \
  op monitor interval=200s timeout=40s
primitive rsc_ip_HA1_ERS10 ocf:suse:aws-vpc-move-ip \
  params address=192.168.201.117 routing_table=rtb-table \
  interface=eth0 profile=cluster \
  op start interval=0 timeout=180 \
  op stop interval=0 timeout=180 \
  op monitor interval=60 timeout=60
primitive rsc_sap_HA1_ERS10 SAPInstance \
  operations $id=rsc_sap_HA1_ERS10-operations \
  op monitor interval=120 timeout=60 on-fail=restart \
  params InstanceName=HA1_ERS10_sapha1er \
         START_PROFILE="/sapmnt/HA1/profile/HA1_ERS10_sapha1er" \
         AUTOMATIC_RECOVER=false IS_ERS=true \
  meta priority=1000

Replace the variable rtb-table with the identifier of the appropriate AWS routing table for the subnets. The name of the AWS CLI profile (cluster in this example) will need to match the previously configured AWS profile.

group grp_HA1_ERS10 \
  rsc_ip_HA1_ERS10 rsc_fs_HA1_ERS10 rsc_sap_HA1_ERS10

Create a txt file (like aws_crm_ers.txt) with your preferred text editor, enter both examples (primitives and group) to that file and load the configuration to the cluster manager configuration.

As user root

# crm configure load update aws_crm_ers.txt

The constraints between the ASCS and ERS instance are needed to define that the ASCS instance should start up exactly on the cluster node running the ERS instance after a failure (loc_sap_HA1_failover_to_ers). This constraint is needed to ensure that the locks are not lost after an ASCS instance (or node) failure.

If the ASCS instance has been started by the cluster the ERS instance should be moved to an "other" cluster node (col_sap_HA1_no_both). This constraint is needed to ensure that the ERS will synchronize the locks again and the cluster is ready for an additional take-over.

colocation col_sap_HA1_no_both -5000: grp_HA1_ERS10 grp_HA1_ASCS00
location loc_sap_HA1_failover_to_ers rsc_sap_HA1_ASCS00 \
         rule 2000: runs_ers_HA1 eq 1
order ord_sap_HA1_first_start_ascs Optional: rsc_sap_HA1_ASCS00:start \
      rsc_sap_HA1_ERS10:stop symmetrical=false

Create a text file (like crm_col.txt) with your preferred text editor, enter all three constraints to that file and load the configuration to the cluster manager configuration.

Issue the following command as root:

# crm configure load update crm_col.txt

Now the last step is to end the cluster maintenance mode and to allow the cluster to detect already running resources.

Issue the following command as root:

# crm configure property maintenance-mode="false"

The cluster will now start the ASCS and the ERS system. This can take a few minutes. Check progress with the command:

# crm status

For normal operation do not stop the ASCS SAP instance with any tool such as cluster tools or SAP tools. The stop of the ASCS instance might lead to a loss of enqueue locks. Because following the new SAP NW-HA-CLU 7.40 certification the cluster must allow local restarts of the ASCS. This feature is needed to allow Rolling Kernel Switch (RKS) updates without reconfiguring the cluster.

Warning

Stopping the ASCS instance might lead into the loss of SAP enqueue locks during the start of the ASCS on the same node.

To migrate the ASCS SAP instance you should use the SAP tools such as the SAP management console. This will trigger sapstartsrv to use the sap-suse-cluster-connector to migrate the ASCS instance. As user ha1adm you might call the following command to migrate-away the ASCS. The migrate-away will always migrate the ASCS to the ERS side which will keep the SAP enqueue locks.

As ha1adm

# sapcontrol -nr 00 -function HAFailoverToNode ""

With SAP NW-HA-CLU 7.40 it is not longer allowed to block resources from being controlled manually. This using the variable BLOCK_RESOURCES in /etc/sysconfig/sap_suse_cluster_connector is not allowed anymore.

Currently all SAP instance numbers controlled by the cluster must be unique. If you need to have multiple dialog instances such as D00 running on different systems they should be not controlled by the cluster.

Procedure to set the cluster into maintenance mode can be done as root or sidadm.

As user root

# crm configure properties maintenance-mode="true"

As user ha1adm (the full path is needed)

# /usr/sbin/crm configure properties maintenance-mode="true"

As user root

# crm configure properties maintenance-mode="false"

How to cleanup resource failures? Failures of the ASCS will be automatically deleted to allow a failback after the configured period of time. For all other resources you can cleanup the status including the failures:

As user root

# crm resource cleanup RESOURCE-NAME

Warning

You should not cleanup the complete group of the ASCS resource as this might lead into an unwanted cluster action to take over the complete group onto the node where ERS instance is running.

Check if the name of the SUSE cluster solution is shown in the output of sapcontrol or SAP management console. This test checks the status of the SAP NetWeaver cluster integration.

As user ha1adm

# sapcontrol -nr 00 -function HAGetFailoverConfig

Check if the HA configuration tests are showing no errors.

As user ha1adm

# sapcontrol -nr 00 -function HACheckConfig
# sapcontrol -nr 00 -function HACheckFailoverConfig

Check if manually migrating the ASCS using HA tools works properly.

As user root

# crm resource migrate rsc_sap_HA1_ASCS00 force
## wait till the ASCS is been migrated to the ERS host
# crm resource unmigrate rsc_sap_HA1_ASCS00

Check if moving the ASCS instance using SAP tools like sapcontrol does work properly.

As user ha1adm

# sapcontrol -nr 00 -function HAFailoverToNode ""

Check if the ASCS instance moves correctly after a node failure.

As user root

## on the ASCS host
# echo b >/proc/sysrq-trigger

Check if the inplace re-start of the SAP resources have been processed correctly. The SAP instance should not failover to an other node, it must start on the same node where it has been stopped.

Warning

This test will force the SAP system to lose the enqueue locks. This test should not be processed during production.

As user ha1adm

## example for ASCS
# sapcontrol -nr 00 -function Stop
## wait till the ASCS is completely down
# sapcontrol -nr 00 -function Start

Standard SAP on AWS for an HA setup is Multi-AZ deployment with ASCS, Primary DB running in one AZ and their counterpart ERS and Secondary DB running in the second AZ of the same region. The Primary Application Server & Additional Application servers based on the load can be distributed in both AZ’s as well to provide resiliency. Considering a scenario where SAP NetWeaver or Business Suite system is running on SAP Sybase ASE. The completely automated HA for the ABAP Stack (ASCS) is provided by the SUSE Linux Enterprise High Availability Extension Cluster. For the Sybase ASE DB the HA feature is provided with the Always On configuration and the failover orchestration is done by the Fault Manager (FM) utility which traditionally was installed on a third host (other than the Primary & Secondary DB). In an SAP world the FM utility comes along with an SAP DB dependent Kernel and gets installed in the ASCS Work directory /usr/sap/<SID>/ASCS<instnr>/exe/. The failover of the ASCS instance along with the associated directories (provided they are installed on a shared file system using either Amazon EFS or NFS) is taken care by the SUSE Linux Enterprise High Availability Extension Cluster.

SAP Sybase ASE comes with an Always On feature which provides native HA & DR capability. The always-on option is a high availability and disaster recovery (HADR) system that consists of two SAP ASE servers: One is designated as the primary server, on which all transaction processing takes place. The other acts as a warm standby (called "standby server" in DR mode, and as a "companion" in HA mode) for the primary server, and contains copies of designated databases from the primary server. The failover orchestration is carried out by ASE provided utility called Fault Manager. The Fault Manager monitors the various components of the HADR environment – Replication Management Agent (RMA), ASE, Replication Server, applications, databases, and the operating system. Its primary job is to ensure the high-availability (zero data loss during failover) of the ASE cluster by initiating automatic failover with minimal manual intervention. In an SAP Stack, the fault manager utility (sybdbfm) comes as part of the DB (Sybase ASE) dependent SAP Kernel. Refer to the SAP Standard ASE HA-DR guide (https://help.sap.com/viewer/efe56ad3cad0467d837c8ff1ac6ba75c/16.0.3.6/en-US/a6645e28bc2b1014b54b8815a64b87ba.html) for setting up the Sybase ASE DB in HA mode.

Important

In the following section we use sometimes examples and sometimes general examples. In the general are terms like <SID>; <instance nr>. They must be adapted to your environment. As an example, su - <sid>adm means su - ha7adm or in capital letters cd /usr/sap/<SID>/ASCS<instance nr>/work means cd /usr/sap/HA7/ASCS00/work

This guide does not duplicate the official HADR documentation. The following procedure describes the key points which you need to take care of.

# zypper install glibc-32bit libgcc_s1-32bit

For the example this software stack is used:

Note

Very useful is that short table of installation information which helps to be prepared for the next steps. SAP Adaptive Server Enterprise - Installation Worksheet https://help.sap.com/viewer/efe56ad3cad0467d837c8ff1ac6ba75c/16.0.3.6/en-US/3fe35550f3814b2bb411d5494976e25a.html

Important

The Fault Manager is enhanced to work in this setup. The minimal version’s which support this scenario are * SAP Kernel 749 PL632 * SAP Kernel 753 PL421

The installation can be done with the SWPM which is provided by SAP.

The following information is requested from the wizard:

After the basis installation is finished the primary database must be prepared for the replication. First the user sa must be unlocked.

# su - syb<sid>
# isql -Usapsso -P <secure password> -S<SID> -X -w1900
# 1> go
# 1> exec sp_locklogin sa, 'unlock'
# 2> go
# Account unlocked.
# (return status = 0)
# 1> quit

In the next step, install the SRS software with a response file and enter the following command as user syb<sid>:

# /sapcd/ase-16.0.03.06/BD_SYBASE_ASE_16.0.03.06_RDBMS_for_BS_/SYBASE_LINUX_X86_64/setup.bin -f /sybase/HA7/srs-setup.txt -i silent

Activate HADR on primary node with a response file and enter the following command as user syb<sid>:

# setuphadr /sybase/HA7/HA7_primary_lin.rs.txt

Note

If the installation stops with an error message as displayed here, perform the steps explained below:

Clean up environment.
Environment cleaned up.
Error: Fail to connect to "PRIM" site SAP ASE at "<hostname>:4901".

Check if the host name and port number are correct and the database server is up and running. If everything is correct and network connection should be available, it might help to modify the interface file. Try to add a new line in the /sybase/<SID>/interfaces file for the <SID> section with the IP address of the corresponding host name.

# vi /sybase/<SID>/interfaces
...
	master tcp ether <hostname> 4901
	master tcp ether 172.17.1.21 4901
...

Create a secure store key entry for the database:

# /usr/sap/hostctrl/exe/saphostctrl -user sapadm <secure password> -function LiveDatabaseUpdate -dbname <SID> -dbtype syb -dbuser DR_admin -dbpass <Secure password> -updatemethod Execute -updateoption TASK=SET_USER_PASSWORD -updateoption USER=DR_ADMIN

The following information is requested from the wizard:

After the basis installation is finished the companion database must be prepared for the replication. First the user sa must be unlocked.

# su - syb<sid>
# isql -Usapsso -P <secure password> -S<SID> -X -w1900
# 1> go
# 1> exec sp_locklogin sa, 'unlock'
# 2> go
# Account unlocked.
# (return status = 0)
# 1> quit

Next step installing the SRS software with a response file on the companion site and enter the following command as user syb<sid>:

# /sapcd/ase-16.0.03.06/BD_SYBASE_ASE_16.0.03.06_RDBMS_for_BS_/SYBASE_LINUX_X86_64/setup.bin -f /sybase/HA7/srs-setup.txt -i silent

Activate HADR on companion node with a response file and enter the following command as user syb<sid>:

# setuphadr /sybase/HA7/HA7_companion_lin.rs.txt

Note

In certain circumstances the installation is not successful. It could help to set up the primary system again and install the companion afterward.

Note

If the system is reinstalled and the companion system reports Missing read/write permissions for this directory /tmp/.SQLAnywhere, check the permission on both node. In case the ownership must be changed run the setup again on both nodes. Start with the Primary.

Creating a secure store key entry for the database:

# /usr/sap/hostctrl/exe/saphostctrl -user sapadm <secure password> -function LiveDatabaseUpdate -dbname <SID> -dbtype syb -dbuser DR_admin -dbpass <Secure password> -updatemethod Execute -updateoption TASK=SET_USER_PASSWORD -updateoption USER=DR_ADMIN

The Fault Manager is configured on the ASCS host. The benefit from this setup is that the sybdbfm service can be monitored and tracked with the existing pacemaker for the ASCS / ERS replication.

# su - <sid>adm
# cd /usr/sap/<SID>/ASCS00/exe/
# sybdbfm install

replication manager agent user DR_admin and password set in Secure Store.
Keep existing values (yes/no)? (yes)
SAPHostAgent connect user sapadm and password set in Secure Store.
Keep existing values (yes/no)? (yes)
Enter value for primary database host: (suse7cl1)
suse7db1
Enter value for primary database name: (HA7)
Enter value for primary database port: (4901)
Enter value for primary site name: (Site1)
Enter value for primary database heart beat port: (13777)
Enter value for standby database host: (suse7cl1)
suse7db2
Enter value for standby database name: (HA7)
Enter value for standby database port: (4901)
Enter value for standby site name : (Site2)
Enter value for standby database heart beat port: (13787)
Enter value for fault manager host: (s7as-service)
Enter value for heart beat to heart beat port: (13797)
Enter value for support for floating database ip: (no)
Enter value for use SAP ASE Cockpit if it is installed and running: (no)

Update the values as per your environment for the Primary DB & companion DB host name, SID & Site Name. Make sure to use the virtual host name for the ASCS host. When the Fault Manager is installed, profile for it will be created in the /sapmnt/<SID>/profile by the name SYBHA.PFL and will have the configuration details. Restart the ASCS Instance which will also start the Fault Manager that has been added to the start profile as below:

Note

In case of a reinstallation it might be better to overwrite the existing user name and password in the secure store for the sapadm and DR_admin if the old values are not 100% known.

# cat /sapmnt/<SID>/profile/<SID>_ASCS00_<virt. ASCS hostname>
....
#-----------------------------------------------------------------------
# copy sybdbfm and dependent
#-----------------------------------------------------------------------
_CP_SYBDBFM_ARG1 = list:$(DIR_CT_RUN)/instancedb.lst
Execute_06 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CP_SYBDBFM_ARG1)
_CP_SYBDBFM_ARG2 = list:$(DIR_GLOBAL)/syb/linuxx86_64/cpe_sybodbc.lst
_CP_SYBDBFM_ARG3 = source:$(DIR_GLOBAL)/syb/linuxx86_64/sybodbc
Execute_07 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CP_SYBDBFM_ARG2) $(_CP_SYBDBFM_ARG3)
#-----------------------------------------------------------------------
# Start sybha
#-----------------------------------------------------------------------
_SYBHAD = sybdbfm.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)
_SYBHA_PF = $(DIR_PROFILE)/SYBHA.PFL
Execute_08 = local rm -f $(_SYBHAD)
Execute_09 = local ln -s -f $(DIR_EXECUTABLE)/sybdbfm$(FT_EXE) $(_SYBHAD)
Restart_Program_02 = local $(_SYBHAD) hadm pf=$(_SYBHA_PF)
#-----------------------------------------------------------------------
....

The status of the FM can be checked as below. Navigate to the ASCS work directory and then run sybdbfm.sap.<SID>_ASCS<instance number> status :

As user ha1adm

# cd /usr/sap/<SID>/ASCS<instance number>/work
# ./sybdbfm.sap<SID>_ASCS<instance number> status

fault manager running, pid = 23234, fault manager overall status = OK, currently executing in mode PAUSING
sanity check report (208).
node 1: server <DB server1>, site <site name one>.
db host status: OK.
db status OK hadr status PRIMARY.
node 2: server <DB server2>, site <site name two>.
db host status: OK.
db status OK hadr status STANDBY.
replication status: SYNC_OK.

Checking the log file is also a suitable meassure to validate the status.

As user ha1adm

# cd /usr/sap/<SID>/ASCS<instance number>/work
# tail -f dev_sybdbfm
# ...

2019 09/11 15:34:30.523 (23234) ----- Log messages ----

2019 09/11 15:34:30.523 (23234) Info: saphostcontrol: Executing LiveDatabaseUpdate

2019 09/11 15:34:30.523 (23234) Info: saphostcontrol: LiveDatabaseUpdate successfully executed

2019 09/11 15:34:30.524 (23234) call is running.
2019 09/11 15:34:30.534 (23234) call exited (exit code 0).
2019 09/11 15:34:30.534 (23234) db status is:
 DB_OK.
2019 09/11 15:34:42.561 (23234) *** sanity check report (136)***.
2019 09/11 15:34:42.562 (23234) node 1: server <DB server1>, site <site name one>.
2019 09/11 15:34:42.562 (23234) db host status: OK.
2019 09/11 15:34:42.562 (23234) db status OK hadr status PRIMARY.
2019 09/11 15:34:42.562 (23234) node 2: server <DB server2>, site <site name two>.
2019 09/11 15:34:42.562 (23234) db host status: OK.
2019 09/11 15:34:42.562 (23234) db status OK hadr status STANDBY.
2019 09/11 15:34:42.562 (23234) replication status: SYNC_OK.
2019 09/11 15:34:57.688 (23234) *** sanity check report (137)***.
2019 09/11 15:34:57.688 (23234) node 1: server <DB server1>, site <site name one>.
2019 09/11 15:34:57.688 (23234) db host status: OK.
2019 09/11 15:34:57.688 (23234) db status OK hadr status PRIMARY.
2019 09/11 15:34:57.688 (23234) node 2: server <DB server2>, site <site name two>.
2019 09/11 15:34:57.688 (23234) db host status: OK.
2019 09/11 15:34:57.688 (23234) db status OK hadr status STANDBY.
2019 09/11 15:34:57.688 (23234) replication status: SYNC_OK.
2019 09/11 15:35:12.827 (23234) *** sanity check report (138)***.
2019 09/11 15:35:12.827 (23234) node 1: server <DB server1>, site <site name one>.
2019 09/11 15:35:12.827 (23234) db host status: OK.
2019 09/11 15:35:12.827 (23234) db status OK hadr status PRIMARY.
2019 09/11 15:35:12.827 (23234) node 2: server <DB server2>, site <site name two>.
2019 09/11 15:35:12.827 (23234) db host status: OK.
2019 09/11 15:35:12.827 (23234) db status OK hadr status STANDBY.
2019 09/11 15:35:12.827 (23234) replication status: SYNC_OK.
# ...

Below are a few parameters that need to be updated in the SYBHA.PFL to make the failover working.

ha/syb/support_cluster = 1
ha/syb/failover_if_unresponsive = 1
ha/syb/allow_restart_companion = 1
ha/syb/set_standby_available_after_failover = 1
ha/syb/chk_restart_repserver = 1
ha/syb/cluster_fmhost1 = Hostname for Node 1 of the ASCS HA Setup
ha/syb/cluster_fmhost2 = Hostname for Node 2 of the ASCS HA Setup
ha/syb/use_boot_file_always = 1
ha/syb/dbfmhost = virtual hostname of ASCS instance

Details of all the FM parameters can be found in the SAP ASE HA DR User Guide. Those highlighted in bold are of interest for the setup. Since the FM is installed with the ASCS which can failover from Node 1 to Node 2, the parameters ha/syb/cluster_fmhost1 and ha/syb/cluster_fmhost2 provide the physical host names of both nodes where the FM can potentially run.

In a scenario where the complete Availability Zone (AZ1) goes down and the ASCS and Primary database are running there, the DB failover is not triggered until the ASCS failover is complete and the FM is up and running in the 2nd Availability Zone (AZ2). The FM then needs to read the boot file to get the prior state of the ASE DB. This is mandatory to ensure that FM can trigger the failover correctly. The parameter ha/syb/use_boot_file_always=1 makes sure that the FM always reads from the boot file which is part of the work directory (the same for ASCS and FM) and failover along with FM.

Important for each high availability solution is an extensive testing procedure. That ensures that the solution is working as expected in case of a failure.

# cd /usr/sap/<SID>/ASCS<instance nr>/work
# ./sybdbfm.sap<SID>_ASCS<instance nr> status

2019 09/13 09:58:52.200 (3290) *** sanity check report (2)***.
2019 09/13 09:58:52.200 (3290) node 1: server sapdb1, site LeonRot.
2019 09/13 09:58:52.200 (3290) db host status: OK.
2019 09/13 09:58:52.200 (3290) db status OK hadr status STANDBY.
2019 09/13 09:58:52.200 (3290) node 2: server sapdb2, site Orlando.
2019 09/13 09:58:52.201 (3290) db host status: OK.
2019 09/13 09dbs_syb_server sapdb1:sapdb2:58:52.201 (3290) db status OK hadr status PRIMARY.
2019 09/13 09:58:52.201 (3290) replication status: SYNC_OK.

# cd /usr/sap/<SID>/ASCS<instance nr>/work
# tail -f  dev_sybdbfm

...
    2019 09/13 11:08:38.301 (3290)  * sanity check report (270)* .
    2019 09/13 11:08:38.301 (3290) node 1: server sapdb1, site LeonRot.
    2019 09/13 11:08:38.301 (3290) db host status: OK.
    2019 09/13 11:08:38.301 (3290) db status OK hadr status STANDBY.
    2019 09/13 11:08:38.301 (3290) node 2: server sapdb2, site Orlando.
    2019 09/13 11:08:38.301 (3290) db host status: OK.
    2019 09/13 11:08:38.301 (3290) db status OK hadr status PRIMARY.
    2019 09/13 11:08:38.301 (3290) replication status: SYNC_OK.
    2019 09/13 11:08:50.416 (3290) ERROR in function SimpleFetch (1832) (SQLExecDirect failed): (30046) [08S01] [SAP][ASE ODBC Driver]Connection to the server has been lost. Unresponsive Connection was disconnected during command timeout. Check the server to determine the status of any open transactions.
    2019 09/13 11:08:50.416 (3290) ERROR in function SimpleFetch (1832) (SQLExecDirect failed): (30149) [HYT00] [SAP][ASE ODBC Driver]The command has timed out.
    2019 09/13 11:08:50.416 (3290) execution of statement master..sp_hadr_admin get_request, '1' failed.
    2019 09/13 11:08:50.416 (3290) ERROR in function SimpleFetch (1824) (SQLAllocStmt failed): (30102) [HY010] [SAP][ASE ODBC Driver]Function sequence error
    2019 09/13 11:08:50.416 (3290) execution of statement select top 1 convert( varchar(10), @@hadr_mode ) || ' ' || convert( varchar(10), @@hadr_state ) from sysobjects failed.
    2019 09/13 11:08:50.416 (3290) disconnect connection
    2019 09/13 11:09:22.505 (3290) ERROR in function SQLConnectWithRetry (1341) (SQLConnectWithRetry failed): (30293) [HY000] [SAP][ASE ODBC Driver]The socket failed to connect within the timeout specified.
    2019 09/13 11:09:22.505 (3290) ERROR in function SQLConnectWithRetry (1341) (SQLConnectWithRetry failed): (30012) [08001] [SAP][ASE ODBC Driver]Client unable to establish a connection
    2019 09/13 11:09:22.505 (3290) connected with warnings (555E69805100)
    2019 09/13 11:09:22.505 (3290) ERROR in function SimpleFetch (1824) (SQLAllocStmt failed): (30293) [HY000] [SAP][ASE ODBC Driver]The socket failed to connect within the timeout specified.
    2019 09/13 11:09:22.505 (3290) ERROR in function SimpleFetch (1824) (SQLAllocStmt failed): (30012) [08001] [SAP][ASE ODBC Driver]Client unable to establish a connection
    2019 09/13 11:09:22.505 (3290) execution of statement select top 1 convert( varchar(10), @@hadr_mode ) || ' ' || convert( varchar(10), @@hadr_state ) from sysobjects failed.
    2019 09/13 11:09:22.505 (3290) disconnect connection
    2019 09/13 11:09:22.505 (3290) primary site unusable.
...
    2019 09/13 11:09:22.984 (3290) primary site unusable.
    2019 09/13 11:09:22.984 (3290)  * sanity check report (271)* .
    2019 09/13 11:09:22.984 (3290) node 1: server sapdb1, site LeonRot.
    2019 09/13 11:09:22.984 (3290) db host status: OK.
    2019 09/13 11:09:22.984 (3290) db status OK hadr status STANDBY.
    2019 09/13 11:09:22.984 (3290) node 2: server sapdb2, site Orlando.
    2019 09/13 11:09:22.984 (3290) db host status: UNUSABLE.
    2019 09/13 11:09:22.984 (3290) db status DB INDOUBT hadr status UNREACHABLE.
    2019 09/13 11:09:22.984 (3290) replication status: SYNC_OK.
    2019 09/13 11:09:23.047 (3290) doAction: Primary database is declared dead or unusable.
    2019 09/13 11:09:23.047 (3290) disconnect connection
    2019 09/13 11:09:23.047 (3290) database host cannot be reached.
    2019 09/13 11:09:23.047 (3290) doAction: failover.
...
    2019 09/13 11:11:55.497 (3290)  * sanity check report (273)* .
    2019 09/13 11:11:55.497 (3290) node 1: server sapdb1, site LeonRot.
    2019 09/13 11:11:55.497 (3290) db host status: OK.
    2019 09/13 11:11:55.497 (3290) db status OK hadr status PRIMARY.
    2019 09/13 11:11:55.497 (3290) node 2: server sapdb2, site Orlando.
    2019 09/13 11:11:55.497 (3290) db host status: UNUSABLE.
    2019 09/13 11:11:55.498 (3290) db status DB INDOUBT hadr status UNREACHABLE.
    2019 09/13 11:11:55.498 (3290) replication status: UNKNOWN.
    2019 09/13 11:11:55.555 (3290) doAction: Standby database is declared dead or unusable.
    2019 09/13 11:11:55.555 (3290) disconnect connection
    2019 09/13 11:11:55.555 (3290) doAction: Companion db host is declared unusable.
    2019 09/13 11:11:55.555 (3290) doAction: no action defined.
    2019 09/13 11:11:58.568 (3290) Error: NIECONN_REFUSED (No route to host), NiRawConnect failed in plugin_fopen()
...
 host is coming back online ##
    2019 09/13 11:18:45.579 (3290) call is running.
    2019 09/13 11:18:45.589 (3290) call exited (exit code 0).
    2019 09/13 11:18:45.589 (3290) db status is: DB_OK.
    2019 09/13 11:18:45.589 (3290) doAction: Standby database is declared dead or unusable.
    2019 09/13 11:18:45.589 (3290) disconnect connection
    2019 09/13 11:18:45.589 (3290) doAction: Companion db host is declared ok.
    2019 09/13 11:18:45.589 (3290) doAction: restart database.
    2019 09/13 11:18:45.805 (3290) Webmethod returned successfully
...
    2019 09/13 11:22:43.677 (3290)  * sanity check report (286)* .
    2019 09/13 11:22:43.677 (3290) node 1: server sapdb1, site LeonRot.
    2019 09/13 11:22:43.677 (3290) db host status: OK.
    2019 09/13 11:22:43.677 (3290) db status OK hadr status PRIMARY.
    2019 09/13 11:22:43.677 (3290) node 2: server sapdb2, site Orlando.
    2019 09/13 11:22:43.677 (3290) db host status: OK.
    2019 09/13 11:22:43.677 (3290) db status OK hadr status STANDBY.
    2019 09/13 11:22:43.677 (3290) replication status: SYNC_OK.
...

# /usr/sap/hostctrl/exe/saphostctrl -user sapadm <secure password> -dbname <SID> -dbtype syb -function LiveDatabaseUpdate -updatemethod Check -updateoption TASK=REPLICATION_STATUS

Webmethod returned successfully
Operation ID: 5254001F87CB1EE9B5C34755C99DDDFA

----- Response data ----
TASK_NAME=REPLICATION_STATUS
REPLICATION_STATUS=active
PRIMARY_SITE=<site1>
STANDBY_SITE=<site2>
REPLICATION_MODE=sync
ASE transaction log backlog (MB)=0
Replication queue backlog (MB)=0
TASK_STATUS=OK
----- Log messages ----
Info: saphostcontrol: Executing LiveDatabaseUpdate
Info: saphostcontrol: LiveDatabaseUpdate successfully executed

Example 21: Triggering an FM Failure #

 

Killing the FM process more than five times will bring pacemaker in action. Up to five times the saphostagent will take care of the SAP process. If this fail-count is reached in a specific time window, the service will not be restarted.

As user ha1adm

# pkill -9 sybdbfm
## check that the PID has changed
# sapcontrol -nr 00 -function GetProcessList
# pkill -9 sybdbfm
...
# sapcontrol -nr 00 -function GetProcessList
...
sybdbfm, , GRAY, Stopped, , , 11154
...

Now pacemaker will restart the FM instance locally first. As user root

# crm_mon -1rfn
...
Migration Summary:
* Node <hostname>:
rsc_sap_WAS_ASCS00: migration-threshold=3 fail-count=1 last-failure='Fri Sep 13 13:46:39 2019
...

Note

If the fail-count reaches the defined threshold, the ASCS is moved away from that host.