Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Linux Enterprise Server for SAP Applications 15 SP2

8 Tuning Edit source

This chapter presents information about tuning SUSE Linux Enterprise Server for SAP Applications to work optimally with SAP applications.

On SUSE Linux Enterprise Server for SAP Applications you have the choice between sapconf and saptune. However, saptune is the more elaborate tool that offers more features.

Note
Note: The sapconf command has been removed

In SUSE Linux Enterprise Server and SUSE Linux Enterprise Server for SAP Applications 11 and 12, the sapconf command was included in the package with the same name.

For SUSE Linux Enterprise Server and SUSE Linux Enterprise Server for SAP Applications 15 this has been changed: the command sapconf have been removed from the sapconf package. The package contains a systemd service only. There is no sapconf command line tool anymore, no sapconf/tuned profiles, and no tuned.

8.1 Tuning systems with sapconf 4 Edit source

The package sapconf is available in SUSE Linux Enterprise Server and SUSE Linux Enterprise Server for SAP Applications. This package contains the tuned profile sapconf. This single tuning profile sets recommended parameters for the following types of SAP applications: SAP NetWeaver, SAP HANA and SAP HANA-based applications.

Overview of sapconf4 in SUSE® Linux Enterprise Server 12
sapconf4 (tuned based)
  • sap-netweaver (tuned profile)

  • sap-hana (tuned profile)

  • sap-bobj (tuned profile)

  • sap-ase (tuned profile)

Overview of sapconf4 in SUSE® Linux Enterprise Server 15
sapconf4 (tuned based)

sapconf (tuned profile)

Note that if you previously made changes to the system tuning, those changes may be overwritten by the sapconf profile.

sapconf consists of two primary parts:

  • A systemd service that ensures tuned and related services are running and the sapconf profile is applied.

  • The tuned profile sapconf that applies configured sapconf tuning parameters using a script and configuration files.

To use sapconf, make sure that the packages tuned and sapconf are installed on your system.

Note
Note: Unified profiles in SUSE Linux Enterprise Server and SUSE Linux Enterprise Server for SAP Applications 15 SP2

In SUSE Linux Enterprise Server and SUSE Linux Enterprise Server for SAP Applications 15 and above, only a single tuned profile, sapconf, is shipped. It is equivalent to the profiles sap-hana/sap-netweaver shipped in earlier versions of SUSE Linux Enterprise Server for SAP Applications.

8.1.1 Enabling and disabling sapconf and viewing its status Edit source

After the installation of sapconf, tuned is enabled and the sapconf profile is activated. However, if another tuned profile is already enabled, sapconf will not enable its own tuned profile.

To make sure sapconf applies all tuning parameters, reboot the machine after installation.

You can inspect or change the status of sapconf as described in the following:

  • To see the status of the service sapconf:

    root # systemctl status sapconf

    The service should be displayed as active (exited), as it is only responsible for starting tuned and will exit afterward.

  • To start the service sapconf and with it the service tuned:

    root # systemctl start sapconf
  • Should sapconf be disabled, enable and start it with:

    root # systemctl enable --now sapconf
  • To stop the service sapconf and with it the service tuned:

    root # systemctl stop sapconf

    This will terminate tuned as well, therefore the vast majority of optimizations will be disabled immediately. The only exceptions from that are options that require a system reboot to enable/disable.

  • To disable sapconf, use:

    root # systemctl disable sapconf

    If you have not specifically enabled any of the services that sapconf depends on yourself, this will also disable most tuning parameters and all services used by sapconf.

Similarly, you can inspect and change the status of the underlying service tuned:

  • To see the status of the service tuned:

    root # systemctl status tuned
  • To see which tuned profile is currently in use:

    root # tuned-adm active

    If this command does not return the name of the currently active profile as sapconf, enable that profile:

    root # tuned-adm profile sapconf
Tip
Tip: Additional services that sapconf relies on

In addition to the sapconf service itself and the tuned service, sapconf also relies on the following two services:

  • sysstat which collects data on system activity.

  • uuidd which generates time-based UUIDs that are guaranteed to be unique even in settings where many processor cores are involved. This is necessary for SAP applications.

8.1.2 Configuring sapconf4 Edit source

In general, the default configuration of sapconf already uses the parameter values recommended by SAP. However, if you have special needs, you can configure the tool to better suit those.

The configuration of sapconf is split into two parts that can be configured in different ways:

/usr/lib/tuned/PROFILE/tuned.conf

Any file that adheres to this pattern can be edited like in Procedure 8.1, “Configuring sapconf4 profiles”. To configure parameters from this file, copy it to the custom profile directory of tuned under /etc/tuned first and then change values in it. If you change the file in place instead, you will lose the changes you make on the next update of the sapconf package.

The following procedure shows an example how to adapt the file /usr/lib/tuned/sapconf/tuned.conf. However, as written before, this is possible with any profile. Configure the file as described in the following procedure:

Procedure 8.1: Configuring sapconf4 profiles
  1. Create a new custom tuned profile directory and copy the file tuned.conf:

    root # mkdir /etc/tuned/sapconf
    root # cp /usr/lib/tuned/sapconf/tuned.conf /etc/tuned/sapconf/
  2. Within the newly copied tuned.conf, fix the reference to script.sh to use an absolute path that points to the script from the original profile:

    script = /usr/lib/tuned/sapconf/script.sh

    Do not instead copy script.sh, as that provokes update compatibility issues for sapconf.

  3. Edit the parameters in /etc/tuned/sapconf/tuned.conf.

After each update to sapconf, make sure to compare the contents of the original and the custom tuned.conf.

Log messages related to this file are written to /var/log/tuned/tuned.log.

/etc/sysconfig/sapconf

This file contains most parameters of sapconf. The parameters from this file are applied using the aforementioned script /usr/lib/tuned/sapconf/script.sh.

This file can be edited directly. All parameters in this file are explained by means of comments and references to SAP Notes which can be viewed at https://launchpad.support.sap.com/.

When sapconf is updated, all customized parameters from this file will be preserved as much as possible. However, sometimes parameters cannot be transferred cleanly to the new configuration file. Therefore, after updating it is advisable to check the difference between the previous custom configuration which during the update is moved to /etc/sysconfig/sapconf.rpmsave and the new version at /etc/sysconfig/sapconf.

Log messages related to this file are written to /var/log/sapconf.log.

When editing either of these files, you will find that some values are commented by means of a # character at the beginning of the line. This means that while the parameter is relevant for tuning, there is no suitable default for it.

Conversely, you can add # characters to the beginning of the line to comment specific parameters. However, you should avoid this practice, as it can lead to sapconf not properly applying the profile.

To apply edited configuration, restart sapconf:

root # systemctl restart sapconf

Confirming that a certain parameter value was applied correctly works differently for different parameters. Hence, the following serves as an example only:

Example 8.1: Checking parameters

To confirm that the setting for TCP_SLOW_START was applied, do the following:

  • View the log file of sapconf to see whether it applied the value. Within /var/log/sapconf.log, check for a line containing this text:

    Change net.ipv4.tcp_slow_start_after_idle from 1 to 0

    Alternatively, the parameter may have already been set correctly before sapconf was started. In this case, sapconf will not change its value:

    Leaving net.ipv4.tcp_slow_start_after_idle unchanged at 1
  • The underlying option behind TCP_SLOW_START can be manually configured at /proc/sys/net.ipv4.tcp_slow_start_after_idle. To check its actual current value, use:

    root # sysctl net.ipv4.tcp_slow_start_after_idle

8.1.3 Removing sapconf Edit source

To remove sapconf from a system, uninstall its package with:

root # zypper rm sapconf

Note that when doing this, dependencies of sapconf will remain installed. However, the services sysstat and tuned will go into a disabled state. If either is still relevant to you, make sure to enable it again.

Certain parameters and files are not removed when sapconf is uninstalled. For more information, see the man page man 7 sapconf, section PACKAGE REQUIREMENTS.

8.1.4 For more information Edit source

The following man pages provide additional information about sapconf:

  • High-level overview of tuning parameters used by sapconf: man 7 tuned-profiles-sapconf

  • Detailed description of all tuning parameters set by sapconf: man 5 sapconf

  • Information about configuring and customizing the sapconf profile: man 7 sapconf

Also see the blog series detailing the updated version of sapconf at https://www.suse.com/c/a-new-sapconf-is-available/.

8.2 Tuning systems with sapconf 5 Edit source

The package sapconf is available in SUSE Linux Enterprise Server and SUSE Linux Enterprise Server for SAP Applications. It sets recommended parameters for the following types of SAP applications: SAP NetWeaver, SAP HANA and SAP HANA-based applications.

Overview of sapconf5 in SUSE® Linux Enterprise Server 12
sapconf5 (without tuned)
  • sapconf-netweaver (sapconf profile as a replacement for tuned profile)

  • sapconf-hana (sapconf profile as a replacement for tuned profile)

  • sapconf-bobj (sapconf profile as a replacement for tuned profile)

  • sapconf-ase (sapconf profile as a replacement for tuned profile)

Overview of sapconf5 in SUSE® Linux Enterprise Server 15
sapconf5 (without tuned)

no profiles anymore

Note that if you previously made changes to the system tuning, those changes may be overwritten by sapconf.

sapconf 5 ships a systemd service which applies the tuning and ensures that related services are running.

To use sapconf, make sure that the package sapconf is installed on your system.

Note
Note: No profiles in SUSE Linux Enterprise Server and SUSE Linux Enterprise Server for SAP Applications 15 SP2

In SUSE Linux Enterprise Server and SUSE Linux Enterprise Server for SAP Applications 15, sapconf no longer supports profiles.

8.2.1 Verifying sapconf setup Edit source

With sapconf 5.0.2 onwards the check tool sapconf_check is available, which verifies the correct setup of sapconf. For example:

root # sapconf_check
This is sapconf_check v1.0.
It verifies if sapconf is set up correctly and will give advice to do so.
Please keep in mind:
- This tool does not check, if the tuning itself works correctly.
- Follow the hints from top to down to minimize side effects.
Checking sapconf
================
[ OK ] sapconf package has version 5.0.2
[ OK ] saptune.service is inactive
[ OK ] saptune.service is disabled
[WARN] tuned.service is enabled/active with profile 'virtual-guest -> Sapconf does not require tuned! Run 'systemctl stop tuned.service', if not needed otherwise.
[FAIL] sapconf.service is inactive -> Run 'systemctl start sapconf.service' to activate the tuning now.
[FAIL] sapconf.service is disabled -> Run 'systemctl enable sapconf.service' to activate sapconf at boot.1 warning(s) have been found.
2 error(s) have been found.
Sapconf will not work properly!

If sapconf_check finds problems, it will give hints how to resolve the issue. The tool will not verify if the system has been tuned correctly. It only checks that sapconf is setup correctly and has been started.

8.2.2 Enabling and disabling sapconf and viewing its status Edit source

After the installation of sapconf, the sapconf service is enabled.

You can inspect or change the status of sapconf as described in the following:

  • To see the status of the service sapconf:

    root # systemctl status sapconf

    The service should be displayed as active (exited).

  • To start the service sapconf:

    root # systemctl start sapconf
  • Should sapconf be disabled, enable and start it with:

    root # systemctl enable --now sapconf
  • To stop the service sapconf:

    root # systemctl stop sapconf

    This command will disable the vast majority of optimizations immediately. The only exceptions from this rule are options that require a system reboot to enable/disable.

  • To disable sapconf, use:

    root # systemctl disable sapconf

    If you have not specifically enabled any of the services that sapconf depends on yourself, this will also disable most tuning parameters and all services used by sapconf.

Tip
Tip: Additional services that sapconf relies on

In addition to the sapconf service it also relies on the following two services:

  • sysstat which collects data on system activity.

  • uuidd which generates time-based UUIDs that are guaranteed to be unique even in settings where many processor cores are involved. This is necessary for SAP applications.

8.2.3 Configuring sapconf5 Edit source

In general, the default configuration of sapconf already uses the parameter values recommended by SAP. However, if you have special needs, you can configure the tool to better suit those.

All parameters of sapconf can be found in the file /etc/sysconfig/sapconf. The file can be edited directly. All parameters in this file are explained by means of comments and references to SAP Notes which can be viewed at https://launchpad.support.sap.com/.

When sapconf is updated, all customized parameters from this file will be preserved as much as possible. However, sometimes parameters cannot be transferred cleanly to the new configuration file. Therefore, after updating it is advisable to check the difference between the previous custom configuration which during the update is moved to /etc/sysconfig/sapconf.rpmsave and the new version at /etc/sysconfig/sapconf.

Log messages related to this file are written to /var/log/sapconf.log.

When editing either of these files, you will find that some values are commented by means of a # character at the beginning of the line. This means that while the parameter is relevant for tuning, there is no suitable default for it.

Conversely, you can add # characters to the beginning of the line to comment specific parameters. However, you should avoid this practice, as it can lead to sapconf not properly applying the profile.

To apply edited configuration, restart sapconf:

root # systemctl restart sapconf

Confirming that a certain parameter value was applied correctly works differently for different parameters. Hence, the following serves as an example only:

Example 8.2: Checking parameters

To confirm that the setting for TCP_SLOW_START was applied, do the following:

  • View the log file of sapconf to see whether it applied the value. Within /var/log/sapconf.log, check for a line containing this text:

    Change net.ipv4.tcp_slow_start_after_idle from 1 to 0

    Alternatively, the parameter may have already been set correctly before sapconf was started. In this case, sapconf will not change its value:

    Leaving net.ipv4.tcp_slow_start_after_idle unchanged at 1
  • The underlying option behind TCP_SLOW_START can be manually configured at /proc/sys/net.ipv4.tcp_slow_start_after_idle. To check its actual current value, use:

    root # sysctl net.ipv4.tcp_slow_start_after_idle

8.2.4 Removing sapconf Edit source

To remove sapconf from a system, uninstall its package with:

root # zypper rm sapconf

Note that when doing this, dependencies of sapconf will remain installed. However, the service sysstat will go into a disabled state. If it is still relevant to you, make sure to enable it again.

8.2.5 For more information Edit source

The following man pages provide additional information about sapconf:

  • Detailed description of all tuning parameters set by sapconf: man 5 sapconf

  • Information about configuring and customizing the sapconf profile: man 7 sapconf

Also see the blog series detailing the updated version of sapconf at https://www.suse.com/c/a-new-sapconf-is-available/.

8.2.6 Using tuned together with sapconf Edit source

With version 5 sapconf does not rely on tuned anymore. This means both tools can be used independently. sapconf will print a warning in it's log if tuned service is started.

Note
Note: Important: using tuned and sapconf together

If you are going to use tuned and sapconf simultaneously, be very careful, that bot tools do not configure the same system parameters.

8.3 Tuning systems with saptune Edit source

Using saptune, you can tune a system for SAP NetWeaver, SAP HANA/SAP BusinessObjects, and SAP S/4HANA applications. This method relies on the system tuning service tuned.

To use saptune, make sure that the packages tuned and saptune are installed on your system.

Note
Note: tuned daemon

sapconf (only version 4) and saptune both rely on the daemon tuned to set tuning configuration but they use different (though very similar) tuning profiles. Therefore, only one of sapconf or saptune can be enabled at a time.

8.3.1 Enabling saptune to tune for an SAP application Edit source

  1. To tune a system, first find a tuning solution. To find the appropriate solution, use:

    tux > saptune solution list

    saptune knows the following tuning solutions (groups of SAP notes):

    • BOBJ Solution for running SAP BusinessObjects.

    • HANA Solution for running an SAP HANA database.

    • MAXDB Solution for running an SAP MaxDB database.

    • NETWEAVER Solution for running SAP NetWeaver application servers.

    • S4HANA-APPSERVER Solution for running SAP S/4HANA application servers .

    • S4HANA-APP+DBSolution for running both SAP S/4HANA application servers and SAP HANA on the same host.

    • S4HANA-DBSERVER Solution for running the SAP HANA database of an SAP S/4HANA installation.

    • SAP-ASE Solution for running an SAP Adaptive Server Enterprise database.

    • NETWEAVER+HANA Solution for running both SAP application servers and SAP HANA on the same host.

    Alternatively, you can tune the computer according to recommendations from specific SAP Notes. A list of notes that you can tune for is available via:

    root # saptune note list
    • To set up saptune with a preconfigured solution, use:

      root # saptune solution apply SOLUTION
    • To set up saptune for the recommendations of a specific SAP Note, use:

      root # saptune note apply NOTE
    Note
    Note: Combining optimizations

    You can combine solutions and notes. However, only one solution can be active at a time. In rare cases, notes can have conflicting options or parameters. To avoid conflicts, order your notes, keeping in mind that the last note always overrides conflicting options or parameters of previous notes.

  2. To start saptune and enable it at boot, make sure to run the following command:

    root # saptune daemon start

In the background, saptune applies a tuned profile also named saptune that is dynamically customized according to selected solutions and notes. Using tuned-adm list, you can also see this profile.

8.3.2 Customizing an SAP note Edit source

Every SAP note can be configured freely with:

root # saptune note customise

The command includes changing a value or disabling a parameter.

8.3.3 Creating a new SAP note Edit source

It is possible to create a new SAP note with:

root # saptune note create

All features of saptune are available.

8.3.4 Deleting an SAP note Edit source

This command allows to delete a created note, including the corresponding override file if available:

root # saptune note delete test
Note to delete is a customer/vendor specific Note.
Do you really want to delete this Note (test2)? [y/n]: y

The note may not be applied at the time. Keep in mind the following points:

  • A confirmation is needed to finish the action.

  • Internal SAP notes shipped by saptune cannot be deleted. Instead, the override file is removed when available.

  • If the note is already applied, the command will be terminated with the information, that the note first needs to be reverted before it can be deleted.

8.3.5 Renaming an SAP note Edit source

This command allows to rename a created note to a new name. If a corresponding override file is available, this file will be renamed too:

root # saptune note rename test test2
Note to rename is a customer/vendor specific Note.
Do you really want to rename this Note (test) to the new name 'test2'? [y/n]: y

The note may not be applied at the time. Keep in mind the following points:

  • A confirmation is needed to finish the action.

  • Internal SAP notes shipped by saptune cannot be renamed.

  • If the note is already applied, the command will be terminated with the information, that the note first needs to be reverted before it can be deleted.

8.3.6 Showing the configuration of an SAP note Edit source

The shipped configuration of a note can be listed with:

root # saptune note show

8.3.7 Verifying an SAP note or an SAP solution Edit source

The commands saptune note verify NOTE and saptune solution verify SOLUTION list the following data for each active or requested note:

  • The parameter name

  • The expected value (default)

  • A configured override (created using saptune customise)

  • The current system value

  • Whether the current state is in accordance with the SAP recommendation

8.3.8 Simulating the application of an SAP note or an SAP solution Edit source

To show each parameter of a note, use:

root # saptune note simulate

To show each parameter of a solution, use:

root # saptune solution simulate

It lists the current system value and the expected values (default and override).

8.3.9 Disabling saptune Edit source

To disable saptune and to stop and disable tuned run:

root # saptune daemon stop

8.3.10 For more information Edit source

See the following man pages:

  • man 8 saptune

  • man 8 saptune_v1

  • man 8 saptune_v2

  • man 8 saptune-migrate

  • man 8 saptune-note

Also see the project home page https://github.com/SUSE/saptune/.

8.4 Tuning kernel parameters manually using sysctl Edit source

In addition to or instead of tuning kernel parameters using sapconf/saptune, you can also use sysctl to make manual adjustments to kernel parameters. However, such changes using sysctl do not persist across reboots by default. To make them persist across reboots, add them to one of the configuration files read by sysctl.

Tip
Tip: sysctl and saptune

If you plan to configure sysctl parameters for your SAP system, consider using saptune as the central tool for managing such configurations.

For more information about sysctl, see the man pages sysctl(8), sysctl.conf(5), and sysctl.d(5).

8.5 Tuning Workload Memory Protection Edit source

Keeping SAP applications in physical memory is essential for their performance. With SUSE Linux Enterprise Server for SAP Applications 11 SP1 onwards and SUSE Linux Enterprise Server for SAP Applications 12 the Page Cache Limit prevented a swap out to disk by a growing page cache. In SUSE Linux Enterprise Server for SAP Applications 15 the Page Cache Limit has been replaced with the more advanced Workload Memory Protection.

Workload Memory Protection puts SAP instances into a dedicated cgroup (v2) and tells the kernel by the memory.low parameter the amount of memory to keep in physical memory. This protects the processes in this cgroup against any form of memory pressure outside that cgroup, including a growing page cache. Workload Memory Protection can not protect against memory pressure inside this cgroup. It covers the memory of all instances together on one host.

The value for memory.low depends on the kind of SAP instance and the workload and has to be configured manually. If the system is under extreme pressure the Linux kernel will ignore the memory.low value and try to stabilize the whole system, even by swapping or invoking the OOM killer.

For more information about cgroups, see https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-tuning-cgroups.html.

8.5.1 Architecture Edit source

WMP relies on three components:

cgroup2 memory controller (Linux kernel)

The cgroup2 memory controller parameter memory.low allows to define an amount of memory, which the Linux kernel will keep in physical memory. This amount of memory will be excluded from the reclaiming process except the entire system is in a critical memory situation.

WMP uses memory.low to prevent memory of SAP processes to be paged or swapped out to disk. Cgroup1 controllers, except the memory controller, still are available, but not mounted anymore.

systemd

Systemd provides the infrastructure to create and maintain the cgroup hierarchy and also allows the configuration of cgroup parameters. WMP ships systemd configuration files to allow easy configuration of memory.low via systemd methods.

SAP start service

The SAP Start Service manages the start and stop of SAP instances. An important feature for WMP is the configurable execution of programs before the instance itself gets started in the instance profile. WMP uses this method to call a program to move the sapstart process into a designated cgroup, so the SAP instance will be started inside that cgroup.

8.5.2 Support for Workload Memory Protection Edit source

WMP is supported for SUSE Linux Enterprise Server for SAP Applications 15 SP2 on Intel 64/AMD64 and POWER for one or multiple SAP systems on one host, such as:

  • App Server (SAP NetWeaver, SAP S/4HANA) or

  • SAP HANA 1.0/2.0

Workload Memory Protection does not cover databases other than SAP HANA. Depending on their start method the processes might run inside or outside the dedicated cgroup. If they run inside, the memory consumption has to be taken into account when determine memory.low.

Important
Important: Restrictions of WMP

Using WMP comes with benefits, but you should be aware of some restrictions:

  • WMP cannot protect against memory pressure inside the dedicated cgroup.

  • WMP cannot protect SAP systems or their instances from each other. All SAP processes share the same memory limit. If you have multiple SAP systems (for example, SAP NetWeaver and SAP S/4HANA), WMP cannot shield one SAP application from the other.

  • Support for SUSE’s HA cluster solution is not yet available.

8.5.3 Setting up Workload Memory Protection Edit source

8.5.3.1 Preparing for Workload Memory Protection Edit source

  1. Check if your SAP software (SAP HANA, SAP NetWeaver etc) is installed. The group sapsys is needed during the package installation of sapwmp later. If you skip that part, you will get a warning message (see Important: Watch out order of package).

  2. Stop the SAP system:

    root # systemctl stop sapinit

    The service can be enabled, but all SAP processes have to be terminated.

  3. Install the package sapwmp:

    tux > sudo zypper install sapwmp
    Important
    Important: Watch out order of package

    The following message should only appear if no SAP software has been installed on the system:

    Warning: sapsys group not found warning: group sapsys does not exist - using root

    Remove the package sapwmp and install the SAP software first before installing it again.

    As an alternative you can fix ownership and permission after installing the SAP software with:

    tux > sudo chgrp sapsys /usr/lib/sapwmp/sapwmp-capture && \
    chmod +s /usr/lib/sapwmp/sapwmp-capture

    The following message can be ignored:

    Warning: Found memory controller on v1 hierarchy. Make sure unified hierarchy only is used.

    Switching to unified hierarchy is done in the next step.

  4. Add systemd.unified_cgroup_hierarchy=true to the kernel command line by adding it to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub like:

    GRUB_CMDLINE_LINUX_DEFAULT="... systemd.unified_cgroup_hierarchy=true swapaccount=1"

    With this change, only cgroup2 controllers will be mounted on /sys/fs/cgroup. Cgroup1 controllers, except the memory controller, are still available and can be used though. Tools using cgroup1 might not work anymore out of the box and need reconfiguration. Also the required mount structure for cgroup1 has to be provided.

    The parameter swapaccount=1 is not needed for WMP to work, but it aids the analysis in support cases to show the amount of swapped out memory for each cgroup.

  5. Rewrite the GRUB2 configuration:

    tux > sudo grub2-mkconfig -o /boot/grub2/grub.cfg

    After reboot (will be done later), the cgroup hierarchy is switched to v2 (unified hierarchy) only.

  6. Configure MemoryLow for the SAP.slice:

    tux > sudo systemctl set-property SAP.slice MemoryLow=...

    This command creates a drop-in in /etc/systemd/system.control/SAP.slice.d/ to set MemoryLow.

    The sapwmp package includes the systemd configuration SAP.slice which creates the cgroup of the same name for the SAP instances. MemoryLow is the systemd equivalent of the cgroup parameter memory.low mentioned in the introduction. The value for MemoryLow depends on the type of the SAP application and the workload.

    For SAP HANA

    Since SAP HANA has a Global Allocation Limit its value can be used directly.

    SAP application server (SAP NetWeaver, SAP S/4HANA)

    For the Application Server the sizing for the workload should indicate the value for MemoryLow. The sapwmp package contains a monitoring part which might be useful to determine MemoryLow. See Section 8.5.6, “Monitoring memory usage”.

    Keep in mind:

    • All SAP instances on one host are inside the SAP.slice. MemoryLow must cover the amount of memory of all instances together on that host. You cannot protect SAP systems or their instances from each other.

    • If you are using a database other than SAP HANA some database processes might be part of SAP.slice. Their memory consumption has to be taken into account when determine the MemoryLow value.

    • Never chose a value for MemoryLow very close or larger than your physical memory. System services and additional installed software require memory too. If they are forced to use swap to extensively in expense of the SAP application, your system can become unresponsive.

    Note
    Note: Correctly calculate MemoryLow value

    MemoryLow takes the memory size in bytes. If the value is suffixed with K, M, G or T, the specified memory size is parsed as Kibibytes, Mebibytes, Gibibytes, or Tebibytes (with the base 1024 instead of 1000, see https://en.wikipedia.org/wiki/Binary_prefix), respectively. Alternatively, a percentage value may be specified, which is taken relative to the installed physical memory on the system.

    The underlying cgroup memory controller will round up the value to a multiple of the page size. To avoid confusion already set a multiple of the page sizes as value for MemoryLow.

  7. Create a backup of each SAP instance profile. Errors in a profile can prevent a SAP system from starting.

  8. For each SAP instance, add the following line to the instance profile (usually located in /usr/sap/SID/SYS/profile/) after the last Execute_ line:

    Execute_20 = local /usr/lib/sapwmp/sapwmp-capture -a

    Increase the number of the Execute statement when necessary, to be the highest one and the line is executed last.

    Important
    Important

    Edit the instance profiles directly only if you do not have imported the profiles into the database to manage them by the SAP GUI (transaction RZ11). If you do so, use the SAP GUI to add the lines. The profile files located in the file system are getting overwritten and any manual changes would get lost!

Now the system is ready for a reboot.

8.5.3.2 Reboot and verification Edit source

  1. Reboot the system.

  2. After Reboot verify that indeed cgroups v2 has been used:

    root # grep  cgroup /proc/mounts
    cgroup /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
  3. Verify that the cgroup was created successfully and the low memory value has been set:

    tux > systemctl show -p MemoryLow SAP.slice
    MemoryLow=18487889920    <- Should be your chosen value (always in bytes)!
    
    # cat /sys/fs/cgroup/SAP.slice/memory.low
    18487889920    <- Should be your chosen value!

    The variable MemoryLow can be set to any value, but the content of the variable is always be a multiple of the page size. Keep this in mind, when you notice a slight difference between both values.

  4. Check that all SAP instance processes are in the correct system slices/cgroup.

    If you have not enabled sapinit.service start the service now. If autostart is not enabled in the instance profiles, start the instances before you check.

    Example:

    root # systemd-cgls -a /sys/fs/cgroup/SAP.slice
    Directory /sys/fs/cgroup/SAP.slice:
    |-wmp-rd91fd6b3ca0d4c1183659ef4f9a092fa.scope
    | |-3349 sapstart pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er
    | `-3375 er.sapHA0_ERS10 pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er N...
    |-wmp-r360ebfe09bcd4df4873ef69898576199.scope
    | |-3572 sapstart pf=/usr/sap/HA0/SYS/profile/HA0_D01_sapha0ci
    | |-3624 dw.sapHA0_D01 pf=/usr/sap/HA0/SYS/profile/HA0_D01_sapha0ci
    ...

    The sapstartsrv process of an instance remains always in the user slice of SIDadm. Only the sapstart process and its children will be moved to the target cgroup.

    For each instance a directory wmp-rSCOPEID.scope exists with all processes of this instance. The SCOPEID is a hexadecimal 128bit random value.

    The SAP HostAgent is not covered by WMP and remains partly in sapinit.slice and partly in the user slice of sapadm.

  5. If the processes are not in the cgroup, check if the Execute lines in the instance profiles are correct. Also each instance start should now be logged in the system log /var/log/messages:

    ...
    2020-06-16T18:41:28.317233+02:00 server-03 sapwmp-capture: Found PIDs:
    2020-06-16T18:41:28.317624+02:00 server-03 sapwmp-capture:      17001
    2020-06-16T18:41:28.317813+02:00 server-03 sapwmp-capture:      16994
    2020-06-16T18:41:28.317959+02:00 server-03 sapwmp-capture:      16551
    2020-06-16T18:41:28.319423+02:00 server-03 sapwmp-capture: Successful capture into SAP.slice/wmp-r07a27e12d7f2491f8ccb9aeb0e080aaa.scope
    2020-06-16T18:41:28.319672+02:00 server-03 systemd[1]: Started wmp-r07a27e12d7f2491f8ccb9aeb0e080aaa.scope.
    ...

To verify the correct setup, run wmp-check. The script checks the setup of Workload Memory Protection:

  • Correct setup of cgroup2.

  • Ownership and permission of the capture program.

  • WMP entries of SAP instance profiles.

  • Correct cgrop of running SAP instance processes.

  • Correct setup of SAP.slice.

  • Sane configuration of MemoryLow. However, it cannot determine if the MemoryLow value has been chosen wisely.

  • Setup of the optional memory sampler.

  • Setup of optional swap accounting.

It assumes SAP instances profiles can be found beneath /usr/sap/SID/SYS/profile/.

8.5.4 Configuring Workload Memory Protection Edit source

To configure WMP edit /etc/sapwmp.conf:

# NOTE: Local changes may be reverted after update of WMP package. Check for
#      .rpmsave file to restore & merge changes.

## Description: Slice unit name where workload is put into
## Type:        string
## Default:     "SAP.slice"
DEFAULT_SLICE="SAP.slice"

## Description: Comma-separated list of command names to which capture is
##              applied (matching against /proc/$PID/stat)
## Type:        string
## Default:     sapstart
PARENT_COMMANDS=sapstart

After a change restart all SAP instances.

Warning
Warning

Altering /etc/sapwmp.conf should not be necessary. Don’t do it until you know exactly what you do!

8.5.5 Changing the value of memoryLow Edit source

To change the value of MemoryLow run:

root # systemctl set-property SAP.slice MemoryLow=...

The changes will take effect immediately.

The underlying cgroup memory controller will round up the value to a multiple of the page size. To avoid confusion set a multiple of the page sizes as value for MemoryLow.

Important
Important

Never set MemoryLow to a value lower than the memory already accounted to SAP.slice. To check run:

root # systemctl show -p MemoryCurrent SAP.slice

8.5.6 Monitoring memory usage Edit source

Logging the memory usage can either be necessary to determine the value for memory.low, but also to monitor the correct operation of WMP.

To enable monitoring activate the shipped timer unit:

root # systemctl enable --now wmp-sample-memory.timer

Now the timer should be listed by systemctl list-timers:

root # systemctl list-timers
NEXT    LEFT       LAST    PASSED  UNIT                     ACTIVATES
...
Tue...  9min left  Tue...  4s ago  wmp-sample-memory.timer  wmp-sample-memory.service
...

If you check the current configuration, you can see that memory data gets collected every 10 minutes with a randomized delay of 3 minutes:

root # systemctl cat wmp-sample-memory.timer
# /usr/lib/systemd/system/wmp-sample-memory.timer
[Unit]
Description=WMP periodic log of memory consumption

[Timer]
OnCalendar=*:0/10
RandomizedDelaySec=180
AccuracySec=60

[Install]
WantedBy=timers.target

To change this, create a drop-in file and reload systemd (example with increasing the interval to 30 minutes):

root # mkdir /etc/systemd/system/wmp-sample-memory.timer.d

# cat <<EOF >/etc/systemd/system/wmp-sample-memory.timer.d/override.conf
[Timer]
OnCalendar=
OnCalendar=*:0/30
EOF

# systemctl daemon-reload

(The first OnCalendar= line is important to delete previously defined OnCalendar= settings.)

To see the memory consumption check the system log for lines written by wmp_memory_current:

root # grep wmp_memory_current /var/log/messages
...


2020-09-14T12:02:40.337266+02:00 server-03 wmp_memory_current: SAP.slice : memory.low=21474836480 memory.current=2294059008 memory.swap.current=0 , user.slice : memory.low=0 memory.current=5499219968 memory.swap.current=0 , init.scope : memory.low=0 memory.current=8364032 memory.swap.current=0 , system.slice : memory.low=0 memory.current=1863335936 memory.swap.current=0
2020-09-14T12:03:00.767838+02:00 server-03 wmp_memory_current: SAP.slice : memory.low=21474836480 memory.current=2294022144 memory.swap.current=0 , user.slice : memory.low=0 memory.current=5499473920 memory.swap.current=0 , init.scope : memory.low=0 memory.current=8364032 memory.swap.current=0 , system.slice : memory.low=0 memory.current=1862586368 memory.swap.current=0
2020-09-14T12:04:00.337315+02:00 server-03 wmp_memory_current: SAP.slice : memory.low=21474836480 memory.current=2294022144 memory.swap.current=0 , user.slice : memory.low=0 memory.current=5499207680 memory.swap.current=0 , init.scope : memory.low=0 memory.current=8355840 memory.swap.current=0 , system.slice : memory.low=0 memory.current=1862746112 memory.swap.current=0
...

Here a reformatted log line to get a better impression:

2020-09-14T12:02:40.337266+02:00 server-03 wmp_memory_current:
SAP.slice    : memory.low=21474836480 memory.current=2294059008 memory.swap.current=0 ,
user.slice   : memory.low=0           memory.current=5499219968 memory.swap.current=0 ,
init.scope   : memory.low=0           memory.current=8364032    memory.swap.current=0 ,
system.slice : memory.low=0           memory.current=1863335936 memory.swap.current=0

For each cgroup directly below /sys/fs/cgroup/ one block exists separated by comma. On a normal system you should at least find user.slice, system.slice, and init.scope. WMP adds SAP.slice.

Each block contains the information about the current value of memory.low and memory.current, the currently allocated amount of physical memory of processes in this cgroup.

If you have enabled swap accounting (swapaccount=1) during setup you also have memory.swap.current, the amount of swapped out memory of the cgroup.

All values are in bytes. See Step 6 in Section 8.5.3.1, “Preparing for Workload Memory Protection”.

Tip
Tip

You can find a script to print the information as table or CSV here: https://github.com/scmschmidt/wmp_log_extract

8.5.7 Verifying correct operation Edit source

Besides monitoring memory consumption and swapping (see Section 8.5.6, “Monitoring memory usage”) you also should check regularly that all SAP instance processes are in their scopes below SAP.slice.

To do so, run 'systemd-cgls and check each instance process.

Example:

root # systemd-cgls -a /sys/fs/cgroup/SAP.slice
Directory /sys/fs/cgroup/SAP.slice:
|-wmp-rd91fd6b3ca0d4c1183659ef4f9a092fa.scope
| |-3349 sapstart pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er
| `-3375 er.sapHA0_ERS10 pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er N...
|-wmp-r360ebfe09bcd4df4873ef69898576199.scope
| |-3572 sapstart pf=/usr/sap/HA0/SYS/profile/HA0_D01_sapha0ci
| |-3624 dw.sapHA0_D01 pf=/usr/sap/HA0/SYS/profile/HA0_D01_sapha0ci
...

A simpler test would be listing all processes including cgroups for all <SID>s used on the system.

Example:

tux > ps -eo user,pid,cgroup:60,args | grep -e [h]a0adm
ha0adm    2062 0::/user.slice/user-1001.slice/user@1001.service/init.scope  /usr/lib/systemd/systemd --user
ha0adm    2065 0::/user.slice/user-1001.slice/user@1001.service/init.scope  (sd-pam)
ha0adm    3081 0::/SAP.slice/wmp-r73c594e050904c9c922a312dd9a28fd4.scope    sapstart pf=/usr/sap/HA0/SYS/profile/HA0_ASCS00_sapha0as
ha0adm    3133 0::/SAP.slice/wmp-r73c594e050904c9c922a312dd9a28fd4.scope    ms.sapHA0_ASCS00 pf=/usr/sap/HA0/SYS/profile/HA0_ASCS00_sapha0as
ha0adm    3134 0::/SAP.slice/wmp-r73c594e050904c9c922a312dd9a28fd4.scope    en.sapHA0_ASCS00 pf=/usr/sap/HA0/SYS/profile/HA0_ASCS00_sapha0as
ha0adm    3327 0::/SAP.slice/wmp-ra42489517eb846c282c57681e627a496.scope    sapstart pf=/usr/sap/HA0/ERS10/profile/HA0_ERS10_sapha0er
...

All instance processes except sapstartsrv have to be in a scope below 0::/SAP.slice/.

To verify the correct setup, use the wmp-check tool. See Section 8.5.3.2, “Reboot and verification” for more details.

8.5.8 Deinstalling Workload Memory Protection Edit source

  1. Stop the SAP system completely. The sapinit.service has to be stopped, but can stay enabled. All SAP processes have to be terminated.

  2. Remove any changes made to SAP.slice like setting MemoryLow:

    root # systemctl revert SAP.slice
  3. (Optional) Remove the package sapwmp:

    root # zypper remove sapwmp

    This step is optional. The package can stay on the system without having an influence.

  4. (Optional) Remove systemd.unified_cgroup_hierarchy=true from GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub.

    This step is optional. You can keep cgroup2 without using WMP.

  5. Rewrite the GRUB2 configuration:

    root # grub2-mkconfig -o /boot/grub2/grub.cfg

    After the next boot, the system is switched back to the hybrid cgroup hierarchy.

  6. Remove the line to call sapwmp-capture from each SAP instance profile (usually located in /usr/sap/SID/SYS/profile/):

    Execute_20 = local /usr/lib/sapwmp/sapwmp-capture -a
    Important
    Important: Backup is necessary

    Before editing an instance profile create a backup! Errors in a profile can prevent a SAP system from starting!

    Important
    Important: About editing profiles directly

    Edit the instance profiles directly only if you do not have imported the profiles into the database to manage them by the SAP GUI (transaction RZ11). If you do so, use the SAP GUI to add the lines. The profile files located in the file system are getting overwritten and any manual changes would get lost!

  7. Reboot the system and verify that your SAP system has been started successfully.

Print this page