Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
SUSE Linux Enterprise Server for SAP Applications 15 SP3

Hardware Monitoring for SAP Systems

Abstract

This guide provides detailed information about how to install and customize SUSE Linux Enterprise Server for SAP Applications to monitor hardware related metrics to provide insights that can help increase uptime of critical SAP applications. It is based on SUSE Linux Enterprise Server for SAP Applications 15 SP3. The concept however can also be used starting with SUSE Linux Enterprise Server for SAP Applications 15 SP1.

Disclaimer: Documents published as part of the SUSE Best Practices series have been contributed voluntarily by SUSE employees and third parties. They are meant to serve as examples of how particular actions can be performed. They have been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. SUSE cannot verify that actions described in these documents do what is claimed or whether actions described have unintended consequences. SUSE LLC, its affiliates, the authors, and the translators may not be held liable for possible errors or the consequences thereof.

Author: Bernd Schubert, SAP Solution Architect, SUSE
Author: Thomas Schlosser, SAP Solution Architect, SUSE
Author: Gereon Vey, SAP Solution Architect, SUSE
Publication Date: 2022-02-15

1 Introduction

Many customers deploy SAP systems such as SAP S/4HANA for their global operations, to support mission-critical business functions. This means the need for maximized system availability becomes crucial. Accordingly, IT departments are faced with very demanding SLAs: many companies now require 24x7 reliability for their SAP systems.

The base for every SAP system is a solid infrastructure supporting it.

Operating System

SUSE Linux Enterprise Server for SAP Applications is the leading Linux platform for SAP HANA, SAP NetWeaver and SAP S/4HANA solutions. It helps reduce downtime with the flexibility to configure and deploy a choice of multiple HA/DR scenarios for SAP HANA and NetWeaver-based applications. System data monitoring enables proactive problem avoidance.

Hardware

Most modern hardware platforms running SAP systems rely on Intel’s system architecture. The combination of SUSE Linux Enterprise Server on the latest generation Intel Xeon Scalable processors and Intel Optane DC persistent memory help deliver fast, innovative, and secure IT services and to provide resilient enterprise S/4HANA platforms. The Intel platform allows to monitor deep into the hardware, to gain insights in what the system is doing on a hardware level. Monitoring on a hardware level can help reduce downtime for SAP systems in several ways:

Failure prediction

Identifying any hardware failure in advance allows customers to react early and in an scheduled manner, reducing the risk of errors that usually occur on operations executed during system outages.

Failure remediation

Having hardware metrics at hand when looking for the root cause of an issue can help speed up the analysis and therefore reduce the time until the system(s) return into operation. It can also reduce the reaction time, providing more precise information about problems, especially for big customers that usually have operations outsourced to many service providers and does not control the environment directly.

This paper describes a hardware monitoring solution for SAP systems that allows to use hardware metrics provided by the Intel hardware platform to be analyzed in an SAP context.

2 Hardware monitoring for SAP systems overview

The solution presented in this document consists of several open source tools that are combined to collect logs and metrics from server systems, store them in a queriable database, and present them in a visual and easy-to-consume way. In the following sections, we will give an overview of the components and how they work together.

2.1 Components

The monitoring solution proposed in this document consists of several components.

Hardware Monitoring Components
Figure 1: Hardware Monitoring Components

These components can be categorized by their use:

Data Sources

Components that simplify the collection of monitoring data, providing measurements or collected data in a way that the data storage components can pick them up.

Data Storage

Components that store the data coming from the data sources, and provide a mechanism to query the data.

Data Visualization

Components that allow a visual representation of the data stored in the data storage components, to make the (possibly aggregated) data easy to understand and analyze.

The following sections describe these components.

2.1.1 Data sources

The data source components collect data from the operating system or hardware interfaces, and provide them to the data storage layer.

2.1.1.1 Processor Counter Monitor (PCM)

Processor Counter Monitor (PCM) is an application programming interface (API) and a set of tools based on the API to monitor performance and energy metrics of Intel® Core™, Xeon®, Atom™ and Xeon Phi™ processors. PCM works on Linux, Windows, macOS X, FreeBSD and DragonFlyBSD operating systems.

2.1.1.2 collectd - System information collection daemon

collectd is a small daemon which collects system information periodically and provides mechanisms to store and monitor the values in a variety of ways.

2.1.1.3 Prometheus Node Exporter

The Prometheus Node Exporter is an exporter for hardware and OS metrics exposed by *NIX kernels. It is written in Go with pluggable metric collectors.

2.1.1.4 Prometheus IPMI Exporter

The Prometheus IPMI Exporter supports both

  • the regular /metrics endpoint for Prometheus, exposing metrics from the host that the exporter is running on,

  • and an /ipmi endpoint that supports IPMI over RMCP.

One exporter instance running on one host can be used to monitor a large number of IPMI interfaces by passing the target parameter to a scrape.

2.1.1.5 Promtail

Promtail is a Loki agent responsible for shipping the contents of local logs to a Loki instance. It is usually deployed to every machine that needed to be monitored.

2.1.2 Data collection

On the data collection layer, we use two tools, covering different kinds of data: metrics and logs.

2.1.2.1 Prometheus

Prometheus is an open source systems monitoring and alerting toolkit. It is storing time series data like metrics locally and runs rules over this data to aggregate and record new time series from existing data. Prometheus it also able to generate alerts. The project has a very active developer and user community. It is now a stand-alone open source project and maintained independently of any company. This makes it very attractive for SUSE as open source company and fits into our culture. To emphasize this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation 5 years ago (2016). Prometheus works well for recording any purely numeric time series.

Prometheus is designed for reliability. It is the system to go to during an outage as it allows you to quickly analyze a situation. Each Prometheus server is a stand-alone server, not depending on network storage or other remote services. You can rely on it when other parts of your infrastructure are broken, and you do not need to set up extensive infrastructure to use it.

2.1.2.2 Loki

Loki is a log aggregation system, inspired by Prometheus and designed to be cost effective and easy to operate. Unlike other logging systems, Loki is built around the idea of only indexing a set of metadata (labels) for logs and leaving the original log message unindexed. Log data itself is then compressed and stored in chunks in object stores, for example locally on the file system. A small index and highly compressed chunks simplify the operation and significantly lower the cost of Loki.

2.1.3 Data visualization

With the wealth of data collected in the previous steps, tooling is needed to make the data accessible. Through aggregation and visualization data becomes meaningful and consumable information.

2.1.3.1 Grafana

Grafana is an open source visualization and analytics platform. Grafana’s plug-in architecture allows interaction with a variety of data sources without creating data copies. Its graphical browser-based user interface visualizes the data through highly customizable views, providing an interactive diagnostic workspace.

Grafana can display metrics data from Prometheus and log data from Loki side-by-side, correlating events from log files with metrics. This can provide helpful insights when trying to identify the cause for an issue. Also, Grafana can trigger alerts based on metrics or log entries, and thus help identify potential issues early.

3 Implementing hardware monitoring for SAP systems

The following sections show how to set up a monitoring solution based on the tools that have been introduced in the solution overview.

3.1 Node exporter

The prometheus-node_exporter can be installed directly from the SUSE repository. It is part of SUSE Linux Enterprise Server and all derived products.

zypper -n in golang-github-prometheus-node_exporter

Start and enable the node exporter for automatic start at system boot.

systemctl enable --now prometheus-node_exporter
Tip
Tip

To check if the exporter is running, you can use the following commands:

systemctl status prometheus-node_exporter
ss -tulpan |grep exporter

Configure the node exporter depending on your needs. Arguments to be passed to prometheus-node_exporter can be provided in the configuration file /etc/sysconfig/prometheus-node_exporter, for example to modify which metrics the node_exporter will collect and expose.

Example 1: Arguments provided in /etc/sysconfig/prometheus-node_exporter
...
ARGS="--collector.systemd --no-collector.mdadm --collector.ksmd --no-collector.rapl --collector.meminfo_numa --no-collector.zfs --no-collector.udp_queues --no-collector.softnet --no-collector.sockstat --no-collector.nfsd --no-collector.netdev --no-collector.infiniband --no-collector.arp"
...

By default, the node exporter is listening for incoming connections on port 9100.

3.2 collectd

The collectd packages can be installed from the SUSE repositories as well. For the example at hand, we have used a newer version from the openSUSE repository.

Create a file /etc/zypp/repos.d/server_monitoring.repo and add the following content to it:

Example 2: Content for /etc/zypp/repos.d/server_monitoring.repo
[server_monitoring]
name=Server Monitoring Software (SLE_15_SP3)
type=rpm-md
baseurl=https://download.opensuse.org/repositories/server:/monitoring/SLE_15_SP3/
gpgcheck=1
gpgkey=https://download.opensuse.org/repositories/server:/monitoring/SLE_15_SP3/repodata/repomd.xml.key
enabled=1

Afterward refresh the repository metadata and install collectd and its plugins.

zypper ref
zypper in collectd collectd-plugins-all

Now the collectd must be adapted to collect the information you want to get and export it in the format you need. For example, when looking for network latency, use the ping plugin and expose the data in a Prometheus format.

Example 3: Configuration of collectd in /etc/collectd.conf (excerpts)
...
LoadPlugin ping
...
<Plugin ping>
        Host "10.162.63.254"
        Interval 1.0
        Timeout 0.9
        TTL 255
#       SourceAddress "1.2.3.4"
#       AddressFamily "any"
        Device "eth0"
        MaxMissed -1
</Plugin>
...
LoadPlugin write_prometheus
...
<Plugin write_prometheus>
        Port "9103"
</Plugin>
...

Uncomment the LoadPlugin line and check the <Plugin ping> section in the file.

Modify the systemd unit that collectd works as expected. First, create a copy from the system-provided service file.

cp /usr/lib/systemd/system/collectd.service /etc/systemd/system/collectd.service

Second, adapt this local copy. Add the required CapabilityBoundingSet parameters in our local copy /etc/systemd/system/collectd.service.

...
# Here's a (incomplete) list of the plugins known capability requirements:
#   ping            CAP_NET_RAW
CapabilityBoundingSet=CAP_NET_RAW
...

Activate the changes and start the collectd function.

systemctl daemon-reload
systemctl enable --now collectd

All collectd metrics are accessible at port 9103.

With a quick test, you can see if the metrics can be scraped.

curl localhost:9103/metrics

3.3 Processor Counter Monitor (PCM)

Processor Counter Monitor (PCM) can be installed from its GitHub project pages.

Make sure the required tools are installed for building.

Example 4: Installing PCM from source
zypper in -y git cmake gcc-c++

Clone the Git repository and build the tool using the following commands.

Example 5: Installing PCM from source
git clone https://github.com/opcm/pcm.git
cd pcm
mkdir build
cd build
cmake ..
cmake --build .
cd bin

To start PCM on the observed host, first start a new screen session, and then start PCM.[1]

Example 6: Starting PCM
screen -S pcm
./pcm-sensor-server -d

The PCM sensor server binary pcm-sensor-server has been started in a screen session which can be detached (type Ctrl+a d). This lets the PCM sensor server continue running in the background.

The PCM metrics can be queried from port 9738.

3.4 Prometheus IPMI Exporter

The IPMI exporter can be used to scrape information like temperature, power supply information and fan information.

Create a directory, download and extract the IPMI exporter.

mkdir ipmi_exporter
cd ipmi_exporter
curl -OL https://github.com/prometheus-community/ipmi_exporter/releases/download/v1.4.0/ipmi_exporter-1.4.0.linux-amd64.tar.gz
tar xzvf ipmi_exporter-1.4.0.linux-amd64.tar.gz
Note
Note

We have been using the version 1.4.0 of the IPMI exporter. For a different release, the URL used in the curl command above needs to be adapted. Current releases can be found at the IPMI exporter GitHub repository.

Some additional packages are required and need to be installed.

zypper in freeipmi monitoring-plugins-ipmi-sensor1 libipmimonitoring6 monitoring-plugins-ipmi-sensor1

To start the IPMI exporter on the observed host, first start a new screen session, and then start the exporter.[2]

Example 7: Starting PCM
screen -S ipmi
cd ipmi_exporter-1.4.0.linux-amd64
./ipmi_exporter

The IPMI exporter binary ipmi_exporter has been started in a screen session which can be detached (type Ctrl+a d). This lets the exporter continue running in the background.

The metrics of the ipmi_exporter are accessible port 9290.

3.5 Prometheus

The Prometheus RPM packages can be found in the PackageHub repository. This repository needs to be activated via the SUSEConnect command first.

SUSEConnect --product PackageHub/15.3/x86_64

Prometheus can then easily be installed using the zypper command:

zypper in golang-github-prometheus-prometheus

Edit the Prometheus configuration file /etc/prometheus/prometheus.yml to include the scrape job configurations you want to add.

Example 8: Job definition for Node Exporter
  - job_name: node-export
    static_configs:
      - targets:
        - monint1:9100
Example 9: Job definition for Collectd
  - job_name: intel-collectd
    static_configs:
      - targets:
        - monint2:9103
        - monint1:9103
Example 10: Job definition for PCM
  - job_name: intel-pcm
    scrape_interval: 2s
    static_configs:
      - targets:
        - monint1:9738
Example 11: Prometheus IPMI Exporter
  - job_name: ipmi
    scrape_interval: 1m
    scrape_timeout: 30s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets:
        - monint1:9290
        - monint2:9104

Finally start and enable the Prometheus service:

systemctl enable --now prometheus.service

3.6 Loki

The Loki RPM packages can be found in the PackageHub repository. The repository needs to be activated via the SUSEConnect command first, unless you have activated it in the previous steps already.

SUSEConnect --product PackageHub/15.3/x86_64

Loki can then be installed via the zypper command:

zypper in loki

Edit the Loki configuration file /etc/loki/loki.yaml and change the following lines:

chunk_store_config:
  max_look_back_period: 240h

table_manager:
  retention_deletes_enabled: true
  retention_period: 240h

Start and enable Loki service:

systemctl enable --now loki.service

3.6.1 Promtail (Loki agent)

The Promtail RPM packages can be found in the PackageHub repository. The repository has to be activated via the SUSEConnect command first, unless you have activated it in the previous steps already.

SUSEConnect --product PackageHub/15.3/x86_64

Promtail can then be installed via the zypper command.

zypper in promtail

Edit the Promtail configuration file /etc/loki/promtail.yaml to include the scrape configurations you want to add.

Example 12: To include the systemd-journal, add the following:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
Important
Important

If you are using systemd-journal, do not forget to add the loki user to the systemd-journal group: usermod -G systemd-journal -a loki

Example 13: To include the HANA alert trace files, add the following:
  - job_name: HANA
    static_configs:
    - targets:
        - localhost
      labels:
        job: hana-trace
        host: monint1
        __path__: /usr/sap/IN1/HDB11/monint1/trace/*_alert_monint1.trc
Important
Important

If you are using SAP logs like the HANA traces, do not forget to add the loki user to the sapsys group: usermod -G sapsys -a loki

Start and enable the Promtail service:

systemctl enable --now promtail.service

3.7 Grafana

The Grafana RPM packages can be found in the PackageHub repository. The repository has to be activated via the SUSEConnect command first, unless you have activated it in the previous steps already.

SUSEConnect --product PackageHub/15.3/x86_64

Grafana can then be installed via zypper command:

zypper in grafana

Start and enable the Grafana server service:

systemctl enable --now grafana-server.service

Now connect from a browser to your Grafana instance and log in:

Grafana Login page
Figure 2: Grafana welcome page

3.7.1 Grafana data sources

After the login, the data source must be added. On the right hand there is a wheel where a new data source can be added.

Grafana add a new data source
Figure 3: Adding a new Grafana data source

Add a data source for the Prometheus service.

Prometheus data source
Figure 4: Grafana data source for Prometheus DB

Also add a data source for Loki.

Loki data source
Figure 5: Grafana data source for LOKI DB

Now Grafana can access both the metrics stored in Prometheus and the log data collected by Loki, to visualize them.

3.7.2 Grafana dashboards

Dashboards are how Grafana presents information to the user. Prepared dashboards can be downloaded from https://grafana.com/dashboards, or imported using the Grafana ID.

Dashboard overview
Figure 6: Grafana dashboard import option

The dashboards can also be created from scratch. Information from all data sources can be merged into one dashboard.

Dashboard create a new dashboard
Figure 7: Build your own dashboard

3.8 Putting it all together

The picture below shows a dashboard displaying detailed information about the SAP HANA cluster, orchestrated by pacemaker.

SUSE HANA cluster dashboard example
Figure 8: SUSE cluster exporter dashboard

4 Practical use cases

The following sections describe some practical use cases of the tooling set up in the previous chapter.

4.1 CPU

I/O performance is very importand on SAP systems. By looking for the iowait metric in command line tools like top or sar, you can only see a single value without any relation. The picture below is showing such a value in a certain timeframe.

iowait over a certain time
Figure 9: iowait over certain timeframe

An iowait of 2% might not show any problem at first glance. But if you look at iowait as part of the whole CPU load, the picture is completely different to what you saw before. The reason is that the total CPU load in the example is only a little higher then iowait.

iowait in Percent of the total CPU load
Figure 10: iowait in Percent of total CPU load

In our example, you now see an iowait value of about 90% of the total CPU load.

To get the percent of iowait of the total CPU load, use the following formula:

100 / (1-CPU_IDLE) * CPU_IOWAIT

The metrics used are:

  • node_cpu_seconds_total{mode="idle"}

  • node_cpu_seconds_total{mode="iowait"}

    Conclusion

    A high iowait in relation to the overall CPU load is indicating a low throughput. As a result, the IO performance might be very bad. An alert could be triggert by setting a proper threshold if the iowait is going through a certain value.

4.2 Memory

Memory performance in modern servers is not only influenced by its speed, but mainly by the way it is accessed. The Non-Uniform Memory Access (NUMA) architecture used in modern systems is a way of building very large multi-processor systems so that every CPU (that is a group of CPU cores) has a certain amount of memory directly attached to it. Multiple CPUs (multiple groups of processors cores) are then connected together using special bus systems (for example UPI) to provide processor data coherency. Memory that is "local" to a CPU can be accessed with maximum performance and minimal latency. If a process running on a CPU core needs to access memory that is attached to another CPU, it can do so. However, this comes at the price of added latency, because it needs to go through the bus system connecting the CPUs.

4.2.1 Non-Uniform Memory Access (NUMA) example

There are two exporters at hand which can help to provide the metrics data. The node_exporter has an option --collector.meminfo_numa which must be enabled in the configuration file /etc/sysconfig/prometheus-node_exporter. In the example below the collectd plugin numa was used.

We are focusing on two metrics:

  • numa_hit: A process wanted to allocate memory attached to a certain NUMA node (mostly the one it is running on), and succeeded.

  • numa_miss: A process wanted to allocate memory attached to a certain NUMA node, but ended up with memory attached to a different NUMA node.

NUMA ratio NUMA nodes
Figure 11: NUMA miss ratio for both NUMA nodes

The metric used is collectd_numa_vmpage_action_total.

Conclusion

If a process attempts to get a page from its local node, but this node is out of free pages, the numa_miss of that node will be incremented (indicating that the node is out of memory) and another node will accomodate the process’s request. To know which nodes are "lending memory" to the out-of-memory node, you need to look at numa_foreign. Having a high value for numa_foreign for a particular node indicates that this node’s memory is underutilized so the node is frequently accommodating memory allocation requests that failed on other nodes. A high amout of numa_miss indicates a performance degradation for memory based applications like SAP HANA.

4.2.2 Memory module observation

Today more and more application data are hold in the main memory. Examples are Dynamic random access memory (DRAM) or Persistent memory PMEM. The observation of this component became quite important. This is because systems with a high amount of main memory, for example multiple terabytes, are populated with a corresponding number of modules. The example below represents a memory hardware failure which was not effecting the system with a downtime, but a maintenance window should now be scheduled to replace faulty modules.

Memory errors
Figure 12: Memory module failure

The example shows a reduction of available space at the same time as the hardware count is increasing.

The metric used here is node_memory_HardwareCorrupted_bytes.

Memory errors (correctable) also correlate with the CPU performance as shown in the example below. For each of the captured memory failure event, an increase of the CPU I/O is shown.

Memory failure metrix
Figure 13: Memory failure
Conclusion

The risk that one of the modules becomes faulty increases with the total amount of modules per system. The observation of memory correctable errors and uncorrectable errors is essential. Features like Intel RAS can help to avoid application downtime if the failure could be handled by the hardware.

4.3 Network

Beside the fact that the network work must be available and the throughput must be fit, the network latency is very important, too. For cluster setups and applications which are working in a sync mode, like SAP HANA with HANA system replication, the network latency becomes even more relevant. The collectd plugin ping can help here to observe the network latency over time. The Grafana dashboard below visualizes the network latency over the past one hour.

collectd ping
Figure 14: collectd latency check

The red line is a threshold which can be used to trigger an alert.

The metrics used here are collectd_ping and collectd_ping_ping_droprate.

Conclusion

A high value or peak over a long period (multiple time stamps) indicates network response time issues at least to the ping destination. An increasing amount of the ping_droprate points to some issues with the ping destination in regards to responding to the ping request.

4.4 Storage

4.4.1 Storage capacity

Monitoring disk space utilization of the server is critical and important for maximizing application availability. Detecting and monitoring - unexpected or expected - growth of data on the disk will help preventing disk full situations, and therefore application unavailability.

growing filesystem
Figure 15: disk free capacity is droping

The example above represents a continuously growing file system.

The metrics used are node_filesystem_free_bytes and node_filesystem_size_bytes.

After a disk full situation many side effects are shown:

  • System load is going high

  • Disk IOps dropping down

filesystem full
Figure 16: side effects after disk full
Conclusion

Predictive alerting can avoid a situation where the file system runs out of space and the system becomes unavailable. Setting up a simple alerting is a great way to help ensure that you do not encounter any surprises.

5 Summary

With SAP systems such as SAP S/4HANA supporting mission-critical business functions, the need for maximized system availability becomes crucial. The solution described in this document provides the tooling necessary to enable detection and potentially prevention of causes for downtime of those systems. We have also provided some practical use cases highlighting how this tooling can be used to detect and prevent some common issues that are usually hard to detect.

7 GNU Free Documentation License

Copyright © 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

0. PREAMBLE

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

1. APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work’s title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

2. VERBATIM COPYING

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

3. COPYING IN QUANTITY

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

4. MODIFICATIONS

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

  1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.

  2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.

  3. State on the Title page the name of the publisher of the Modified Version, as the publisher.

  4. Preserve all the copyright notices of the Document.

  5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.

  6. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.

  7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document’s license notice.

  8. Include an unaltered copy of this License.

  9. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.

  10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.

  11. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.

  12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.

  13. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.

  14. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.

  15. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’s license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties—​for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

5. COMBINING DOCUMENTS

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

6. COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

7. AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

8. TRANSLATION

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

9. TERMINATION

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

10. FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

ADDENDUM: How to use this License for your documents

Copyright (c) YEAR YOUR NAME.
   Permission is granted to copy, distribute and/or modify this document
   under the terms of the GNU Free Documentation License, Version 1.2
   or any later version published by the Free Software Foundation;
   with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
   A copy of the license is included in the section entitled “GNU
   Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “ with…​Texts.” line with this:

with the Invariant Sections being LIST THEIR TITLES, with the
   Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.



[1] Starting PCM should really be done by creating a systemd unit.

[2] Starting the IPMI exporter should really be done by creating a systemd unit.

Print this page