Pacemaker 远程快速入门
SUSE Linux Enterprise High Availability Extension 15 SP4

# Pacemaker 远程快速入门 #

出版日期：2025 年 12 月 11 日

本文档将指导您完成具有远程节点或 Guest 节点的高可用性群集设置，这些节点由 Pacemaker 和 `pacemaker_remote` 管理。`pacemaker_remote` 中的 Remote 并不表示实际距离的远近，而是表示节点的特殊状态，即这些节点不运行完整的群集堆栈，因此不属于群集的常规成员。

作者: Tanja Roth、Thomas Schraitle
修订历史: Pacemaker 远程快速入门

## 1 概念概述和术语 #

一个常规群集最多只能包含 32 个节点。借助 `pacemaker_remote` 服务，可以将高可用性群集进行扩展，使其包含超出此限制的额外节点。

`pacemaker_remote` 服务可作为物理节点（称为远程节点）运行，也可作为虚拟节点（称为 Guest 节点）运行。与常规群集节点不同，远程节点和 Guest 节点都作为资源由群集进行管理。因此，它们不会计入群集堆栈的 32 个节点限制。不过，从资源管理的角度而言，它们的行为与常规群集节点无异。

对远程节点不需要安装完整的群集堆栈，因为它们只运行 `pacemaker_remote` 服务。该服务充当代理，可让“常规”群集节点上的群集堆栈连接到该服务。因此，运行 `pacemaker_remote` 服务的节点可作为远程节点（请参见术语）有效集成到群集中。

术语 #

群集节点

  运行完整群集堆栈的节点。

  图 1︰ 常规群集堆栈（双节点群集） #

  常规群集节点可以执行以下任务：

  - 运行群集资源。

  - 运行所有命令行工具，例如 `crm`、`crm_mon`。

  - 执行屏蔽操作。

  - 计入群集仲裁。

  - 充当群集的指定协调员 (DC)。

Pacemaker 远程（systemd 服务：`pacemaker_remote`）

  可以使节点用作 Pacemaker 节点而无需部署完整群集堆栈的服务守护程序。请注意，`pacemaker_remote` 是 systemd 服务的名称，pacemaker-remoted （服务名称后带一个 d）才是该守护程序的名称。

远程节点

  运行 `pacemaker_remote` 守护程序的物理机。需要在其中一个群集节点上运行的特殊资源 (`ocf:pacemaker:remote`)，用于管理群集节点与远程节点之间的通讯（请参见第 3 节 “用例 1：设置包含远程节点的群集”）。

Guest 节点

  运行 `pacemaker_remote` 守护程序的虚拟机。通过使用资源代理（如 `ocf:pacemaker:VirtualDomain`）并指定 `remote-node` 元属性来创建 Guest 节点（请参见第 4 节 “用例 2：设置包含 Guest 节点的群集”）。

  对于包含数个 Guest 节点的物理机，创建过程如下所示：

  1. 在群集节点上，虚拟机由 Pacemaker 启动。

  2. 群集会连接到虚拟机的 `pacemaker_remote` 服务。

  3. 通过 `pacemaker_remote` 将虚拟机集成到群集中。

  了解虚拟机在高可用性群集中可以具有的数个角色的不同之处很重要：

  - 虚拟机可以运行完整的群集堆栈。在此情况下，虚拟机是常规群集节点，并非以自我管理的方式由群集进行管理。

  - 虚拟机可作为资源由群集进行管理，群集无需感知到虚拟机内运行的服务。在此情况下，虚拟机对于群集不透明。

  - 虚拟机可以是群集资源并运行 `pacemaker_remote`，这样群集便能管理虚拟机内的服务。在此情况下，虚拟机是 Guest 节点并对群集透明。

远程节点和 Guest 节点可以运行群集资源和大部分命令行工具，但它们具有以下限制：

- 无法执行屏蔽操作。

- 不会影响仲裁。

- 无法充当指定协调员 (DC)。

## 2 使用情形 #

本文中的过程介绍了如何设置具有以下特征的最小群集：

- 两个运行 SUSE Linux Enterprise High Availability Extension 12 GA 或更高版本的群集节点。在本指南中，它们的主机名分别为 `alice` 和 `bob`。

- 根据您选择的设置，群集最终包含以下节点之一：

  - 一个运行 `pacemaker_remote` 的远程节点（本文中该远程节点名为 `charlie`）。

    或：

  - 一个运行 `pacemaker_remote` 的 Guest 节点（本文中该 Guest 节点名为 `doro`）。

- 用于管理 Guest 节点和远程节点的 Pacemaker。

- 当活动的主机发生故障（主动/被动设置）时，资源从一个节点故障转移至另一个节点。

## 3 用例 1：设置包含远程节点的群集 #

下面的示例设置中使用了远程节点 `charlie`。

1. 第 3.1 节 “准备群集节点和远程节点”

2. 第 3.2 节 “配置身份验证密钥”

3. 第 3.3 节 “配置远程节点”

4. 第 3.4 节 “将远程节点集成到群集中”

### 3.1 准备群集节点和远程节点 #

要准备群集节点和远程节点，请执行以下操作：

1. 按Article “安装和设置快速入门”中所述安装并设置一个基本的双节点群集。如此双节点群集将包含两个物理主机：`alice` 和 `bob`。

2. 在您要用作远程节点的物理主机 (`charlie`) 上，安装 SUSE Linux Enterprise Server 15 SP4，并以扩展的形式添加 SUSE Linux Enterprise High Availability Extension 15 SP4。但不要安装高可用性安装软件集，因为远程节点只需要单个软件包（请参见第 3.3 节）。

3. 在所有群集节点上，查看 `/etc/hosts` 并添加 `charlie` 对应的项。

### 3.2 配置身份验证密钥 #

在群集节点 `alice` 上，执行以下操作：

1. 为 `pacemaker_remote` 服务创建特定的身份验证密钥：

       root # dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4k count=1

   `pacemaker_remote` 服务的密钥与您在 YaST 群集模块中创建的群集身份验证密钥不同。

2. 使用 `scp` 在所有群集节点和您未来的远程节点之间同步身份验证密钥：

       root # scp -r -p /etc/pacemaker/ bob:/etc
       root # scp -r -p /etc/pacemaker/ charlie:/etc

   密钥需始终保持同步。

### 3.3 配置远程节点 #

下面的过程将物理主机 `charlie` 配置为远程节点：

1. 在 `charlie` 上执行如下操作：

   1. 在防火墙设置中，为 `pacemaker_remote` 打开 TCP 端口 3121。

   2. 安装 pacemaker-remote 和 crmsh 软件包。

          root # zypper in pacemaker-remote crmsh

   3. 在 `charlie` 上启用并启动 `pacemaker_remote` 服务：

          root # systemctl enable pacemaker_remote
          root # systemctl start pacemaker_remote

2. 在 `alice` 或 `bob` 上，使用 `ssh` 校验连到远程节点的主机连接：

       root # ssh -p 3121 charlie

   此 SSH 连接将失败，但其失败过程会指出设置是否有效：

   有效设置

         ssh_exhange_identification: read: Connection reset by peer.

   无效设置

         ssh: connect to host charlie port 3121: No route to host
         ssh: connect to host charlie port 3121: Connection refused

     如果您看到上面两条消息中的任意一条，则表示设置无效。对 `ssh` 使用 `-v` 选项并再次执行命令，以查看调试消息。这有助于您找出连接、身份验证或配置问题。使用多个 `-v` 选项可提高详细程度。

   视需要添加更多远程节点并按上面所述进行配置。

### 3.4 将远程节点集成到群集中 #

要将远程节点集成到群集中，请执行以下操作：

1. 确保所有群集节点上均已启动群集服务。如果情况并非如此，请运行以下命令在所有节点上启动群集服务：

       root # crm cluster start --all

2. 在节点 `alice` 上，创建 `ocf:pacemaker:remote` 原始资源：

       root # crm configure
       crm(live)configure# primitive charlie ocf:pacemaker:remote \
            params server=charlie reconnect_interval=15m \
            op monitor interval=30s
       crm(live)configure# commit
       crm(live)configure# quit

3. 使用 `crm status` 命令检查群集的状态：命令结果应包含一个所含节点全部都可访问的运行中群集：

       root # crm status
       [...]
       Online: [ alice bob ]
       RemoteOnline: [ charlie ]

       Full list of resources:
       charlie (ocf:pacemaker:remote): Started alice
        [...]

### 3.5 在远程节点上启动资源 #

远程节点集成到群集中后，您便可以像在任意群集节点上启动资源一样在远程节点上启动资源。

警告：有关组和约束的限制

请勿在资源组、共置约束或顺序约束中包含远程节点连接资源，否则可能导致在群集转换时出现非预期的行为。

屏蔽远程节点： 屏蔽远程节点的方式与屏蔽群集节点的方式相同。请用为群集节点配置屏蔽资源的相同方式配置用于远程节点的屏蔽资源。

远程节点不会发起屏蔽操作，只有群集节点才可对其他节点执行屏蔽操作。

## 4 用例 2：设置包含 Guest 节点的群集 #

下面的示例设置中使用 KVM 来设置虚拟 Guest 节点 (`doro`)。

1. 第 4.1 节 “准备群集节点和 Guest 节点”

2. 第 4.2 节 “配置身份验证密钥”

3. 第 4.3 节 “配置 Guest 节点”

4. 第 4.4 节 “将 Guest 节点集成到群集中”

### 4.1 准备群集节点和 Guest 节点 #

要准备群集节点和 Guest 节点，请执行以下操作：

1. 按Article “安装和设置快速入门”中所述安装并设置一个基本的双节点群集。如此双节点群集将包含两个物理主机：`alice` 和 `bob`。

2. 在 `alice` 上创建 KVM Guest。有关详细信息，请参见 SUSE Linux Enterprise Server 15 SP4 的《虚拟化指南》。

3. 在您要用作 Guest 节点的 KVM Guest (`doro`) 上，安装 SUSE Linux Enterprise Server 15 SP4，并以扩展的形式添加 SUSE Linux Enterprise High Availability Extension 15 SP4。但不要安装高可用性安装软件集，因为远程节点只需要单个软件包（请参见第 4.3 节）。

4. 在所有群集节点上，查看 `/etc/hosts` 并添加 `doro` 对应的项。

### 4.2 配置身份验证密钥 #

在群集节点 `alice` 上，执行以下操作：

1. 为 `pacemaker_remote` 服务创建特定的身份验证密钥：

       root # mkdir -p --mode=0755 /etc/pacemaker
       root # dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4k count=1

   `pacemaker_remote` 服务的密钥与您在 YaST 群集模块中创建的群集身份验证密钥不同。

2. 使用 `scp` 在所有群集节点和您的 Guest 节点之间同步身份验证密钥：

       root # scp -r -p /etc/pacemaker/ bob:/etc
       root # scp -p /etc/pacemaker/ doro:/etc

   密钥需始终保持同步。

### 4.3 配置 Guest 节点 #

下面的过程会将 `doro` 配置为群集节点 `alice` 上的 Guest 节点：

1. 在 `doro` 上执行如下操作：

   1. 在防火墙设置中，为 `pacemaker_remote` 打开 TCP 端口 3121。

   2. 安装 pacemaker-remote 和 crmsh 软件包。

          root # zypper in pacemaker-remote crmsh

   3. 在 `alice` 上启用并启动 `pacemaker_remote` 服务：

          root # systemctl enable pacemaker_remote
          root # systemctl start pacemaker_remote

2. 在 `alice` 或 `bob` 上，运行 `ssh` 以校验连到 Guest 的主机连接：

       root # ssh -p 3121 doro

   此 SSH 连接将失败，但其失败过程会指出设置是否有效：

   有效设置

         ssh_exhange_identification: read: Connection reset by peer.

   无效设置

         ssh: connect to host doro port 3121: No route to host
         ssh: connect to host doro port 3121: Connection refused

     如果您看到上面两条消息中的任意一条，则表示设置无效。对 `ssh` 使用 `-v` 选项并再次执行命令，以查看调试消息。这有助于您找出连接、身份验证或配置问题。使用多个 `-v` 选项可提高详细程度。

   视需要添加更多 Guest 节点并按上面所述进行配置。

3. 关闭 Guest 节点并继续第 4.4 节 “将 Guest 节点集成到群集中”。

### 4.4 将 Guest 节点集成到群集中 #

要将 Guest 节点集成到群集中，请执行以下操作：

1. 确保所有群集节点上均已启动群集服务。如果情况并非如此，请运行以下命令在所有节点上启动群集服务：

       root # crm cluster start --all

2. 转储下一步中需要用到的 KVM Guest 的 XML 配置：

       root # virsh list --all
        Id    Name         State
       -----------------------------------
        -     doro       shut off
       root # virsh dumpxml doro > /etc/pacemaker/doro.xml

3. 在节点 `alice` 上，创建用于启动虚拟机的 `VirtualDomain` 资源。使用步骤 2 中转储的配置：

       root # crm configure
       crm(live)configure# primitive vm-doro ocf:heartbeat:VirtualDomain \
         params hypervisor="qemu:///system" \
                config="/etc/pacemaker/doro.xml" \
                meta remote-node=doro

   Pacemaker 将自动监视 pacemaker_remote 连接是否失败，因此无需针对 `VirtualDomain` 资源创建定期监视器。

4. 使用 `crm status` 命令检查群集的状态：命令结果应包含一个所含节点全部都可访问的运行中群集。

### 4.5 测试设置 #

要演示资源是如何执行的，请使用一个虚设资源。该资源仅作测试之用。

1. 创建虚设资源：

       root # crm configure primitive fake1 ocf:pacemaker:Dummy

2. 使用 `crm status` 命令检查群集状态。您应该会看到如下内容：

       root # crm status
       [...]
       Online: [ alice bob ]
       GuestOnline: [ doro@alice ]

       Full list of resources:
       vm-doro (ocf:heartbeat:VirtualDomain): Started alice
       fake1           (ocf:pacemaker:Dummy): Started bob

3. 要将虚设原始资源转移到 Guest 节点 (`doro`)，请使用以下命令：

       root # crm resource move fake1 doro

   此时状态将更改为：

       root # crm status
       [...]
       Online: [ alice bob ]
       GuestOnline: [ doro@alice ]

       Full list of resources:
       vm-doro (ocf:heartbeat:VirtualDomain): Started alice
       fake1           (ocf:pacemaker:Dummy): Started doro

4. 要测试屏蔽是否有效果，请终止 `doro` 上的 `pacemaker-remoted` 守护程序：

       root # kill -9 $(pidof pacemaker-remoted)

5. 几秒后，再次检查群集状态。它应显示为：

       root # crm status
       [...]
       Online: [ alice bob ]

       Full list of resources:
       vm-doro (ocf::heartbeat:VirtualDomain): Started alice
       fake1           (ocf:pacemaker:Dummy): Stopped

       Failed Actions:
       * doro_monitor_30000 on alice 'unknown error' (1): call=8, status=Error, exitreason='none',
           last-rc-change='Tue Jul 18 13:11:51 2017', queued=0ms, exec=0ms

## 5 升级群集和 pacemaker_remote 节点 #

有关不同情形和受支持升级路径的完整信息，请参见Book “管理指南”, Chapter 5 “升级群集和更新软件包”。有关您要升级到的产品的任何更改和新功能的详细信息，请参见其发行说明，所在网址为 https://www.suse.com/releasenotes/。

## 6 更多信息 #

https://documentation.suse.com/sle-ha-15/ 上提供了更多有关此产品的文档。有关其他配置和管理任务，请参见详尽的 《管理指南》。

http://www.clusterlabs.org/pacemaker/doc/ 上提供了上游文档。请参见文档《Pacemaker Remote—Scaling High Availability Clusters》（Pacemaker Remote — 扩缩高可用性群集）。

## 7 法律声明 #

版权所有 © 2006– 2025 SUSE LLC 和贡献者。保留所有权利。

根据 GNU 自由文档许可 (GNU Free Documentation License) 版本 1.2 或（根据您的选择）版本 1.3 中的条款，在此授予您复制、分发和/或修改本文档的权限；本版权声明和许可附带不可变部分。许可版本 1.2 的副本包含在题为“GNU Free Documentation License”的部分。

有关 SUSE 商标，请参见 http://www.suse.com/company/legal/。所有其他第三方商标分别为相应所有者的财产。商标符号（®、™ 等）代表 SUSE 及其关联公司的商标。星号 (*) 代表第三方商标。

本指南力求涵盖所有细节，但这不能确保本指南准确无误。SUSE LLC 及其关联公司、作者和译者对于可能出现的错误或由此造成的后果皆不承担责任。

# A GNU licenses #

This appendix contains the GNU Free Documentation License version 1.2.

## GNU Free Documentation License #

Copyright (C) 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

##### 0. PREAMBLE #

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or non-commercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

##### 1. APPLICABILITY AND DEFINITIONS #

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

##### 2. VERBATIM COPYING #

You may copy and distribute the Document in any medium, either commercially or non-commercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

##### 3. COPYING IN QUANTITY #

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

##### 4. MODIFICATIONS #

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.

2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.

3. State on the Title page the name of the publisher of the Modified Version, as the publisher.

4. Preserve all the copyright notices of the Document.

5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.

6. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.

7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.

8. Include an unaltered copy of this License.

9. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.

10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.

11. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.

12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.

13. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.

14. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.

15. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

##### 5. COMBINING DOCUMENTS #

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

##### 6. COLLECTIONS OF DOCUMENTS #

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

##### 7. AGGREGATION WITH INDEPENDENT WORKS #

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

##### 8. TRANSLATION #

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

##### 9. TERMINATION #

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

##### 10. FUTURE REVISIONS OF THIS LICENSE #

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See https://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

##### ADDENDUM: How to use this License for your documents #

    Copyright (c) YEAR YOUR NAME.
    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.2
    or any later version published by the Free Software Foundation;
    with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
    A copy of the license is included in the section entitled “GNU
    Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.” line with this:

    with the Invariant Sections being LIST THEIR TITLES, with the
    Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.