Geo 群集快速入门
SUSE Linux Enterprise High Availability Extension 15 SP5

# Geo 群集快速入门 #

出版日期：2025 年 12 月 11 日

Geo 群集可保护全局分布式数据中心范围的工作负载。本文档会指导您使用 crm 外壳提供的 Geo 引导脚本完成 Geo 群集的基本设置。

修订历史: Geo 群集快速入门

版权所有 © 2006–2025 SUSE LLC 和贡献者。保留所有权利。

根据 GNU 自由文档许可 (GNU Free Documentation License) 版本 1.2 或（根据您的选择）版本 1.3 中的条款，在此授予您复制、分发和/或修改本文档的权限；本版权声明和许可附带不可变部分。许可版本 1.2 的副本包含在题为“GNU Free Documentation License”的部分。

有关 SUSE 商标，请参见 http://www.suse.com/company/legal/。所有第三方商标均是其各自所有者的财产。商标符号（®、™ 等）代表 SUSE 及其关联公司的商标。星号 (*) 代表第三方商标。

本指南力求涵盖所有细节，但这不能确保本指南准确无误。SUSE LLC 及其关联公司、作者和译者对于可能出现的错误或由此造成的后果皆不承担责任。

## 1 概念概述 #

基于 SUSE® Linux Enterprise High Availability Extension 的 Geo 群集可视为“覆盖”群集，其中每个群集站点对应传统群集中的一个群集节点。覆盖群集由投票间群集票据管理器（以下简称“投票间”）管理。Geo 群集中的各方都会运行一个服务，即 `boothd`。它连接到其余站点上运行的投票间守护程序，并交换连接性细节。投票间依靠称为“票据”的群集对象来确保群集资源在各站点之间的高度可用性。票据授予在特定群集站点上运行某些资源的权限。投票间可保证每个票据一次只会授予给一个站点。

如果两个投票间实例之间的通讯断开，则可能是因为群集站点之间的网络发生故障或其中一个群集站点中断。在此情况下，您需要使用一个额外的实例（第三个群集站点或 `arbitrator`）来就各决策（例如在站点间进行资源的故障转移）达成一致意见。仲裁方是一台以特殊模式运行投票间实例的计算机（不属于群集）。每个 Geo 群集都可以有一个或多个仲裁方。

图 1︰ 双站点群集 - 2x2 个节点 + 仲裁方（可选） #

也可以运行无仲裁方的双站点 Geo 群集。在此情况下，Geo 群集管理员需要手动管理票据。如果同时将一个票据授予多个站点，投票间会显示一条警告。

有关用于 Geo 群集的概念、组件和票据管理的更多细节，请参见Book “管理指南”。

## 2 使用情形 #

下面，我们将会设置一个包含两个群集站点和一个仲裁方的基本 Geo 群集：

- 假设群集站点名为 `amsterdam` 和 `berlin`。

- 我们假设每个站点分别包含两个节点。节点 `alice` 和 `bob` 属于群集 `amsterdam`。节点 `charlie` 和 `doro` 属于群集 `berlin`。

- 站点 `amsterdam` 将分配到以下虚拟 IP 地址：`192.168.201.100`。

- 站点 `berlin` 将分配到以下虚拟 IP 地址：`192.168.202.100`。

- 我们假设仲裁方的 IP 地址为 `192.168.203.100`。

继续操作前，请确保已满足以下要求：

要求 #

两个现有的群集

  您至少拥有两个要合并到 Geo 群集中的现有群集（如果需要先设置两个群集，请按照Article “安装和设置快速入门”中的指导操作。）

有意义的群集名称

  每个群集都有一个有意义的名称，该名称可反映其位置，定义于 `/etc/corosync/corosync.conf` 中。

仲裁方

  您已安装不属于任何现有群集且将用作仲裁方的第三台计算机。

有关各项的详细要求，另请参见第 3 节 “要求”。

## 3 要求 #

将加入群集的所有计算机（群集节点和仲裁方）上都至少需安装以下模块和扩展：

- Basesystem Module 15 SP5

- Server Applications Module 15 SP5

- SUSE Linux Enterprise High Availability Extension15 SP5

安装计算机时，请选择 `HA GEO Node` 作为 `system role`。如此将安装最小系统，这种情况下，默认会安装软件集 `Geo Clustering for High Availability (ha_geo)` 中的软件包。

网络需求 #

- 要用于每个群集站点的虚拟 IP 在 Geo 群集中必须可访问。

- 必须可以通过每个投票间实例上的一个 UDP 和 TCP 端口访问站点。这意味着，必须相应地配置通讯双方之间的所有防火墙和 IPsec 隧道。

- 其他设置决定可能需要打开更多端口（例如，为 DRBD 或进行数据库复制）。

其他要求和建议 #

- 所有站点上的所有群集节点应同步到群集外部的 NTP 服务器。有关详细信息，请参见 Administration Guide for SUSE Linux Enterprise Server 15 SP5。

  如果节点未同步，将难以分析日志文件和群集报告。

- 在 Geo 群集中使用奇数数量的站点。这样可确保在网络连接断开时仍存有大部分站点（从而避免出现节点分裂情况）。如果群集站点的数量为偶数，请使用一个仲裁方来处理票据的自动故障转移。如果您不使用仲裁方，则需要手动处理票据故障转移。

- 每个站点上的群集都有一个有意义的名称，例如 `amsterdam` 和 `berlin`。

  每个站点的群集名称在相应的 `/etc/corosync/corosync.conf` 文件中定义：

      totem {
          [...]
          cluster_name: amsterdam
          }

  使用下面的 crmsh 命令来更改名称：

      # crm cluster rename NEW_NAME

  停止群集服务然后再将其启动，以使更改生效：

      # crm cluster restart

- 不支持在一个群集中采用混合体系结构。但是，对于 Geo 群集而言，Geo 群集的每个成员（群集站点或仲裁方）可以采用不同的体系结构。例如，您可以运行一个包含三个成员（两个群集站点和一个仲裁方）的 Geo 群集，其中的一个群集站点在 IBM Z 上运行，另一个群集站点在 x86 上运行，仲裁方在 POWER 上运行。

## 4 Geo 引导脚本概述 #

- 使用 `crm cluster geo_init` 可将一个群集变为 Geo 群集的第一个站点。该脚本会接受参数（例如群集的名称、仲裁方，以及一个或多个票据）并基于它们创建 `/etc/booth/booth.conf`。它会将投票间配置复制到当前群集站点上的所有节点。它还会为当前群集站点上的投票间配置所需的群集资源。

  有关详细信息，请参见第 6 节 “设置 Geo 群集的第一个站点”。

- 使用 `crm cluster geo_join` 可将当前群集添加到现有的 Geo 群集中。该脚本会从现有群集站点复制投票间配置，并将其写入到当前群集站点上所有节点的 `/etc/booth/booth.conf` 中。它还会为当前群集站点上的投票间配置所需的群集资源。

  有关详细信息，请参见第 7 节 “向 Geo 群集添加另一个站点”。

- 使用 `crm cluster geo_init_arbitrator` 可将当前计算机变为 Geo 群集的仲裁方。该脚本会从现有群集站点复制投票间配置，并将其写入 `/etc/booth/booth.conf`。

  有关详细信息，请参见第 8 节 “添加仲裁方”。

所有引导脚本都会记录到 `/var/log/crmsh/crmsh.log` 中。如需引导过程的任何细节，请查看日志文件。在引导过程中设置的任何选项都可在稍后加以修改（通过修改投票间设置、修改资源等）。有关详细信息，请参见Book “管理指南”。

## 5 安装高可用性和 Geo 群集软件包 #

`High Availability` 和 `Geo Clustering for High Availability` 安装软件集中包含用于配置和管理 Geo 群集的软件包。只有在安装 SUSE Linux Enterprise High Availability Extension 后，才能使用这些软件集。

在安装 SUSE Linux Enterprise Server 期间或之后，您可以注册到 SUSE Customer Center 并安装 High Availability Extension。有关详细信息，请参见 SUSE Linux Enterprise Server 的 Deployment Guide。

过程 1︰ 安装高可用性和 Geo 群集软件集 #

1. 从命令行安装高可用性和 Geo 群集软件集：

       # zypper install -t pattern ha_sles ha_geo

2. 请在要加入群集的所有计算机上安装高可用性和 Geo 群集软件集。

   注意：在所有节点上安装软件包

   如果要自动安装 SUSE Linux Enterprise Server 15 SP5 和 High Availability Extension，请使用 AutoYaST 克隆现有节点。有关详细信息，请参见Book “管理指南”, Chapter 3 “安装 High Availability Extension”, Section 3.2 “使用 AutoYaST 进行批量安装和部署”。

## 6 设置 Geo 群集的第一个站点 #

使用 `crm cluster geo_init` 命令可将一个现有群集变为 Geo 群集的第一个站点。

过程 2︰ 使用 `crm cluster geo_init` 设置第一个站点 (`amsterdam`) #

1. 为每个群集站点定义一个虚拟 IP，以便访问该站点。在此，我们假设使用 `192.168.201.100` 和 `192.168.202.100`。您并不需要将虚拟 IP 配置为群集资源。此工作将由引导脚本完成。

2. 至少定义一个票据的名称，该票据用于授予在群集站点上运行特定资源的权限。使用一个有意义的名称来反映将依赖于该票据的资源，例如 `ticket-nfs`。引导脚本只需要票据名称，您可以稍后再按第 10 节 “后续步骤”中所述定义其余的细节（资源的票据依赖项）。

3. 登录现有群集的节点（例如，群集 `alice` 的节点 `amsterdam`）。

4. 运行 `crm cluster geo_init`.例如，使用以下选项：

       # crm cluster geo_init \
         --clusters "amsterdam=192.168.201.100 berlin=192.168.202.100" \1
         --tickets ticket-nfs \2
         --arbitrator 192.168.203.1003

   1 | 群集站点的名称（定义于 `/etc/corosync/corosync.conf` 中）和您想要为每个群集站点使用的虚拟 IP 地址。在本例中，我们有两个群集站点（`amsterdam` 和 `berlin`），每个站点各有一个虚拟 IP 地址。
   2 | 一个或多个票据的名称。
   3 | 不属于群集的计算机的主机名或 IP 地址。

该引导脚本会创建投票间配置文件，并在群集站点间同步该文件。它还会创建投票间所需的的基本群集资源。执行过程 2 的步骤 4 将创建以下投票间配置和群集资源：

例 1︰ `crm cluster geo_init` 创建的投票间配置 #

    # The booth configuration file is "/etc/booth/booth.conf". You need to
    # prepare the same booth configuration file on each arbitrator and
    # each node in the cluster sites where the booth daemon can be launched.

    # "transport" means which transport layer booth daemon will use.
    # Currently only "UDP" is supported.
    transport="UDP"
    port="9929"

    arbitrator="192.168.203.100"
    site="192.168.201.100"
    site="192.168.202.100"
    authfile="/etc/booth/authkey"
    ticket="ticket-nfs"
    expire="600"

例 2︰ `crm cluster geo_init` 创建的群集资源 #

    primitive1 booth-ip IPaddr2 \
      params rule #cluster-name eq amsterdam ip=192.168.201.100 \
      params rule #cluster-name eq berlin ip=192.168.202.100 \
    primitive2 booth-site ocf:pacemaker:booth-site \
      meta resource-stickiness=INFINITY \
      params config=booth \
      op monitor interval=10s
    group3 g-booth booth-ip booth-site \
    meta target-role=Stopped4

1 | 每个群集站点的虚拟 IP 地址。必须为投票间守护程序提供该虚拟 IP 地址，因为其需要每个群集站点上都有永久的 IP 地址。
2 | 投票间守护程序的原始资源。它会与其余群集站点上的投票间守护程序通讯。该守护程序可以在站点的任何节点上启动。为了使资源分布在同一节点上（如果可能），resource-stickiness 设置为 `INFINITY`。
3 | 两个原始资源的群集资源组。通过这种配置，每个投票间守护程序都将在各自的 IP 地址处可用，独立于运行守护程序的节点。
4 | 该群集资源组默认不会启动。校验群集资源的配置（并添加完成设置所需的资源）后，您需要启动该资源组。有关详细信息，请参见 完成 Geo 群集设置所需的步骤。

## 7 向 Geo 群集添加另一个站点 #

初始化 Geo 群集的第一个站点后，请按`crm cluster geo_join` 中所述使用 过程 3 添加第二个群集。该脚本需要通过 SSH 访问已配置的群集站点，并且会将当前的群集添加到 Geo 群集。

过程 3︰ 使用 `crm cluster geo_join` 添加第二个站点 (`berlin`) #

1. 登录要添加的群集站点的某个节点（例如，群集 `charlie` 的节点 `berlin`）。

2. 运行 `crm cluster geo_join` 命令。例如：

       # crm cluster geo_join \
         --cluster-node 192.168.201.100\1
         --clusters "amsterdam=192.168.201.100 berlin=192.168.202.100"2

   1 | 指定要从何处复制投票间配置。请使用已配置的 Geo 群集站点中某节点的 IP 地址或主机名。也可以使用现有群集站点的虚拟 IP 地址（如本示例中所示）。或者，使用已为 Geo 群集配置的仲裁方的 IP 地址或主机名。
   2 | 群集站点的名称（定义于 `/etc/corosync/corosync.conf` 中）和您想要为每个群集站点使用的虚拟 IP 地址。在本例中，我们有两个群集站点（`amsterdam` 和 `berlin`），每个站点各有一个虚拟 IP 地址。

`crm cluster geo_join` 命令会从 1 复制投票间配置，请参见例 1。此外，它还会创建投票间所需的群集资源（请参见例 2）。

## 8 添加仲裁方 #

使用 `crm cluster geo_init` 和 `crm cluster geo_join` 设置 Geo 群集的所有站点后，请使用 `crm cluster geo_init_arbitrator` 设置仲裁方。

过程 4︰ 使用 `crm cluster geo_init_arbitrator` 设置仲裁方 #

1. 登录要用作仲裁方的计算机。

2. 运行以下命令。例如：

       # crm cluster geo_init_arbitrator --cluster-node 192.168.201.1001

   1 | 指定要从何处复制投票间配置。请使用已配置的 Geo 群集站点中某节点的 IP 地址或主机名。或者，使用现有群集站点的虚拟 IP 地址（如本示例中所示）。

`crm cluster geo_init_arbitrator` 脚本会从 1 复制投票间配置，如例 1 所示。它还会在仲裁方上启用并启动投票间服务。因此，当群集站点上也在运行投票间服务时，仲裁方便可以与这些群集站点上的投票间实例通讯。

## 9 监视群集站点 #

要查看您已在引导过程中创建的群集站点及资源与票据，请使用 Hawk2。通过 Hawk2 Web 界面，您可以监视和管理多个（不相关的）群集和 Geo 群集。

先决条件 #

- 要通过 Hawk5 的仪表板监视的所有群集必须运行 SUSE Linux Enterprise High Availability Extension 15 SP5。

- 如果您之前未在每个群集节点上用自己的证书（或官方证书颁发机构签名的证书）替换 Hawk2 的自我签名证书，请执行以下操作：在每个群集的每个节点上至少登录 Hawk2 一次。验证证书（或在浏览器中添加例外以绕过警告）。否则，Hawk2 将无法连接到群集。

过程 5︰ 使用 Hawk2 仪表板 #

1. 启动 Web 浏览器并输入第一个群集站点 `amsterdam` 的虚拟 IP。

       https://192.168.201.100:7630/

   或者，使用 `alice` 或 `bob` 的 IP 地址或主机名。如果您已使用引导脚本设置两个节点，则 `hawk` 服务在这两个节点上应当都会运行。

2. 登录到 Hawk2 Web 界面。

3. 在左侧导航栏中，选择仪表板。

   Hawk2 会显示当前群集站点的资源和节点的概述。此外，它还会显示已为 Geo 群集配置的所有票据。如需有关此视图中所用图标的信息，请单击图例。

   图 2︰ 包含一个群集站点 (`amsterdam`) 的 Hawk2 仪表板 #

4. 要添加第二个群集站点的仪表板，请单击添加群集。

   1. 输入用于在仪表板中标识该群集的群集名称。在本例中为 `berlin`。

   2. 输入其中一个群集节点（在本例中为 `charlie` 或 `doro`）的完全限定的主机名。

   3. 单击添加。Hawk2 会为新添加的群集站点显示另一个选项卡，提供该群集站点的节点和资源概览。

      图 3︰ 包含两个群集站点的 Hawk2 仪表板 #

5. 要查看群集站点的更多细节或管理群集站点，请切换到站点的选项卡并单击锁链图标。

   Hawk2 会在新的浏览器窗口或选项卡中打开此站点的状态视图。在此视图中，您可以管理 Geo 群集的这部分内容。

## 10 后续步骤 #

使用 Geo 群集引导脚本，您可以快速地设置可用于测试用途的基本 Geo 群集。不过，要将创建的 Geo 群集迁移到正常运行且可用于生产环境的 Geo 群集，则需执行更多步骤。

完成 Geo 群集设置所需的步骤 #

在群集站点上启动投票间服务

  引导过程完成后，仲裁方投票间服务还不能与群集站点上的投票间服务通讯，因为这些服务默认不会启动。

  每个群集站点的投票间服务都由投票间资源组 `g-booth` 管理（请参见例 2 “`crm cluster geo_init` 创建的群集资源”）。要为每个站点启动一个投票间服务实例，请在每个群集站点上启动相应的投票间资源组。这样可让所有投票间实例互相通讯。

配置票据依赖项和顺序约束

  要使资源依赖您已在 Geo 群集引导过程中创建的票据，请配置约束。请为每个约束设置 `loss-policy`，它用于定义当从群集站点撤消票据时相应资源应发生的情况。

  有关详细信息，请参见Book “Geo 群集指南”, Chapter 6 “配置群集资源和约束”。

在开始时向站点授予票据

  在投票间可以管理 Geo 群集中的特定票据前，您首先需要手动将该票据授予给站点。您可以使用投票间客户端命令行工具或 Hawk2 授予票据。

  有关细节，请参见Book “Geo 群集指南”, Chapter 8 “管理 Geo 群集”。

引导脚本会在两个群集站点上创建相同的投票间资源，并在所有站点（包括仲裁方）上创建相同的投票间配置文件。如果您扩展 Geo 群集设置（以便迁移到生产环境），可能需要调整投票间配置，并更改与投票间有关的群集资源的配置。之后，您需要将更改同步到 Geo 群集的其余站点才能使其生效。

注意：跨群集站点同步更改

- 要将投票间配置中的更改同步到所有群集站点（包括仲裁方），请使用 Csync2。有关详细信息，请参见Book “Geo 群集指南”, Chapter 5 “跨所有站点和仲裁方同步配置文件”。

- CIB（群集信息数据库）不会跨 Geo 群集的群集站点自动同步。这意味着需要将所有群集站点上必需的资源配置中的所有更改手动传输到其余站点。要执行此操作，请标记相应资源，从当前 CIB 导出这些资源并将其导入到其余群集站点上的 CIB。有关详细信息，请参见Book “Geo 群集指南”, Chapter 6 “配置群集资源和约束”, Section 6.4 “将资源配置传输到其他群集站点”。

## 11 更多信息 #

- https://documentation.suse.com/sle-ha/ 上提供了更多有关此产品的文档。有关其他配置和管理任务，请参见详尽的 Geo Clustering Guide。

- 有关通过 DRBD 在 Geo 群集中复制数据的信息，请参见以下 SUSE Best Practices document。

# A GNU licenses #

修订历史: Geo 群集快速入门

This appendix contains the GNU Free Documentation License version 1.2.

## GNU Free Documentation License #

修订历史: Geo 群集快速入门

Copyright (C) 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

##### 0. PREAMBLE #

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or non-commercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

##### 1. APPLICABILITY AND DEFINITIONS #

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

##### 2. VERBATIM COPYING #

You may copy and distribute the Document in any medium, either commercially or non-commercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

##### 3. COPYING IN QUANTITY #

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

##### 4. MODIFICATIONS #

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.

2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.

3. State on the Title page the name of the publisher of the Modified Version, as the publisher.

4. Preserve all the copyright notices of the Document.

5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.

6. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.

7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.

8. Include an unaltered copy of this License.

9. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.

10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.

11. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.

12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.

13. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.

14. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.

15. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

##### 5. COMBINING DOCUMENTS #

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

##### 6. COLLECTIONS OF DOCUMENTS #

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

##### 7. AGGREGATION WITH INDEPENDENT WORKS #

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

##### 8. TRANSLATION #

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

##### 9. TERMINATION #

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

##### 10. FUTURE REVISIONS OF THIS LICENSE #

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See https://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

##### ADDENDUM: How to use this License for your documents #

    Copyright (c) YEAR YOUR NAME.
    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.2
    or any later version published by the Free Software Foundation;
    with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
    A copy of the license is included in the section entitled “GNU
    Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.” line with this:

    with the Invariant Sections being LIST THEIR TITLES, with the
    Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.