使用 DRBD 和 Pacemaker 的高度可用 NFS 存储系统
SUSE Linux Enterprise High Availability Extension 15 SP5

# 使用 DRBD 和 Pacemaker 的高度可用 NFS 存储系统 #

出版日期：2025 年 12 月 11 日

此文档介绍了如何使用以下组件在双节点群集中设置高度可用的 NFS 存储系统：DRBD*（Distributed Replicated Block Device，分布式复制块设备）、LVM（Logical Volume Manager，逻辑卷管理器）和群集资源管理器 Pacemaker。

修订历史: 使用 DRBD 和 Pacemaker 的高度可用 NFS 存储系统

版权所有 © 2006–2025 SUSE LLC 和贡献者。保留所有权利。

根据 GNU 自由文档许可 (GNU Free Documentation License) 版本 1.2 或（根据您的选择）版本 1.3 中的条款，在此授予您复制、分发和/或修改本文档的权限；本版权声明和许可附带不可变部分。许可版本 1.2 的副本包含在题为“GNU Free Documentation License”的部分。

有关 SUSE 商标，请参见 http://www.suse.com/company/legal/。所有第三方商标均是其各自所有者的财产。商标符号（®、™ 等）代表 SUSE 及其关联公司的商标。星号 (*) 代表第三方商标。

本指南力求涵盖所有细节，但这不能确保本指南准确无误。SUSE LLC 及其关联公司、作者和译者对于可能出现的错误或由此造成的后果皆不承担责任。

## 1 使用情形 #

本文档将帮助您设置高度可用的 NFS 服务器。用于实现高度可用的 NFS 存储的群集具有以下属性：

- 两个节点：`alice`（IP：`192.168.1.1`）和 `bob`（IP：`192.168.1.2`），两者之间通过网络彼此相连。

- 两个浮动虚拟 IP 地址（`192.168.1.10` 和 `192.168.1.11`），这样无论服务在哪个物理节点上运行，客户端都可连接到服务。一个 IP 地址用于使用 Hawk2 进行群集管理，另一个 IP 地址专用于 NFS 导出。

- SBD 用作 STONITH 屏蔽设备，以避免分裂情况。STONITH 对于 HA 群集而言是必需的。

- 当活动的主机发生故障（主动/被动设置）时，资源从一个节点故障转移至另一个节点。

- 每个节点上的本地存储。使用 LVM 上的 DRBD 在节点之间同步数据。

- 通过 NFS 导出的文件系统，以及一个用于跟踪 NFS 客户端状态的单独文件系统。

安装并设置好基本的双节点群集后，再使用 NFS 的存储和群集资源对其进行扩展，即可获得一个高度可用的 NFS 存储服务器。

## 2 准备双节点群集 #

在设置高度可用的 NFS 存储之前，您必须先准备好一个高可用性群集：

过程 1︰ 为 NFS 存储准备两节点群集 #

1. 按《安装和设置快速入门》中所述安装并设置一个基本的双节点群集。

2. 在两个节点上安装软件包 nfs-kernel-server：

       # zypper install nfs-kernel-server

## 3 创建 LVM 设备 #

使用 LVM（逻辑卷管理器）可在多个文件系统上灵活分配存储空间。

使用 `crm cluster run` 同时在两个节点上运行这些命令。

过程 2︰ 为 DRBD 创建 LVM 设备 #

1. 创建 LVM 物理卷，并将 `/dev/sdb1` 替换为 LVM 的相应设备：

       # crm cluster run "pvcreate /dev/sdb1"

2. 创建包含此物理卷的 LVM 卷组 `nfs`：

       # crm cluster run "vgcreate nfs /dev/sdb1"

3. 在卷组 `nfs` 中创建一个名为 `share` 的逻辑卷：

       # crm cluster run "lvcreate -n share -L 20G nfs"

   此卷用于 NFS 导出。

4. 在卷组 `nfs` 中创建一个名为 `state` 的逻辑卷：

       # crm cluster run "lvcreate -n state -L 8G nfs"

   此卷用于跟踪 NFS 客户端状态。本示例中使用的 8 GB 卷大小应支持数千个并发 NFS 客户端。

5. 激活卷组：

       # crm cluster run "vgchange -ay nfs"

现在应会在系统上看到以下设备：`/dev/nfs/share` 和 `/dev/nfs/state`。

## 4 创建 DRBD 设备 #

本章介绍如何在 LVM 上设置 DRBD 设备。使用 LVM 作为 DRBD 的后端可获得以下好处：

- 比在 DRBD 上使用 LVM 的配置更容易设置。

- 当需要调整 LVM 磁盘的大小或有更多磁盘添加到卷组时，管理起来更容易。

以下过程会创建两个 DRBD 设备：一个设备用于 NFS 导出，另一个设备用于跟踪 NFS 客户端状态。

### 4.1 创建 DRBD 配置 #

DRBD 配置文件保存在 `/etc/drbd.d/` 目录中，必须以 `.res` 扩展名结尾。在此过程中，配置文件命名为 `/etc/drbd.d/nfs.res`。

过程 3︰ 创建 DRBD 配置 #

1. 创建包含以下内容的 `/etc/drbd.d/nfs.res` 文件：

       resource nfs {
          volume 0 { 1
             device           /dev/drbd0; 2
             disk             /dev/nfs/state; 3
             meta-disk        internal; 4
          }
          volume 1 {
             device           /dev/drbd1;
             disk             /dev/nfs/share;
             meta-disk        internal;
          }

          net {
             protocol  C; 5
             fencing resource-and-stonith; 6
          }

          handlers { 7
             fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
             after-resync-target "/usr/lib/drbd/crm-unfence-peer.9.sh";
          }

          connection-mesh { 8
             hosts     alice bob;
          }
          on alice { 9
             address   192.168.1.1:7790;
             node-id   0;
          }
          on bob { 9
             address   192.168.1.2:7790;
             node-id   1;
          }
       }

   1 | 要创建的每个 DRBD 设备的卷号。
   2 | 应用程序要访问的 DRBD 设备。
   3 | DRBD 用于存储实际数据的较低级别块设备。这是在第 3 节 “创建 LVM 设备”中创建的 LVM 设备。
   4 | 元数据的存储位置。如果使用 `internal`，元数据将与用户数据一起存储在同一设备上。有关更多信息，请参见手册页。
   5 | 此连接使用的协议。协议 `C` 提供更好的数据可用性，只有在写入内容进入所有本地和远程磁盘之后，该协议才将写入操作视为已完成。
   6 | 指定屏蔽策略。对于配置了 STONITH 设备的群集，请使用 `resource-and-stonith`。
   7 | 启用资源级别屏蔽。如果 DRBD 复制链路的连接中断，Pacemaker 会尝试将 DRBD 资源升级到另一个节点。有关详细信息，请参见Book “管理指南”, Chapter 22 “DRBD”, Section 22.6 “搭配使用资源级屏蔽与 STONITH”。
   8 | 定义网格的所有节点。`hosts` 参数包含共享相同 DRBD 设置的所有主机名。
   9 | 包含每个节点的 IP 地址和唯一标识符。

2. 打开 `/etc/csync2/csync2.cfg` 并检查其中是否包含以下两行：

       include /etc/drbd.conf;
       include /etc/drbd.d;

   如果不存在，请将这两行添加到文件中。

3. 将文件复制到其他节点上：

       # csync2 -xv

   有关 Csync2 的信息，请参见Book “管理指南”, Chapter 4 “使用 YaST 群集模块”, Section 4.7 “将配置传输到所有节点”。

### 4.2 激活 DRBD 设备 #

准备好 DRBD 配置后，激活设备：

过程 4︰ 激活 DRBD 设备 #

1. 如果您在群集中使用防火墙，请在防火墙配置中打开端口 `7790`。

2. 初始化元数据存储：

       # crm cluster run "drbdadm create-md nfs"

3. 创建 DRBD 设备：

       # crm cluster run "drbdadm up nfs"

4. 设备中尚不包含数据，因此您可以运行以下命令来跳过初始同步：

       # drbdadm new-current-uuid --clear-bitmap nfs/0
       # drbdadm new-current-uuid --clear-bitmap nfs/1

5. 将 `alice` 设为主节点：

       # drbdadm primary --force nfs

6. 检查 `nfs` 的 DRBD 状态：

       # drbdadm status nfs

   此命令将返回以下消息：

       nfs role:Primary
         volume:0 disk:UpToDate
         volume:1 disk:UpToDate
         bob role:Secondary
           volume:0 peer-disk:UpToDate
           volume:1 peer-disk:UpToDate

您可以访问块设备 `/dev/drbd0` 和 `/dev/drbd1` 上的 DRBD 资源。

### 4.3 创建文件系统 #

激活 DRBD 设备后，在其上创建文件系统：

过程 5︰ 为 DRBD 创建文件系统 #

1. 在 `/dev/drbd0` 上创建 `ext4` 文件系统：

       # mkfs.ext4 /dev/drbd0

2. 在 `/dev/drbd1` 上创建 `ext4` 文件系统：

       # mkfs.ext4 /dev/drbd1

## 5 创建群集资源 #

以下过程说明如何配置高度可用的 NFS 群集所需的资源。

群集资源概述 #

DRBD 原始资源和可升级克隆资源

  这些资源用于复制数据。群集资源管理器认为有必要时，会将可升级克隆资源在主次角色之间切换。

文件系统资源

  这些资源将管理要导出的文件系统，以及要跟踪 NFS 客户端状态的文件系统。

NFS 内核服务器资源

  此资源管理 NFS 服务器守护程序。

NFS 导出

  此资源用于将目录 `/srv/nfs/share` 导出到客户端。

虚拟 IP 地址

  初始安装会创建用于 Hawk2 的管理虚拟 IP 地址。创建另一个专用于 NFS 导出的虚拟 IP 地址。这样，以后便可更轻松地应用安全限制。

示例 NFS 情形 #

- 下列配置示例假设 `192.168.1.11` 是用于为 `192.168.1.x/24` 子网中的客户端传递数据的 NFS 服务器的虚拟 IP 地址。

- 服务导出从 `/srv/nfs/share` 传递的数据。

- 群集将 DRBD 设备 `/dev/drbd1` 中的 `ext4` 文件系统挂载到此导出目录中。此 DRBD 设备位于名为 `/dev/nfs/share` 的 LVM 逻辑卷上。

- DRBD 设备 `/dev/drbd0` 用于共享 `/var/lib/nfs` 中的 NFS 客户端状态。此 DRBD 设备位于名为 `/dev/nfs/state` 的 LVM 逻辑卷上。

### 5.1 创建 DRBD 原始资源和可升级克隆资源 #

创建一个群集资源用于管理 DRBD 设备，并创建一个可升级克隆资源，使此群集资源可在两个节点上运行：

过程 6︰ 为 NFS 创建 DRBD 资源 #

1. 启动 `crm` 交互外壳：

       # crm configure

2. 为 DRBD 配置 `nfs` 创建原始资源：

       crm(live)configure# primitive drbd-nfs ocf:linbit:drbd \
         params drbd_resource="nfs" \
         op monitor interval=15 role=Promoted \
         op monitor interval=30 role=Unpromoted

3. 为 `drbd-nfs` 原始资源创建一个可升级克隆资源：

       crm(live)configure# clone cl-drbd-nfs drbd-nfs \
         meta promotable="true" promoted-max="1" promoted-node-max="1" \
         clone-max="2" clone-node-max="1" notify="true" interleave=true

4. 提交此配置：

       crm(live)configure# commit

Pacemaker 将激活两个节点上的 DRBD 资源，并将其升级为其中一个节点上的主要角色。使用 `crm status` 命令检查群集的状态，或运行 `drbdadm status`。

### 5.2 创建文件系统资源 #

创建群集资源来管理要导出的文件系统和用于状态跟踪的文件系统：

过程 7︰ 为 NFS 创建文件系统资源 #

1. 在 `/dev/drbd0` 上创建一个用于跟踪 NFS 客户端状态的原始资源：

       crm(live)configure# primitive fs-nfs-state Filesystem \
         params device=/dev/drbd0 directory=/var/lib/nfs fstype=ext4

2. 在 `/dev/drbd1` 上为要导出的文件系统创建一个原始资源：

       crm(live)configure# primitive fs-nfs-share Filesystem \
         params device=/dev/drbd1 directory=/srv/nfs/share fstype=ext4

   请仅在添加并置约束和顺序约束之后才提交此配置。

3. 将这两个资源添加到名为 `g-nfs` 的资源组：

       crm(live)configure# group g-nfs fs-nfs-state fs-nfs-share

   资源按照它们在组中的添加顺序启动，并按照相反的顺序停止。

4. 添加共置约束，以确保资源组始终在 DRBD 可升级克隆资源充当主要角色的节点上启动：

       crm(live)configure# colocation col-nfs-on-drbd inf: g-nfs cl-drbd-nfs:Promoted

5. 添加顺序约束，以确保 DRBD 可升级克隆资源始终在资源组之前启动：

       crm(live)configure# order o-drbd-before-nfs Mandatory: cl-drbd-nfs:promote g-nfs:start

6. 提交此配置：

       crm(live)configure# commit

Pacemaker 将 `/dev/drbd0` 挂载到 `/var/lib/nfs`，将 `/dev/drbd1` 挂载到 `srv/nfs/share`。使用 `mount` 或通过查看 `/proc/mounts` 来确认是否如此。

### 5.3 创建 NFS 内核服务器资源 #

创建一个群集资源来管理 NFS 服务器守护程序：

过程 8︰ 创建 NFS 内核服务器资源 #

1. 创建一个原始资源来管理 NFS 服务器守护程序：

       crm(live)configure# primitive nfsserver nfsserver \
         params nfs_server_scope=SUSE nfs_shared_infodir="/var/lib/nfs"

   在群集中运行 NFS 服务器的所有节点上，`nfs_server_scope` 必须相同，但默认情况下并非采用这种设置。所有使用 SUSE 软件的群集都可以使用相同的范围，因此我们建议将该值设置为 `SUSE`。

   警告：过短的租用时间可能会导致文件状态丢失

   NFS 客户端定期与 NFS 服务器续订其状态。如果租用时间过短，系统或网络延迟可能会导致计时器在续订完成之前失效，从而导致发生 I/O 错误并丢失文件状态。

   `NFSV4LEASETIME` 是在 NFS 服务器上的 `/etc/sysconfig/nfs` 文件中设置的。默认值为 90 秒。如果有必要缩短租用时间，我们建议将值设置为 60 或更大。强烈反对设置小于 30 的值。

2. 将此资源追加到现有的 `g-nfs` 资源组：

       crm(live)configure# modgroup g-nfs add nfsserver

3. 提交此配置：

       crm(live)configure# commit

### 5.4 创建 NFS 导出资源 #

创建一个群集资源来管理 NFS 导出：

过程 9︰ 创建 NFS 导出资源 #

1. 为 NFS 导出创建原始资源：

       crm(live)configure# primitive exportfs-nfs exportfs \
         params directory="/srv/nfs/share" \
         options="rw,mountpoint" clientspec="192.168.1.0/24" fsid=101 \1
         op monitor interval=30s timeout=90s2

   1 | `fsid` 对于每个 NFS 导出资源必须是唯一的。
   2 | `op monitor timeout` 的值必须大于 `stonith-timeout` 的值。要查找 `stonith-timeout` 值，请运行 `crm configure show` 并查看 `property` 部分。

   重要：不要设置 `wait_for_leasetime_on_stop=true`

   在高度可用的 NFS 设置中将此选项设置为 `true` 可能会导致不必要的延迟和锁丢失。

   `wait_for_leasetime_on_stop` 的默认值为 `false`。如果已按本指南中所述配置了 `/var/lib/nfs` 和 `nfsserver`，则无需将其设置为 `true`。

2. 将此资源追加到现有的 `g-nfs` 资源组：

       crm(live)configure# modgroup g-nfs add exportfs-nfs

3. 提交此配置：

       crm(live)configure# commit

4. 确认 NFS 导出设置正确：

       # exportfs -v
       /srv/nfs/share   IP_ADDRESS_OF_CLIENT(OPTIONS)

### 5.5 为 NFS 导出创建虚拟 IP 地址 #

创建一个群集资源来管理 NFS 导出的虚拟 IP 地址：

过程 10︰ 为 NFS 导出创建虚拟 IP 地址 #

1. 为虚拟 IP 地址创建原始资源：

       crm(live)configure# primitive vip-nfs IPaddr2 params ip=192.168.1.11

2. 将此资源追加到现有的 `g-nfs` 资源组：

       crm(live)configure# modgroup g-nfs add vip-nfs

3. 提交此配置：

       crm(live)configure# commit

4. 退出 `crm` 交互外壳：

       crm(live)configure# quit

5. 检查集群的状态。`g-nfs` 组中的资源应按以下顺序显示：

       # crm status
         [...]
         Full List of Resources
           [...]
           * Resource Group: g-nfs:
             * fs-nfs-state    (ocf:heartbeat:Filesystem):   Started alice
             * fs-nfs-share    (ocf:heartbeat:Filesystem):   Started alice
             * nfsserver       (ocf:heartbeat:nfsserver):    Started alice
             * exportfs-nfs    (ocf:heartbeat:exportfs):     Started alice
             * vip-nfs         (ocf:heartbeat:IPaddr2):      Started alice

## 6 使用 NFS 服务 #

本章概述如何从 NFS 客户端使用高度可用的 NFS 服务。

要连接到 NFS 服务，请务必使用虚拟 IP 地址来连接群集，而不要使用其中一个群集节点的网络接口上配置的物理 IP。出于兼容性原因，请使用服务器上 NFS 导出的完整路径。

用于挂载 NFS 导出的命令如下：

    # mount 192.168.1.11:/srv/nfs/share /home/share

如果您需要配置其他挂载选项，例如特定的传输协议 (`proto`)、最大读写请求大小（`rsize` 和 `wsize`）或特定的 NFS 版本 (`vers`)，请使用 `-o` 选项。例如：

    # mount -o proto=tcp,rsize=32768,wsize=32768,vers=3 \
    192.168.1.11:/srv/nfs/share /home/share

要了解更多 NFS 挂载选项，请参见 `nfs` 手册页。

注意：回写挂载

只有 NFS 版本 3 支持回写挂载，而 NFS 版本 4 则不支持。有关详细信息，请参见 https://www.suse.com/support/kb/doc/?id=000018709。

## 7 将更多 NFS 共享添加到群集 #

如果需要增加可用存储，您可以将更多 NFS 共享添加到群集。

在此示例中，名为 `/dev/drbd2` 的新 DRBD 设备位于名为 `/dev/nfs/share2` 的 LVM 逻辑卷上。

过程 11︰ 将更多 NFS 共享添加到群集 #

1. 为新共享创建 LVM 逻辑卷：

       # crm cluster run "lvcreate -n share2 -L 20G nfs"

2. 更新文件 `/etc/drbd.d/nfs.res` 以在现有卷下添加新卷：

          volume 2 {
             device           /dev/drbd2;
             disk             /dev/nfs/share2;
             meta-disk        internal;
          }

3. 将已更新的文件复制到其他节点：

       # csync2 -xv

4. 初始化新卷的元数据存储：

       # crm cluster run "drbdadm create-md nfs/2 --force"

5. 更新 `nfs` 配置以创建新设备：

       # crm cluster run "drbdadm adjust nfs"

6. 跳过新设备的初始同步：

       # drbdadm new-current-uuid --clear-bitmap nfs/2

7. NFS 群集资源自创建后可能已转移到其他节点。使用 `drbdadm status nfs` 检查 DRBD 状态，并记下哪个节点充当 `Primary` 角色。

8. 在充当 `Primary` 角色的节点上的 `/dev/drbd2` 中创建 `ext4` 文件系统：

       # mkfs.ext4 /dev/drbd2

9. 启动 `crm` 交互外壳：

       # crm configure

10. 在 `/dev/drbd2` 上为要导出的文件系统创建一个原始资源：

        crm(live)configure# primitive fs-nfs-share2 Filesystem \
          params device="/dev/drbd2" directory="/srv/nfs/share2" fstype=ext4

11. 将新文件系统资源添加到 `g-nfs` 组中的 `nfsserver` 资源之前：

        crm(live)configure# modgroup g-nfs add fs-nfs-share2 before nfsserver

12. 为新共享中的 NFS 导出创建原始资源：

        crm(live)configure# primitive exportfs-nfs2 exportfs \
          params directory="/srv/nfs/share2" \
          options="rw,mountpoint" clientspec="192.168.1.0/24" fsid=102 \
          op monitor interval=30s timeout=90s

13. 将新的 NFS 导出资源添加到 `g-nfs` 组中的 `vip-nfs` 资源之前：

        crm(live)configure# modgroup g-nfs add exportfs-nfs2 before vip-nfs

14. 提交此配置：

        crm(live)configure# commit

15. 退出 `crm` 交互外壳：

        crm(live)configure# quit

16. 检查集群的状态。`g-nfs` 组中的资源应按以下顺序显示：

        # crm status
        [...]
        Full List of Resources
          [...]
          * Resource Group: g-nfs:
            * fs-nfs-state    (ocf:heartbeat:Filesystem):   Started alice
            * fs-nfs-share    (ocf:heartbeat:Filesystem):   Started alice
            * fs-nfs-share2   (ocf:heartbeat:Filesystem):   Started alice
            * nfsserver       (ocf:heartbeat:nfsserver):    Started alice
            * exportfs-nfs    (ocf:heartbeat:exportfs):     Started alice
            * exportfs-nfs2   (ocf:heartbeat:exportfs):     Started alice
            * vip-nfs         (ocf:heartbeat:IPaddr2):      Started alice

17. 确认 NFS 导出设置正确：

        # exportfs -v
        /srv/nfs/share   IP_ADDRESS_OF_CLIENT(OPTIONS)
        /srv/nfs/share2  IP_ADDRESS_OF_CLIENT(OPTIONS)

## 8 更多信息 #

- 有关本指南中所述步骤的更多细节，请参见 https://www.suse.com/support/kb/doc/?id=000020396。

- 有关 NFS 和 LVM 的详细信息，请参见 《SUSE Linux Enterprise Server 存储管理指南》。

- 有关 DRBD 的详细信息，请参见Book “管理指南”, Chapter 22 “DRBD”。

- 有关群集资源的详细信息，请参见Book “管理指南”, Chapter 6 “配置群集资源”。

# A GNU licenses #

修订历史: 使用 DRBD 和 Pacemaker 的高度可用 NFS 存储系统

This appendix contains the GNU Free Documentation License version 1.2.

## GNU Free Documentation License #

修订历史: 使用 DRBD 和 Pacemaker 的高度可用 NFS 存储系统

Copyright (C) 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

##### 0. PREAMBLE #

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or non-commercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

##### 1. APPLICABILITY AND DEFINITIONS #

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

##### 2. VERBATIM COPYING #

You may copy and distribute the Document in any medium, either commercially or non-commercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

##### 3. COPYING IN QUANTITY #

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

##### 4. MODIFICATIONS #

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.

2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.

3. State on the Title page the name of the publisher of the Modified Version, as the publisher.

4. Preserve all the copyright notices of the Document.

5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.

6. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.

7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.

8. Include an unaltered copy of this License.

9. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.

10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.

11. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.

12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.

13. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.

14. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.

15. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

##### 5. COMBINING DOCUMENTS #

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

##### 6. COLLECTIONS OF DOCUMENTS #

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

##### 7. AGGREGATION WITH INDEPENDENT WORKS #

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

##### 8. TRANSLATION #

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

##### 9. TERMINATION #

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

##### 10. FUTURE REVISIONS OF THIS LICENSE #

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See https://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

##### ADDENDUM: How to use this License for your documents #

    Copyright (c) YEAR YOUR NAME.
    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.2
    or any later version published by the Free Software Foundation;
    with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
    A copy of the license is included in the section entitled “GNU
    Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.” line with this:

    with the Invariant Sections being LIST THEIR TITLES, with the
    Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.