使用 DRBD 和 Pacemaker 的高度可用 NFS 储存
SUSE Linux Enterprise High Availability Extension 15 SP4

# 使用 DRBD 和 Pacemaker 的高度可用 NFS 储存 #

出版日期：2025 年 12 月 11 日

此文档介绍了如何使用以下组件在双节点群集中设置高度可用的 NFS 储存：DRBD*（Distributed Replicated Block Device，分布式复制块设备）、LVM（Logical Volume Manager，逻辑卷管理器）和群集资源管理器 Pacemaker。

作者: Tanja Roth、Thomas Schraitle
修订历史: 使用 DRBD 和 Pacemaker 的高度可用 NFS 储存

## 1 使用情形 #

本文档将帮助您设置高度可用的 NFS 服务器。用于实现高度可用的 NFS 储存的群集具有以下属性：

- 两个节点：`alice`（IP：`192.168.1.1`）和 `bob`（IP：`192.168.1.2`），两者之间通过网络彼此相连。

- 两个浮动虚拟 IP 地址（`192.168.1.10` 和 `192.168.2.1`），这样无论服务在哪个物理节点上运行，客户端都可连接到服务。一个 IP 地址用于使用 Hawk2 进行群集管理，另一个 IP 地址专用于 NFS 导出。

- 一台共享储存设备，用作 SBD 屏蔽机制。可避免节点分裂的情况。

- 当活动的主机发生故障（主动/被动设置）时，资源从一个节点故障转移至另一个节点。

- 每台主机上的本地储存。使用 LVM 上的 DRBD 在主机之间同步数据。

- 通过 NFS 导出的文件系统。

安装并设置好基本的双节点群集后，再使用 NFS 的储存和群集资源对其进行扩展，即可获得一个高度可用的 NFS 储存服务器。

## 2 安装基本的双节点群集 #

继续前，您需要安装并设置一个基本的双节点群集。《Installation and Setup Quick Start》（安装和设置快速入门）中介绍了此任务。《Installation and Setup Quick Start》（安装和设置快速入门）介绍了如何使用 crm 外壳花费极少的精力设置群集。

## 3 创建 LVM 设备 #

LVM（逻辑卷管理器）支持在多个文件系统上灵活分配硬盘空间。

要准备供 LVM 使用的磁盘，请执行以下操作：

1. 创建 LVM 卷组，然后使用要用于 LVM 的相应设备替换 `/dev/sdb1`：

       root # pvcreate /dev/sdb1

2. 创建包含此物理卷的 LVM 卷组 `nfs`：

       root # vgcreate nfs /dev/sdb1

3. 在卷组 `nfs` 中创建一个或多个逻辑卷。下面的示例会创建一个名为 `work` 的 20 GB 逻辑卷：

       root # lvcreate -n work -L 20G nfs

4. 激活卷组：

       root # vgchange -ay nfs

成功执行上述步骤后，您的系统将会显示以下设备：`/dev/VOLGROUP/LOGICAL_VOLUME`。在本示例中，该设备为 `/dev/nfs/work`。

## 4 创建 DRBD 设备 #

本章介绍如何在 LVM 上设置 DRBD 设备。使用 LVM 作为 DRBD 后端的配置具有以下优点：

- 比在 DRBD 上使用 LVM 的配置更容易设置。

- 当需要调整 LVM 磁盘的大小或有更多磁盘添加到卷组时，管理起来更容易。

由于 LVM 卷组名为 `nfs`，DRBD 资源也会使用该名称。

### 4.1 创建 DRBD 配置 #

出于一致性原因，强烈建议您遵循以下建议：

- 为您的配置使用 `/etc/drbd.d/` 目录。

- 根据资源的用途为文件命名。

- 将您的资源配置置于扩展名为 `.res` 的文件中。以下示例中使用 `/etc/drbd.d/nfs.res` 文件。

按如下所示继续：

过程 1︰ 创建 DRBD 配置 #

1. 创建包含以下内容的 `/etc/drbd.d/nfs.res` 文件：

       resource nfs {
          device /dev/drbd0; 1
          disk   /dev/nfs/work; 2
          meta-disk internal; 3

          net {
             protocol  C; 4
             fencing resource-and-stonith;
          }

          handlers { 5
             fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
             after-resync-target "/usr/lib/drbd/crm-unfence-peer.9.sh";
             # ...
          }

          connection-mesh { 6
             hosts     alice bob;
          }
          on alice { 7
             address   192.168.1.1:7790;
             node-id   0;
          }
          on bob { 7
             address   192.168.1.2:7790;
             node-id   1;
          }
       }

   1 | 应用程序预期要访问的 DRBD 设备。
   2 | DRBD 用于储存实际数据的较低级别块设备。这是在第 3 节 “创建 LVM 设备”中创建的 LVM 设备。
   3 | 储存元数据格式的位置。使用 `internal` 的情况下，元数据将与用户数据一起储存在同一设备上。有关更多信息，请参见手册页。
   4 | 所指定要用于此连接的协议。如果是协议 `C`，则会将已到达所有磁盘（本地或远程）的写入操作视为完成。
   5 | 启用资源级别屏蔽。如果 DRBD 复制链路的连接中断，Pacemaker 会尝试将 DRBD 资源升级到另一个节点。在此过程中会调用脚本。有关更多信息，请参见Book “管理指南”, Chapter 21 “DRBD”, Section 21.6 “搭配使用资源级屏蔽与 STONITH”。
   6 | 定义网格的所有节点。`hosts` 参数包含共享相同 DRBD 设置的所有主机名。
   7 | 包含每个节点的 IP 地址和唯一标识符。

2. 打开 `/etc/csync2/csync2.cfg` 并检查是否存在以下两行：

       include /etc/drbd.conf;
       include /etc/drbd.d/*.res;

   如果不存在，请将这两行添加到文件中。

3. 将文件复制到其他节点上：

       root # csync2 -xv

   有关 Csync2 的信息，请参见Book “管理指南”, Chapter 4 “使用 YaST 群集模块”, Section 4.5 “将配置传输到所有节点”。

### 4.2 激活 DRBD 设备 #

准备好 DRBD 配置后，请执行以下操作：

1. 如果您在群集中使用防火墙，请在防火墙配置中打开端口 `7790`。

2. 如果是第一次执行此操作，请在两个节点（在本示例中为 `alice` 和 `bob`）上执行以下命令：

       root # drbdadm create-md nfs
       root # drbdadm up nfs

   此命令会初始化元数据储存并创建 `/dev/drbd0` 设备。

3. 如果所有节点上的 DRBD 设备的数据都相同，请跳过初始的重新同步过程。使用以下命令：

       root # drbdadm new-current-uuid --clear-bitmap nfs/0

4. 将 `alice` 设为主节点：

       root # drbdadm primary --force nfs

5. 检查 DRBD 状态：

       root # drbdadm status nfs

   此命令将返回以下消息：

       nfs role:Primary
         disk:UpToDate
         alice role:Secondary
           peer-disk:UpToDate

同步完成后，您便可访问块设备 `/dev/drbd0` 上的 DRBD 资源。将使用此设备来创建您的文件系统。有关 DRBD 的详细信息，请参见Book “管理指南”, Chapter 21 “DRBD”。

### 4.3 创建文件系统 #

完成第 4.2 节 “激活 DRBD 设备”后，您应该能在 `/dev/drbd0` 上看到 DRBD 设备：

    root # mkfs.ext3 /dev/drbd0

## 5 调整 Pacemaker 的配置 #

当原始节点恢复联机并位于群集中时，资源可能会故障回复到该节点。为防止资源故障回复到之前运行它的节点，或者要指定让该资源故障回复到其他节点，请更改其资源粘性值。可以在创建资源时或之后指定资源粘性。

要调整选项，请以 `root` 身份（或任何属于 `haclient` 组的非 `root` 用户身份）打开 crm 外壳并运行以下命令：

    root # crm configure
    crm(live)configure# rsc_defaults resource-stickiness="200"
    crm(live)configure# commit

有关全局群集选项的详细信息，请参见Book “管理指南”, Chapter 6 “配置和管理基础”, Section 6.2 “仲裁判定”。

## 6 创建群集资源 #

下面几节介绍高度可用的 NFS 群集所需资源的配置。使用 crm 外壳来执行配置步骤。下面的列表列出了必要的群集资源：

群集资源概述 #

DRBD 原始资源和可升级克隆资源

  这些资源用于复制数据。群集资源管理器认为有必要时，会将可升级克隆资源在主次角色之间切换。

NFS 内核服务器资源

  使用此资源，Pacemaker 可确保 NFS 服务器守护程序始终可用。

NFS 导出

  一或多个 NFS 导出，通常对应于文件系统。

示例 NFS 情形 #

- 下列配置示例假设 `192.168.2.1` 是用于为 `192.168.2.x/24` 子网中的客户端传递数据的 NFS 服务器的虚拟 IP 地址。

- 服务从 `/srv/nfs/work` 导出传递的数据。

- 在此导出目录中，群集将从 DRBD 设备 `/dev/drbd0` 挂载 `ext3` 文件系统。此 DRBD 设备位于名为 `nfs` 的 LVM 逻辑卷上。

### 6.1 DRBD 原始资源和可升级克隆资源 #

要配置这些资源，请从 crm 外壳运行以下命令：

    crm(live)# configure
    crm(live)configure# primitive drbd_nfs \
      ocf:linbit:drbd \
        params drbd_resource="nfs" \
      op monitor interval="15" role="Promoted" \
      op monitor interval="30" role="Unpromoted"
    crm(live)configure# clone clone-drbd_nfs drbd_nfs \
      meta promotable="true" promoted-max="1" promoted-node-max="1" clone-max="2" \
      clone-node-max="1" notify="true"
    crm(live)configure# commit

此命令会创建与 DRBD 资源 `nfs` 对应的 Pacemaker 可提升克隆资源。Pacemaker 现在应该会在两个节点上激活您的 DRBD 资源，并在其中一个节点上将该资源提升为主要角色。

使用 `crm status` 命令或运行 `drbdadm status` 检查群集的状态。

### 6.2 NFS 内核服务器资源 #

在 crm 外壳中，NFS 服务器守护程序的资源必须配置为 `systemd` 资源类型的克隆。

    crm(live)configure# primitive nfsserver \
      systemd:nfs-server \
      op monitor interval="30s"
    crm(live)configure# clone cl-nfsserver nfsserver \
       meta interleave=true
    crm(live)configure# commit

提交此配置后，Pacemaker 应该就会在两个节点上启动 NFS 内核服务器进程。

### 6.3 文件系统资源 #

1. 按如下所示配置文件系统类型资源（但先不要提交此配置）：

       crm(live)configure# primitive fs_work \
         ocf:heartbeat:Filesystem \
         params device=/dev/drbd0 \
           directory=/srv/nfs/work \
           fstype=ext3 \
         op monitor interval="10s"

2. 将这些资源合并到一个 Pacemaker 资源组中：

       crm(live)configure# group g-nfs fs_work

3. 添加以下约束，以确保该组在 DRBD 可提升克隆资源为主要角色的节点上启动：

       crm(live)configure# order o-drbd_before_nfs Mandatory: \
         clone-drbd_nfs:promote g-nfs:start
       crm(live)configure# colocation col-nfs_on_drbd inf: \
         g-nfs clone-drbd_nfs:Promoted

4. 提交此配置：

       crm(live)configure# commit

提交这些更改后，Pacemaker 会将 DRBD 设备挂载到同一节点上的 `/srv/nfs/work`。请使用 `mount`（或查看 `/proc/mounts`）确认这一点。

### 6.4 NFS 导出资源 #

当您的 DRBD、LVM 和文件系统资源正常工作时，继续使用用于管理 NFS 导出的资源。要创建高度可用的 NFS 导出资源，请使用 `exportfs` 资源类型。

要将 `/srv/nfs/work` 目录导出到客户端，请使用以下原始资源：

1. 使用以下命令创建 NFS 导出：

       crm(live)configure# primitive exportfs_work \
         ocf:heartbeat:exportfs \
           params directory="/srv/nfs/work" \
             options="rw,mountpoint" \
             clientspec="192.168.2.0/24" \
             wait_for_leasetime_on_stop=true \
             fsid=100 \
         op monitor interval="30s"

2. 创建这些资源后，将它们追加到现有的 `g-nfs` 资源组：

       crm(live)configure# modgroup g-nfs add exportfs_work

3. 提交此配置：

       crm(live)configure# commit

   Pacemaker 将导出 NFS 虚拟根文件系统及两个其他导出。

4. 确认 NFS 导出设置正确：

       root # exportfs -v
       /srv/nfs/work   IP_ADDRESS_OF_CLIENT(OPTIONS)

### 6.5 NFS 导出的虚拟 IP 地址 #

初始安装会创建用于 Hawk2 的管理虚拟 IP 地址。虽然您也可将此 IP 地址用于 NFS 导出，但最好另外创建一个专用于 NFS 导出的 IP 地址。这样，以后便可更轻松地应用安全限制。在 crm 外壳中使用以下命令：

    crm(live)configure# primitive vip_nfs IPaddr2 \
       params ip=192.168.2.1 cidr_netmask=24 \
       op monitor interval=10 timeout=20
    crm(live)configure# modgroup g-nfs add vip_nfs
    crm(live)configure# commit

## 7 使用 NFS 服务 #

本章概述如何从 NFS 客户端使用高度可用的 NFS 服务。

要连接到 NFS 服务，请务必使用虚拟 IP 地址来连接群集，而不要使用其中一个群集节点的网络接口上配置的物理 IP。出于兼容性原因，请使用服务器上 NFS 导出的完整路径。

最简单的形式是，使用如下所示的命令挂载 NFS 导出：

    root # mount -t nfs 192.168.2.1:/srv/nfs/work /home/work

要配置特定传输协议 (`proto`) 和最大读取和写入请求大小（`rsize` 和 `wsize`），请使用以下命令：

    root # mount -o rsize=32768,wsize=32768 \
        192.168.2.1:/srv/nfs/work /home/work

如果您需要与 NFS 版本 3 兼容，请在 `-o` 选项后添加值 `vers=3`。

要了解更多 NFS 挂载选项，请参见 `nfs` 手册页。

## 8 法律声明 #

版权所有 © 2006– 2025 SUSE LLC 和贡献者。保留所有权利。

根据 GNU 自由文档许可 (GNU Free Documentation License) 版本 1.2 或（根据您的选择）版本 1.3 中的条款，在此授予您复制、分发和/或修改本文档的权限；本版权声明和许可附带不可变部分。许可版本 1.2 的副本包含在题为“GNU Free Documentation License”的部分。

有关 SUSE 商标，请参见 http://www.suse.com/company/legal/。所有其他第三方商标分别为相应所有者的财产。商标符号（®、™ 等）代表 SUSE 及其关联公司的商标。星号 (*) 代表第三方商标。

本指南力求涵盖所有细节，但这不能确保本指南准确无误。SUSE LLC 及其关联公司、作者和译者对于可能出现的错误或由此造成的后果皆不承担责任。

# A GNU licenses #

This appendix contains the GNU Free Documentation License version 1.2.

## GNU Free Documentation License #

Copyright (C) 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

##### 0. PREAMBLE #

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or non-commercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

##### 1. APPLICABILITY AND DEFINITIONS #

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

##### 2. VERBATIM COPYING #

You may copy and distribute the Document in any medium, either commercially or non-commercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

##### 3. COPYING IN QUANTITY #

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

##### 4. MODIFICATIONS #

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.

2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.

3. State on the Title page the name of the publisher of the Modified Version, as the publisher.

4. Preserve all the copyright notices of the Document.

5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.

6. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.

7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.

8. Include an unaltered copy of this License.

9. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.

10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.

11. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.

12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.

13. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.

14. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.

15. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

##### 5. COMBINING DOCUMENTS #

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

##### 6. COLLECTIONS OF DOCUMENTS #

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

##### 7. AGGREGATION WITH INDEPENDENT WORKS #

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

##### 8. TRANSLATION #

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

##### 9. TERMINATION #

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

##### 10. FUTURE REVISIONS OF THIS LICENSE #

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See https://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

##### ADDENDUM: How to use this License for your documents #

    Copyright (c) YEAR YOUR NAME.
    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.2
    or any later version published by the Free Software Foundation;
    with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
    A copy of the license is included in the section entitled “GNU
    Free Documentation License”.

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “with...Texts.” line with this:

    with the Invariant Sections being LIST THEIR TITLES, with the
    Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.