Jump to content
SUSE CaaS Platform 4.5.1

Air gap Deployment Guide

This guide describes deployment for SUSE CaaS Platform 4.5.1 in an air gap environment.

Authors: Markus Napp and Nora Kořánová
Publication Date: 2020-10-20

Warning
Warning

This document is a work in progress.

The content in this document is subject to change without notice.

Note
Note

This guide assumes a configured SUSE Linux Enterprise Server 15 SP2 environment.

Copyright © 2006 — 2020 SUSE LLC and contributors. All rights reserved.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled GNU Free Documentation License.

For SUSE trademarks, see http://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™, etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.

About This Guide

1 Required Background

To keep the scope of these guidelines manageable, certain technical assumptions have been made. These documents are not aimed at beginners in Kubernetes usage and require that:

  • You have some computer experience and are familiar with common technical terms.

  • You are familiar with the documentation for your system and the network on which it runs.

  • You have a basic understanding of Linux systems.

  • You have an understanding of how to follow instructions aimed at experienced Linux administrators and can fill in gaps with your own research.

  • You understand how to plan, deploy and manage Kubernetes applications.

2 Available Documentation

We provide HTML and PDF versions of our books in different languages. Documentation for our products is available at https://documentation.suse.com/, where you can also find the latest updates and browse or download the documentation in various formats.

The following documentation is available for this product:

Deployment Guide

The SUSE CaaS Platform Deployment Guide gives you details about installation and configuration of SUSE CaaS Platform along with a description of architecture and minimum system requirements.

Quick Start Guide

The SUSE CaaS Platform Quick Start guides you through the installation of a minimum cluster in the fastest way possible.

Admin Guide

The SUSE CaaS Platform Admin Guide discusses authorization, updating clusters and individual nodes, monitoring, logging, use of Helm and Tiller, troubleshooting and integration with SUSE Enterprise Storage and SUSE Cloud Application Platform.

3 Feedback

Several feedback channels are available:

Bugs and Enhancement Requests

For services and support options available for your product, refer to http://www.suse.com/support/.

To report bugs for a product component, go to https://scc.suse.com/support/requests, log in, and click Create New.

User Comments

We want to hear your comments about and suggestions for this manual and the other documentation included with this product. Use the User Comments feature at the bottom of each page in the online documentation or go to https://documentation.suse.com/, click Feedback at the bottom of the page and enter your comments in the Feedback Form.

Mail

For feedback on the documentation of this product, you can also send a mail to doc-team@suse.com. Make sure to include the document title, the product version and the publication date of the documentation. To report errors or suggest enhancements, provide a concise description of the problem and refer to the respective section number and page (or URL).

4 Documentation Conventions

The following notices and typographical conventions are used in this documentation:

  • /etc/passwd : directory names and file names

  • <PLACEHOLDER>: replace <PLACEHOLDER> with the actual value

  • PATH: the environment variable PATH

  • ls, --help: commands, options, and parameters

  • user : users or groups

  • package name : name of a package

  • Alt, AltF1 : a key to press or a key combination; keys are shown in uppercase as on a keyboard

  • File › Save As : menu items, buttons

  • Dancing Penguins (Chapter Penguins, ↑Another Manual): This is a reference to a chapter in another manual.

  • Commands that must be run with root privileges. Often you can also prefix these commands with the sudo command to run them as non-privileged user.

    sudo command
  • Commands that can be run by non-privileged users.

    command
  • Notices:

    Warning
    Warning

    Vital information you must be aware of before proceeding. Warns you about security issues, potential loss of data, damage to hardware, or physical hazards.

    Important
    Important

    Important information you should be aware of before proceeding.

    Note
    Note

    Additional information, for example about differences in software versions.

    Tip
    Tip

    Helpful information, like a guideline or a piece of practical advice.

1 Air gapped deployment

An air gapped deployment is defined by not allowing any direct connection to the Internet or external networks from the cluster during setup or runtime.

All data that is transferred to the cluster must be transferred in a secure fashion.

Note
Note

Air gapped deployment can be performed with any of the other deployment types and includes a set of steps that need to be performed before, or during the deployment steps of the concrete deployment.

Important
Important: Scope Of This Document

This document focuses on providing mirrors for the resources provided by SUSE and required for basic SUSE CaaS Platform functionality. If you require additional functionality, you can use these instructions as an example on how to provide additional mirrors.

Providing a full set of mirroring instructions, for all usage scenarios, is beyond the scope of this document.

1.1 Process Checklist

The steps that must be performed for an air gapped installation are:

  1. Read the concepts section.

    Section 1.2, “Concepts”

  2. Deploy mirror servers on external and internal networks.

    Section 1.3.1, “Mirror Servers”

  3. Install Repository Mirroring Tool (RMT) on servers.

    Section 1.4, “RPM Repository Mirror”

  4. Configure container image registry on servers.

    Section 1.6, “Container Registry Mirror”

  5. Configure Helm Chart repository on internal mirror.

    Section 1.7, “Helm Chart Repository Mirror”

  6. Perform the Repository Mirroring Tool (RMT) update procedure to populate the RPM repository.

    Section 1.5, “Updating RPM Repository Mirror”

  7. Perform the shared update procedure to populate the Helm chart repository and registry services.

    Section 1.5, “Updating RPM Repository Mirror”

  8. Deploy SUSE CaaS Platform and configure the nodes to use the respective services on the internal network.

    Section 1.9, “Deploying SUSE CaaS Platform”

    RPM Packages: Section 1.4.2, “Client Configuration”

    Helm Charts: Section 1.7.2, “Client Configuration”

    Container Images: Section 1.6.2, “Client Configuration”

1.2 Concepts

1.2.1 Network Separation

For an air gapped scenario we assume a network separation into three logical parts.

caasp cluster airgap network
Upstream

Outside the controlled network.

External

Inside the controlled network, outside the air gapped network.

Internal

Inside the air gapped network.

The following instructions will use these three terms to refer to parts of the infrastructure. For example: "internal mirror" refers to the mirroring server on the air gapped network. The terms air gapped and internal will be used interchangeably.

1.2.2 Mirrored Resources

In order to disconnect SUSE CaaS Platform from the external network, we provide ways for the components to retrieve data from alternative sources inside the internal (air gapped) network.

You will need to create a mirror server inside the internal network; which acts as a replacement for the default sources.

The three main sources that must be replaced are:

  • SUSE Linux Enterprise Server RPM packages

    Provided by the SUSE package repositories

  • Helm installation charts

    Provided by the SUSE helm chart repository (https://kubernetes-charts.suse.com/)

  • Container images

    Provided by the SUSE container registry (https://registry.suse.com)

You will provide replacements for these resources on a dedicated server inside your internal (air gapped) network.

The internal mirror must be updated with data retrieved from the original upstream sources; in a trusted and secure fashion. To achieve this, you will need an additional mirroring server outside of the air gapped network which acts as a first stage mirror and allows retrieving data from the internet.

Updating of mirrors happens in three stages.

  1. Update the external mirror from upstream.

  2. Transfer the updated data onto a trusted storage device.

  3. Update the internal mirror from the trusted storage device.

Once the replacement sources are in place, the key components are reconfigured to use the mirrors as their main sources.

1.2.3 RPM Package Repository Mirroring

Mirroring of the RPM repositories is handled by the Repository Mirroring Tool for SUSE Linux Enterprise Server 15. The tool provides functionality that mirrors the upstream SUSE package repositories on the local network. This is intended to minimize reliance on SUSE infrastructure for updating large volumes of machines. The air gapped deployment uses the same technology to provide the packages locally for the air gapped environment.

SUSE Linux Enterprise Server bundles software packages into so called modules. You must enable the SUSE CaaS Platform, SUSE Linux Enterprise Server and Containers Module modules in addition to the modules enabled by default. All enabled modules need to be mirrored inside the air gapped network in order to provide the necessary software for other parts of this scenario.

Repository Mirroring Tool (RMT) will provide a repository server that holds the packages and related metadata for SUSE Linux Enterprise Server; to install them like from the upstream repository. Data is synchronized once a day to the external mirror automatically or can be forced via the CLI.

You can copy this data to your trusted storage at any point and update the internal mirror.

1.2.4 Helm Chart and Container Image Mirroring

SUSE CaaS Platform uses Helm as one method to install additional software on the cluster. The logic behind this relies on Charts, which are configuration files that tell Kubernetes how to deploy software and its dependencies. The actual software installed using this method is delivered as container images. The download location of the container image is stored inside the Helm chart.

Container images are provided by SUSE and others on so called registries. The SUSE container registry is used to update the SUSE CaaS Platform components.

To mirror container images inside the air gapped environment, you will run two container image registry services that are used to pull and in turn serve these images. The registry service is shipped as a container image itself.

Helm charts are provided independently from container images and can be developed by any number of sources. Please make sure that you trust the origin of container images referenced in the helm charts.

We provide helm-mirror to allow downloading all charts present in a chart repository in bulk and moreover to extract all container image URLs from the charts. skopeo is used to download all the images referred to in the Helm charts from their respective registry.

Helm charts will be provided to the internal network by a webserver and refer to the container images hosted on the internal registry mirror.

Once mirroring is configured, you will not have to modify Dockerfile(s) or Kubernetes manifests to use the mirrors. The requests are passed through the container engine which forwards them to the configured mirrors. For example: All images with a prefix registry.suse.com/ will be automatically pulled from the configured (internal) mirror instead.

For further information on registry mirror configuration, refer to https://documentation.suse.com/suse-caasp/4.5/html/caasp-admin/_miscellaneous.html#_configuring_container_registries_for_cri_o.

1.3 Requirements

1.3.1 Mirror Servers

Note
Note: Shared Mirror Server

If you have multiple SUSE CaaS Platform clusters or a very large number of nodes accessing the mirrors, you should increase the sizing of CPU/RAM.

Storage sizing depends on your intended update frequency and data retention model. If you want to keep snapshots or images of repository states at various points, you must increase storage size accordingly.

You will need to provide and maintain at least two machines in addition to your SUSE CaaS Platform cluster. These mirror servers will reside on the external part of your network and the internal (air gapped) network respectively.

For more information on the requirements of a SUSE Linux Enterprise 15 server, refer to: Installation Preparation.

External

This machine will host the Repository Mirroring Tool (RMT) for RPM packages and the container image registry for container images.

  • 1 Host machines for the mirror servers.

    • SLES 15

    • 2 (v)CPU

    • 4 GB RAM

    • 250 GB Storage

Internal (Air gapped)

This machine will host the Repository Mirroring Tool (RMT) for RPM packages, and container image registry for container images as well as the Helm chart repository files.

  • 1 Host machines for the mirror servers.

    • SLES 15

    • 2 (v)CPU

    • 8 GB RAM

    • 500 GB Storage

Important
Important: Adjust Number Of Mirror Servers

This scenario description does not contain any fallback contingencies for the mirror servers. Add additional mirror servers (behind a load balancer) if you require additional reliability/availability.

Procedure: Provision Mirror Servers
  1. Set up two SUSE Linux Enterprise Server 15 machines one on the internal network and one on the air gapped network.

  2. Make sure you have enabled the Containers module on both servers.

  3. Make sure you have Repository Mirroring Tool installed on both server.

1.3.2 Networking

Note
Note: Additional Port Configuration

If you choose to add more container image registries to your internal network, these must run on different ports than the standard registry running on 5000. Configure your network to allow for this communication accordingly.

1.3.2.1 Ports

The external mirror server must be able to exchange outgoing traffic with upstream sources on ports 80 and 443.

All members of the SUSE CaaS Platform cluster must be able to communicate with the internal mirror server(s) within the air gapped network. You must configure at least these ports in all firewalls between the cluster and the internal mirror:

  • 80 HTTP - Repository Mirroring Tool (RMT) Server and Helm chart repository mirror

  • 443 HTTPS - Repository Mirroring Tool (RMT) Server and Helm chart repository mirror

  • 5000 HTTPS - Container image registry

1.3.2.2 Hostnames / FQDN

You need to define fully qualified domain names (FQDN) for both of the mirror servers in their respective network. These hostnames are the basis for the required SSL certificates and are used by the components to access the respective mirror sources.

1.3.2.3 SSL Certificates

You will need SSL/TLS certificates to secure services on each server.

On the air gapped network, certificates need to cover the hostname of your server and the subdomains for the registry (registry.) and helm chart repository (charts.). You must add corresponding aliases to the certificate.

Tip
Tip

You can use wildcard certificates to cover the entire hostname.

The certificates can be replaced with the self-signed certificate, or you can re-use the certificates created by Repository Mirroring Tool (RMT) during the setup of the mirror servers.

Place the certificate, CA certificate and key file in /etc/rmt/ssl/ as rmt-server.crt, rmt-ca.cert, and rmt-server.key.

These certificates can be re-used by all three mirror services.

Make sure the CA certificate is available to SUSE CaaS Platform system wide; so they can be used by the deployed components.

You can add system wide certificates with following commands on all nodes:

sudo cp /etc/rmt/ssl/rmt-ca.crt /etc/pki/trust/anchors/
sudo update-ca-certificates

1.3.3 Trusted Storage

Transferring data from the external network mirror to the internal mirror can be performed in many ways. The most common way is portable storage (USB keys or external hard drives).

Sizing of the storage is dependent on the number of data sources that need to be stored. Container images can easily measure several Gigabytes per item; although they are generally smaller for Kubernetes related applications. The overall size of any given RPM repository is at least tens of Gigabytes. For example: At the time of writing, the package repository for SUSE Linux Enterprise Server contains approximately 36 GB of data.

The storage must be formatted to a file system type supporting files larger than 4 GB.

We recommend external storage with at least 128 GB.

Note
Note: Mount Point For Storage In Examples

In the following procedures, we will assume the storage (when connected) is mounted on /mnt/storage . Please make sure to adjust the mountpoint in the respective command to where the device is actually available.

Note
Note: Handling Of Trusted Storage

Data integrity checks, duplication, backup, and secure handling procedures of trusted storage are beyond the scope of this document.

1.4 RPM Repository Mirror

1.4.1 Mirror Configuration

Note
Note: Deploy The Mirror Before SUSE CaaS PlatformCluster Deployment

The mirror on the air gapped network must be running and populated before

Procedure: Configure The External Mirror
  1. Connect the external mirror to SUSE Customer Center as described in these instructions.

    Important
    Important: Mirror Registration

    During the installation of Repository Mirroring Tool (RMT) you will be asked for login credentials. On the external mirror, you need to enter your SUSE Customer Center login credentials to register. On the internal mirror, you can skip the SUSE Customer Center login since the registration will not be possible without an internet connection to SUSE Customer Center .

  2. You need to disable the automatic repository sync on the internal server. Otherwise it will attempt to download information from SUSE Customer Center which can not be reached from inside the air gapped network.

    sudo systemctl stop rmt-server-sync.timer
    sudo systemctl disable rmt-server-sync.timer

Now you need to perform the update procedure to do an initial sync of data between the upstream sources and the external mirror and the external and internal mirrors. Refer to: Section 1.5, “Updating RPM Repository Mirror”.

1.4.2 Client Configuration

Follow these instructions to configure all SUSE CaaS Platform nodes to use the package repository mirror server in the air gapped network.

1.5 Updating RPM Repository Mirror

Follow these instructions to update the external server, transfer the data to a storage device, and use that device to update the air gapped server.

1.6 Container Registry Mirror

Note
Note: Mirroring Multiple Image Registries / Chart Repositories

You can mirror images and charts from multiple registries in one shared internal registry. We do not recommend mirroring multiple registries in a shared registry due to the potential conflicts.

We highly recommend running separate helm chart and container registry mirrors for each source registry.

Additional mirror registries must be run on separate mirror servers for technical reasons.

1.6.1 Mirror Configuration

The container image registry is provided as a container image itself. You must download the registry container from SUSE and run it on the respective server.

Note
Note: Which images to Mirror

CaaS Platform requires a base set of images to be mirrored, as they contain the core services needed to run the cluster.

This list of base images can be found under the following link: https://documentation.suse.com/external-tree/en-us/suse-caasp/4/skuba-cluster-images.txt

Alternatively, the list can be obtained from skuba - just run this command on the machine you have skuba installed on:

skuba cluster images

This will print out a list of the images skuba is expecting to use on the cluster to be bootstrapped.

Mirror those and setup the crio-registries to point to the location they are mirrored at.

In addition to the base images used by skuba, Helm 2 also requires the helm-tiller image. When using Helm 3 in an air-gapped deployment the helm-tiller image is not required.

Important
Important

These images need to be available in the external and internal mirrors at the time you try to deploy SUSE CaaS Platform. Image tags will vary depending on the version of kubernetes you install.

CATEGORY

IMAGE

URL

KUBEADM

hyperkube

registry.suse.com/caasp/v4.5/hyperkube

KUBEADM

pause

registry.suse.com/caasp/v4.5/pause

KUBEADM

coredns

registry.suse.com/caasp/v4.5/coredns

KUBEADM

etcd

registry.suse.com/caasp/v4.5/etcd

ADDONS

cilium

registry.suse.com/caasp/v4.5/cilium-init

ADDONS

cilium

registry.suse.com/caasp/v4.5/cilium-operator

ADDONS

cilium

registry.suse.com/caasp/v4.5/cilium

ADDONS

dex

registry.suse.com/caasp/v4.5/caasp-dex

ADDONS

gangway

registry.suse.com/caasp/v4.5/gangway

ADDONS

kured

registry.suse.com/caasp/v4.5/kured

MISC

skuba-tooling

registry.suse.com/caasp/v4.5/skuba-tooling

HELM

helm-tiller

registry.suse.com/caasp/v4.5/helm-tiller

Note
Note: Internal Registry Mirror Is Read Only

For security reasons, the internal registry mirror is configured in read-only mode. Therefore, pushing container images to this mirror will not be possible. It can only serve images that were previously pulled and cached by the external mirror and then uploaded to the internal mirror.

You can modify and store your own container images on the external registry and transfer them with the other container images using the same process. If you need to be able to modify and store container images on the internal network, we recommend creating a new registry that will hold these images. The steps needed to run your own full container image registry are not part of this document.

For more information you can refer to: SLES15 - Docker Open Source Engine Guide: What is Docker Registry?.

We will re-use the nginx webserver that is running as part of Repository Mirroring Tool (RMT) to act as a reverse proxy for the container image registry service and to serve the chart repository files. This step is not necessary for the external host.

Procedure: Set Up Reverse Proxy and Virtual Host
  1. SSH into the internal mirror server.

  2. Create a virtual host configuration file /etc/nginx/vhosts.d/registry-server-https.conf .

    Replace mymirror.local with the hostname of your mirror server for which you created the SSL certificates.

    upstream docker-registry {
        server 127.0.0.1:5000;
    }
    
    map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
      '' 'registry/2.0';
    }
    
    server {
        listen 443   ssl;
        server_name  registry.`mymirror.local`;
    
        access_log  /var/log/nginx/registry_https_access.log;
        error_log   /var/log/nginx/registry_https_error.log;
        root        /usr/share/rmt/public;
    
        ssl_certificate     /etc/rmt/ssl/rmt-server.crt;
        ssl_certificate_key /etc/rmt/ssl/rmt-server.key;
        ssl_protocols       TLSv1.2 TLSv1.3;
    
        # disable any limits to avoid HTTP 413 for large image uploads
        client_max_body_size 0;
    
        location /v2/ {
          # Do not allow connections from docker 1.5 and earlier
          # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
          if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
            return 404;
          }
    
          ## If $docker_distribution_api_version is empty, the header is not added.
          ## See the map directive above where this variable is defined.
          add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;
    
          proxy_pass                          http://docker-registry;
          proxy_set_header  Host              $http_host;   # required for docker client's sake
          proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
          proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
          proxy_set_header  X-Forwarded-Proto $scheme;
          proxy_read_timeout                  900;
        }
    }
  3. Create a virtual host configuration file /etc/nginx/vhosts.d/charts-server-https.conf .

    Replace mymirror.local with the hostname of your mirror server for which you created the SSL certificates.

    server {
      listen 443   ssl;
      server_name  charts.`mymirror.local`;
    
      access_log  /var/log/nginx/charts_https_access.log;
      error_log   /var/log/nginx/charts_https_error.log;
      root        /srv/www/;
    
      ssl_certificate     /etc/rmt/ssl/rmt-server.crt;
      ssl_certificate_key /etc/rmt/ssl/rmt-server.key;
      ssl_protocols       TLSv1.2 TLSv1.3;
    
      location /charts {
        autoindex on;
      }
    }
  4. Restart nginx for the changes to take effect.

    sudo systemctl restart nginx
Procedure: Set Up The External Mirror
  1. SSH into the external mirror server.

  2. Install docker , helm-mirror and skopeo .

    sudo zypper in docker helm-mirror skopeo
  3. Start the docker service and enable it at boot time:

    sudo systemctl enable --now docker.service
  4. Pull the registry container image from SUSE .

    sudo docker pull registry.suse.com/sles12/registry:2.6.2
  5. Save the pulled image to a .tar file.

    sudo docker save -o /tmp/registry.tar registry.suse.com/sles12/registry:2.6.2
  6. Connect the trusted storage to the external mirror. Copy the registry image onto the storage.

    mv /tmp/registry.tar /mnt/storage/registry.tar
  7. Create basic authentication credentials for the container image registry.

    Replace USERNAME and PASSWORD with proper credentials of your choosing.

    sudo mkdir -p /etc/docker/registry/{auth,certs}
    sudo docker run --entrypoint htpasswd registry.suse.com/sles12/registry:2.6.2 -Bbn <USERNAME> <PASSWORD> | sudo tee /etc/docker/registry/auth/htpasswd
  8. Create the /etc/docker/registry/config.yml configuration file.

    Note
    Note

    Setting up a required authentication seems to break, when using CRI-O as the client, so the internal registry does not use any authentication.

    version: 0.1
    log:
      fields:
        service: registry
    storage:
      cache:
        blobdescriptor: inmemory
      filesystem:
        rootdirectory: /var/lib/registry
    http:
      addr: 0.0.0.0:5000
      headers:
        X-Content-Type-Options: [nosniff]
    health:
      storagedriver:
        enabled: true
        interval: 10s
    threshold: 3

    For more details on the configuration, refer to: Docker Registry: Configuration

  9. Start the registry container.

    sudo docker run -d -p 5000:5000 -v /etc/rmt/ssl:/etc/rmt/ssl:ro --restart=always --name registry \
    -v /etc/docker/registry:/etc/docker/registry:ro \
    -v /var/lib/registry:/var/lib/registry registry.suse.com/sles12/registry:2.6.2
Procedure: Set Up Internal Mirror
  1. SSH into the internal mirror server.

  2. Install docker .

    sudo zypper in docker
  3. Start the docker service and enable it at boot time:

    sudo systemctl enable --now docker.service
  4. Connect the trusted storage to the internal mirror and load the registry container image to the local file system.

    sudo docker load -i /mnt/storage/registry.tar
  5. Create the /etc/docker/registry/config.yml configuration file.

    sudo mkdir -p /etc/docker/registry/
    version: 0.1
    log:
      fields:
        service: registry
    storage:
      cache:
        blobdescriptor: inmemory
      filesystem:
        rootdirectory: /var/lib/registry
      maintenance:
        readonly:
          enabled: true
    http:
      addr: 0.0.0.0:5000
      headers:
        X-Content-Type-Options: [nosniff]
      tls:
        certificate: /etc/rmt/ssl/rmt-server.crt
        key: /etc/rmt/ssl/rmt-server.key
    health:
      storagedriver:
        enabled: true
        interval: 10s
    threshold: 3

    For more details on the configuration, refer to: Docker Registry: Configuration

  6. Start the registry container.

    sudo docker run -d -p 5000:5000 -v /etc/rmt/ssl:/etc/rmt/ssl:ro --restart=always --name registry \
    -v /etc/docker/registry:/etc/docker/registry:ro \
    -v /var/lib/registry:/var/lib/registry registry.suse.com/sles12/registry:2.6.2

Now, you should have the registries set up and listening on port 5000 on their respective servers.

1.6.2 Client Configuration

Important
Important

The example provided with the installation is in the old v1 format of the CRI-O registries syntax. You must replace/remove all content from the example file and build a new file based on the v2 syntax.

The example below is written in the correct v2 syntax. registries.conf is written using TOML.

Configure /etc/containers/registries.conf to setup the mirroring from registry.suse.com to the internal mirror. This needs to be done on all cluster nodes. Make sure to adjust all the correct domain name for your local registry:

[[registry]]
prefix = "registry.suse.com"
location = "registry01.mydomain.local:5000/registry.suse.com"
[[registry]]
prefix = "docker.io"
location = "registry01.mydomain.local:5000/docker.io"
[[registry]]
prefix = "docker.io/library"
location = "registry01.mydomain.local:5000/docker.io"
[[registry]]
prefix = "quay.io"
location = "registry01.mydomain.local:5000/quay.io"
[[registry]]
prefix = "k8s.gcr.io"
location = "registry01.mydomain.local:5000/k8s.gcr.io"
[[registry]]
prefix = "gcr.io"
location = "registry01.mydomain.local:5000/gcr.io"

For detailed information about the configuration format see html/caasp-admin/_miscellaneous.html#_configuring_container_registries_for_cri_o.

1.7 Helm Chart Repository Mirror

Important
Important

To make use of the helm charts, you must complete Section 1.6, “Container Registry Mirror”.

The helm charts will require images available from a registry mirror. The charts themselves are served on a simple webserver and do not require any particular configuration apart from basic networking availability and a hostname.

1.7.1 Mirror Configuration

Update the Helm chart repository by following the shared update procedure Section 1.8, “Updating Registry Mirror For Helm Charts”.

1.7.2 Client Configuration

Add the webserver as a repo to helm.

This step needs to be performed on a machine where Helm is installed and configured to talk to the Tiller server in the SUSE CaaS Platform cluster. For steps to install Helm 2 and Tiller, reference https://documentation.suse.com/suse-caasp/4.5/html/caasp-admin/_software_management.html#_installing_tiller and be sure that the helm-tiller image has been mirrored as described in Section 1.8, “Updating Registry Mirror For Helm Charts”.

To initialize Helm 2 before the helm-tiller image is ready, use the command:

helm init --client-only --skip-refresh

<SUSE_MIRROR> will be the user-defined name for this repository listed by Helm. The name of the repository must adhere to Helm Chart naming conventions.

helm repo add <SUSE_MIRROR> https://charts.<MYMIRROR.LOCAL>

1.8 Updating Registry Mirror For Helm Charts

Note
Note: Live Update Of Registry

There is no need to stop the container image registry services while doing the update procedures. All changed images will be re-indexed automatically.

Helm charts and container images must be refreshed in the same procedure, otherwise charts might refer to image versions that are not mirrored or you are mirroring outdated image versions that cause the chart deployment to fail.

Procedure: Pull Data From Upstream Sources
  1. SSH into the mirror server on the external network.

  2. Download all charts from the repository to the file system (e.g. /tmp/charts ).

    This action will download all charts and overwrite the existing Helm chart repository URL. Replace http://charts.mymirror.local with the hostname of the webserver providing the Helm chart repository on the internal network.

    mkdir /tmp/charts
    cd /tmp/charts
    helm-mirror --new-root-url http://charts.mymirror.local https://kubernetes-charts.suse.com /tmp/charts
  3. Translate the chart information into the skopeo format.

    helm-mirror inspect-images /tmp/charts -o skopeo=sync.yaml --ignore-errors
    Note
    Note: Ignoring Chart Errors

    The helm-mirror tool will attempt to render and inspect all downloaded charts. Some charts will have values that are filled from environment data on their source repository and produce errors. You can still proceed with this step by using the --ignore-errors flag.

  4. Add helm-tiller to the image list.

    Edit the sync.yaml file and add an entry for the helm-tiller image after any other entries. Verify the version of the image matches the version used in helm init as documented in https://documentation.suse.com/suse-caasp/4.5/html/caasp-admin/_software_management.html#_installing_tiller and adjust the entry if needed.

    registry.suse.com:
        images:
            caasp/v4.5/helm-tiller:
              - 2.16.9
  5. Download all the referenced images using skopeo.

    mkdir /tmp/skopeodata
    skopeo sync --src yaml --dest dir sync.yaml /tmp/skopeodata

    skopeo will automatically create a directory named after the hostname of the registry from which you are downloading the images. The final path will be something like /tmp/skopeodata/registry.suse.com/ .

  6. Populate the local registry with the downloaded data.

    For --dest-creds you must use the credentials you created during Section 1.6.1, “Mirror Configuration”.

    skopeo sync --dest-creds USERNAME:PASSWORD \
    --src dir --dest docker \
    /tmp/skopeodata/registry.suse.com/ mymirror.local:5000
  7. After the synchronization is done, you can remove the skopeodata directory.

    rm -rf /tmp/skopeodata
Procedure: Transfer Data To Secure Storage
  1. Connect the trusted storage to the external mirror.

  2. Transfer the container image data to the trusted storage. This will remove all files and directories that are no longer present on the external host from the trusted storage.

    rsync -aP /var/lib/registry/ /mnt/storage/registry/ --delete
  3. Transfer the helm chart data to the trusted storage.

    rsync -aP /tmp/charts/ /mnt/storage/charts --delete
  4. Connect the trusted storage to the internal mirror.

  5. Transfer the container image data to the internal mirror. This will remove all files and directories that are no longer present on the trusted storage from the internal mirror.

    The target directory is /var/lib/registry.

    rsync -aP /mnt/storage/registry/ /var/lib/registry/ --delete
  6. Transfer the helm chart data to the internal mirror. This will remove all charts that do not exist on the trusted storage. If you have added any charts to the location manually, please back up these first and restore after the sync from the trusted storage is done.

    rsync -aP /mnt/storage/charts/ /srv/www/charts/ --delete
  7. Set the file permissions and ownership to 555 and nginx:nginx.

    sudo chown -R nginx:nginx /srv/www/charts sudo chmod -R 555 /srv/www/charts/
Procedure: Refresh information on the SUSE CaaS Platformcluster
  1. Update the repository information on the machine on which you are using Helm to install software to the cluster.

    helm repo update

    You can now deploy additional software on your SUSE CaaS Platform Refer to: https://documentation.suse.com/suse-caasp/4.5/html/caasp-admin/_software_management.html#software-installation.

1.9 Deploying SUSE CaaS Platform

Use the SUSE CaaS Platform Deployment Guide as usual. Some of the considerations below apply; depending of the chosen installation medium.

Make sure to add the CA certificate of your Repository Mirroring Tool (RMT) server during deployment. Refer to: Section 1.3.2.3, “SSL Certificates”.

1.9.1 Using the ISO

From YaST register the node against the Repository Mirroring Tool (RMT) server. This will ensure the node zypper repositories are pointed against Repository Mirroring Tool (RMT). Moreover, all the available updates are going to be installed and there is no need to manually install updates right after the installation.

1.9.2 Using AutoYaST

Ensure the admin node is registered against Repository Mirroring Tool (RMT), that will ensure the nodes that are provisioned by AutoYaST are registered against Repository Mirroring Tool (RMT) to have all the updates applied.

1.10 Troubleshooting

1.10.1 Skopeo Fails Because Of Self Signed Certificate

If you are using a self-signed certificate for the registry you can use the --dest-cert-dir /path/to/the/cert parameter to provide the certificate.

1.10.2 Registering An Existing Node against Repository Mirroring Tool (RMT)

Refer to: Section 1.4.2, “Client Configuration”.

1.10.3 Helm chart connection terminated by HTTPS TO HTTP

When registry mirror is using virtual repository URL. You may need to manually modify the Helm chart index.yaml and point the correct HTTPS base URL.

1.10.4 Helm Tiller container fails to start

If the URL for the helm-tiller image is not available when helm init is invoked, the pod for Tiller may be stuck in the "ImagePullBackOff" state. To clear out the broken pod and prepare to attempt helm init again, use the following command.

helm reset --force

1.10.5 Glossary

AWS

Amazon Web Services. A broadly adopted cloud platform run by Amazon.

BPF

Berkeley Packet Filter. Technology used by Cilium to filter network traffic at the level of packet processing in the kernel.

CA

Certificate or Certification Authority. An entity that issues digital certificates.

CIDR

Classless Inter-Domain Routing. Method for allocating IP addresses and IP routing.

CNI

Container Networking Interface. Creates a generic plugin-based networking solution for containers based on spec files in JSON format.

CRD

Custom Resource Definition. Functionality to define non-default resources for Kubernetes pods.

FQDN

Fully Qualified Domain Name. The complete domain name for a specific computer, or host, on the internet, consisting of two parts: the hostname and the domain name.

GKE

Google Kubernetes Engine. Manager for container orchestration built on Kubernetes by Google. Similar for example to Amazon Elastic Kubernetes Service (Amazon EKS) and Azure Kubernetes Service (AKS).

HPA

Horizontal Pod Autoscaler. Based on CPU usage, HPA controls the number of pods in a deployment/replica or stateful set or a replication controller.

KVM

Kernel-based Virtual Machine. Linux native virtualization tool that allows the kernel to function as a hypervisor.

LDAP

Lightweight Directory Access Protocol. A client/server protocol used to access and manage directory information. It reads and edits directories over IP networks and runs directly over TCP/IP using simple string formats for data transfer.

OCI

Open Containers Initiative. A project under the Linux Foundation with the goal of creating open industry standards around container formats and runtime.

OIDC

OpenID Connect. Identity layer on top of the OAuth 2.0 protocol.

OLM

Operator Lifecycle Manager. Open Source tool for managing operators in a Kubernetes cluster.

POC

Proof of Concept. Pioneering project directed at proving the feasibility of a design concept.

PSP

Pod Security Policy. PSPs are cluster-level resources that control security-sensitive aspects of pod specification.

PVC

Persistent Volume Claim. A request for storage by a user.

RBAC

Role-based Access Control. An approach to restrict authorized user access based on defined roles.

RMT

Repository Mirroring Tool. Successor of the SMT. Helps optimize the management of SUSE Linux Enterprise software updates and subscription entitlements.

RPO

Recovery Point Objective. Defines the interval of time that can occur between to backup points before normal business can no longer be resumed.

RTO

Recovery Time Objective. This defines the time (and typically service level from SLA) with which backup relevant incidents must be handled within.

RSA

Rivest-Shamir-Adleman. Asymmetric encryption technique that uses two different keys as public and private keys to perform the encryption and decryption.

SLA

Service Level Agreement. A contractual clause or set of clauses that determines the guaranteed handling of support or incidents by a software vendor or supplier.

SMT

SUSE Subscription Management Tool. Helps to manage software updates, maintain corporate firewall policy and meet regulatory compliance requirements in SUSE Linux Enterprise 11 and 12. Has been replaced by the RMT and SUSE Manager in newer SUSE Linux Enterprise versions.

STS

StatefulSet. Manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods for a "stateful" application.

SMTP

Simple Mail Transfer Protocol. A communication protocol for electronic mail transmission.

TOML

Tom’s Obvious, Minimal Language. Configuration file format used for configuring container registries for CRI-O.

VPA

Vertical Pod Autoscaler. VPA automatically sets the values for resource requests and container limits based on usage.

VPC

Virtual Private Cloud. Division of a public cloud, which supports private cloud computing and thus offers more control over virtual networks and an isolated environment for sensitive workloads.

A Contributors

The contents of these documents are edited by the technical writers for SUSE CaaS Platform and original works created by its contributors.

B GNU Licenses

This appendix contains the GNU Free Documentation License version 1.2.

B.1 GNU Free Documentation License

Copyright © 2000, 2001, 2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

B.1.1 0. PREAMBLE

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or non-commercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

B.1.2 1. APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work’s title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

B.1.3 2. VERBATIM COPYING

You may copy and distribute the Document in any medium, either commercially or non-commercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.

B.1.4 3. COPYING IN QUANTITY

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

B.1.5 4. MODIFICATIONS

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

  1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.

  2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.

  3. State on the Title page the name of the publisher of the Modified Version, as the publisher.

  4. Preserve all the copyright notices of the Document.

  5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.

  6. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.

  7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document’s license notice.

  8. Include an unaltered copy of this License.

  9. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.

  10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.

  11. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.

  12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.

  13. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.

  14. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.

  15. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’s license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties—​for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

B.1.6 5. COMBINING DOCUMENTS

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

B.1.7 6. COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

B.1.8 7. AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

B.1.9 8. TRANSLATION

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

B.1.10 9. TERMINATION

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.

B.1.10.1 10. FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

B.1.10.2 ADDENDUM: How to use this License for your documents

Copyright (c) YEAR YOUR NAME.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU
Free Documentation License".

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the “ with…​Texts.” line with this:

with the Invariant Sections being LIST THEIR TITLES, with the
Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.

Print this page