Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com › SUSE Enterprise Storage 7 Documentation › Deployment Guide › Deploying Ceph Cluster › Deploy with cephadm
ContentsContents
Deployment Guide
  1. About this guide
  2. I Introducing SUSE Enterprise Storage (SES)
    1. 1 SES and Ceph
    2. 2 Hardware requirements and recommendations
    3. 3 Admin Node HA setup
  3. II Deploying Ceph Cluster
    1. 4 Introduction and common tasks
    2. 5 Deploy with cephadm
  4. III Installation of Additional Services
    1. 6 Installation of iSCSI gateway
  5. IV Upgrading from Previous Releases
    1. 7 Upgrade from a previous release
  6. A Ceph maintenance updates based on upstream 'Octopus' point releases
  7. Glossary
Navigation←→
documentation.suse.com › SUSE Enterprise Storage 7 Documentation › Deployment Guide › Deploying Ceph Cluster › Deploy with cephadm
Top
←→
Applies to SUSE Enterprise Storage 7

5 Deploy with cephadm Edit source

5.1 Installing and configuring SUSE Linux Enterprise Server
5.2 Deploying Salt
5.3 Deploying the Ceph cluster
5.4 Deploying services and gateways

SUSE Enterprise Storage 7 uses the Salt-based ceph-salt tool to prepare the operating system on each participating cluster node for deployment via cephadm. cephadm deploys and manages a Ceph cluster by connecting to hosts from the Ceph Manager daemon via SSH. cephadm manages the full life-cycle of a Ceph cluster. It starts by bootstrapping a tiny cluster on a single node (one MON and MGR service) and then uses the orchestration interface to expand the cluster to include all hosts and to provision all Ceph services. You can perform this via the Ceph command line interface (CLI) or partially via Ceph Dashboard (GUI).

Important
Important

Note that the Ceph community documentation uses the cephadm bootstrap command during initial deployment. The ceph-salt calls the cephadm bootstrap command and should not be run directly. Any Ceph cluster deployment manually using the cephadm bootstrap will be unsupported.

To deploy a Ceph cluster by using cephadm, you need to complete the following tasks:

  1. Install and do basic configuration of the underlying operating system—SUSE Linux Enterprise Server 15 SP2—on all cluster nodes.

  2. Deploy the Salt infrastructure on all cluster nodes for performing the initial deployment preparations via ceph-salt.

  3. Configure the basic properties of the cluster via ceph-salt and deploy it.

  4. Add new nodes and roles to the cluster and deploy services to them using cephadm.

5.1 Installing and configuring SUSE Linux Enterprise Server Edit source

  1. Install and register SUSE Linux Enterprise Server 15 SP2 on each cluster node. During installation of SUSE Enterprise Storage, access to the update repositories is required, therefore registration is mandatory. Include at least the following modules:

    • Basesystem Module

    • Server Applications Module

    Find more details on how to install SUSE Linux Enterprise Server in https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-install.html.

  2. Install the SUSE Enterprise Storage 7 extension on each cluster node.

    Tip
    Tip: Install SUSE Enterprise Storage together with SUSE Linux Enterprise Server

    You can either install the SUSE Enterprise Storage 7 extension separately after you have installed SUSE Linux Enterprise Server 15 SP2, or you can add it during the SUSE Linux Enterprise Server 15 SP2 installation procedure.

    Find more details on how to install extensions in https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-register-sle.html.

  3. Configure network settings including proper DNS name resolution on each node. For more information on configuring a network, see https://documentation.suse.com/sles/15-SP2/single-html/SLES-admin/#sec-network-yast For more information on configuring a DNS server, see https://documentation.suse.com/sles/15-SP2/single-html/SLES-admin/#cha-dns.

5.2 Deploying Salt Edit source

SUSE Enterprise Storage uses Salt and ceph-salt for the initial cluster preparation. Salt helps you configure and run commands on multiple cluster nodes simultaneously from one dedicated host called the Salt Master. Before deploying Salt, consider the following important points:

  • Salt Minions are the nodes controlled by a dedicated node called Salt Master.

  • If the Salt Master host should be part of the Ceph cluster, it needs to run its own Salt Minion, but this is not a requirement.

    Tip
    Tip: Sharing multiple roles per server

    You will get the best performance from your Ceph cluster when each role is deployed on a separate node. But real deployments sometimes require sharing one node for multiple roles. To avoid trouble with performance and the upgrade procedure, do not deploy the Ceph OSD, Metadata Server, or Ceph Monitor role to the Admin Node.

  • Salt Minions need to correctly resolve the Salt Master's host name over the network. By default, they look for the salt host name, but you can specify any other network-reachable host name in the /etc/salt/minion file.

  1. Install the salt-master on the Salt Master node: 

    root@master # zypper in salt-master

  2. Check that the salt-master service is enabled and started, and enable and start it if needed: 

    root@master # systemctl enable salt-master.service
root@master # systemctl start salt-master.service

  3. If you intend to use the firewall, verify that the Salt Master node has ports 4505 and 4506 open to all Salt Minion nodes. If the ports are closed, you can open them using the yast2 firewall command by allowing the salt-master service for the appropriate zone. For example, public.

  4. Install the package salt-minion on all minion nodes. 

    root@minion > zypper in salt-minion

  5. Edit /etc/salt/minion and uncomment the following line: 

    #log_level_logfile: warning

    Change the warning log level to info.

    Note
    Note: log_level_logfile and log_level

    While log_level controls which log messages will be displayed on the screen, log_level_logfile controls which log messages will be written to /var/log/salt/minion.

    Note
    Note

    Ensure you change the log level on all cluster (minion) nodes.

  6. Make sure that the fully qualified domain name of each node can be resolved to an IP address on the public cluster network by all the other nodes.

  7. Configure all minions to connect to the master. If your Salt Master is not reachable by the host name salt, edit the file /etc/salt/minion or create a new file /etc/salt/minion.d/master.conf with the following content: 

    master: host_name_of_salt_master

    If you performed any changes to the configuration files mentioned above, restart the Salt service on all related Salt Minions: 

    root@minion > systemctl restart salt-minion.service

  8. Check that the salt-minion service is enabled and started on all nodes. Enable and start it if needed: 

    root # systemctl enable salt-minion.service
root # systemctl start salt-minion.service

  9. Verify each Salt Minion's fingerprint and accept all salt keys on the Salt Master if the fingerprints match.

    Note
    Note

    If the Salt Minion fingerprint comes back empty, make sure the Salt Minion has a Salt Master configuration and that it can communicate with the Salt Master.

    View each minion's fingerprint: 

    root@minion > salt-call --local key.finger
local:
3f:a3:2f:3f:b4:d3:d9:24:49:ca:6b:2c:e1:6c:3f:c3:83:37:f0:aa:87:42:e8:ff...

    After gathering fingerprints of all the Salt Minions, list fingerprints of all unaccepted minion keys on the Salt Master: 

    root@master # salt-key -F
[...]
Unaccepted Keys:
minion1:
3f:a3:2f:3f:b4:d3:d9:24:49:ca:6b:2c:e1:6c:3f:c3:83:37:f0:aa:87:42:e8:ff...

    If the minions' fingerprints match, accept them: 

    root@master # salt-key --accept-all

  10. Verify that the keys have been accepted: 

    root@master # salt-key --list-all

  11. Test whether all Salt Minions respond: 

    root@master # salt-run manage.status

5.3 Deploying the Ceph cluster Edit source

This section guides you through the process of deploying a basic Ceph cluster. Read the following subsections carefully and execute the included commands in the given order.

5.3.1 Installing ceph-salt Edit source

ceph-salt provides tools for deploying Ceph clusters managed by cephadm. ceph-salt uses the Salt infrastructure to perform OS management—for example, software updates or time synchronization—and defining roles for Salt Minions.

On the Salt Master, install the ceph-salt package: 

root@master # zypper install ceph-salt

The above command installed ceph-salt-formula as a dependency which modified the Salt Master configuration by inserting additional files in the /etc/salt/master.d directory. To apply the changes, restart salt-master.service and synchronize Salt modules: 

root@master # systemctl restart salt-master.service
root@master # salt \* saltutil.sync_all

5.3.2 Configuring cluster properties Edit source

Use the ceph-salt config command to configure the basic properties of the cluster.

Important
Important

The /etc/ceph/ceph.conf file is managed by cephadm and users should not edit it. Ceph configuration parameters should be set using the new ceph config command. See Section 28.2, “Configuration database” for more information.

5.3.2.1 Using the ceph-salt shell Edit source

If you run ceph-salt config without any path or subcommand, you will enter an interactive ceph-salt shell. The shell is convenient if you need to configure multiple properties in one batch and do not want type the full command syntax. 

root@master # ceph-salt config
/> ls
o- / ............................................................... [...]
  o- ceph_cluster .................................................. [...]
  | o- minions .............................................. [no minions]
  | o- roles ....................................................... [...]
  |   o- admin .............................................. [no minions]
  |   o- bootstrap ........................................... [no minion]
  |   o- cephadm ............................................ [no minions]
  |   o- tuned ..................................................... [...]
  |     o- latency .......................................... [no minions]
  |     o- throughput ....................................... [no minions]
  o- cephadm_bootstrap ............................................. [...]
  | o- advanced .................................................... [...]
  | o- ceph_conf ................................................... [...]
  | o- ceph_image_path .................................. [ no image path]
  | o- dashboard ................................................... [...]
  | | o- force_password_update ................................. [enabled]
  | | o- password ................................................ [admin]
  | | o- ssl_certificate ....................................... [not set]
  | | o- ssl_certificate_key ................................... [not set]
  | | o- username ................................................ [admin]
  | o- mon_ip .................................................. [not set]
  o- containers .................................................... [...]
  | o- registries_conf ......................................... [enabled]
  | | o- registries .............................................. [empty]
  | o- registry_auth ............................................... [...]
  |   o- password .............................................. [not set]
  |   o- registry .............................................. [not set]
  |   o- username .............................................. [not set]
  o- ssh ............................................... [no key pair set]
  | o- private_key .................................. [no private key set]
  | o- public_key .................................... [no public key set]
  o- time_server ........................... [enabled, no server host set]
    o- external_servers .......................................... [empty]
    o- servers ................................................... [empty]
    o- subnet .................................................. [not set]

As you can see from the output of ceph-salt's ls command, the cluster configuration is organized in a tree structure. To configure a specific property of the cluster in the ceph-salt shell, you have two options:

  • Run the command from the current position and enter the absolute path to the property as the first argument: 

    /> /cephadm_bootstrap/dashboard ls
o- dashboard ....................................................... [...]
  o- force_password_update ..................................... [enabled]
  o- password .................................................... [admin]
  o- ssl_certificate ........................................... [not set]
  o- ssl_certificate_key ....................................... [not set]
  o- username .................................................... [admin]
/> /cephadm_bootstrap/dashboard/username set ceph-admin
Value set.

  • Change to the path whose property you need to configure and run the command: 

    /> cd /cephadm_bootstrap/dashboard/
/ceph_cluster/minions> ls
o- dashboard ....................................................... [...]
  o- force_password_update ..................................... [enabled]
  o- password .................................................... [admin]
  o- ssl_certificate ........................................... [not set]
  o- ssl_certificate_key ....................................... [not set]
  o- username ................................................[ceph-admin]
Tip
Tip: Autocompletion of configuration snippets

While in a ceph-salt shell, you can use the autocompletion feature similar to a normal Linux shell (Bash) autocompletion. It completes configuration paths, subcommands, or Salt Minion names. When autocompleting a configuration path, you have two options:

  • To let the shell finish a path relative to your current position,press the TAB key →| twice.

  • To let the shell finish an absolute path, enter / and press the TAB key →| twice.

Tip
Tip: Navigating with the cursor keys

If you enter cd from the ceph-salt shell without any path, the command will print a tree structure of the cluster configuration with the line of the current path active. You can use the up and down cursor keys to navigate through individual lines. After you confirm with Enter, the configuration path will change to the last active one.

Important
Important: Convention

To keep the documentation consistent, we will use a single command syntax without entering the ceph-salt shell. For example, you can list the cluster configuration tree by using the following command: 

root@master # ceph-salt config ls

5.3.2.2 Adding Salt Minions Edit source

Include all or a subset of Salt Minions that we deployed and accepted in Section 5.2, “Deploying Salt” to the Ceph cluster configuration. You can either specify the Salt Minions by their full names, or use a glob expressions '*' and '?' to include multiple Salt Minions at once. Use the add subcommand under the /ceph_cluster/minions path. The following command includes all accepted Salt Minions: 

root@master # ceph-salt config /ceph_cluster/minions add '*'

Verify that the specified Salt Minions were added: 

root@master # ceph-salt config /ceph_cluster/minions ls
o- minions ................................................. [Minions: 5]
  o- ses-master.example.com .................................. [no roles]
  o- ses-min1.example.com .................................... [no roles]
  o- ses-min2.example.com .................................... [no roles]
  o- ses-min3.example.com .................................... [no roles]
  o- ses-min4.example.com .................................... [no roles]

5.3.2.3 Specifying Salt Minions managed by cephadm Edit source

Specify which nodes will belong to the Ceph cluster and will be managed by cephadm. Include all nodes that will run Ceph services as well as the Admin Node: 

root@master # ceph-salt config /ceph_cluster/roles/cephadm add '*'

5.3.2.4 Specifying Admin Node Edit source

The Admin Node is the node where the ceph.conf configuration file and the Ceph admin keyring is installed. You usually run Ceph related commands on the Admin Node.

Tip
Tip: Salt Master and Admin Node on the Same Node

In a homogeneous environment where all or most hosts belong to SUSE Enterprise Storage, we recommend having the Admin Node on the same host as the Salt Master.

In a heterogeneous environment where one Salt infrastructure hosts more than one cluster, for example, SUSE Enterprise Storage together with SUSE Manager, do not place the Admin Node on the same host as Salt Master.

To specify the Admin Node, run the following command: 

root@master # ceph-salt config /ceph_cluster/roles/admin add ses-master.example.com
1 minion added.
root@master # ceph-salt config /ceph_cluster/roles/admin ls
o- admin ................................................... [Minions: 1]
  o- ses-master.example.com ...................... [Other roles: cephadm]
Tip
Tip: Install ceph.conf and the admin keyring on multiple nodes

You can install the Ceph configuration file and admin keyring on multiple nodes if your deployment requires it. For security reasons, avoid installing them on all the cluster's nodes.

5.3.2.5 Specifying first MON/MGR node Edit source

You need to specify which of the cluster's Salt Minions will bootstrap the cluster. This minion will become the first one running Ceph Monitor and Ceph Manager services. 

root@master # ceph-salt config /ceph_cluster/roles/bootstrap set ses-min1.example.com
Value set.
root@master # ceph-salt config /ceph_cluster/roles/bootstrap ls
o- bootstrap ..................................... [ses-min1.example.com]

Additionally, you need to specify the bootstrap MON's IP address on the public network to ensure that the public_network parameter is set correctly, for example: 

root@master # ceph-salt config /cephadm_bootstrap/mon_ip set 192.168.10.20

5.3.2.6 Specifying tuned profiles Edit source

You need to specify which of the cluster's minions have actively tuned profiles. To do so, add these roles explicitly with the following commands:

Note
Note

One minion cannot have both the latency and throughput roles. 

root@master # ceph-salt config /ceph_cluster/roles/tuned/latency add ses-min1.example.com
Adding ses-min1.example.com...
1 minion added.
root@master # ceph-salt config /ceph_cluster/roles/tuned/throughput add ses-min2.example.com
Adding ses-min2.example.com...
1 minion added.

5.3.2.7 Generating an SSH key pair Edit source

cephadm uses the SSH protocol to communicate with cluster nodes. A user account named cephadm is automatically created and used for SSH communication.

You need to generate the private and public part of the SSH key pair: 

root@master # ceph-salt config /ssh generate
Key pair generated.
root@master # ceph-salt config /ssh ls
o- ssh .................................................. [Key Pair set]
  o- private_key ..... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  o- public_key ...... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]

5.3.2.8 Configuring the time server Edit source

All cluster nodes need to have their time synchronized with a reliable time source. There are several scenarios to approach time synchronization:

  • If all cluster nodes are already configured to synchronize their time using an NTP service of choice, disable time server handling completely: 

    root@master # ceph-salt config /time_server disable

  • If your site already has a single source of time, specify the host name of the time source: 

     root@master # ceph-salt config /time_server/servers add time-server.example.com

  • Alternatively, ceph-salt has the ability to configure one of the Salt Minion to serve as the time server for the rest of the cluster. This is sometimes referred to as an "internal time server". In this scenario, ceph-salt will configure the internal time server (which should be one of the Salt Minion) to synchronize its time with an external time server, such as pool.ntp.org, and configure all the other minions to get their time from the internal time server. This can be achieved as follows: 

    root@master # ceph-salt config /time_server/servers add ses-master.example.com
root@master # ceph-salt config /time_server/external_servers add pool.ntp.org

    The /time_server/subnet option specifies the subnet from which NTP clients are allowed to access the NTP server. It is automatically set when you specify /time_server/servers. If you need to change it or specify it manually, run: 

    root@master # ceph-salt config /time_server/subnet set 10.20.6.0/24

Check the time server settings: 

root@master # ceph-salt config /time_server ls
o- time_server ................................................ [enabled]
  o- external_servers ............................................... [1]
  | o- pool.ntp.org ............................................... [...]
  o- servers ........................................................ [1]
  | o- ses-master.example.com ..................................... [...]
  o- subnet .............................................. [10.20.6.0/24]

Find more information on setting up time synchronization in https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-ntp.html#sec-ntp-yast.

5.3.2.9 Configuring the Ceph Dashboard login credentials Edit source

Ceph Dashboard will be available after the basic cluster is deployed. To access it, you need to set a valid user name and password, for example: 

root@master # ceph-salt config /cephadm_bootstrap/dashboard/username set admin
root@master # ceph-salt config /cephadm_bootstrap/dashboard/password set PWD
Tip
Tip: Forcing password update

By default, the first dashboard user will be forced to change their password on first login to the dashboard. To disable this feature, run the following command: 

root@master # ceph-salt config /cephadm_bootstrap/dashboard/force_password_update disable

5.3.2.10 Using the container registry Edit source

The Ceph cluster needs to have access to a container registry so that it can download and deploy containerized Ceph services. There are two ways to access the registry:

  • If your cluster can access the default registry at registry.suse.com (directly or via proxy), you can point ceph-salt directly to this URL without creating a local registry. Continue by following the steps in Section 5.3.2.10.2, “Configuring the path to container images”.

  • If your cluster cannot access the default registry—for example, for an air-gapped deployment—you need to configure a local container registry. After the local registry is created and configured, you need to point ceph-salt to it.

5.3.2.10.1 Creating and configuring the local registry (optional) Edit source
Important
Important

There are numerous methods of creating a local registry. The instructions in this section are examples of creating secure and insecure registries.

Tip
Tip: Placement and port usage

Deploy the registry on a machine accessible by all nodes in the cluster. We recommend the Admin Node. By default, the registry listens on port 5000.

On the registry node, use the following command to ensure that the port is free: 

ss -tulpn | grep :5000

If other processes (such as iscsi-tcmu) are already listening on port 5000, determine another free port which can be used to map to port 5000 in the registry container.

Procedure 5.1: Creating the local registry

  1. Verify that the Containers Module extension is enabled: 

    tux > SUSEConnect --list-extensions | grep -A2 "Containers Module"
Containers Module 15 SP2 x86_64 (Activated)

  2. Verify that the following packages are installed: apache2-utils (if enabling a secure registry), cni, cni-plugins, podman, podman-cni-config, and skopeo.

  3. Gather the following information:

    • Fully qualified domain name of the registry host (REG_HOST_FQDN).

    • An available port number used to map to the registry container port of 5000 (REG_HOST_PORT).

    • Whether the registry will be secure or insecure (insecure=[true|false]).

  4. To start an insecure registry (without SSL encryption), follow these steps:

    1. Configure ceph-salt for the insecure registry: 

      cephuser@adm > ceph-salt config containers/registries_conf enable
cephuser@adm > ceph-salt config containers/registries_conf/registries \
 add prefix=REG_HOST_FQDN insecure=true \
 location=REG_HOST_PORT:5000

    2. Start the insecure registry by creating the necessary directory (for example, /var/lib/registry) and starting the registry with the podman command: 

      root # mkdir -p /var/lib/registry
root # podman run --privileged -d --name registry \
 -p REG_HOST_PORT:5000 -v /var/lib/registry:/var/lib/registry \
 --restart=always registry:2

    3. To have the registry start after a reboot, create a systemd unit file for it and enable it: 

      tux > sudo podman generate systemd --files --name registry
tux > sudo mv container-registry.service /etc/systemd/system/
tux > sudo systemctl enable container-registry.service

  5. To start a secure registry, follow these steps:

    1. Create the necessary directories: 

      root # mkdir -p /var/lib/registry/{auth,certs}

    2. Generate an SSL certificate: 

      root # openssl req -newkey rsa:4096 -nodes -sha256 \
 -keyout /var/lib/registry/certs/domain.key -x509 -days 365 \
 -out /var/lib/registry/certs/domain.crt
      Note
      Note

      Set the CN=[value] value to the fully qualified domain name of the host ([REG_HOST_FQDN]).

    3. Copy the certificate to all cluster nodes and refresh the certificate cache: 

      root # salt-cp '*' /var/lib/registry/certs/domain.crt \
 /etc/pki/trust/anchors/
root # salt '*' cmd.shell "update-ca-certificates"

    4. Generate a username and password combination for authentication to the registry: 

      root # htpasswd2 -bBc /var/lib/registry/auth/htpasswd \
 REG_USERNAME REG_PASSWORD

    5. Start the secure registry: 

      podman run --name myregistry -p REG_HOST_PORT:5000 \
 -v /var/lib/registry:/var/lib/registry \
 -v /var/lib/registry/auth:/auth:z \
 -e "REGISTRY_AUTH=htpasswd" \
 -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
 -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
 -v /var/lib/registry/certs:/certs:z \
 -e "REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt" \
 -e "REGISTRY_HTTP_TLS_KEY=/certs/domain.key" \
 -e REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true -d registry:2

    6. Test secure access to the registry: 

      tux > curl https://REG_HOST_FQDN:REG_HOST_PORT/v2/_catalog \
 -u REG_USERNAME:REG_PASSWORD

  6. When the local registry is created, you need to synchronize container images from the official SUSE registry at registry.suse.com to the local one. You can use the skopeo sync command found in the skopeo package for that purpose. For more details, refer to the manual page (man 1 skopeo-sync).

Procedure 5.2: Configure the local registry and access credentials

  1. Configure the URL of the local registry: 

    cephuser@adm > ceph-salt config /containers/registry_auth/registry set REG_HOST_URL

  2. Configure the user name and password to access the local registry: 

    cephuser@adm > ceph-salt config /containers/registry_auth/username set REG_USERNAME
    cephuser@adm > ceph-salt config /containers/registry_auth/password set REG_PASSWORD
Tip
Tip: Registry cache

To avoid re-syncing the local registry when new updated containers appear, you can configure a registry cache.

5.3.2.10.2 Configuring the path to container images Edit source
Tip
Tip: Configuring HTTPS proxy

If you need to use a proxy to communicate with the container registry server, perform the following configuration steps on all cluster nodes:

  1. Copy the configuration file for containers: 

    tux > sudo cp /usr/share/containers/containers.conf /etc/containers/containers.conf

  2. Edit the newly-copied file and add the http_proxy setting to its [engine] section, for example: 

    tux > cat /etc/containers/containers.conf
 [engine]
 http_proxy=proxy.example.com
 [...]

cephadm needs to know a valid URI path to container images. Verify the default setting by executing 

root@master # ceph-salt config /cephadm_bootstrap/ceph_image_path ls

If you do not need an alternative or local registry, specify the default SUSE container registry: 

root@master # ceph-salt config /cephadm_bootstrap/ceph_image_path set registry.suse.com/ses/7/ceph/ceph

If your deployment requires a specific path, for example, a path to a local registry, configure it as follows: 

root@master # ceph-salt config /cephadm_bootstrap/ceph_image_path set LOCAL_REGISTRY_PATH
Note
Note

Note that these container images will not be used by ceph-salt for the deployment. This is a preparatory step for a later one, in which cephadm will be used for the deployment or migration of monitoring components.

For more information about the images used by the monitoring stack and how to customize them, visit the Section 16.1, “Configuring custom or local images” page.

5.3.2.11 Enabling data in-flight encryption (msgr2) Edit source

The Messenger v2 protocol (MSGR2) is Ceph's on-wire protocol. It provides a security mode that encrypts all data passing over the network, encapsulation of authentication payloads, and the enabling of future integration of new authentication modes (such as Kerberos).

Important
Important

msgr2 is not currently supported by Linux kernel Ceph clients, such as CephFS and RADOS Block Device.

Ceph daemons can bind to multiple ports, allowing both legacy Ceph clients and new v2-capable clients to connect to the same cluster. By default, MONs now bind to the new IANA-assigned port 3300 (CE4h or 0xCE4) for the new v2 protocol, while also binding to the old default port 6789 for the legacy v1 protocol.

The v2 protocol (MSGR2) supports two connection modes:

crc mode

A strong initial authentication when the connection is established and a CRC32C integrity check.

secure mode

A strong initial authentication when the connection is established and full encryption of all post-authentication traffic, including a cryptographic integrity check.

For most connections, there are options that control which modes are used:

ms_cluster_mode

The connection mode (or permitted modes) used for intra-cluster communication between Ceph daemons. If multiple modes are listed, the modes listed first are preferred.

ms_service_mode

A list of permitted modes for clients to use when connecting to the cluster.

ms_client_mode

A list of connection modes, in order of preference, for clients to use (or allow) when talking to a Ceph cluster.

There are a parallel set of options that apply specifically to monitors, allowing administrators to set different (usually more secure) requirements on communication with the monitors.

ms_mon_cluster_mode

The connection mode (or permitted modes) to use between monitors.

ms_mon_service_mode

A list of permitted modes for clients or other Ceph daemons to use when connecting to monitors.

ms_mon_client_mode

A list of connection modes, in order of preference, for clients or non-monitor daemons to use when connecting to monitors.

In order to enable MSGR2 encryption mode during the deployment, you need to add some configuration options to the ceph-salt configuration before running ceph-salt apply.

To use secure mode, run the following commands.

Add the global section to ceph_conf in the ceph-salt configuration tool: 

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf add global

Set the following options: 

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_cluster_mode "secure crc"
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_service_mode "secure crc"
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_client_mode "secure crc"
Note
Note

Ensure secure precedes crc.

To force secure mode, run the following commands: 

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_cluster_mode secure
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_service_mode secure
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_client_mode secure
Tip
Tip: Updating settings

If you want to change any of the above settings, set the configuration changes in the monitor configuration store. This is achieved using the ceph config set command. 

root@master # ceph config set global CONNECTION_OPTION CONNECTION_MODE [--force]

For example: 

root@master # ceph config set global ms_cluster_mode "secure crc"

If you want to check the current value, including default value, run the following command: 

root@master # ceph config get CEPH_COMPONENT CONNECTION_OPTION

For example, to get the ms_cluster_mode for OSD's, run: 

root@master # ceph config get osd ms_cluster_mode

5.3.2.12 Configuring the cluster network Edit source

Optionally, if you are running a separate cluster network, you may need to set the cluster network IP address followed by the subnet mask part after the slash sign, for example 192.168.10.22/24.

Run the following commands to enable cluster_network: 

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf add global
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set cluster_network NETWORK_ADDR

5.3.2.13 Verifying the cluster configuration Edit source

The minimal cluster configuration is finished. Inspect it for obvious errors: 

root@master # ceph-salt config ls
o- / ............................................................... [...]
  o- ceph_cluster .................................................. [...]
  | o- minions .............................................. [Minions: 5]
  | | o- ses-master.example.com .................................. [admin]
  | | o- ses-min1.example.com ......................... [bootstrap, admin]
  | | o- ses-min2.example.com ................................. [no roles]
  | | o- ses-min3.example.com ................................. [no roles]
  | | o- ses-min4.example.com ................................. [no roles]
  | o- roles ....................................................... [...]
  |   o- admin .............................................. [Minions: 2]
  |   | o- ses-master.example.com ....................... [no other roles]
  |   | o- ses-min1.example.com ................. [other roles: bootstrap]
  |   o- bootstrap ................................ [ses-min1.example.com]
  |   o- cephadm ............................................ [Minions: 5]
  |   o- tuned ..................................................... [...]
  |     o- latency .......................................... [no minions]
  |     o- throughput ....................................... [no minions]
  o- cephadm_bootstrap ............................................. [...]
  | o- advanced .................................................... [...]
  | o- ceph_conf ................................................... [...]
  | o- ceph_image_path ............... [registry.suse.com/ses/7/ceph/ceph]
  | o- dashboard ................................................... [...]
  |   o- force_password_update ................................. [enabled]
  |   o- password ................................... [randomly generated]
  |   o- username ................................................ [admin]
  | o- mon_ip ............................................ [192.168.10.20]
  o- containers .................................................... [...]
  | o- registries_conf ......................................... [enabled]
  | | o- registries .............................................. [empty]
  | o- registry_auth ............................................... [...]
  |   o- password .............................................. [not set]
  |   o- registry .............................................. [not set]
  |   o- username .............................................. [not set]
  o- ssh .................................................. [Key Pair set]
  | o- private_key ..... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  | o- public_key ...... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  o- time_server ............................................... [enabled]
    o- external_servers .............................................. [1]
    | o- 0.pt.pool.ntp.org ......................................... [...]
    o- servers ....................................................... [1]
    | o- ses-master.example.com .................................... [...]
    o- subnet ............................................. [10.20.6.0/24]
Tip
Tip: Status of cluster configuration

You can check if the configuration of the cluster is valid by running the following command: 

root@master # ceph-salt status
cluster: 5 minions, 0 hosts managed by cephadm
config: OK

5.3.2.14 Exporting cluster configurations Edit source

After you have configured the basic cluster and its configuration is valid, it is a good idea to export its configuration to a file: 

root@master # ceph-salt export > cluster.json
Warning
Warning

The output of the ceph-salt export includes the SSH private key. If you are concerned about the security implications, do not execute this command without taking appropriate precautions.

In case you break the cluster configuration and need to revert to a backup state, run: 

root@master # ceph-salt import cluster.json

5.3.3 Updating nodes and bootstrap minimal cluster Edit source

Before you deploy the cluster, update all software packages on all nodes: 

root@master # ceph-salt update

If a node reports Reboot is needed during the update, important OS packages—such as the kernel—were updated to a newer version and you need to reboot the node to apply the changes.

To reboot all nodes that require rebooting, either append the --reboot option 

root@master # ceph-salt update --reboot

Or, reboot them in a separate step: 

root@master # ceph-salt reboot
Important
Important

The Salt Master is never rebooted by ceph-salt update --reboot or ceph-salt reboot commands. If the Salt Master needs rebooting, you need to reboot it manually.

After the nodes are updated, bootstrap the minimal cluster: 

root@master # ceph-salt apply
Note
Note

When bootstrapping is complete, the cluster will have one Ceph Monitor and one Ceph Manager.

The above command will open an interactive user interface that shows the current progress of each minion.

Deployment of a minimal cluster
Figure 5.1: Deployment of a minimal cluster
Tip
Tip: Non-interactive mode

If you need to apply the configuration from a script, there is also a non-interactive mode of deployment. This is also useful when deploying the cluster from a remote machine because constant updating of the progress information on the screen over the network may become distracting: 

root@master # ceph-salt apply --non-interactive

5.3.4 Reviewing final steps Edit source

After the ceph-salt apply command has completed, you should have one Ceph Monitor and one Ceph Manager. You should be able to run the ceph status command successfully on any of the minions that were given the admin role as root or the cephadm user using sudo.

The next steps involve using the cephadm to deploy additional Ceph Monitor, Ceph Manager, OSDs, the Monitoring Stack, and Gateways.

Before you continue, review your new cluster's network settings. At this point, the public_network setting has been populated based on what was entered for /cephadm_bootstrap/mon_ip in the ceph-salt configuration. However, this setting was only applied to Ceph Monitor. You can review this setting with the following command: 

root@master # ceph config get mon public_network

This is the minimum that Ceph requires to work, but we recommend making this public_network setting global, which means it will apply to all types of Ceph daemons, and not only to MONs: 

root@master # ceph config set global public_network "$(ceph config get mon public_network)"
Note
Note

This step is not required. However, if you do not use this setting, the Ceph OSDs and other daemons (except Ceph Monitor) will listen on all addresses.

If you want your OSDs to communicate amongst themselves using a completely separate network, run the following command: 

root@master # ceph config set global cluster_network "cluster_network_in_cidr_notation"

Executing this command will ensure that the OSDs created in your deployment will use the intended cluster network from the start.

If your cluster is set to have dense nodes (greater than 62 OSDs per host), make sure to assign sufficient ports for Ceph OSDs. The default range (6800-7300) currently allows for no more than 62 OSDs per host. For a cluster with dense nodes, adjust the setting ms_bind_port_max to a suitable value. Each OSD will consume eight additional ports. For example, given a host that is set to run 96 OSDs, 768 ports will be needed. ms_bind_port_max should be set at least to 7568 by running the following command: 

root@master # ceph config set osd.* ms_bind_port_max 7568

You will need to adjust your firewall settings accordingly for this to work. See Section 13.7, “Firewall settings for Ceph” for more information.

5.3.5 Disable insecure clients Edit source

Since Octopus v15.2.11, a new health warning was introduced that informs you that insecure clients are allowed to join the cluster. This warning is on by default. The Ceph Dashboard will show the cluster in the HEALTH_WARN status and verifying the cluster status on the command line informs you as follows: 

cephuser@adm > ceph status
cluster:
  id:     3fe8b35a-689f-4970-819d-0e6b11f6707c
  health: HEALTH_WARN
  mons are allowing insecure global_id reclaim
[...]

This warning means that the Ceph Monitors are still allowing old, unpatched clients to connect to the cluster. This ensures existing clients can still connect while the cluster is being upgraded, but warns you that there is a problem that needs to be addressed. When the cluster and all clients are upgraded to the latest version of Ceph, disallow unpatched clients by running the following command: 

cephuser@adm > ceph config set mon auth_allow_insecure_global_id_reclaim false

5.4 Deploying services and gateways Edit source

After deploying the basic Ceph cluster, deploy core services to more cluster nodes. To make the cluster data accessible to clients, deploy additional services as well.

Currently, we support deployment of Ceph services on the command line by using the Ceph orchestrator (ceph orch subcommands).

5.4.1 The ceph orch command Edit source

The Ceph orchestrator command ceph orch—which is an interface to the cephadm module—will take care of listing cluster components and deploying Ceph services on new cluster nodes.

5.4.1.1 Displaying the orchestrator status Edit source

The following command shows the current mode and status of the Ceph orchestrator. 

cephuser@adm > ceph orch status

5.4.1.2 Listing devices, services, and daemons Edit source

To list all disk devices, run the following: 

cephuser@adm > ceph orch device ls
Hostname   Path      Type  Serial  Size   Health   Ident  Fault  Available
ses-master /dev/vdb  hdd   0d8a... 10.7G  Unknown  N/A    N/A    No
ses-min1   /dev/vdc  hdd   8304... 10.7G  Unknown  N/A    N/A    No
ses-min1   /dev/vdd  hdd   7b81... 10.7G  Unknown  N/A    N/A    No
[...]
Tip
Tip: Services and daemons

Service is a general term for a Ceph service of a specific type, for example Ceph Manager.

Daemon is a specific instance of a service, for example a process mgr.ses-min1.gdlcik running on a node called ses-min1.

To list all services known to cephadm, run: 

cephuser@adm > ceph orch ls
NAME  RUNNING  REFRESHED  AGE  PLACEMENT  IMAGE NAME                  IMAGE ID
mgr       1/0  5m ago     -    <no spec>  registry.example.com/[...]  5bf12403d0bd
mon       1/0  5m ago     -    <no spec>  registry.example.com/[...]  5bf12403d0bd
Tip
Tip

You can limit the list to services on a particular node with the optional -–host parameter, and services of a particular type with the optional --service-type parameter. Acceptable types are mon, osd, mgr, mds, and rgw.

To list all running daemons deployed by cephadm, run: 

cephuser@adm > ceph orch ps
NAME            HOST     STATUS   REFRESHED AGE VERSION    IMAGE ID     CONTAINER ID
mgr.ses-min1.gd ses-min1 running) 8m ago    12d 15.2.0.108 5bf12403d0bd b8104e09814c
mon.ses-min1    ses-min1 running) 8m ago    12d 15.2.0.108 5bf12403d0bd a719e0087369
Tip
Tip

To query the status of a particular daemon, use --daemon_type and --daemon_id. For OSDs, the ID is the numeric OSD ID. For MDS, the ID is the file system name: 

cephuser@adm > ceph orch ps --daemon_type osd --daemon_id 0
cephuser@adm > ceph orch ps --daemon_type mds --daemon_id my_cephfs

5.4.2 Service and placement specification Edit source

The recommended way to specify the deployment of Ceph services is to create a YAML-formatted file with the specification of the services that you intend to deploy.

5.4.2.1 Creating service specifications Edit source

You can create a separate specification file for each type of service, for example: 

root@master # cat nfs.yml
service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-min1
  - ses-min2
spec:
  pool: EXAMPLE_POOL
  namespace: EXAMPLE_NAMESPACE

Alternatively, you can specify multiple (or all) service types in one file—for example, cluster.yml—that describes which nodes will run specific services. Remember to separate individual service types with three dashes (---): 

cephuser@adm > cat cluster.yml
service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-min1
  - ses-min2
spec:
  pool: EXAMPLE_POOL
  namespace: EXAMPLE_NAMESPACE
---
service_type: rgw
service_id: REALM_NAME.ZONE_NAME
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
---
[...]

The aforementioned properties have the following meaning:

service_type

The type of the service. It can be either a Ceph service (mon, mgr, mds, crash, osd, or rbd-mirror), a gateway (nfs or rgw), or part of the monitoring stack (alertmanager, grafana, node-exporter, or prometheus).

service_id

The name of the service. Specifications of type mon, mgr, alertmanager, grafana, node-exporter, and prometheus do not require the service_id property.

placement

Specifies which nodes will be running the service. Refer to Section 5.4.2.2, “Creating placement specification” for more details.

spec

Additional specification relevant for the service type.

Tip
Tip: Applying specific services

Ceph cluster services have usually a number of properties specific to them. For examples and details of individual services' specification, refer to Section 5.4.3, “Deploy Ceph services”.

5.4.2.2 Creating placement specification Edit source

To deploy Ceph services, cephadm needs to know on which nodes to deploy them. Use the placement property and list the short host names of the nodes that the service applies to: 

cephuser@adm > cat cluster.yml
[...]
placement:
  hosts:
  - host1
  - host2
  - host3
[...]

5.4.2.3 Applying cluster specification Edit source

After you have created a full cluster.yml file with specifications of all services and their placement, you can apply the cluster by running the following command: 

cephuser@adm > ceph orch apply -i cluster.yml

To view the status of the cluster, run the ceph orch status command. For more details, see Section 5.4.1.1, “Displaying the orchestrator status”.

5.4.2.4 Exporting the specification of a running cluster Edit source

Although you deployed services to the Ceph cluster by using the specification files as described in Section 5.4.2, “Service and placement specification”, the configuration of the cluster may diverge from the original specification during its operation. Also, you may have removed the specification files accidentally.

To retrieve a complete specification of a running cluster, run: 

cephuser@adm > ceph orch ls --export
placement:
  hosts:
  - hostname: ses-min1
    name: ''
    network: ''
service_id: my_cephfs
service_name: mds.my_cephfs
service_type: mds
---
placement:
  count: 2
service_name: mgr
service_type: mgr
---
[...]
Tip
Tip

You can append the --format option to change the default yaml output format. You can select from json, json-pretty, or yaml. For example: 

ceph orch ls --export --format json

5.4.3 Deploy Ceph services Edit source

After the basic cluster is running, you can deploy Ceph services to additional nodes.

5.4.3.1 Deploying Ceph Monitors and Ceph Managers Edit source

Ceph cluster has three or five MONs deployed across different nodes. If there are five or more nodes in the cluster, we recommend deploying five MONs. A good practice is to have MGRs deployed on the same nodes as MONs.

Important
Important: Include Bootstrap MON

When deploying MONs and MGRs, remember to include the first MON that you added when configuring the basic cluster in Section 5.3.2.5, “Specifying first MON/MGR node”.

To deploy MONs, apply the following specification: 

service_type: mon
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
Note
Note

If you need to add another node, append the host name to the same YAML list. For example: 

service_type: mon
placement:
 hosts:
 - ses-min1
 - ses-min2
 - ses-min3
 - ses-min4

Similarly, to deploy MGRs, apply the following specification:

Important
Important

Ensure your deployment has at least three Ceph Managers in each deployment. 

service_type: mgr
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
Tip
Tip

If MONs or MGRs are not on the same subnet, you need to append the subnet addresses. For example: 

service_type: mon
placement:
  hosts:
  - ses-min1:10.1.2.0/24
  - ses-min2:10.1.5.0/24
  - ses-min3:10.1.10.0/24

5.4.3.2 Deploying Ceph OSDs Edit source

Important
Important: When Storage Device is Available

A storage device is considered available if all of the following conditions are met:

  • The device has no partitions.

  • The device does not have any LVM state.

  • The device is not be mounted.

  • The device does not contain a file system.

  • The device does not contain a BlueStore OSD.

  • The device is larger than 5 GB.

If the above conditions are not met, Ceph refuses to provision such OSDs.

There are two ways you can deploy OSDs:

  • Tell Ceph to consume all available and unused storage devices: 

    cephuser@adm > ceph orch apply osd --all-available-devices

  • Use DriveGroups (see Section 13.4.3, “Adding OSDs using DriveGroups specification”) to create OSD specification describing devices that will be deployed based on their properties, such as device type (SSD or HDD), device model names, size, or the nodes on which the devices exist. Then apply the specification by running the following command: 

    cephuser@adm > ceph orch apply osd -i drive_groups.yml

5.4.3.3 Deploying Metadata Servers Edit source

CephFS requires one or more Metadata Server (MDS) services. To create a CephFS, first create MDS servers by applying the following specification:

Note
Note

Ensure you have at least two pools, one for CephFS data and one for CephFS metadata, created before applying the following specification. 

service_type: mds
service_id: CEPHFS_NAME
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3

After MDSs are functional, create the CephFS: 

ceph fs new CEPHFS_NAME metadata_pool data_pool

5.4.3.4 Deploying Object Gateways Edit source

cephadm deploys an Object Gateway as a collection of daemons that manage a particular realm and zone.

You can either relate an Object Gateway service to already existing realm and zone, (refer to Section 21.13, “Multisite Object Gateways” for more details), or you can specify a non-existing REALM_NAME and ZONE_NAME and they will be created automatically after you apply the following configuration: 

service_type: rgw
service_id: REALM_NAME.ZONE_NAME
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
spec:
  rgw_realm: RGW_REALM
  rgw_zone: RGW_ZONE
5.4.3.4.1 Using secure SSL access Edit source

To use a secure SSL connection to the Object Gateway, you need a pair of valid SSL certificate and key files (see Section 21.7, “Enable HTTPS/SSL for Object Gateways” for more details). You need to enable SSL, specify a port number for SSL connections, and the SSL certificate and key files.

To enable SSL and specify the port number, include the following in your specification: 

spec:
  ssl: true
  rgw_frontend_port: 443

To specify the SSL certificate and key, you can paste their contents directly into the YAML specification file. The pipe sign (|) at the end of line tells the parser to expect a multi-line string as a value. For example: 

spec:
  ssl: true
  rgw_frontend_port: 443
  rgw_frontend_ssl_certificate: |
   -----BEGIN CERTIFICATE-----
   MIIFmjCCA4KgAwIBAgIJAIZ2n35bmwXTMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
   BAYTAkFVMQwwCgYDVQQIDANOU1cxHTAbBgNVBAoMFEV4YW1wbGUgUkdXIFNTTCBp
   [...]
   -----END CERTIFICATE-----
   rgw_frontend_ssl_key: |
   -----BEGIN PRIVATE KEY-----
   MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQDLtFwg6LLl2j4Z
   BDV+iL4AO7VZ9KbmWIt37Ml2W6y2YeKX3Qwf+3eBz7TVHR1dm6iPpCpqpQjXUsT9
   [...]
   -----END PRIVATE KEY-----
Tip
Tip

Instead of pasting the content of SSL certificate and key files, you can omit the rgw_frontend_ssl_certificate: and rgw_frontend_ssl_key: keywords and upload them to the configuration database: 

cephuser@adm > ceph config-key set rgw/cert/REALM_NAME/ZONE_NAME.crt \
 -i SSL_CERT_FILE
cephuser@adm > ceph config-key set rgw/cert/REALM_NAME/ZONE_NAME.key \
 -i SSL_KEY_FILE
5.4.3.4.2 Deploying with a subcluster Edit source

Subclusters help you organize the nodes in your clusters to isolate workloads and make elastic scaling easier. If you are deploying with a subcluster, apply the following configuration: 

service_type: rgw
service_id: REALM_NAME.ZONE_NAME.SUBCLUSTER
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
spec:
  rgw_realm: RGW_REALM
  rgw_zone: RGW_ZONE
  subcluster: SUBCLUSTER

5.4.3.5 Deploying iSCSI Gateways Edit source

cephadm deploys an iSCSI Gateway which is a storage area network (SAN) protocol that allows clients (called initiators) to send SCSI commands to SCSI storage devices (targets) on remote servers.

Apply the following configuration to deploy. Ensure trusted_ip_list contains the IP addresses of all iSCSI Gateway and Ceph Manager nodes (see the example output below).

Note
Note

Ensure the pool is created before applying the following specification. 

service_type: iscsi
service_id: EXAMPLE_ISCSI
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
spec:
  pool: EXAMPLE_POOL
  api_user: EXAMPLE_USER
  api_password: EXAMPLE_PASSWORD
  trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2"
Note
Note

Ensure the IPs listed for trusted_ip_list do not have a space after the comma separation.

5.4.3.5.1 Secure SSL configuration Edit source

To use a secure SSL connection between the Ceph Dashboard and the iSCSI target API, you need a pair of valid SSL certificate and key files. These can be either CA-issued or self-signed (see Section 10.1.1, “Creating self-signed certificates”). To enable SSL, include the api_secure: true setting in your specification file: 

spec:
  api_secure: true

To specify the SSL certificate and key, you can paste the content directly into the YAML specification file. The pipe sign (|) at the end of line tells the parser to expect a multi-line string as a value. For example: 

spec:
  pool: EXAMPLE_POOL
  api_user: EXAMPLE_USER
  api_password: EXAMPLE_PASSWORD
  trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2"
  api_secure: true
  ssl_cert: |
    -----BEGIN CERTIFICATE-----
    MIIDtTCCAp2gAwIBAgIYMC4xNzc1NDQxNjEzMzc2MjMyXzxvQ7EcMA0GCSqGSIb3
    DQEBCwUAMG0xCzAJBgNVBAYTAlVTMQ0wCwYDVQQIDARVdGFoMRcwFQYDVQQHDA5T
    [...]
    -----END CERTIFICATE-----
  ssl_key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC5jdYbjtNTAKW4
    /CwQr/7wOiLGzVxChn3mmCIF3DwbL/qvTFTX2d8bDf6LjGwLYloXHscRfxszX/4h
    [...]
    -----END PRIVATE KEY-----

5.4.3.6 Deploying NFS Ganesha Edit source

Important
Important

NFS Ganesha supports NFS version 4.1 and newer. It does not support NFS version 3.

cephadm deploys NFS Ganesha using a pre-defined RADOS pool and an optional name-space. To deploy NFS Ganesha, apply the following specification:

Note
Note

You need to have a pre-defined RADOS pool otherwise the ceph orch apply operation will fail. For more information on creating a pool, see Section 18.1, “Creating a pool”. 

service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-min1
  - ses-min2
spec:
  pool: EXAMPLE_POOL
  namespace: EXAMPLE_NAMESPACE

  • EXAMPLE_NFS with an arbitrary string that identifies the NFS export.

  • EXAMPLE_POOL with the name of the pool where the NFS Ganesha RADOS configuration object will be stored.

  • EXAMPLE_NAMESPACE (optional) with the desired Object Gateway NFS namespace (for example, ganesha).

5.4.3.7 Deploying rbd-mirror Edit source

The rbd-mirror service takes care of synchronizing RADOS Block Device images between two Ceph clusters (for more details, see Section 20.4, “RBD image mirrors”). To deploy rbd-mirror, use the following specification: 

service_type: rbd-mirror
service_id: EXAMPLE_RBD_MIRROR
placement:
  hosts:
  - ses-min3

5.4.3.8 Deploying the monitoring stack Edit source

The monitoring stack consists of Prometheus, Prometheus exporters, Prometheus Alertmanager, and Grafana. Ceph Dashboard makes use of these components to store and visualize detailed metrics on cluster usage and performance.

Tip
Tip

If your deployment requires custom or locally served container images of the monitoring stack services, refer to Section 16.1, “Configuring custom or local images”.

To deploy the monitoring stack, follow these steps:

  1. Enable the prometheus module in the Ceph Manager daemon. This exposes the internal Ceph metrics so that Prometheus can read them: 

    cephuser@adm > ceph mgr module enable prometheus
    Note
    Note

    Ensure this command is run before Prometheus is deployed. If the command was not run before the deployment, you must redeploy Prometheus to update Prometheus' configuration: 

    cephuser@adm > ceph orch redeploy prometheus

  2. Create a specification file (for example monitoring.yaml) with a content similar to the following: 

    service_type: prometheus
placement:
  hosts:
  - ses-min2
---
service_type: node-exporter
---
service_type: alertmanager
placement:
  hosts:
  - ses-min4
---
service_type: grafana
placement:
  hosts:
  - ses-min3

  3. Apply monitoring services by running: 

    cephuser@adm > ceph orch apply -i monitoring.yaml

    It may take a minute or two for the monitoring services to be deployed.

Important
Important

Prometheus, Grafana, and the Ceph Dashboard are all automatically configured to talk to each other, resulting in a fully functional Grafana integration in the Ceph Dashboard when deployed as described above.

The only exception to this rule is monitoring with RBD images. See Section 16.5.4, “Enabling RBD-image monitoring” for more information.

→Part III Installation of Additional Services←Chapter 4 Introduction and common tasks
Print this page

© 2021 SUSE