The basic component of Ceph is called RADOS (Reliable Autonomic Distributed Object Store). It is responsible for managing the data stored in the cluster. Data in Ceph is usually stored as objects. Each object consists of an identifier and the data.

RADOS provides the following access methods to the stored objects that cover many use cases:

librados is used by Object Gateway and RBD while CephFS directly interfaces with RADOS Figure 1.1, “Interfaces to the Ceph object store”.

Figure 1.1: Interfaces to the Ceph object store #

  

At the core of a Ceph cluster is the CRUSH algorithm. CRUSH is the acronym for Controlled Replication Under Scalable Hashing. CRUSH is a function that handles the storage allocation and needs comparably few parameters. That means only a small amount of information is necessary to calculate the storage position of an object. The parameters are a current map of the cluster including the health state, some administrator-defined placement rules and the name of the object that needs to be stored or retrieved. With this information, all nodes in the Ceph cluster are able to calculate where an object and its replicas are stored. This makes writing or reading data very efficient. CRUSH tries to evenly distribute data over all nodes in the cluster.

The CRUSH Map contains all storage nodes and administrator-defined placement rules for storing objects in the cluster. It defines a hierarchical structure that usually corresponds to the physical structure of the cluster. For example, the data-containing disks are in hosts, hosts are in racks, racks in rows and rows in data centers. This structure can be used to define failure domains. Ceph then ensures that replications are stored on different branches of a specific failure domain.

If the failure domain is set to rack, replications of objects are distributed over different racks. This can mitigate outages caused by a failed switch in a rack. If one power distribution unit supplies a row of racks, the failure domain can be set to row. When the power distribution unit fails, the replicated data is still available on other rows.

In Ceph, nodes are servers working for the cluster. They can run several different types of daemons. We recommend running only one type of daemon on each node, except for Ceph Manager daemons which can be co-located with Ceph Monitors. Each cluster requires at least Ceph Monitor, Ceph Manager, and Ceph OSD daemons:

To use CephFS, Object Gateway, NFS Ganesha, or iSCSI Gateway, additional nodes are required:

Objects that are stored in a Ceph cluster are put into pools. Pools represent logical partitions of the cluster to the outside world. For each pool a set of rules can be defined, for example, how many replications of each object must exist. The standard configuration of pools is called replicated pool.

Pools usually contain objects but can also be configured to act similar to a RAID 5. In this configuration, objects are stored in chunks along with additional coding chunks. The coding chunks contain the redundant information. The number of data and coding chunks can be defined by the administrator. In this configuration, pools are referred to as erasure coded pools or EC pools.

Placement Groups (PGs) are used for the distribution of data within a pool. When creating a pool, a certain number of placement groups is set. The placement groups are used internally to group objects and are an important factor for the performance of a Ceph cluster. The PG for an object is determined by the object's name.

This section provides a simplified example of how Ceph manages data (see Figure 1.2, “Small scale Ceph example”). This example does not represent a recommended configuration for a Ceph cluster. The hardware setup consists of three storage nodes or Ceph OSDs (Host 1, Host 2, Host 3). Each node has three hard disks which are used as OSDs (osd.1 to osd.9). The Ceph Monitor nodes are neglected in this example.

Note: Difference between Ceph OSD and OSD

While Ceph OSD or Ceph OSD daemon refers to a daemon that is run on a node, the word OSD refers to the logical disk that the daemon interacts with.

The cluster has two pools, Pool A and Pool B. While Pool A replicates objects only two times, resilience for Pool B is more important and it has three replications for each object.

When an application puts an object into a pool, for example via the REST API, a Placement Group (PG1 to PG4) is selected based on the pool and the object name. The CRUSH algorithm then calculates on which OSDs the object is stored, based on the Placement Group that contains the object.

In this example the failure domain is set to host. This ensures that replications of objects are stored on different hosts. Depending on the replication level set for a pool, the object is stored on two or three OSDs that are used by the Placement Group.

An application that writes an object only interacts with one Ceph OSD, the primary Ceph OSD. The primary Ceph OSD takes care of replication and confirms the completion of the write process after all other OSDs have stored the object.

If osd.5 fails, all object in PG1 are still available on osd.1. As soon as the cluster recognizes that an OSD has failed, another OSD takes over. In this example osd.4 is used as a replacement for osd.5. The objects stored on osd.1 are then replicated to osd.4 to restore the replication level.

Figure 1.2: Small scale Ceph example #

  

If a new node with new OSDs is added to the cluster, the cluster map is going to change. The CRUSH function then returns different locations for objects. Objects that receive new locations will be relocated. This process results in a balanced usage of all OSDs.

We recommend a single fault-tolerant network with enough bandwidth to fulfil your requirements. For the Ceph public network environment, we recommend two bonded 25 GbE (or faster) network interfaces bonded using 802.3ad (LACP). This is considered the minimal setup for Ceph. If you are also using a cluster network, we recommend four bonded 25 GbE network interfaces. Bonding two or more network interfaces provides better throughput via link aggregation and, given redundant links and switches, improved fault tolerance and maintainability.

You can also create VLANs to isolate different types of traffic over a bond. For example, you can create a bond to provide two VLAN interfaces, one for the public network, and the second for the cluster network. However, this is not required when setting up Ceph networking. Details on bonding the interfaces can be found in https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-network.html#sec-network-iface-bonding.

Fault tolerance can be enhanced through isolating the components into failure domains. To improve fault tolerance of the network, bonding one interface from two separate Network Interface Cards (NIC) offers protection against failure of a single NIC. Similarly, creating a bond across two switches protects against failure of a switch. We recommend consulting with the network equipment vendor in order to architect the level of fault tolerance required.

Important: Administration network not supported

Additional administration network setup—that enables for example separating SSH, Salt, or DNS networking—is neither tested nor supported.

Tip: Nodes configured via DHCP

If your storage nodes are configured via DHCP, the default timeouts may not be sufficient for the network to be configured correctly before the various Ceph daemons start. If this happens, the Ceph MONs and OSDs will not start correctly (running systemctl status ceph\* will result in "unable to bind" errors). To avoid this issue, we recommend increasing the DHCP client timeout to at least 30 seconds on each node in your storage cluster. This can be done by changing the following settings on each node:

In /etc/sysconfig/network/dhcp, set

DHCLIENT_WAIT_AT_BOOT="30"

In /etc/sysconfig/network/config, set

WAIT_FOR_INTERFACES="60"

2.1.1.2 Monitoring nodes on different subnets #

  

If the monitor nodes are on multiple subnets, for example they are located in different rooms and served by different switches, you need to specify their public network address in CIDR notation:

cephuser@adm > ceph config set mon public_network "MON_NETWORK_1, MON_NETWORK_2, MON_NETWORK_N

For example:

cephuser@adm > ceph config set mon public_network "192.168.1.0/24, 10.10.0.0/16"

Warning

If you do specify more than one network segment for the public (or cluster) network as described in this section, each of these subnets must be capable of routing to all the others - otherwise, the MONs and other Ceph daemons on different network segments will not be able to communicate and a split cluster will ensue. Additionally, if you are using a firewall, make sure you include each IP address or subnet in your iptables and open ports for them on all nodes as necessary.

A minimal product cluster configuration consists of:

A detailed configuration is:

Figure 2.2: Minimum cluster configuration #

  

Once you grow your cluster, we recommend relocating Ceph Monitors, Metadata Servers, and Gateways to separate nodes for better fault tolerance.

If you want to use multipath hardware, ensure that LVM sees multipath_component_detection = 1 in the configuration file under the devices section. This can be checked via the lvm config command.

Alternatively, ensure that LVM filters a device's mpath components via the LVM filter configuration. This will be host specific.

Note

This is not recommended and should only ever be considered if multipath_component_detection = 1 cannot be set.

For more information on multipath configuration, see https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-multipath.html#sec-multipath-lvm.

There are two types of disk space needed to run on OSD: the space for the WAL/DB device, and the primary space for the stored data. The minimum (and default) value for the WAL/DB is 6 GB. The minimum space for data is 5 GB, as partitions smaller than 5 GB are automatically assigned the weight of 0.

So although the minimum disk space for an OSD is 11 GB, we do not recommend a disk smaller than 20 GB, even for testing purposes.

Solid-state drives (SSD) have no moving parts. This reduces random access time and read latency while accelerating data throughput. Because their price per 1MB is significantly higher than the price of spinning hard disks, SSDs are only suitable for smaller storage.

OSDs may see a significant performance improvement by storing their WAL/DB partitions on an SSD and the object data on a separate hard disk.

Tip: Sharing an SSD for Multiple WAL/DB Partitions

As WAL/DB partitions occupy relatively little space, you can share one SSD disk with multiple WAL/DB partitions. Keep in mind that with each WAL/DB partition, the performance of the SSD disk degrades. We do not recommend sharing more than six WAL/DB partitions on the same SSD disk and 12 on NVMe disks.

You can have as many disks in one server as it allows. There are a few things to consider when planning the number of disks per server:

SUSE Manager and SUSE Enterprise Storage are not integrated, therefore SUSE Manager cannot currently manage an SES cluster.

If you run ceph-salt config without any path or subcommand, you will enter an interactive ceph-salt shell. The shell is convenient if you need to configure multiple properties in one batch and do not want type the full command syntax.

root@master # ceph-salt config
/> ls
o- / ............................................................... [...]
  o- ceph_cluster .................................................. [...]
  | o- minions .............................................. [no minions]
  | o- roles ....................................................... [...]
  |   o- admin .............................................. [no minions]
  |   o- bootstrap ........................................... [no minion]
  |   o- cephadm ............................................ [no minions]
  |   o- tuned ..................................................... [...]
  |     o- latency .......................................... [no minions]
  |     o- throughput ....................................... [no minions]
  o- cephadm_bootstrap ............................................. [...]
  | o- advanced .................................................... [...]
  | o- ceph_conf ................................................... [...]
  | o- ceph_image_path .................................. [ no image path]
  | o- dashboard ................................................... [...]
  | | o- force_password_update ................................. [enabled]
  | | o- password ................................................ [admin]
  | | o- ssl_certificate ....................................... [not set]
  | | o- ssl_certificate_key ................................... [not set]
  | | o- username ................................................ [admin]
  | o- mon_ip .................................................. [not set]
  o- containers .................................................... [...]
  | o- registries_conf ......................................... [enabled]
  | | o- registries .............................................. [empty]
  | o- registry_auth ............................................... [...]
  |   o- password .............................................. [not set]
  |   o- registry .............................................. [not set]
  |   o- username .............................................. [not set]
  o- ssh ............................................... [no key pair set]
  | o- private_key .................................. [no private key set]
  | o- public_key .................................... [no public key set]
  o- time_server ........................... [enabled, no server host set]
    o- external_servers .......................................... [empty]
    o- servers ................................................... [empty]
    o- subnet .................................................. [not set]

As you can see from the output of ceph-salt's ls command, the cluster configuration is organized in a tree structure. To configure a specific property of the cluster in the ceph-salt shell, you have two options:

Tip: Autocompletion of configuration snippets

While in a ceph-salt shell, you can use the autocompletion feature similar to a normal Linux shell (Bash) autocompletion. It completes configuration paths, subcommands, or Salt Minion names. When autocompleting a configuration path, you have two options:

• To let the shell finish a path relative to your current position,press the TAB key →| twice.

• To let the shell finish an absolute path, enter / and press the TAB key →| twice.

Tip: Navigating with the cursor keys

If you enter cd from the ceph-salt shell without any path, the command will print a tree structure of the cluster configuration with the line of the current path active. You can use the up and down cursor keys to navigate through individual lines. After you confirm with Enter, the configuration path will change to the last active one.

Important: Convention

To keep the documentation consistent, we will use a single command syntax without entering the ceph-salt shell. For example, you can list the cluster configuration tree by using the following command:

root@master # ceph-salt config ls

Include all or a subset of Salt Minions that we deployed and accepted in Chapter 6, Deploying Salt to the Ceph cluster configuration. You can either specify the Salt Minions by their full names, or use a glob expressions '*' and '?' to include multiple Salt Minions at once. Use the add subcommand under the /ceph_cluster/minions path. The following command includes all accepted Salt Minions:

root@master # ceph-salt config /ceph_cluster/minions add '*'

Verify that the specified Salt Minions were added:

root@master # ceph-salt config /ceph_cluster/minions ls
o- minions ................................................. [Minions: 5]
  o- ses-master.example.com .................................. [no roles]
  o- ses-min1.example.com .................................... [no roles]
  o- ses-min2.example.com .................................... [no roles]
  o- ses-min3.example.com .................................... [no roles]
  o- ses-min4.example.com .................................... [no roles]

Specify which nodes will belong to the Ceph cluster and will be managed by cephadm. Include all nodes that will run Ceph services as well as the Admin Node:

root@master # ceph-salt config /ceph_cluster/roles/cephadm add '*'

The Admin Node is the node where the ceph.conf configuration file and the Ceph admin keyring is installed. You usually run Ceph related commands on the Admin Node.

Tip: Salt Master and Admin Node on the Same Node

In a homogeneous environment where all or most hosts belong to SUSE Enterprise Storage, we recommend having the Admin Node on the same host as the Salt Master.

In a heterogeneous environment where one Salt infrastructure hosts more than one cluster, for example, SUSE Enterprise Storage together with SUSE Manager, do not place the Admin Node on the same host as Salt Master.

To specify the Admin Node, run the following command:

root@master # ceph-salt config /ceph_cluster/roles/admin add ses-master.example.com
1 minion added.
root@master # ceph-salt config /ceph_cluster/roles/admin ls
o- admin ................................................... [Minions: 1]
  o- ses-master.example.com ...................... [Other roles: cephadm]

Tip: Install ceph.conf and the admin keyring on multiple nodes

You can install the Ceph configuration file and admin keyring on multiple nodes if your deployment requires it. For security reasons, avoid installing them on all the cluster's nodes.

You need to specify which of the cluster's Salt Minions will bootstrap the cluster. This minion will become the first one running Ceph Monitor and Ceph Manager services.

root@master # ceph-salt config /ceph_cluster/roles/bootstrap set ses-min1.example.com
Value set.
root@master # ceph-salt config /ceph_cluster/roles/bootstrap ls
o- bootstrap ..................................... [ses-min1.example.com]

Additionally, you need to specify the bootstrap MON's IP address on the public network to ensure that the public_network parameter is set correctly, for example:

root@master # ceph-salt config /cephadm_bootstrap/mon_ip set 192.168.10.20

You need to specify which of the cluster's minions have actively tuned profiles. To do so, add these roles explicitly with the following commands:

Note

One minion cannot have both the latency and throughput roles.

root@master # ceph-salt config /ceph_cluster/roles/tuned/latency add ses-min1.example.com
Adding ses-min1.example.com...
1 minion added.
root@master # ceph-salt config /ceph_cluster/roles/tuned/throughput add ses-min2.example.com
Adding ses-min2.example.com...
1 minion added.

cephadm uses the SSH protocol to communicate with cluster nodes. A user account named cephadm is automatically created and used for SSH communication.

You need to generate the private and public part of the SSH key pair:

root@master # ceph-salt config /ssh generate
Key pair generated.
root@master # ceph-salt config /ssh ls
o- ssh .................................................. [Key Pair set]
  o- private_key ..... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  o- public_key ...... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]

All cluster nodes need to have their time synchronized with a reliable time source. There are several scenarios to approach time synchronization:

Check the time server settings:

root@master # ceph-salt config /time_server ls
o- time_server ................................................ [enabled]
  o- external_servers ............................................... [1]
  | o- pool.ntp.org ............................................... [...]
  o- servers ........................................................ [1]
  | o- ses-master.example.com ..................................... [...]
  o- subnet .............................................. [10.20.6.0/24]

Find more information on setting up time synchronization in https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-ntp.html#sec-ntp-yast.

Ceph Dashboard will be available after the basic cluster is deployed. To access it, you need to set a valid user name and password, for example:

root@master # ceph-salt config /cephadm_bootstrap/dashboard/username set admin
root@master # ceph-salt config /cephadm_bootstrap/dashboard/password set PWD

Tip: Forcing password update

By default, the first dashboard user will be forced to change their password on first login to the dashboard. To disable this feature, run the following command:

root@master # ceph-salt config /cephadm_bootstrap/dashboard/force_password_update disable

The Ceph cluster needs to have access to a container registry so that it can download and deploy containerized Ceph services. There are two ways to access the registry:

7.2.10.1 Creating and configuring the local registry (optional) #

  

Important

There are numerous methods of creating a local registry. The instructions in this section are examples of creating secure and insecure registries. For general information on running a container image registry, refer to https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-registry-installation.html#sec-docker-registry-installation.

Tip: Placement and port usage

Deploy the registry on a machine accessible by all nodes in the cluster. We recommend the Admin Node. By default, the registry listens on port 5000.

On the registry node, use the following command to ensure that the port is free:

ss -tulpn | grep :5000

If other processes (such as iscsi-tcmu) are already listening on port 5000, determine another free port which can be used to map to port 5000 in the registry container.

Procedure 7.1: Creating the local registry #

  

1. Verify that the Containers Module extension is enabled:

   > SUSEConnect --list-extensions | grep -A2 "Containers Module"
   Containers Module 15 SP2 x86_64 (Activated)

2. Verify that the following packages are installed: apache2-utils (if enabling a secure registry), cni, cni-plugins, podman, podman-cni-config, and skopeo.

3. Gather the following information:

   • Fully qualified domain name of the registry host (REG_HOST_FQDN).

   • An available port number used to map to the registry container port of 5000 (REG_HOST_PORT).

   • Whether the registry will be secure or insecure (insecure=[true|false]).

4. To start an insecure registry (without SSL encryption), follow these steps:

   1. Configure ceph-salt for the insecure registry:

      cephuser@adm > ceph-salt config containers/registries_conf enable
      cephuser@adm > ceph-salt config containers/registries_conf/registries \
       add prefix=REG_HOST_FQDN insecure=true \
       location=REG_HOST_PORT:5000

   2. Start the insecure registry by creating the necessary directory (for example, /var/lib/registry) and starting the registry with the podman command:

      # mkdir -p /var/lib/registry
      # podman run --privileged -d --name registry \
       -p REG_HOST_PORT:5000 -v /var/lib/registry:/var/lib/registry \
       --restart=always registry:2

   3. To have the registry start after a reboot, create a systemd unit file for it and enable it:

      > sudo podman generate systemd --files --name registry
      > sudo mv container-registry.service /etc/systemd/system/
      > sudo systemctl enable container-registry.service

5. To start a secure registry, follow these steps:

   1. Create the necessary directories:

      # mkdir -p /var/lib/registry/{auth,certs}

   2. Generate an SSL certificate:

      # openssl req -newkey rsa:4096 -nodes -sha256 \
       -keyout /var/lib/registry/certs/domain.key -x509 -days 365 \
       -out /var/lib/registry/certs/domain.crt

      

      Note

      Set the CN=[value] value to the fully qualified domain name of the host ([REG_HOST_FQDN]).

   3. Copy the certificate to all cluster nodes and refresh the certificate cache:

      # salt-cp '*' /var/lib/registry/certs/domain.crt \
       /etc/pki/trust/anchors/
      # salt '*' cmd.shell "update-ca-certificates"

   4. Generate a username and password combination for authentication to the registry:

      # htpasswd2 -bBc /var/lib/registry/auth/htpasswd \
       REG_USERNAME REG_PASSWORD

   5. Start the secure registry. Use the REGISTRY_STORAGE_DELETE_ENABLED=true flag so that you can delete images afterwards with the skopeo delete command.

      podman run --name myregistry -p REG_HOST_PORT:5000 \
       -v /var/lib/registry:/var/lib/registry \
       -v /var/lib/registry/auth:/auth:z \
       -e "REGISTRY_AUTH=htpasswd" \
       -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
       -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
       -v /var/lib/registry/certs:/certs:z \
       -e "REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt" \
       -e "REGISTRY_HTTP_TLS_KEY=/certs/domain.key" \
       -e REGISTRY_STORAGE_DELETE_ENABLED=true \
       -e REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true -d registry:2

   6. Test secure access to the registry:

      > curl https://REG_HOST_FQDN:REG_HOST_PORT/v2/_catalog \
       -u REG_USERNAME:REG_PASSWORD

6. When the local registry is created, you need to synchronize container images from the official SUSE registry at registry.suse.com to the local one. You can use the skopeo sync command found in the skopeo package for that purpose. For more details, refer to the manual page (man 1 skopeo-sync). Consider the following examples:

   Example 7.1: Viewing manifest files #

     

   skopeo inspect docker://registry.suse.com/ses/7/ceph/ceph | jq .RepoTags
   skopeo inspect docker://registry.suse.com/ses/7/ceph/grafana | jq .RepoTags
   skopeo inspect docker://registry.suse.com/ses/7/ceph/prometheus-server:2.27.1 | jq .RepoTags
   skopeo inspect docker://registry.suse.com/ses/7/ceph/prometheus-node-exporter:1.1.2 | jq .RepoTags
   skopeo inspect docker://registry.suse.com/ses/7/ceph/prometheus-alertmanager:0.21.0 | jq .RepoTags

   Example 7.2: Synchronize to a directory #

     

   Synchronize all Ceph images:

   skopeo sync --src docker --dest dir registry.suse.com/ses/7/ceph/ceph /root/images/

   Synchronize just the latest images:

   skopeo sync --src docker --dest dir registry.suse.com/ses/7/ceph/ceph:latest /root/images/

   Example 7.3: Synchronize Grafana images: #

     

   skopeo sync --src docker --dest dir registry.suse.com/ses/7/ceph/grafana /root/images/

   Synchronize the latest Grafana images only:

   skopeo sync --src docker --dest dir registry.suse.com/ses/7/ceph/grafana:latest /root/images/

   Example 7.4: Synchronize latest Prometheus images #

     

   skopeo sync --src docker --dest dir registry.suse.com/ses/7/ceph/prometheus-server:2.27.1 /root/images/
   skopeo sync --src docker --dest dir registry.suse.com/ses/7/ceph/prometheus-node-exporter:1.1.2 /root/images/
   skopeo sync --src docker --dest dir registry.suse.com/ses/7/ceph/prometheus-alertmanager:0.21.0 /root/images/

Procedure 7.2: Configure the local registry and access credentials #

  

1. Configure the URL of the local registry:

   cephuser@adm > ceph-salt config /containers/registry_auth/registry set REG_HOST_URL

2. Configure the user name and password to access the local registry:

   cephuser@adm > ceph-salt config /containers/registry_auth/username set REG_USERNAME

   cephuser@adm > ceph-salt config /containers/registry_auth/password set REG_PASSWORD

Tip: Registry cache

To avoid re-syncing the local registry when new updated containers appear, you can configure a registry cache.

7.2.10.2 Configuring the path to container images #

  

Important

This section helps you configure the path to container images of the bootstrap cluster (deployment of the first Ceph Monitor and Ceph Manager pair). The path does not apply to container images of additional services, for example the monitoring stack.

Tip: Configuring HTTPS proxy

If you need to use a proxy to communicate with the container registry server, perform the following configuration steps on all cluster nodes:

1. Copy the configuration file for containers:

   > sudo cp /usr/share/containers/containers.conf /etc/containers/containers.conf

2. Edit the newly-copied file and add the http_proxy setting to its [engine] section, for example:

   > cat /etc/containers/containers.conf
    [engine]
    http_proxy=proxy.example.com
    [...]

cephadm needs to know a valid URI path to container images. Verify the default setting by executing

root@master # ceph-salt config /cephadm_bootstrap/ceph_image_path ls

If you do not need an alternative or local registry, specify the default SUSE container registry:

root@master # ceph-salt config /cephadm_bootstrap/ceph_image_path set registry.suse.com/ses/7/ceph/ceph

If your deployment requires a specific path, for example, a path to a local registry, configure it as follows:

root@master # ceph-salt config /cephadm_bootstrap/ceph_image_path set LOCAL_REGISTRY_PATH

The Messenger v2 protocol (MSGR2) is Ceph's on-wire protocol. It provides a security mode that encrypts all data passing over the network, encapsulation of authentication payloads, and the enabling of future integration of new authentication modes (such as Kerberos).

Important

msgr2 is not currently supported by Linux kernel Ceph clients, such as CephFS and RADOS Block Device.

Ceph daemons can bind to multiple ports, allowing both legacy Ceph clients and new v2-capable clients to connect to the same cluster. By default, MONs now bind to the new IANA-assigned port 3300 (CE4h or 0xCE4) for the new v2 protocol, while also binding to the old default port 6789 for the legacy v1 protocol.

The v2 protocol (MSGR2) supports two connection modes:

For most connections, there are options that control which modes are used:

There are a parallel set of options that apply specifically to monitors, allowing administrators to set different (usually more secure) requirements on communication with the monitors.

In order to enable MSGR2 encryption mode during the deployment, you need to add some configuration options to the ceph-salt configuration before running ceph-salt apply.

To use secure mode, run the following commands.

Add the global section to ceph_conf in the ceph-salt configuration tool:

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf add global

Set the following options:

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_cluster_mode "secure crc"
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_service_mode "secure crc"
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_client_mode "secure crc"

Note

Ensure secure precedes crc.

To force secure mode, run the following commands:

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_cluster_mode secure
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_service_mode secure
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_client_mode secure

Tip: Updating settings

If you want to change any of the above settings, set the configuration changes in the monitor configuration store. This is achieved using the ceph config set command.

root@master # ceph config set global CONNECTION_OPTION CONNECTION_MODE [--force]

For example:

root@master # ceph config set global ms_cluster_mode "secure crc"

If you want to check the current value, including default value, run the following command:

root@master # ceph config get CEPH_COMPONENT CONNECTION_OPTION

For example, to get the ms_cluster_mode for OSD's, run:

root@master # ceph config get osd ms_cluster_mode

Optionally, if you are running a separate cluster network, you may need to set the cluster network IP address followed by the subnet mask part after the slash sign, for example 192.168.10.22/24.

Run the following commands to enable cluster_network:

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf add global
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set cluster_network NETWORK_ADDR

The minimal cluster configuration is finished. Inspect it for obvious errors:

root@master # ceph-salt config ls
o- / ............................................................... [...]
  o- ceph_cluster .................................................. [...]
  | o- minions .............................................. [Minions: 5]
  | | o- ses-master.example.com .................................. [admin]
  | | o- ses-min1.example.com ......................... [bootstrap, admin]
  | | o- ses-min2.example.com ................................. [no roles]
  | | o- ses-min3.example.com ................................. [no roles]
  | | o- ses-min4.example.com ................................. [no roles]
  | o- roles ....................................................... [...]
  |   o- admin .............................................. [Minions: 2]
  |   | o- ses-master.example.com ....................... [no other roles]
  |   | o- ses-min1.example.com ................. [other roles: bootstrap]
  |   o- bootstrap ................................ [ses-min1.example.com]
  |   o- cephadm ............................................ [Minions: 5]
  |   o- tuned ..................................................... [...]
  |     o- latency .......................................... [no minions]
  |     o- throughput ....................................... [no minions]
  o- cephadm_bootstrap ............................................. [...]
  | o- advanced .................................................... [...]
  | o- ceph_conf ................................................... [...]
  | o- ceph_image_path ............... [registry.suse.com/ses/7/ceph/ceph]
  | o- dashboard ................................................... [...]
  |   o- force_password_update ................................. [enabled]
  |   o- password ................................... [randomly generated]
  |   o- username ................................................ [admin]
  | o- mon_ip ............................................ [192.168.10.20]
  o- containers .................................................... [...]
  | o- registries_conf ......................................... [enabled]
  | | o- registries .............................................. [empty]
  | o- registry_auth ............................................... [...]
  |   o- password .............................................. [not set]
  |   o- registry .............................................. [not set]
  |   o- username .............................................. [not set]
  o- ssh .................................................. [Key Pair set]
  | o- private_key ..... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  | o- public_key ...... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  o- time_server ............................................... [enabled]
    o- external_servers .............................................. [1]
    | o- 0.pt.pool.ntp.org ......................................... [...]
    o- servers ....................................................... [1]
    | o- ses-master.example.com .................................... [...]
    o- subnet ............................................. [10.20.6.0/24]

Tip: Status of cluster configuration

You can check if the configuration of the cluster is valid by running the following command:

root@master # ceph-salt status
cluster: 5 minions, 0 hosts managed by cephadm
config: OK

After you have configured the basic cluster and its configuration is valid, it is a good idea to export its configuration to a file:

root@master # ceph-salt export > cluster.json

Warning

The output of the ceph-salt export includes the SSH private key. If you are concerned about the security implications, do not execute this command without taking appropriate precautions.

In case you break the cluster configuration and need to revert to a backup state, run:

root@master # ceph-salt import cluster.json

The following command shows the current mode and status of the Ceph orchestrator.

cephuser@adm > ceph orch status

To list all disk devices, run the following:

cephuser@adm > ceph orch device ls
Hostname   Path      Type  Serial  Size   Health   Ident  Fault  Available
ses-master /dev/vdb  hdd   0d8a... 10.7G  Unknown  N/A    N/A    No
ses-min1   /dev/vdc  hdd   8304... 10.7G  Unknown  N/A    N/A    No
ses-min1   /dev/vdd  hdd   7b81... 10.7G  Unknown  N/A    N/A    No
[...]

Tip: Services and daemons

Service is a general term for a Ceph service of a specific type, for example Ceph Manager.

Daemon is a specific instance of a service, for example a process mgr.ses-min1.gdlcik running on a node called ses-min1.

To list all services known to cephadm, run:

cephuser@adm > ceph orch ls
NAME  RUNNING  REFRESHED  AGE  PLACEMENT  IMAGE NAME                  IMAGE ID
mgr       1/0  5m ago     -    <no spec>  registry.example.com/[...]  5bf12403d0bd
mon       1/0  5m ago     -    <no spec>  registry.example.com/[...]  5bf12403d0bd

Tip

You can limit the list to services on a particular node with the optional -–host parameter, and services of a particular type with the optional --service-type parameter. Acceptable types are mon, osd, mgr, mds, and rgw.

To list all running daemons deployed by cephadm, run:

cephuser@adm > ceph orch ps
NAME            HOST     STATUS   REFRESHED AGE VERSION    IMAGE ID     CONTAINER ID
mgr.ses-min1.gd ses-min1 running) 8m ago    12d 15.2.0.108 5bf12403d0bd b8104e09814c
mon.ses-min1    ses-min1 running) 8m ago    12d 15.2.0.108 5bf12403d0bd a719e0087369

Tip

To query the status of a particular daemon, use --daemon_type and --daemon_id. For OSDs, the ID is the numeric OSD ID. For MDS, the ID is the file system name:

cephuser@adm > ceph orch ps --daemon_type osd --daemon_id 0
cephuser@adm > ceph orch ps --daemon_type mds --daemon_id my_cephfs

You can create a separate specification file for each type of service, for example:

root@master # cat nfs.yml
service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-min1
  - ses-min2
spec:
  pool: EXAMPLE_POOL
  namespace: EXAMPLE_NAMESPACE

Alternatively, you can specify multiple (or all) service types in one file—for example, cluster.yml—that describes which nodes will run specific services. Remember to separate individual service types with three dashes (---):

cephuser@adm > cat cluster.yml
service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-min1
  - ses-min2
spec:
  pool: EXAMPLE_POOL
  namespace: EXAMPLE_NAMESPACE
---
service_type: rgw
service_id: REALM_NAME.ZONE_NAME
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
---
[...]

The aforementioned properties have the following meaning:

Tip: Applying specific services

Ceph cluster services have usually a number of properties specific to them. For examples and details of individual services' specification, refer to Section 8.3, “Deploy Ceph services”.

To deploy Ceph services, cephadm needs to know on which nodes to deploy them. Use the placement property and list the short host names of the nodes that the service applies to:

cephuser@adm > cat cluster.yml
[...]
placement:
  hosts:
  - host1
  - host2
  - host3
[...]

After you have created a full cluster.yml file with specifications of all services and their placement, you can apply the cluster by running the following command:

cephuser@adm > ceph orch apply -i cluster.yml

To view the status of the cluster, run the ceph orch status command. For more details, see Section 8.1.1, “Displaying the orchestrator status”.

Although you deployed services to the Ceph cluster by using the specification files as described in Section 8.2, “Service and placement specification”, the configuration of the cluster may diverge from the original specification during its operation. Also, you may have removed the specification files accidentally.

To retrieve a complete specification of a running cluster, run:

cephuser@adm > ceph orch ls --export
placement:
  hosts:
  - hostname: ses-min1
    name: ''
    network: ''
service_id: my_cephfs
service_name: mds.my_cephfs
service_type: mds
---
placement:
  count: 2
service_name: mgr
service_type: mgr
---
[...]

Tip

You can append the --format option to change the default yaml output format. You can select from json, json-pretty, or yaml. For example:

ceph orch ls --export --format json

Ceph cluster has three or five MONs deployed across different nodes. If there are five or more nodes in the cluster, we recommend deploying five MONs. A good practice is to have MGRs deployed on the same nodes as MONs.

Important: Include Bootstrap MON

When deploying MONs and MGRs, remember to include the first MON that you added when configuring the basic cluster in Section 7.2.5, “Specifying first MON/MGR node”.

To deploy MONs, apply the following specification:

service_type: mon
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3

Note

If you need to add another node, append the host name to the same YAML list. For example:

service_type: mon
placement:
 hosts:
 - ses-min1
 - ses-min2
 - ses-min3
 - ses-min4

Similarly, to deploy MGRs, apply the following specification:

Important

Ensure your deployment has at least three Ceph Managers in each deployment.

service_type: mgr
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3

Tip

If MONs or MGRs are not on the same subnet, you need to append the subnet addresses. For example:

service_type: mon
placement:
  hosts:
  - ses-min1:10.1.2.0/24
  - ses-min2:10.1.5.0/24
  - ses-min3:10.1.10.0/24

Important: When Storage Device is Available

A storage device is considered available if all of the following conditions are met:

• The device has no partitions.

• The device does not have any LVM state.

• The device is not be mounted.

• The device does not contain a file system.

• The device does not contain a BlueStore OSD.

• The device is larger than 5 GB.

If the above conditions are not met, Ceph refuses to provision such OSDs.

There are two ways you can deploy OSDs:

CephFS requires one or more Metadata Server (MDS) services. To create a CephFS, first create MDS servers by applying the following specification:

Note

Ensure you have at least two pools, one for CephFS data and one for CephFS metadata, created before applying the following specification.

service_type: mds
service_id: CEPHFS_NAME
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3

After MDSs are functional, create the CephFS:

ceph fs new CEPHFS_NAME metadata_pool data_pool

cephadm deploys an Object Gateway as a collection of daemons that manage a particular realm and zone.

You can either relate an Object Gateway service to already existing realm and zone, (refer to Book “Administration and Operations Guide”, Chapter 21 “Ceph Object Gateway”, Section 21.13 “Multisite Object Gateways” for more details), or you can specify a non-existing REALM_NAME and ZONE_NAME and they will be created automatically after you apply the following configuration:

service_type: rgw
service_id: REALM_NAME.ZONE_NAME
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
spec:
  rgw_realm: RGW_REALM
  rgw_zone: RGW_ZONE

8.3.4.1 Using secure SSL access #

  

To use a secure SSL connection to the Object Gateway, you need a pair of valid SSL certificate and key files (see Book “Administration and Operations Guide”, Chapter 21 “Ceph Object Gateway”, Section 21.7 “Enable HTTPS/SSL for Object Gateways” for more details). You need to enable SSL, specify a port number for SSL connections, and the SSL certificate and key files.

To enable SSL and specify the port number, include the following in your specification:

spec:
  ssl: true
  rgw_frontend_port: 443

To specify the SSL certificate and key, you can paste their contents directly into the YAML specification file. The pipe sign (|) at the end of line tells the parser to expect a multi-line string as a value. For example:

spec:
  ssl: true
  rgw_frontend_port: 443
  rgw_frontend_ssl_certificate: |
   -----BEGIN CERTIFICATE-----
   MIIFmjCCA4KgAwIBAgIJAIZ2n35bmwXTMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
   BAYTAkFVMQwwCgYDVQQIDANOU1cxHTAbBgNVBAoMFEV4YW1wbGUgUkdXIFNTTCBp
   [...]
   -----END CERTIFICATE-----
   rgw_frontend_ssl_key: |
   -----BEGIN PRIVATE KEY-----
   MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQDLtFwg6LLl2j4Z
   BDV+iL4AO7VZ9KbmWIt37Ml2W6y2YeKX3Qwf+3eBz7TVHR1dm6iPpCpqpQjXUsT9
   [...]
   -----END PRIVATE KEY-----

Tip

Instead of pasting the content of SSL certificate and key files, you can omit the rgw_frontend_ssl_certificate: and rgw_frontend_ssl_key: keywords and upload them to the configuration database:

cephuser@adm > ceph config-key set rgw/cert/REALM_NAME/ZONE_NAME.crt \
 -i SSL_CERT_FILE
cephuser@adm > ceph config-key set rgw/cert/REALM_NAME/ZONE_NAME.key \
 -i SSL_KEY_FILE

8.3.4.1.1 Configure the Object Gateway to listen on both ports 443 and 80 #

  

To configure the Object Gateway to listen on both ports 443 (HTTPS) and 80 (HTTP), follow these steps:

Note

The commands in the procedure use realm and zone default.

1. Deploy the Object Gateway by supplying a specification file. Refer to Section 8.3.4, “Deploying Object Gateways” for more details on the Object Gateway specification. Use the following command:

   cephuser@adm > ceph orch apply -i SPEC_FILE

2. If SSL certificates are not supplied in the specification file, add them by using the following command:

   cephuser@adm > ceph config-key set rgw/cert/default/default.crt -i certificate.pem
   cephuser@adm > ceph config-key set rgw/cert/default/default.key -i key.pem

3. Change the default value of the rgw_frontends option:

   cephuser@adm > ceph config set client.rgw.default.default rgw_frontends \
    "beast port=80 ssl_port=443"

4. Restart Object Gateways:

   cephuser@adm > ceph orch restart rgw.default.default

service_type: rgw
service_id: REALM_NAME.ZONE_NAME.SUBCLUSTER
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
spec:
  rgw_realm: RGW_REALM
  rgw_zone: RGW_ZONE
  subcluster: SUBCLUSTER

cephadm deploys an iSCSI Gateway which is a storage area network (SAN) protocol that allows clients (called initiators) to send SCSI commands to SCSI storage devices (targets) on remote servers.

Apply the following configuration to deploy. Ensure trusted_ip_list contains the IP addresses of all iSCSI Gateway and Ceph Manager nodes (see the example output below).

Note

Ensure the pool is created before applying the following specification.

service_type: iscsi
service_id: EXAMPLE_ISCSI
placement:
  hosts:
  - ses-min1
  - ses-min2
  - ses-min3
spec:
  pool: EXAMPLE_POOL
  api_user: EXAMPLE_USER
  api_password: EXAMPLE_PASSWORD
  trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2"

Note

Ensure the IPs listed for trusted_ip_list do not have a space after the comma separation.

spec:
  api_secure: true

spec:
  pool: EXAMPLE_POOL
  api_user: EXAMPLE_USER
  api_password: EXAMPLE_PASSWORD
  trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2"
  api_secure: true
  ssl_cert: |
    -----BEGIN CERTIFICATE-----
    MIIDtTCCAp2gAwIBAgIYMC4xNzc1NDQxNjEzMzc2MjMyXzxvQ7EcMA0GCSqGSIb3
    DQEBCwUAMG0xCzAJBgNVBAYTAlVTMQ0wCwYDVQQIDARVdGFoMRcwFQYDVQQHDA5T
    [...]
    -----END CERTIFICATE-----
  ssl_key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC5jdYbjtNTAKW4
    /CwQr/7wOiLGzVxChn3mmCIF3DwbL/qvTFTX2d8bDf6LjGwLYloXHscRfxszX/4h
    [...]
    -----END PRIVATE KEY-----

Important

NFS Ganesha supports NFS version 4.1 and newer. It does not support NFS version 3.

cephadm deploys NFS Ganesha using a pre-defined RADOS pool and an optional name-space. To deploy NFS Ganesha, apply the following specification:

Note

You need to have a pre-defined RADOS pool otherwise the ceph orch apply operation will fail. For more information on creating a pool, see Book “Administration and Operations Guide”, Chapter 18 “Manage storage pools”, Section 18.1 “Creating a pool”.

service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-min1
  - ses-min2
spec:
  pool: EXAMPLE_POOL
  namespace: EXAMPLE_NAMESPACE

The rbd-mirror service takes care of synchronizing RADOS Block Device images between two Ceph clusters (for more details, see Book “Administration and Operations Guide”, Chapter 20 “RADOS Block Device”, Section 20.4 “RBD image mirrors”). To deploy rbd-mirror, use the following specification:

service_type: rbd-mirror
service_id: EXAMPLE_RBD_MIRROR
placement:
  hosts:
  - ses-min3

The monitoring stack consists of Prometheus, Prometheus exporters, Prometheus Alertmanager, and Grafana. Ceph Dashboard makes use of these components to store and visualize detailed metrics on cluster usage and performance.

Tip

If your deployment requires custom or locally served container images of the monitoring stack services, refer to Book “Administration and Operations Guide”, Chapter 16 “Monitoring and alerting”, Section 16.1 “Configuring custom or local images”.

To deploy the monitoring stack, follow these steps:

Important

Prometheus, Grafana, and the Ceph Dashboard are all automatically configured to talk to each other, resulting in a fully functional Grafana integration in the Ceph Dashboard when deployed as described above.

The only exception to this rule is monitoring with RBD images. See Book “Administration and Operations Guide”, Chapter 16 “Monitoring and alerting”, Section 16.5.4 “Enabling RBD-image monitoring” for more information.

iSCSI is an implementation of the Small Computer System Interface (SCSI) command set using the Internet Protocol (IP), specified in RFC 3720. iSCSI is implemented as a service where a client (the initiator) talks to a server (the target) via a session on TCP port 3260. An iSCSI target's IP address and port are called an iSCSI portal, where a target can be exposed through one or more portals. The combination of a target and one or more portals is called the target portal group (TPG).

The underlying data link layer protocol for iSCSI is most often Ethernet. More specifically, modern iSCSI infrastructures use 10 GigE Ethernet or faster networks for optimal throughput. 10 Gigabit Ethernet connectivity between the iSCSI gateway and the back-end Ceph cluster is strongly recommended.

ceph-iscsi combines the benefits of RADOS Block Devices with the ubiquitous versatility of iSCSI. By employing ceph-iscsi on an iSCSI target host (known as the iSCSI Gateway), any application that needs to make use of block storage can benefit from Ceph, even if it does not speak any Ceph client protocol. Instead, users can use iSCSI or any other target front-end protocol to connect to an LIO target, which translates all target I/O to RBD storage operations.

Figure 9.1: Ceph Cluster with a Single iSCSI Gateway #

  

ceph-iscsi is inherently highly-available and supports multipath operations. Thus, downstream initiator hosts can use multiple iSCSI gateways for both high availability and scalability. When communicating with an iSCSI configuration with more than one gateway, initiators may load-balance iSCSI requests across multiple gateways. In the event of a gateway failing, being temporarily unreachable, or being disabled for maintenance, I/O will transparently continue via another gateway.

Figure 9.2: Ceph cluster with multiple iSCSI gateways #

  

A minimum configuration of SUSE Enterprise Storage 7 with ceph-iscsi consists of the following components:

A recommended production configuration of SUSE Enterprise Storage 7 with ceph-iscsi consists of:

This section describes steps to install and configure an iSCSI Gateway on top of SUSE Enterprise Storage.

cephuser@adm > rbd --pool iscsi-images create --size=102400 testvol

9.1.4.3 Exporting RBD images via iSCSI #

  

To export RBD images via iSCSI, you can use either Ceph Dashboard Web interface or the ceph-iscsi gwcli utility. In this section, we will focus on gwcli only, demonstrating how to create an iSCSI target that exports an RBD image using the command line.

Note

RBD images with the following properties cannot be exported via iSCSI:

• images with the journaling feature enabled

• images with a stripe unit less than 4096 bytes

As root, enter the iSCSI Gateway container:

# cephadm enter --name CONTAINER_NAME

As root, start the iSCSI Gateway command line interface:

# gwcli

Go to iscsi-targets and create a target with the name iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol:

gwcli >  /> cd /iscsi-targets
gwcli >  /iscsi-targets> create iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol

Create the iSCSI gateways by specifying the gateway name and ip address:

gwcli >  /iscsi-targets> cd iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/gateways
gwcli >  /iscsi-target...tvol/gateways> create iscsi1 192.168.124.104
gwcli >  /iscsi-target...tvol/gateways> create iscsi2 192.168.124.105

Tip

Use the help command to show the list of available commands in the current configuration node.

Add the RBD image with the name testvol in the pool iscsi-images::

gwcli >  /iscsi-target...tvol/gateways> cd /disks
gwcli >  /disks> attach iscsi-images/testvol

Map the RBD image to the target:

gwcli >  /disks> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/disks
gwcli >  /iscsi-target...testvol/disks> add iscsi-images/testvol

Note

You can use lower-level tools, such as targetcli, to query the local configuration, but not to modify it.

Tip

You can use the ls command to review the configuration. Some configuration nodes also support the info command, which can be used to display more detailed information.

Note that, by default, ACL authentication is enabled so this target is not accessible yet. Check Section 9.1.4.4, “Authentication and access control” for more information about authentication and access control.

9.1.4.4 Authentication and access control #

  

iSCSI authentication is flexible and covers many authentication possibilities.

9.1.4.4.1 Disabling ACL authentication #

  

No Authentication means that any initiator will be able to access any LUNs on the corresponding target. You can enable No Authentication by disabling the ACL authentication:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts
gwcli >  /iscsi-target...testvol/hosts> auth disable_acl

9.1.4.4.2 Using ACL authentication #

  

When using initiator-name-based ACL authentication, only the defined initiators are allowed to connect. You can define an initiator by doing:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts
gwcli >  /iscsi-target...testvol/hosts> create iqn.1996-04.de.suse:01:e6ca28cc9f20

Defined initiators will be able to connect, but will only have access to the RBD images that were explicitly added to the initiator:

gwcli >  /iscsi-target...:e6ca28cc9f20> disk add rbd/testvol

9.1.4.4.3 Enabling CHAP authentication #

  

In addition to the ACL, you can enable CHAP authentication by specifying a user name and password for each initiator:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts/iqn.1996-04.de.suse:01:e6ca28cc9f20
gwcli >  /iscsi-target...:e6ca28cc9f20> auth username=common12 password=pass12345678

Note

User names must have a length of 8 to 64 characters and can contain alphanumeric characters, ., @, -, _ or :.

Passwords must have a length of 12 to 16 characters and can contain alphanumeric characters, @, -, _ or /..

Optionally, you can also enable CHAP mutual authentication by specifying the mutual_username and mutual_password parameters in the auth command.

9.1.4.4.4 Configuring discovery and mutual authentication #

  

Discovery authentication is independent of the previous authentication methods. It requires credentials for browsing, it is optional, and can be configured by:

gwcli >  /> cd /iscsi-targets
gwcli >  /iscsi-targets> discovery_auth username=du123456 password=dp1234567890

Note

User names must have a length of 8 to 64 characters and can only contain letters, ., @, -, _ or :.

Passwords must have a length of 12 to 16 characters and can only contain letters, @, -, _ or /.

Optionally, you can also specify the mutual_username and mutual_password parameters in the discovery_auth command.

Discovery authentication can be disabled by using the following command:

gwcli >  /iscsi-targets> discovery_auth nochap

9.1.4.5 Configuring advanced settings #

  

ceph-iscsi can be configured with advanced parameters which are subsequently passed on to the LIO I/O target. The parameters are divided up into target and disk parameters.

Warning

Unless otherwise noted, changing these parameters from the default setting is not recommended.

9.1.4.5.1 Viewing target settings #

  

You can view the value of these settings by using the info command:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol
gwcli >  /iscsi-target...i.SYSTEM-ARCH:testvol> info

And change a setting using the reconfigure command:

gwcli >  /iscsi-target...i.SYSTEM-ARCH:testvol> reconfigure login_timeout 20

The available target settings are:

default_cmdsn_depth

Default CmdSN (Command Sequence Number) depth. Limits the amount of requests that an iSCSI initiator can have outstanding at any moment.

default_erl

Default error recovery level.

login_timeout

Login timeout value in seconds.

netif_timeout

NIC failure timeout in seconds.

prod_mode_write_protect

If set to 1, prevents writes to LUNs.

9.1.4.5.2 Viewing disk settings #

  

You can view the value of these settings by using the info command:

gwcli >  /> cd /disks/rbd/testvol
gwcli >  /disks/rbd/testvol> info

And change a setting using the reconfigure command:

gwcli >  /disks/rbd/testvol> reconfigure rbd/testvol emulate_pr 0

The available disk settings are:

block_size

Block size of the underlying device.

emulate_3pc

If set to 1, enables Third Party Copy.

emulate_caw

If set to 1, enables Compare and Write.

emulate_dpo

If set to 1, turns on Disable Page Out.

emulate_fua_read

If set to 1, enables Force Unit Access read.

emulate_fua_write

If set to 1, enables Force Unit Access write.

emulate_model_alias

If set to 1, uses the back-end device name for the model alias.

emulate_pr

If set to 0, support for SCSI Reservations, including Persistent Group Reservations, is disabled. While disabled, the SES iSCSI Gateway can ignore reservation state, resulting in improved request latency.

Tip

Setting backstore_emulate_pr to 0 is recommended if iSCSI initiators do not require SCSI Reservation support.

emulate_rest_reord

If set to 0, the Queue Algorithm Modifier has Restricted Reordering.

emulate_tas

If set to 1, enables Task Aborted Status.

emulate_tpu

If set to 1, enables Thin Provisioning Unmap.

emulate_tpws

If set to 1, enables Thin Provisioning Write Same.

emulate_ua_intlck_ctrl

If set to 1, enables Unit Attention Interlock.

emulate_write_cache

If set to 1, turns on Write Cache Enable.

enforce_pr_isids

If set to 1, enforces persistent reservation ISIDs.

is_nonrot

If set to 1, the backstore is a non-rotational device.

max_unmap_block_desc_count

Maximum number of block descriptors for UNMAP.

max_unmap_lba_count:

Maximum number of LBAs for UNMAP.

max_write_same_len

Maximum length for WRITE_SAME.

optimal_sectors

Optimal request size in sectors.

pi_prot_type

DIF protection type.

queue_depth

Queue depth.

unmap_granularity

UNMAP granularity.

unmap_granularity_alignment

UNMAP granularity alignment.

force_pr_aptpl

When enabled, LIO will always write out the persistent reservation state to persistent storage, regardless of whether the client has requested it via aptpl=1. This has no effect with the kernel RBD back-end for LIO—it always persists PR state. Ideally, the target_core_rbd option should force it to '1' and throw an error if someone tries to disable it via configuration.

unmap_zeroes_data

Affects whether LIO will advertise LBPRZ to SCSI initiators, indicating that zeros will be read back from a region following UNMAP or WRITE SAME with an unmap bit.

The ceph-iscsi supports both rbd (kernel-based) and user:rbd (tcmu-runner) backstores, making all the management transparent and independent of the backstore.

Warning: Technology preview

tcmu-runner based iSCSI Gateway deployments are currently a technology preview.

Unlike kernel-based iSCSI Gateway deployments, tcmu-runner based iSCSI Gateways do not offer support for multipath I/O or SCSI Persistent Reservations.

To export an RADOS Block Device image using tcmu-runner, all you need to do is specify the user:rbd backstore when attaching the disk:

gwcli >  /disks> attach rbd/testvol backstore=user:rbd

Note

When using tcmu-runner, the exported RBD image must have the exclusive-lock feature enabled.