The basic component of Ceph is called RADOS (Reliable Autonomic Distributed Object Store). It is responsible for managing the data stored in the cluster. Data in Ceph is usually stored as objects. Each object consists of an identifier and the data.

RADOS provides the following access methods to the stored objects that cover many use cases:

librados is used by Object Gateway and RBD while CephFS directly interfaces with RADOS Figure 1.1, “Interfaces to the Ceph object store”.

Figure 1.1: Interfaces to the Ceph object store #

  

At the core of a Ceph cluster is the CRUSH algorithm. CRUSH is the acronym for Controlled Replication Under Scalable Hashing. CRUSH is a function that handles the storage allocation and needs comparably few parameters. That means only a small amount of information is necessary to calculate the storage position of an object. The parameters are a current map of the cluster including the health state, some administrator-defined placement rules and the name of the object that needs to be stored or retrieved. With this information, all nodes in the Ceph cluster are able to calculate where an object and its replicas are stored. This makes writing or reading data very efficient. CRUSH tries to evenly distribute data over all nodes in the cluster.

The CRUSH Map contains all storage nodes and administrator-defined placement rules for storing objects in the cluster. It defines a hierarchical structure that usually corresponds to the physical structure of the cluster. For example, the data-containing disks are in hosts, hosts are in racks, racks in rows and rows in data centers. This structure can be used to define failure domains. Ceph then ensures that replications are stored on different branches of a specific failure domain.

If the failure domain is set to rack, replications of objects are distributed over different racks. This can mitigate outages caused by a failed switch in a rack. If one power distribution unit supplies a row of racks, the failure domain can be set to row. When the power distribution unit fails, the replicated data is still available on other rows.

In Ceph, nodes are servers working for the cluster. They can run several different types of daemons. We recommend running only one type of daemon on each node, except for Ceph Manager daemons which can be co-located with Ceph Monitors. Each cluster requires at least Ceph Monitor, Ceph Manager, and Ceph OSD daemons:

To use CephFS, Object Gateway, NFS Ganesha, or iSCSI Gateway, additional nodes are required:

Objects that are stored in a Ceph cluster are put into pools. Pools represent logical partitions of the cluster to the outside world. For each pool a set of rules can be defined, for example, how many replications of each object must exist. The standard configuration of pools is called replicated pool.

Pools usually contain objects but can also be configured to act similar to a RAID 5. In this configuration, objects are stored in chunks along with additional coding chunks. The coding chunks contain the redundant information. The number of data and coding chunks can be defined by the administrator. In this configuration, pools are referred to as erasure coded pools or EC pools.

Placement Groups (PGs) are used for the distribution of data within a pool. When creating a pool, a certain number of placement groups is set. The placement groups are used internally to group objects and are an important factor for the performance of a Ceph cluster. The PG for an object is determined by the object's name.

This section provides a simplified example of how Ceph manages data (see Figure 1.2, “Small scale Ceph example”). This example does not represent a recommended configuration for a Ceph cluster. The hardware setup consists of three storage nodes or Ceph OSDs (Host 1, Host 2, Host 3). Each node has three hard disks which are used as OSDs (osd.1 to osd.9). The Ceph Monitor nodes are neglected in this example.

Note: Difference between Ceph OSD and OSD

While Ceph OSD or Ceph OSD daemon refers to a daemon that is run on a node, the word OSD refers to the logical disk that the daemon interacts with.

The cluster has two pools, Pool A and Pool B. While Pool A replicates objects only two times, resilience for Pool B is more important and it has three replications for each object.

When an application puts an object into a pool, for example via the REST API, a Placement Group (PG1 to PG4) is selected based on the pool and the object name. The CRUSH algorithm then calculates on which OSDs the object is stored, based on the Placement Group that contains the object.

In this example the failure domain is set to host. This ensures that replications of objects are stored on different hosts. Depending on the replication level set for a pool, the object is stored on two or three OSDs that are used by the Placement Group.

An application that writes an object only interacts with one Ceph OSD, the primary Ceph OSD. The primary Ceph OSD takes care of replication and confirms the completion of the write process after all other OSDs have stored the object.

If osd.5 fails, all object in PG1 are still available on osd.1. As soon as the cluster recognizes that an OSD has failed, another OSD takes over. In this example osd.4 is used as a replacement for osd.5. The objects stored on osd.1 are then replicated to osd.4 to restore the replication level.

Figure 1.2: Small scale Ceph example #

  

If a new node with new OSDs is added to the cluster, the cluster map is going to change. The CRUSH function then returns different locations for objects. Objects that receive new locations will be relocated. This process results in a balanced usage of all OSDs.

We recommend a single fault-tolerant network with enough bandwidth to fulfil your requirements. For the Ceph public network environment, we recommend two bonded 25 GbE (or faster) network interfaces bonded using 802.3ad (LACP). This is considered the minimal setup for Ceph. If you are also using a cluster network, we recommend four bonded 25 GbE network interfaces. Bonding two or more network interfaces provides better throughput via link aggregation and, given redundant links and switches, improved fault tolerance and maintainability.

You can also create VLANs to isolate different types of traffic over a bond. For example, you can create a bond to provide two VLAN interfaces, one for the public network, and the second for the cluster network. However, this is not required when setting up Ceph networking. Details on bonding the interfaces can be found in https://documentation.suse.com/sles/15-SP3/html/SLES-all/cha-network.html#sec-network-iface-bonding.

Fault tolerance can be enhanced through isolating the components into failure domains. To improve fault tolerance of the network, bonding one interface from two separate Network Interface Cards (NIC) offers protection against failure of a single NIC. Similarly, creating a bond across two switches protects against failure of a switch. We recommend consulting with the network equipment vendor in order to architect the level of fault tolerance required.

Important: Administration network not supported

Additional administration network setup—that enables for example separating SSH, Salt, or DNS networking—is neither tested nor supported.

Tip: Nodes configured via DHCP

If your storage nodes are configured via DHCP, the default timeouts may not be sufficient for the network to be configured correctly before the various Ceph daemons start. If this happens, the Ceph MONs and OSDs will not start correctly (running systemctl status ceph\* will result in "unable to bind" errors). To avoid this issue, we recommend increasing the DHCP client timeout to at least 30 seconds on each node in your storage cluster. This can be done by changing the following settings on each node:

In /etc/sysconfig/network/dhcp, set

DHCLIENT_WAIT_AT_BOOT="30"

In /etc/sysconfig/network/config, set

WAIT_FOR_INTERFACES="60"

2.1.1.2 Monitoring nodes on different subnets #

  

If the monitor nodes are on multiple subnets, for example they are located in different rooms and served by different switches, you need to specify their public network address in CIDR notation:

cephuser@adm > ceph config set mon public_network "MON_NETWORK_1, MON_NETWORK_2, MON_NETWORK_N

For example:

cephuser@adm > ceph config set mon public_network "192.168.1.0/24, 10.10.0.0/16"

Warning

If you do specify more than one network segment for the public (or cluster) network as described in this section, each of these subnets must be capable of routing to all the others - otherwise, the MONs and other Ceph daemons on different network segments will not be able to communicate and a split cluster will ensue. Additionally, if you are using a firewall, make sure you include each IP address or subnet in your iptables and open ports for them on all nodes as necessary.

A minimal product cluster configuration consists of:

A detailed configuration is:

Figure 2.2: Minimum cluster configuration #

  

Once you grow your cluster, we recommend relocating Ceph Monitors, Metadata Servers, and Gateways to separate nodes for better fault tolerance.

If you want to use multipath hardware, ensure that LVM sees multipath_component_detection = 1 in the configuration file under the devices section. This can be checked via the lvm config command.

Alternatively, ensure that LVM filters a device's mpath components via the LVM filter configuration. This will be host specific.

Note

This is not recommended and should only ever be considered if multipath_component_detection = 1 cannot be set.

For more information on multipath configuration, see https://documentation.suse.com/sles/15-SP3/html/SLES-all/cha-multipath.html#sec-multipath-lvm.

There are two types of disk space needed to run on OSD: the space for the WAL/DB device, and the primary space for the stored data. The minimum (and default) value for the WAL/DB is 6 GB. The minimum space for data is 5 GB, as partitions smaller than 5 GB are automatically assigned the weight of 0.

So although the minimum disk space for an OSD is 11 GB, we do not recommend a disk smaller than 20 GB, even for testing purposes.

Solid-state drives (SSD) have no moving parts. This reduces random access time and read latency while accelerating data throughput. Because their price per 1MB is significantly higher than the price of spinning hard disks, SSDs are only suitable for smaller storage.

OSDs may see a significant performance improvement by storing their WAL/DB partitions on an SSD and the object data on a separate hard disk.

Tip: Sharing an SSD for Multiple WAL/DB Partitions

As WAL/DB partitions occupy relatively little space, you can share one SSD disk with multiple WAL/DB partitions. Keep in mind that with each WAL/DB partition, the performance of the SSD disk degrades. We do not recommend sharing more than six WAL/DB partitions on the same SSD disk and 12 on NVMe disks.

You can have as many disks in one server as it allows. There are a few things to consider when planning the number of disks per server:

SUSE Manager and SUSE Enterprise Storage are not integrated, therefore SUSE Manager cannot currently manage an SES cluster.

If you run ceph-salt config without any path or subcommand, you will enter an interactive ceph-salt shell. The shell is convenient if you need to configure multiple properties in one batch and do not want type the full command syntax.

root@master # ceph-salt config
/> ls
o- / ............................................................... [...]
  o- ceph_cluster .................................................. [...]
  | o- minions .............................................. [no minions]
  | o- roles ....................................................... [...]
  |   o- admin .............................................. [no minions]
  |   o- bootstrap ........................................... [no minion]
  |   o- cephadm ............................................ [no minions]
  |   o- tuned ..................................................... [...]
  |     o- latency .......................................... [no minions]
  |     o- throughput ....................................... [no minions]
  o- cephadm_bootstrap ............................................. [...]
  | o- advanced .................................................... [...]
  | o- ceph_conf ................................................... [...]
  | o- ceph_image_path .................................. [ no image path]
  | o- dashboard ................................................... [...]
  | | o- force_password_update ................................. [enabled]
  | | o- password ................................................ [admin]
  | | o- ssl_certificate ....................................... [not set]
  | | o- ssl_certificate_key ................................... [not set]
  | | o- username ................................................ [admin]
  | o- mon_ip .................................................. [not set]
  o- containers .................................................... [...]
  | o- registries_conf ......................................... [enabled]
  | | o- registries .............................................. [empty]
  | o- registry_auth ............................................... [...]
  |   o- password .............................................. [not set]
  |   o- registry .............................................. [not set]
  |   o- username .............................................. [not set]
  o- ssh ............................................... [no key pair set]
  | o- private_key .................................. [no private key set]
  | o- public_key .................................... [no public key set]
  o- time_server ........................... [enabled, no server host set]
    o- external_servers .......................................... [empty]
    o- servers ................................................... [empty]
    o- subnet .................................................. [not set]

As you can see from the output of ceph-salt's ls command, the cluster configuration is organized in a tree structure. To configure a specific property of the cluster in the ceph-salt shell, you have two options:

Tip: Autocompletion of configuration snippets

While in a ceph-salt shell, you can use the autocompletion feature similar to a normal Linux shell (Bash) autocompletion. It completes configuration paths, subcommands, or Salt Minion names. When autocompleting a configuration path, you have two options:

• To let the shell finish a path relative to your current position,press the TAB key →| twice.

• To let the shell finish an absolute path, enter / and press the TAB key →| twice.

Tip: Navigating with the cursor keys

If you enter cd from the ceph-salt shell without any path, the command will print a tree structure of the cluster configuration with the line of the current path active. You can use the up and down cursor keys to navigate through individual lines. After you confirm with Enter, the configuration path will change to the last active one.

Important: Convention

To keep the documentation consistent, we will use a single command syntax without entering the ceph-salt shell. For example, you can list the cluster configuration tree by using the following command:

root@master # ceph-salt config ls

Include all or a subset of Salt Minions that we deployed and accepted in Chapter 6, Deploying Salt to the Ceph cluster configuration. You can either specify the Salt Minions by their full names, or use a glob expressions '*' and '?' to include multiple Salt Minions at once. Use the add subcommand under the /ceph_cluster/minions path. The following command includes all accepted Salt Minions:

root@master # ceph-salt config /ceph_cluster/minions add '*'

Verify that the specified Salt Minions were added:

root@master # ceph-salt config /ceph_cluster/minions ls
o- minions ............................................... [Minions: 5]
  o- ses-main.example.com .................................. [no roles]
  o- ses-node1.example.com ................................. [no roles]
  o- ses-node2.example.com ................................. [no roles]
  o- ses-node3.example.com ................................. [no roles]
  o- ses-node4.example.com ................................. [no roles]

Specify which nodes will belong to the Ceph cluster and will be managed by cephadm. Include all nodes that will run Ceph services as well as the Admin Node:

root@master # ceph-salt config /ceph_cluster/roles/cephadm add '*'

The Admin Node is the node where the ceph.conf configuration file and the Ceph admin keyring is installed. You usually run Ceph related commands on the Admin Node.

Tip: Salt Master and Admin Node on the Same Node

In a homogeneous environment where all or most hosts belong to SUSE Enterprise Storage, we recommend having the Admin Node on the same host as the Salt Master.

In a heterogeneous environment where one Salt infrastructure hosts more than one cluster, for example, SUSE Enterprise Storage together with SUSE Manager, do not place the Admin Node on the same host as Salt Master.

To specify the Admin Node, run the following command:

root@master # ceph-salt config /ceph_cluster/roles/admin add ses-main.example.com
1 minion added.
root@master # ceph-salt config /ceph_cluster/roles/admin ls
o- admin ................................................... [Minions: 1]
  o- ses-main.example.com ...................... [Other roles: cephadm]

Tip: Install ceph.conf and the admin keyring on multiple nodes

You can install the Ceph configuration file and admin keyring on multiple nodes if your deployment requires it. For security reasons, avoid installing them on all the cluster's nodes.

You need to specify which of the cluster's Salt Minions will bootstrap the cluster. This minion will become the first one running Ceph Monitor and Ceph Manager services.

root@master # ceph-salt config /ceph_cluster/roles/bootstrap set ses-node1.example.com
Value set.
root@master # ceph-salt config /ceph_cluster/roles/bootstrap ls
o- bootstrap ..................................... [ses-node1.example.com]

Additionally, you need to specify the bootstrap MON's IP address on the public network to ensure that the public_network parameter is set correctly, for example:

root@master # ceph-salt config /cephadm_bootstrap/mon_ip set 192.168.10.20

You need to specify which of the cluster's minions have actively tuned profiles. To do so, add these roles explicitly with the following commands:

Note

One minion cannot have both the latency and throughput roles.

root@master # ceph-salt config /ceph_cluster/roles/tuned/latency add ses-node1.example.com
Adding ses-node1.example.com...
1 minion added.
root@master # ceph-salt config /ceph_cluster/roles/tuned/throughput add ses-node2.example.com
Adding ses-node2.example.com...
1 minion added.

cephadm uses the SSH protocol to communicate with cluster nodes. A user account named cephadm is automatically created and used for SSH communication.

You need to generate the private and public part of the SSH key pair:

root@master # ceph-salt config /ssh generate
Key pair generated.
root@master # ceph-salt config /ssh ls
o- ssh .................................................. [Key Pair set]
  o- private_key ..... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  o- public_key ...... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]

All cluster nodes need to have their time synchronized with a reliable time source. There are several scenarios to approach time synchronization:

Check the time server settings:

root@master # ceph-salt config /time_server ls
o- time_server ................................................ [enabled]
  o- external_servers ............................................... [1]
  | o- pool.ntp.org ............................................... [...]
  o- servers ........................................................ [1]
  | o- ses-main.example.com ......  ............................... [...]
  o- subnet .............................................. [10.20.6.0/24]

Find more information on setting up time synchronization in https://documentation.suse.com/sles/15-SP3/html/SLES-all/cha-ntp.html#sec-ntp-yast.

Ceph Dashboard will be available after the basic cluster is deployed. To access it, you need to set a valid user name and password, for example:

root@master # ceph-salt config /cephadm_bootstrap/dashboard/username set admin
root@master # ceph-salt config /cephadm_bootstrap/dashboard/password set PWD

Tip: Forcing password update

By default, the first dashboard user will be forced to change their password on first login to the dashboard. To disable this feature, run the following command:

root@master # ceph-salt config /cephadm_bootstrap/dashboard/force_password_update disable

The Messenger v2 protocol (MSGR2) is Ceph's on-wire protocol. It provides a security mode that encrypts all data passing over the network, encapsulation of authentication payloads, and the enabling of future integration of new authentication modes (such as Kerberos).

Important

msgr2 is not currently supported by Linux kernel Ceph clients, such as CephFS and RADOS Block Device.

Ceph daemons can bind to multiple ports, allowing both legacy Ceph clients and new v2-capable clients to connect to the same cluster. By default, MONs now bind to the new IANA-assigned port 3300 (CE4h or 0xCE4) for the new v2 protocol, while also binding to the old default port 6789 for the legacy v1 protocol.

The v2 protocol (MSGR2) supports two connection modes:

For most connections, there are options that control which modes are used:

There are a parallel set of options that apply specifically to monitors, allowing administrators to set different (usually more secure) requirements on communication with the monitors.

In order to enable MSGR2 encryption mode during the deployment, you need to add some configuration options to the ceph-salt configuration before running ceph-salt apply.

To use secure mode, run the following commands.

Add the global section to ceph_conf in the ceph-salt configuration tool:

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf add global

Set the following options:

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_cluster_mode "secure crc"
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_service_mode "secure crc"
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_client_mode "secure crc"

Note

Ensure secure precedes crc.

To force secure mode, run the following commands:

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_cluster_mode secure
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_service_mode secure
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_client_mode secure

Tip: Updating settings

If you want to change any of the above settings, set the configuration changes in the monitor configuration store. This is achieved using the ceph config set command.

root@master # ceph config set global CONNECTION_OPTION CONNECTION_MODE [--force]

For example:

root@master # ceph config set global ms_cluster_mode "secure crc"

If you want to check the current value, including default value, run the following command:

root@master # ceph config get CEPH_COMPONENT CONNECTION_OPTION

For example, to get the ms_cluster_mode for OSD's, run:

root@master # ceph config get osd ms_cluster_mode

Optionally, if you are running a separate cluster network, you may need to set the cluster network IP address followed by the subnet mask part after the slash sign, for example 192.168.10.22/24.

Run the following commands to enable cluster_network:

root@master # ceph-salt config /cephadm_bootstrap/ceph_conf add global
root@master # ceph-salt config /cephadm_bootstrap/ceph_conf/global set cluster_network NETWORK_ADDR

The minimal cluster configuration is finished. Inspect it for obvious errors:

root@master # ceph-salt config ls
o- / ............................................................... [...]
  o- ceph_cluster .................................................. [...]
  | o- minions .............................................. [Minions: 5]
  | | o- ses-main.example.com .......  ........................... [admin]
  | | o- ses-node1.example.com ........................ [bootstrap, admin]
  | | o- ses-node2.example.com ................................ [no roles]
  | | o- ses-node3.example.com ................................ [no roles]
  | | o- ses-node4.example.com ................................ [no roles]
  | o- roles ....................................................... [...]
  |   o- admin .............................................. [Minions: 2]
  |   | o- ses-main.example.com ......................... [no other roles]
  |   | o- ses-node1.example.com ................ [other roles: bootstrap]
  |   o- bootstrap ............................... [ses-node1.example.com]
  |   o- cephadm ............................................ [Minions: 5]
  |   o- tuned ..................................................... [...]
  |     o- latency .......................................... [no minions]
  |     o- throughput ....................................... [no minions]
  o- cephadm_bootstrap ............................................. [...]
  | o- advanced .................................................... [...]
  | o- ceph_conf ................................................... [...]
  | o- ceph_image_path ............. [registry.suse.com/ses/7.1/ceph/ceph]
  | o- dashboard ................................................... [...]
  |   o- force_password_update ................................. [enabled]
  |   o- password ................................... [randomly generated]
  |   o- username ................................................ [admin]
  | o- mon_ip ............................................ [192.168.10.20]
  o- containers .................................................... [...]
  | o- registries_conf ......................................... [enabled]
  | | o- registries .............................................. [empty]
  | o- registry_auth ............................................... [...]
  |   o- password .............................................. [not set]
  |   o- registry .............................................. [not set]
  |   o- username .............................................. [not set]
  o- ssh .................................................. [Key Pair set]
  | o- private_key ..... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  | o- public_key ...... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
  o- time_server ............................................... [enabled]
    o- external_servers .............................................. [1]
    | o- 0.pt.pool.ntp.org ......................................... [...]
    o- servers ....................................................... [1]
    | o- ses-main.example.com ...................................... [...]
    o- subnet ............................................. [10.20.6.0/24]

Tip: Status of cluster configuration

You can check if the configuration of the cluster is valid by running the following command:

root@master # ceph-salt status
cluster: 5 minions, 0 hosts managed by cephadm
config: OK

After you have configured the basic cluster and its configuration is valid, it is a good idea to export its configuration to a file:

root@master # ceph-salt export > cluster.json

Warning

The output of the ceph-salt export includes the SSH private key. If you are concerned about the security implications, do not execute this command without taking appropriate precautions.

In case you break the cluster configuration and need to revert to a backup state, run:

root@master # ceph-salt import cluster.json

The following command shows the current mode and status of the Ceph orchestrator.

cephuser@adm > ceph orch status

To list all disk devices, run the following:

cephuser@adm > ceph orch device ls
Hostname   Path      Type  Serial  Size   Health   Ident  Fault  Available
ses-main   /dev/vdb  hdd   0d8a... 10.7G  Unknown  N/A    N/A    No
ses-node1  /dev/vdc  hdd   8304... 10.7G  Unknown  N/A    N/A    No
ses-node1  /dev/vdd  hdd   7b81... 10.7G  Unknown  N/A    N/A    No
[...]

Tip: Services and daemons

Service is a general term for a Ceph service of a specific type, for example Ceph Manager.

Daemon is a specific instance of a service, for example, a process mgr.ses-node1.gdlcik running on a node called ses-node1.

To list all services known to cephadm, run:

cephuser@adm > ceph orch ls
NAME  RUNNING  REFRESHED  AGE  PLACEMENT  IMAGE NAME                  IMAGE ID
mgr       1/0  5m ago     -    <no spec>  registry.example.com/[...]  5bf12403d0bd
mon       1/0  5m ago     -    <no spec>  registry.example.com/[...]  5bf12403d0bd

Tip

You can limit the list to services on a particular node with the optional -–host parameter, and services of a particular type with the optional --service-type parameter. Acceptable types are mon, osd, mgr, mds, and rgw.

To list all running daemons deployed by cephadm, run:

cephuser@adm > ceph orch ps
NAME             HOST      STATUS   REFRESHED AGE VERSION    IMAGE ID     CONTAINER ID
mgr.ses-node1.gd ses-node1 running) 8m ago    12d 15.2.0.108 5bf12403d0bd b8104e09814c
mon.ses-node1    ses-node1 running) 8m ago    12d 15.2.0.108 5bf12403d0bd a719e0087369

Tip

To query the status of a particular daemon, use --daemon_type and --daemon_id. For OSDs, the ID is the numeric OSD ID. For MDS, the ID is the file system name:

cephuser@adm > ceph orch ps --daemon_type osd --daemon_id 0
cephuser@adm > ceph orch ps --daemon_type mds --daemon_id my_cephfs

You can create a separate specification file for each type of service, for example:

root@master # cat nfs.yml
service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-node1
  - ses-node2
spec:
  pool: EXAMPLE_POOL
  namespace: EXAMPLE_NAMESPACE

Alternatively, you can specify multiple (or all) service types in one file—for example, cluster.yml—that describes which nodes will run specific services. Remember to separate individual service types with three dashes (---):

cephuser@adm > cat cluster.yml
service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-node1
  - ses-node2
spec:
  pool: EXAMPLE_POOL
  namespace: EXAMPLE_NAMESPACE
---
service_type: rgw
service_id: REALM_NAME.ZONE_NAME
placement:
  hosts:
  - ses-node1
  - ses-node2
  - ses-node3
---
[...]

The aforementioned properties have the following meaning:

Tip: Applying specific services

Ceph cluster services have usually a number of properties specific to them. For examples and details of individual services' specification, refer to Section 8.3, “Deploy Ceph services”.

To deploy Ceph services, cephadm needs to know on which nodes to deploy them. Use the placement property and list the short host names of the nodes that the service applies to:

cephuser@adm > cat cluster.yml
[...]
placement:
  hosts:
  - host1
  - host2
  - host3
[...]

After you have created a full cluster.yml file with specifications of all services and their placement, you can apply the cluster by running the following command:

cephuser@adm > ceph orch apply -i cluster.yml

To view the status of the cluster, run the ceph orch status command. For more details, see Section 8.1.1, “Displaying the orchestrator status”.

Although you deployed services to the Ceph cluster by using the specification files as described in Section 8.2, “Service and placement specification”, the configuration of the cluster may diverge from the original specification during its operation. Also, you may have removed the specification files accidentally.

To retrieve a complete specification of a running cluster, run:

cephuser@adm > ceph orch ls --export
placement:
  hosts:
  - hostname: ses-node1
    name: ''
    network: ''
service_id: my_cephfs
service_name: mds.my_cephfs
service_type: mds
---
placement:
  count: 2
service_name: mgr
service_type: mgr
---
[...]

Tip

You can append the --format option to change the default yaml output format. You can select from json, json-pretty, or yaml. For example:

ceph orch ls --export --format json

Ceph cluster has three or five MONs deployed across different nodes. If there are five or more nodes in the cluster, we recommend deploying five MONs. A good practice is to have MGRs deployed on the same nodes as MONs.

Important: Include Bootstrap MON

When deploying MONs and MGRs, remember to include the first MON that you added when configuring the basic cluster in Section 7.2.5, “Specifying first MON/MGR node”.

To deploy MONs, apply the following specification:

service_type: mon
placement:
  hosts:
  - ses-node1
  - ses-node2
  - ses-node3

Note

If you need to add another node, append the host name to the same YAML list. For example:

service_type: mon
placement:
 hosts:
 - ses-node1
 - ses-node2
 - ses-node3
 - ses-node4

Similarly, to deploy MGRs, apply the following specification:

Important

Ensure your deployment has at least three Ceph Managers in each deployment.

service_type: mgr
placement:
  hosts:
  - ses-node1
  - ses-node2
  - ses-node3

Tip

If MONs or MGRs are not on the same subnet, you need to append the subnet addresses. For example:

service_type: mon
placement:
  hosts:
  - ses-node1:10.1.2.0/24
  - ses-node2:10.1.5.0/24
  - ses-node3:10.1.10.0/24

Important: When Storage Device is Available

A storage device is considered available if all of the following conditions are met:

• The device has no partitions.

• The device does not have any LVM state.

• The device is not be mounted.

• The device does not contain a file system.

• The device does not contain a BlueStore OSD.

• The device is larger than 5 GB.

If the above conditions are not met, Ceph refuses to provision such OSDs.

There are two ways you can deploy OSDs:

CephFS requires one or more Metadata Server (MDS) services. To create a CephFS, first create MDS servers by applying the following specification:

Note

Ensure you have at least two pools, one for CephFS data and one for CephFS metadata, created before applying the following specification.

service_type: mds
service_id: CEPHFS_NAME
placement:
  hosts:
  - ses-node1
  - ses-node2
  - ses-node3

After MDSs are functional, create the CephFS:

ceph fs new CEPHFS_NAME metadata_pool data_pool

cephadm deploys an Object Gateway as a collection of daemons that manage a particular realm and zone.

You can either relate an Object Gateway service to already existing realm and zone, (refer to Book “Administration and Operations Guide”, Chapter 21 “Ceph Object Gateway”, Section 21.13 “Multisite Object Gateways” for more details), or you can specify a non-existing REALM_NAME and ZONE_NAME and they will be created automatically after you apply the following configuration:

service_type: rgw
service_id: REALM_NAME.ZONE_NAME
placement:
  hosts:
  - ses-node1
  - ses-node2
  - ses-node3
spec:
  rgw_realm: RGW_REALM
  rgw_zone: RGW_ZONE

8.3.4.1 Using secure SSL access #

  

To use a secure SSL connection to the Object Gateway, you need a file containing both the valid SSL certificate and the key (see Book “Administration and Operations Guide”, Chapter 21 “Ceph Object Gateway”, Section 21.7 “Enable HTTPS/SSL for Object Gateways” for more details). You need to enable SSL, specify a port number for SSL connections, and the SSL certificate file.

To enable SSL and specify the port number, include the following in your specification:

spec:
  ssl: true
  rgw_frontend_port: 443

To specify the SSL certificate and key, you can paste their contents directly into the YAML specification file. The pipe sign (|) at the end of line tells the parser to expect a multi-line string as a value. For example:

spec:
  ssl: true
  rgw_frontend_port: 443
  rgw_frontend_ssl_certificate: |
   -----BEGIN CERTIFICATE-----
   MIIFmjCCA4KgAwIBAgIJAIZ2n35bmwXTMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
   BAYTAkFVMQwwCgYDVQQIDANOU1cxHTAbBgNVBAoMFEV4YW1wbGUgUkdXIFNTTCBp
   [...]
   -----END CERTIFICATE-----
   -----BEGIN PRIVATE KEY-----
   MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQDLtFwg6LLl2j4Z
   BDV+iL4AO7VZ9KbmWIt37Ml2W6y2YeKX3Qwf+3eBz7TVHR1dm6iPpCpqpQjXUsT9
   [...]
   -----END PRIVATE KEY-----

Tip

Instead of pasting the content of an SSL certificate file, you can omit the rgw_frontend_ssl_certificate: directive and upload the certificate file to the configuration database:

cephuser@adm > ceph config-key set rgw/cert/REALM_NAME/ZONE_NAME.crt \
 -i SSL_CERT_FILE

8.3.4.1.1 Configure the Object Gateway to listen on both ports 443 and 80 #

  

To configure the Object Gateway to listen on both ports 443 (HTTPS) and 80 (HTTP), follow these steps:

Note

The commands in the procedure use realm and zone default.

1. Deploy the Object Gateway by supplying a specification file. Refer to Section 8.3.4, “Deploying Object Gateways” for more details on the Object Gateway specification. Use the following command:

   cephuser@adm > ceph orch apply -i SPEC_FILE

2. If SSL certificates are not supplied in the specification file, add them by using the following command:

   cephuser@adm > ceph config-key set rgw/cert/default/default.crt -i certificate.pem
   cephuser@adm > ceph config-key set rgw/cert/default/default.key -i key.pem

3. Change the default value of the rgw_frontends option:

   cephuser@adm > ceph config set client.rgw.default.default rgw_frontends \
    "beast port=80 ssl_port=443"

4. Remove the specific configuration created by cephadm. Identify for which target the rgw_frontends option was configured by running the command:

   cephuser@adm > ceph config dump | grep rgw

   For example, the target is client.rgw.default.default.node4.yiewdu. Remove the current specific rgw_frontends value:

   cephuser@adm > ceph config rm client.rgw.default.default.node4.yiewdu rgw_frontends

   

   Tip

   Instead of removing a value for rgw_frontends, you can specify it. For example:

   cephuser@adm > ceph config set client.rgw.default.default.node4.yiewdu \
    rgw_frontends "beast port=80 ssl_port=443"

5. Restart Object Gateways:

   cephuser@adm > ceph orch restart rgw.default.default

service_type: rgw
service_id: REALM_NAME.ZONE_NAME.SUBCLUSTER
placement:
  hosts:
  - ses-node1
  - ses-node2
  - ses-node3
spec:
  rgw_realm: RGW_REALM
  rgw_zone: RGW_ZONE
  subcluster: SUBCLUSTER

8.3.4.3 Deploying High Availability for the Object Gateway #

  

You can create high availability (HA) endpoints for the Object Gateway service by deploying Ingress. Ingress includes a combination of HAProxy and Keepalived to provide load balancing on a floating virtual IP address. A virtual IP address is automatically configured only on one of Ingress hosts at a time.

Each Keepalived daemon checks every few seconds whether the HAProxy daemon on the same host is responding. Keepalived also checks whether the master Keepalived daemon is running without problems. If the master Keepalived or the active HAProxy daemon is not responding, one of the remaining Keepalived daemons running in backup mode is elected as the master daemon, and the virtual IP address will be assigned to that node.

Figure 8.1: High Availability setup for Object Gateway #

  

Important

If your Object Gateway deployment uses an SSL connection, disable it and configure Ingress to use the SSL certificate instead of the Object Gateway node.

1. Create specifications for the Object Gateway and Ingress services, for example:

   > cat rgw.yaml
   service_type: rgw
   service_id: myrealm.myzone
   placement:
     hosts:
       - ses-node1
       - ses-node2
       - ses-node3
   [...]

   > cat ingress.yaml
   service_type: ingress
   service_id: rgw.myrealm.myzone
   placement:
     hosts:
       - ingress-host1
       - ingress-host2
       - ingress-host3
   spec:
     backend_service: rgw.myrealm.myzone
     virtual_ip: 192.168.20.1/24
     frontend_port: 8080
     monitor_port: 1967     # used by HAproxy for load balancer status
     ssl_cert: |
       -----BEGIN CERTIFICATE-----
       ...
       -----END CERTIFICATE-----
       -----BEGIN PRIVATE KEY-----
       ...
       -----END PRIVATE KEY-----

2. Apply the specifications:

   cephuser@adm > ceph orch apply -i rgw.yaml
   cephuser@adm > ceph orch apply -i ingress.yaml

cephadm deploys an iSCSI Gateway which is a storage area network (SAN) protocol that allows clients (called initiators) to send SCSI commands to SCSI storage devices (targets) on remote servers.

Apply the following configuration to deploy. Ensure trusted_ip_list contains the IP addresses of all iSCSI Gateway and Ceph Manager nodes (see the example output below).

Note

Ensure the pool is created before applying the following specification.

service_type: iscsi
service_id: EXAMPLE_ISCSI
placement:
  hosts:
  - ses-node1
  - ses-node2
  - ses-node3
spec:
  pool: EXAMPLE_POOL
  api_user: EXAMPLE_USER
  api_password: EXAMPLE_PASSWORD
  trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2"

Note

Ensure the IPs listed for trusted_ip_list do not have a space after the comma separation.

spec:
  api_secure: true

spec:
  pool: EXAMPLE_POOL
  api_user: EXAMPLE_USER
  api_password: EXAMPLE_PASSWORD
  trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2"
  api_secure: true
  ssl_cert: |
    -----BEGIN CERTIFICATE-----
    MIIDtTCCAp2gAwIBAgIYMC4xNzc1NDQxNjEzMzc2MjMyXzxvQ7EcMA0GCSqGSIb3
    DQEBCwUAMG0xCzAJBgNVBAYTAlVTMQ0wCwYDVQQIDARVdGFoMRcwFQYDVQQHDA5T
    [...]
    -----END CERTIFICATE-----
  ssl_key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC5jdYbjtNTAKW4
    /CwQr/7wOiLGzVxChn3mmCIF3DwbL/qvTFTX2d8bDf6LjGwLYloXHscRfxszX/4h
    [...]
    -----END PRIVATE KEY-----

Important

NFS Ganesha supports NFS version 4.1 and newer. It does not support NFS version 3.

cephadm deploys NFS Ganesha using a standard pool .nfs instead of a user-predefined pool. To deploy NFS Ganesha, apply the following specification:

service_type: nfs
service_id: EXAMPLE_NFS
placement:
  hosts:
  - ses-node1
  - ses-node2

EXAMPLE_NFS should be replaced with an arbitrary string that identifies the NFS export.

Enable the nfs module in the Ceph Manager daemon. This allows defining NFS exports through the NFS section in the Ceph Dashboard:

cephuser@adm > ceph mgr module enable nfs

8.3.6.1 Deploying High Availability for the NFS Ganesha #

  

You can create high availability (HA) endpoints for the NFS Ganesha service by deploying Ingress. Ingress includes a combination of HAProxy and Keepalived to provide load balancing on a floating virtual IP address. A virtual IP address is automatically configured only on one of Ingress hosts at a time.

Each Keepalived daemon checks every few seconds whether the HAProxy daemon on the same host is responding. Keepalived also checks whether the master Keepalived daemon is running without problems. If the master Keepalived or the active HAProxy daemon is not responding, one of the remaining Keepalived daemons running in backup mode is elected as the master daemon, and the virtual IP address will be assigned to that node.

Figure 8.2: High Availability setup for NFS Ganesha #

  

1. Ensure that the following sysctl variables are set on each Ingress host:

   net.ipv4.ip_nonlocal_bind=1
   net.ipv4.ip_forward=1

   

   Tip

   You can persistently set the sysctl variables by creating a file /etc/sysctl.d/90-ingress.conf and including the above variables. To apply the changes, run sysctl --system or reboot the host.

2. Create specifications for the Ingress and NFS Ganesha services, for example:

   > cat nfs.yaml
   service_type: nfs
   service_id: nfs-ha
   placement:
     hosts:
     - nfs-node1
     - nfs-node2
     - nfs-node3
   spec:
     port: 12049

   > cat ingress.yaml
   service_type: ingress
   service_id: nfs.nfs-ha
   placement:
     hosts:
     - ingress-node1
     - ingress-node2
     - ingress-node3
   spec:
     backend_service: nfs.nfs-ha
     frontend_port: 2049
     monitor_port: 9049
     virtual_ip: 10.20.27.111

   

   Warning

   We recommend running at least three NFS Ganesha daemons, otherwise, mounted clients may experience excessively long hangs during failovers.

3. Apply the customized service specifications:

   cephuser@adm > ceph orch apply -i nfs.yaml
   cephuser@adm > ceph orch apply -i ingress.yaml

4. Create a NFS Ganesha export. Find more details in Book “Administration and Operations Guide”, Chapter 7 “Manage NFS Ganesha”, Section 7.1 “Creating NFS exports”. Remember to select the High Availability NFS Ganesha cluster and an appropriate storage back-end and pseudo path.

   Alternatively, create the export from the command line on the MGR node:

   cephuser@adm > ceph nfs export create \
    STORAGE_BACKEND \
    NFS_HA_CLUSTER \
    NFS_PSEUDO_PATH \
    STORAGE_BACKEND_INSTANCE \
    EXPORT_PATH

5. Mount the NFS Ganesha share on your clients:

   # mount -t nfs \
   VIRTUAL_IP:PORT/NFS_PSEUDO_PATH \
   /mnt/nfs

The rbd-mirror service takes care of synchronizing RADOS Block Device images between two Ceph clusters (for more details, see Book “Administration and Operations Guide”, Chapter 20 “RADOS Block Device”, Section 20.4 “RBD image mirrors”). To deploy rbd-mirror, use the following specification:

service_type: rbd-mirror
service_id: EXAMPLE_RBD_MIRROR
placement:
  hosts:
  - ses-node3

The monitoring stack consists of Prometheus, Prometheus exporters, Prometheus Alertmanager, and Grafana. Ceph Dashboard makes use of these components to store and visualize detailed metrics on cluster usage and performance.

Tip

If your deployment requires custom or locally served container images of the monitoring stack services, refer to Book “Administration and Operations Guide”, Chapter 16 “Monitoring and alerting”, Section 16.1 “Configuring custom or local images”.

To deploy the monitoring stack, follow these steps:

Important

Prometheus, Grafana, and the Ceph Dashboard are all automatically configured to talk to each other, resulting in a fully functional Grafana integration in the Ceph Dashboard when deployed as described above.

The only exception to this rule is monitoring with RBD images. See Book “Administration and Operations Guide”, Chapter 16 “Monitoring and alerting”, Section 16.5.4 “Enabling RBD-image monitoring” for more information.

iSCSI is an implementation of the Small Computer System Interface (SCSI) command set using the Internet Protocol (IP), specified in RFC 3720. iSCSI is implemented as a service where a client (the initiator) talks to a server (the target) via a session on TCP port 3260. An iSCSI target's IP address and port are called an iSCSI portal, where a target can be exposed through one or more portals. The combination of a target and one or more portals is called the target portal group (TPG).

The underlying data link layer protocol for iSCSI is most often Ethernet. More specifically, modern iSCSI infrastructures use 10 GigE Ethernet or faster networks for optimal throughput. 10 Gigabit Ethernet connectivity between the iSCSI gateway and the back-end Ceph cluster is strongly recommended.

ceph-iscsi combines the benefits of RADOS Block Devices with the ubiquitous versatility of iSCSI. By employing ceph-iscsi on an iSCSI target host (known as the iSCSI Gateway), any application that needs to make use of block storage can benefit from Ceph, even if it does not speak any Ceph client protocol. Instead, users can use iSCSI or any other target front-end protocol to connect to an LIO target, which translates all target I/O to RBD storage operations.

Figure 9.1: Ceph Cluster with a Single iSCSI Gateway #

  

ceph-iscsi is inherently highly-available and supports multipath operations. Thus, downstream initiator hosts can use multiple iSCSI gateways for both high availability and scalability. When communicating with an iSCSI configuration with more than one gateway, initiators may load-balance iSCSI requests across multiple gateways. In the event of a gateway failing, being temporarily unreachable, or being disabled for maintenance, I/O will transparently continue via another gateway.

Figure 9.2: Ceph cluster with multiple iSCSI gateways #

  

A minimum configuration of SUSE Enterprise Storage 7.1 with ceph-iscsi consists of the following components:

A recommended production configuration of SUSE Enterprise Storage 7.1 with ceph-iscsi consists of:

This section describes steps to install and configure an iSCSI Gateway on top of SUSE Enterprise Storage.

cephuser@adm > rbd --pool iscsi-images create --size=102400 testvol

9.1.4.3 Exporting RBD images via iSCSI #

  

To export RBD images via iSCSI, you can use either Ceph Dashboard Web interface or the ceph-iscsi gwcli utility. In this section, we will focus on gwcli only, demonstrating how to create an iSCSI target that exports an RBD image using the command line.

Note

RBD images with the following properties cannot be exported via iSCSI:

• images with the journaling feature enabled

• images with a stripe unit less than 4096 bytes

Important: Incorrectly reported image sizes and the emulate_legacy_capacity option

A bug in older kernels caused iSCSI targets to report LUNs as one block (512 bytes) larger than the underlying RBD image. This allowed iSCSI initiators to write to one additional block of the image. To keep the image data consistent, new kernels support the old behavior by default unless you set the emulate_legacy_capacity attribute to 0.

We recommend disabling emulate_legacy_capacity for newly created RBD images when setting up the iSCSI LUN. If you plan to enable the object-map feature for RBD images, disabling emulate_legacy_capacity is required.

For already existing RBD targets that do not use the object-map feature, you may continue to use the emulate_legacy_capacity enabled. In such case you can safely ignore related target_core_rbd: beyond EOD kernel messages on corresponding gateway nodes.

To avoid data corruption, emulate_legacy_capacity should only be disabled on existing iSCSI LUNs if the RBD image is grown by one 512-byte block at the same time. Disconnect all iSCSI initiators and disable the LUN or iSCSI target, grow the image using the procedure below, disable emulate_legacy_capacity, and then finally re-enable the LUN or iSCSI target.

To see the current image size in bytes, use the following command:

cephuser@adm > rbd info POOL/IMAGE --format json --pretty-format

To resize the image, use the following command:

cephuser@adm > rbd resize POOL/IMAGE -s REPORTED_SIZE + 512

To disable the emulate_legacy_capacity option, enter the gwcli shell and run the following commands:

# gwcli
gwcli > cd /disks/POOL/IMAGE
gwcli > reconfigure emulate_legacy_capacity 0

As root, enter the iSCSI Gateway container:

# cephadm enter --name CONTAINER_NAME

As root, start the iSCSI Gateway command line interface:

# gwcli

Go to iscsi-targets and create a target with the name iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol:

gwcli >  /> cd /iscsi-targets
gwcli >  /iscsi-targets> create iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol

Create the iSCSI gateways by specifying the gateway name and ip address:

gwcli >  /iscsi-targets> cd iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/gateways
gwcli >  /iscsi-target...tvol/gateways> create iscsi1 192.168.124.104
gwcli >  /iscsi-target...tvol/gateways> create iscsi2 192.168.124.105

Tip

Use the help command to show the list of available commands in the current configuration node.

Add the RBD image with the name testvol in the pool iscsi-images::

gwcli >  /iscsi-target...tvol/gateways> cd /disks
gwcli >  /disks> attach iscsi-images/testvol

Map the RBD image to the target:

gwcli >  /disks> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/disks
gwcli >  /iscsi-target...testvol/disks> add iscsi-images/testvol

Note

You can use lower-level tools, such as targetcli, to query the local configuration, but not to modify it.

Tip

You can use the ls command to review the configuration. Some configuration nodes also support the info command, which can be used to display more detailed information.

Note that, by default, ACL authentication is enabled so this target is not accessible yet. Check Section 9.1.4.4, “Authentication and access control” for more information about authentication and access control.

9.1.4.4 Authentication and access control #

  

iSCSI authentication is flexible and covers many authentication possibilities.

9.1.4.4.1 Disabling ACL authentication #

  

No Authentication means that any initiator will be able to access any LUNs on the corresponding target. You can enable No Authentication by disabling the ACL authentication:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts
gwcli >  /iscsi-target...testvol/hosts> auth disable_acl

9.1.4.4.2 Using ACL authentication #

  

When using initiator-name-based ACL authentication, only the defined initiators are allowed to connect. You can define an initiator by doing:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts
gwcli >  /iscsi-target...testvol/hosts> create iqn.1996-04.de.suse:01:e6ca28cc9f20

Defined initiators will be able to connect, but will only have access to the RBD images that were explicitly added to the initiator:

gwcli >  /iscsi-target...:e6ca28cc9f20> disk add rbd/testvol

9.1.4.4.3 Enabling CHAP authentication #

  

In addition to the ACL, you can enable CHAP authentication by specifying a user name and password for each initiator:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts/iqn.1996-04.de.suse:01:e6ca28cc9f20
gwcli >  /iscsi-target...:e6ca28cc9f20> auth username=common12 password=pass12345678

Note

User names must have a length of 8 to 64 characters and can contain alphanumeric characters, ., @, -, _ or :.

Passwords must have a length of 12 to 16 characters and can contain alphanumeric characters, @, -, _ or /..

Optionally, you can also enable CHAP mutual authentication by specifying the mutual_username and mutual_password parameters in the auth command.

9.1.4.4.4 Configuring discovery and mutual authentication #

  

Discovery authentication is independent of the previous authentication methods. It requires credentials for browsing, it is optional, and can be configured by:

gwcli >  /> cd /iscsi-targets
gwcli >  /iscsi-targets> discovery_auth username=du123456 password=dp1234567890

Note

User names must have a length of 8 to 64 characters and can only contain letters, ., @, -, _ or :.

Passwords must have a length of 12 to 16 characters and can only contain letters, @, -, _ or /.

Optionally, you can also specify the mutual_username and mutual_password parameters in the discovery_auth command.

Discovery authentication can be disabled by using the following command:

gwcli >  /iscsi-targets> discovery_auth nochap

9.1.4.5 Configuring advanced settings #

  

ceph-iscsi can be configured with advanced parameters which are subsequently passed on to the LIO I/O target. The parameters are divided up into target and disk parameters.

Warning

Unless otherwise noted, changing these parameters from the default setting is not recommended.

9.1.4.5.1 Viewing target settings #

  

You can view the value of these settings by using the info command:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol
gwcli >  /iscsi-target...i.SYSTEM-ARCH:testvol> info

And change a setting using the reconfigure command:

gwcli >  /iscsi-target...i.SYSTEM-ARCH:testvol> reconfigure login_timeout 20

The available target settings are:

default_cmdsn_depth

Default CmdSN (Command Sequence Number) depth. Limits the amount of requests that an iSCSI initiator can have outstanding at any moment.

default_erl

Default error recovery level.

login_timeout

Login timeout value in seconds.

netif_timeout

NIC failure timeout in seconds.

prod_mode_write_protect

If set to 1, prevents writes to LUNs.

9.1.4.5.2 Viewing disk settings #

  

You can view the value of these settings by using the info command:

gwcli >  /> cd /disks/rbd/testvol
gwcli >  /disks/rbd/testvol> info

And change a setting using the reconfigure command:

gwcli >  /disks/rbd/testvol> reconfigure rbd/testvol emulate_pr 0

The available disk settings are:

block_size

Block size of the underlying device.

emulate_3pc

If set to 1, enables Third Party Copy.

emulate_caw

If set to 1, enables Compare and Write.

emulate_dpo

If set to 1, turns on Disable Page Out.

emulate_fua_read

If set to 1, enables Force Unit Access read.

emulate_fua_write

If set to 1, enables Force Unit Access write.

emulate_model_alias

If set to 1, uses the back-end device name for the model alias.

emulate_pr

If set to 0, support for SCSI Reservations, including Persistent Group Reservations, is disabled. While disabled, the SES iSCSI Gateway can ignore reservation state, resulting in improved request latency.

Tip

Setting backstore_emulate_pr to 0 is recommended if iSCSI initiators do not require SCSI Reservation support.

emulate_rest_reord

If set to 0, the Queue Algorithm Modifier has Restricted Reordering.

emulate_tas

If set to 1, enables Task Aborted Status.

emulate_tpu

If set to 1, enables Thin Provisioning Unmap.

emulate_tpws

If set to 1, enables Thin Provisioning Write Same.

emulate_ua_intlck_ctrl

If set to 1, enables Unit Attention Interlock.

emulate_write_cache

If set to 1, turns on Write Cache Enable.

enforce_pr_isids

If set to 1, enforces persistent reservation ISIDs.

is_nonrot

If set to 1, the backstore is a non-rotational device.

max_unmap_block_desc_count

Maximum number of block descriptors for UNMAP.

max_unmap_lba_count:

Maximum number of LBAs for UNMAP.

max_write_same_len

Maximum length for WRITE_SAME.

optimal_sectors

Optimal request size in sectors.

pi_prot_type

DIF protection type.

queue_depth

Queue depth.

unmap_granularity

UNMAP granularity.

unmap_granularity_alignment

UNMAP granularity alignment.

force_pr_aptpl

When enabled, LIO will always write out the persistent reservation state to persistent storage, regardless of whether the client has requested it via aptpl=1. This has no effect with the kernel RBD back-end for LIO—it always persists PR state. Ideally, the target_core_rbd option should force it to '1' and throw an error if someone tries to disable it via configuration.

unmap_zeroes_data

Affects whether LIO will advertise LBPRZ to SCSI initiators, indicating that zeros will be read back from a region following UNMAP or WRITE SAME with an unmap bit.

The ceph-iscsi supports both rbd (kernel-based) and user:rbd (tcmu-runner) backstores, making all the management transparent and independent of the backstore.

Warning: Technology preview

tcmu-runner based iSCSI Gateway deployments are currently a technology preview.

Unlike kernel-based iSCSI Gateway deployments, tcmu-runner based iSCSI Gateways do not offer support for multipath I/O or SCSI Persistent Reservations.

To export an RADOS Block Device image using tcmu-runner, all you need to do is specify the user:rbd backstore when attaching the disk:

gwcli >  /disks> attach rbd/testvol backstore=user:rbd

Note

When using tcmu-runner, the exported RBD image must have the exclusive-lock feature enabled.

Before upgrading, ensure you read through the following sections to ensure you understand all tasks that need to be executed.

We strongly recommend backing up all cluster configuration and data before starting your upgrade to SUSE Enterprise Storage 7.1. For instructions on how to back up all your data, see Book “Administration and Operations Guide”, Chapter 15 “Backup and restore”.

In case you previously upgraded from version 5, verify that the upgrade to version 6 was completed successfully:

Check for the existence of the /srv/salt/ceph/configuration/files/ceph.conf.import file.

This file is created by the engulf process during the upgrade from SUSE Enterprise Storage 5 to 6. The configuration_init: default-import option is set in /srv/pillar/ceph/proposals/config/stack/default/ceph/cluster.yml.

If configuration_init is still set to default-import, the cluster is using ceph.conf.import as its configuration file and not DeepSea's default ceph.conf, which is compiled from files in /srv/salt/ceph/configuration/files/ceph.conf.d/.

Therefore, you need to inspect ceph.conf.import for any custom configuration, and possibly move the configuration to one of the files in /srv/salt/ceph/configuration/files/ceph.conf.d/.

Then remove the configuration_init: default-import line from /srv/pillar/ceph/proposals/config/stack/default/ceph/cluster.yml.

Verify that all latest updates of SUSE Linux Enterprise Server 15 SP1 and SUSE Enterprise Storage 6 are applied to all cluster nodes:

# zypper refresh && zypper patch

Tip

Refer to https://documentation.suse.com/ses/6/html/ses-all/storage-salt-cluster.html#deepsea-rolling-updates for detailed information about updating the cluster nodes.

After updates are applied, restart the Salt Master, synchronize new Salt modules, and check the cluster health:

root@master # systemctl restart salt-master.service
root@master # salt '*' saltutil.sync_all
cephuser@adm > ceph -s

 cephuser@adm > ceph status
 cluster:
   id:     3fe8b35a-689f-4970-819d-0e6b11f6707c
   health: HEALTH_WARN
   mons are allowing insecure global_id reclaim
 [...]

cephuser@adm > ceph config set mon auth_allow_insecure_global_id_reclaim false

cephuser@adm > ceph config set mon mon_mds_skip_sanity true

Verify that each cluster node has access to the SUSE Linux Enterprise Server 15 SP3 and SUSE Enterprise Storage 7.1 software repositories, as well as the registry of container images.

In SUSE Enterprise Storage 7.1, the Object Gateways are always configured with a realm, which allows for multi-site (see Book “Administration and Operations Guide”, Chapter 21 “Ceph Object Gateway”, Section 21.13 “Multisite Object Gateways” for more details) in the future. If you used a single-site Object Gateway configuration in SUSE Enterprise Storage 6, follow these steps to add a realm. If you do not plan to use the multi-site functionality, you can use default for the realm, zonegroup and zone names.

To have the Object Gateway service containerized, create its specification file as described in Section 8.3.4, “Deploying Object Gateways”, and apply it.

cephuser@adm > ceph orch apply -i RGW.yml

Tip

After applying the new Object Gateway specification, run ceph config dump and inspect the lines that contain client.rgw. to see if there are any old settings that need to be applied to the new Object Gateway instances.

Important

NFS Ganesha supports NFS version 4.1 and newer. It does not support NFS version 3.

Important

The upgrade process disables the nfs module in the Ceph Manager daemon. You can re-enable it by executing the following command from the Admin Node:

cephuser@adm > ceph mgr module enable nfs

The following demonstrates how to migrate an existing NFS Ganesha service running Ceph Nautilus to an NFS Ganesha container running Ceph Octopus.

Warning

The following documentation requires you to have already successfully upgraded the core Ceph services.

NFS Ganesha stores additional per-daemon configuration and exports configuration in a RADOS pool. The configured RADOS pool can be found on the watch_url line of the RADOS_URLS block in the ganesha.conf file. By default, this pool will be named ganesha_config

Before attempting any migration, we strongly recommend making a copy of the export and daemon configuration objects located in the RADOS pool. To locate the configured RADOS pool, run the following command:

cephuser@adm > grep -A5 RADOS_URLS /etc/ganesha/ganesha.conf

To list the contents of the RADOS pool:

cephuser@adm > rados --pool ganesha_config --namespace ganesha ls | sort
  conf-node3
  export-1
  export-2
  export-3
  export-4

To copy the RADOS objects:

cephuser@adm > RADOS_ARGS="--pool ganesha_config --namespace ganesha"
cephuser@adm > OBJS=$(rados $RADOS_ARGS ls)
cephuser@adm > for obj in $OBJS; do rados $RADOS_ARGS get $obj $obj; done
cephuser@adm > ls -lah
total 40K
drwxr-xr-x 2 root root 4.0K Sep 8 03:30 .
drwx------ 9 root root 4.0K Sep 8 03:23 ..
-rw-r--r-- 1 root root 90 Sep 8 03:30 conf-node2
-rw-r--r-- 1 root root 90 Sep 8 03:30 conf-node3
-rw-r--r-- 1 root root 350 Sep 8 03:30 export-1
-rw-r--r-- 1 root root 350 Sep 8 03:30 export-2
-rw-r--r-- 1 root root 350 Sep 8 03:30 export-3
-rw-r--r-- 1 root root 358 Sep 8 03:30 export-4

On a per-node basis, any existing NFS Ganesha service needs to be stopped and then replaced with a container managed by cephadm.

The existing exports can be migrated in two different ways:

After the service has been successfully migrated, the Nautilus-based NFS Ganesha service can be removed.

Unlike MONs, MGRs and OSDs, Metadata Server cannot be adopted in-place. Instead, you need to redeploy them in containers using the Ceph orchestrator.

To upgrade the iSCSI Gateway, you need to redeploy it in containers using the Ceph orchestrator. If you have multiple iSCSI Gateways, you need to redeploy them one-by-one to reduce the service downtime.

Before upgrading, ensure you read through the following sections to ensure you understand all tasks that need to be executed.

We strongly recommend backing up all cluster configuration and data before starting the upgrade. For instructions on how to back up all your data, see Book “Administration and Operations Guide”, Chapter 15 “Backup and restore”.

Verify that each cluster node has access to the SUSE Linux Enterprise Server 15 SP3 and SUSE Enterprise Storage 7.1 software repositories, as well as the registry of container images.

11.1.3.2 Container images #

  

All cluster nodes need access to the container image registry. In most cases, you will use the public SUSE registry at registry.suse.com. You need the following images:

• registry.suse.com/ses/7.1/ceph/ceph

• registry.suse.com/ses/7.1/ceph/grafana

• registry.suse.com/ses/7.1/ceph/prometheus-server

• registry.suse.com/ses/7.1/ceph/prometheus-node-exporter

• registry.suse.com/ses/7.1/ceph/prometheus-alertmanager

Alternatively—for example, for air-gapped deployments—configure a local registry and verify that you have the correct set of container images available. Refer to Section 7.2.10, “Using the container registry” for more details about configuring a local container image registry.

Tip: Remove unused container images

Optionally, remove unused container images remaining on the system after the upgrade:

root@master # salt '*' cmd.shell "podman image prune --all --force"