The basic component of Ceph is called RADOS (Reliable Autonomic Distributed Object Store). It is responsible for managing the data stored in the cluster. Data in Ceph is usually stored as objects. Each object consists of an identifier and the data.

RADOS provides the following access methods to the stored objects that cover many use cases:

librados is used by Object Gateway and RBD while CephFS directly interfaces with RADOS Figure 1.1, “Interfaces to the Ceph Object Store”.

Figure 1.1: Interfaces to the Ceph Object Store #

At the core of a Ceph cluster is the CRUSH algorithm. CRUSH is the acronym for Controlled Replication Under Scalable Hashing. CRUSH is a function that handles the storage allocation and needs comparably few parameters. That means only a small amount of information is necessary to calculate the storage position of an object. The parameters are a current map of the cluster including the health state, some administrator-defined placement rules and the name of the object that needs to be stored or retrieved. With this information, all nodes in the Ceph cluster are able to calculate where an object and its replicas are stored. This makes writing or reading data very efficient. CRUSH tries to evenly distribute data over all nodes in the cluster.

The CRUSH map contains all storage nodes and administrator-defined placement rules for storing objects in the cluster. It defines a hierarchical structure that usually corresponds to the physical structure of the cluster. For example, the data-containing disks are in hosts, hosts are in racks, racks in rows and rows in data centers. This structure can be used to define failure domains. Ceph then ensures that replications are stored on different branches of a specific failure domain.

If the failure domain is set to rack, replications of objects are distributed over different racks. This can mitigate outages caused by a failed switch in a rack. If one power distribution unit supplies a row of racks, the failure domain can be set to row. When the power distribution unit fails, the replicated data is still available on other rows.

In Ceph, nodes are servers working for the cluster. They can run several different types of daemons. We recommend running only one type of daemon on each node, except for Ceph Manager daemons which can be co-located with Ceph Monitors. Each cluster requires at least Ceph Monitor, Ceph Manager, and Ceph OSD daemons:

To use CephFS, Object Gateway, NFS Ganesha, or iSCSI Gateway, additional nodes are required:

Objects that are stored in a Ceph cluster are put into pools. Pools represent logical partitions of the cluster to the outside world. For each pool a set of rules can be defined, for example, how many replications of each object must exist. The standard configuration of pools is called replicated pool.

Pools usually contain objects but can also be configured to act similar to a RAID 5. In this configuration, objects are stored in chunks along with additional coding chunks. The coding chunks contain the redundant information. The number of data and coding chunks can be defined by the administrator. In this configuration, pools are referred to as erasure coded pools.

Placement Groups (PGs) are used for the distribution of data within a pool. When creating a pool, a certain number of placement groups is set. The placement groups are used internally to group objects and are an important factor for the performance of a Ceph cluster. The PG for an object is determined by the object's name.

This section provides a simplified example of how Ceph manages data (see Figure 1.2, “Small Scale Ceph Example”). This example does not represent a recommended configuration for a Ceph cluster. The hardware setup consists of three storage nodes or Ceph OSDs (Host 1, Host 2, Host 3). Each node has three hard disks which are used as OSDs (osd.1 to osd.9). The Ceph Monitor nodes are neglected in this example.

Note: Difference between Ceph OSD and OSD

While Ceph OSD or Ceph OSD daemon refers to a daemon that is run on a node, the word OSD refers to the logical disk that the daemon interacts with.

The cluster has two pools, Pool A and Pool B. While Pool A replicates objects only two times, resilience for Pool B is more important and it has three replications for each object.

When an application puts an object into a pool, for example via the REST API, a Placement Group (PG1 to PG4) is selected based on the pool and the object name. The CRUSH algorithm then calculates on which OSDs the object is stored, based on the Placement Group that contains the object.

In this example the failure domain is set to host. This ensures that replications of objects are stored on different hosts. Depending on the replication level set for a pool, the object is stored on two or three OSDs that are used by the Placement Group.

An application that writes an object only interacts with one Ceph OSD, the primary Ceph OSD. The primary Ceph OSD takes care of replication and confirms the completion of the write process after all other OSDs have stored the object.

If osd.5 fails, all object in PG1 are still available on osd.1. As soon as the cluster recognizes that an OSD has failed, another OSD takes over. In this example osd.4 is used as a replacement for osd.5. The objects stored on osd.1 are then replicated to osd.4 to restore the replication level.

Figure 1.2: Small Scale Ceph Example #

If a new node with new OSDs is added to the cluster, the cluster map is going to change. The CRUSH function then returns different locations for objects. Objects that receive new locations will be relocated. This process results in a balanced usage of all OSDs.

For the Ceph network environment, we recommend two bonded 25 GbE (or faster) network interfaces bonded using 802.3ad (LACP). The use of two network interfaces provides aggregation and fault-tolerance. The bond should then be used to provide two VLAN interfaces, one for the public network, and the second for the cluster network. Details on bonding the interfaces can be found in https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-network.html#sec-network-iface-bonding.

Fault tolerance can be enhanced through isolating the components into failure domains. To improve fault tolerance of the network, bonding one interface from two separate Network Interface Cards (NIC) offers protection against failure of a single NIC. Similarly, creating a bond across two switches protects against failure of a switch. We recommend consulting with the network equipment vendor in order to architect the level of fault tolerance required.

Important: Administration Network not Supported

Additional administration network setup—that enables for example separating SSH, Salt, or DNS networking—is neither tested nor supported.

Tip: Nodes Configured via DHCP

If your storage nodes are configured via DHCP, the default timeouts may not be sufficient for the network to be configured correctly before the various Ceph daemons start. If this happens, the Ceph MONs and OSDs will not start correctly (running systemctl status ceph\* will result in "unable to bind" errors). To avoid this issue, we recommend increasing the DHCP client timeout to at least 30 seconds on each node in your storage cluster. This can be done by changing the following settings on each node:

In /etc/sysconfig/network/dhcp, set

DHCLIENT_WAIT_AT_BOOT="30"

In /etc/sysconfig/network/config, set

WAIT_FOR_INTERFACES="60"

[global]
 [...]
 mon host = 192.168.123.12, 1.2.3.4, 242.12.33.12
 mon initial members = MON1, MON2, MON3
 [...]

[mon.MON1]
 public network = 192.168.123.0/24

 [mon.MON2]
 public network = 1.2.3.0/24

 [mon.MON3]
 public network = 242.12.33.12/0

A minimal product cluster configuration consists of:

A detailed configuration is:

Figure 2.2: Minimum Cluster Configuration #

Once you grow your cluster, we recommend to relocate monitors, Metadata Server, and gateways on separate nodes to ensure better fault tolerance.

There are two types of disk space needed to run on OSD: the space for the disk journal (for FileStore) or WAL/DB device (for BlueStore), and the primary space for the stored data. The minimum (and default) value for the journal/WAL/DB is 6 GB. The minimum space for data is 5 GB, as partitions smaller than 5 GB are automatically assigned the weight of 0.

So although the minimum disk space for an OSD is 11 GB, we do not recommend a disk smaller than 20 GB, even for testing purposes.

Solid-state drives (SSD) have no moving parts. This reduces random access time and read latency while accelerating data throughput. Because their price per 1MB is significantly higher than the price of spinning hard disks, SSDs are only suitable for smaller storage.

OSDs may see a significant performance improvement by storing their journal on an SSD and the object data on a separate hard disk.

Tip: Sharing an SSD for Multiple Journals

As journal data occupies relatively little space, you can mount several journal directories to a single SSD disk. Keep in mind that with each shared journal, the performance of the SSD disk degrades. We do not recommend sharing more than six journals on the same SSD disk and 12 on NVMe disks.

You can have as many disks in one server as it allows. There are a few things to consider when planning the number of disks per server:

SUSE Manager and SUSE Enterprise Storage are not integrated, therefore SUSE Manager cannot currently manage a SUSE Enterprise Storage cluster.

Salt has several standard locations and several naming conventions used on your master node:

DeepSea commands are executed via the Salt infrastructure. When using the salt command, you need to specify a set of Salt minions that the command will affect. We describe the set of the minions as a target for the salt command. The following sections describe possible methods to target the minions.

5.2.2.1 Matching the Minion Name #

You can target a minion or a group of minions by matching their names. A minion's name is usually the short host name of the node where the minion runs. This is a general Salt targeting method, not related to DeepSea. You can use globbing, regular expressions, or lists to limit the range of minion names. The general syntax follows:

root@master # salt target example.module

Tip: Ceph-only Cluster

If all Salt minions in your environment belong to your Ceph cluster, you can safely substitute target with '*' to include all registered minions.

Match all minions in the example.net domain (assuming the minion names are identical to their "full" host names):

root@master # salt '*.example.net' test.ping

Match the 'web1' to 'web5' minions:

root@master # salt 'web[1-5]' test.ping

Match both 'web1-prod' and 'web1-devel' minions using a regular expression:

root@master # salt -E 'web1-(prod|devel)' test.ping

Match a simple list of minions:

root@master # salt -L 'web1,web2,web3' test.ping

Match all minions in the cluster:

root@master # salt '*' test.ping

root@master # salt target grains.append deepsea default

root@master # salt target grains.delval deepsea destructive=True

root@master # salt -G 'deepsea:*' test.ping

root@master # salt -C 'G@deepsea:*' test.ping

5.2.2.3 Set the deepsea_minions Option #

Setting the deepsea_minions option's target is a requirement for DeepSea deployments. DeepSea uses it to instruct minions during the execution of stages (refer to DeepSea Stages Description for details.

To set or change the deepsea_minions option, edit the /srv/pillar/ceph/deepsea_minions.sls file on the Salt master and add or replace the following line:

deepsea_minions: target

Tip: deepsea_minions Target

As the target for the deepsea_minions option, you can use any targeting method: both Matching the Minion Name and Targeting with a DeepSea Grain.

Match all Salt minions in the cluster:

deepsea_minions: '*'

Match all minions with the 'deepsea' grain:

deepsea_minions: 'G@deepsea:*'

The progress monitor provides a detailed, real-time visualization of what is happening during execution of stages using salt-run state.orch commands in other terminal sessions.

Tip: Start Monitor in a New Terminal Session

You need to start the monitor in a new terminal window before running any salt-run state.orch so that the monitor can detect the start of the stage's execution.

If you start the monitor after issuing the salt-run state.orch command, then no execution progress will be shown.

You can start the monitor mode by running the following command:

root@master # deepsea monitor

For more information about the available command line options of the deepsea monitor command, check its manual page:

root@master # man deepsea-monitor

In the stand-alone mode, DeepSea CLI can be used to run a DeepSea stage, showing its execution in real-time.

The command to run a DeepSea stage from the DeepSea CLI has the following form:

root@master # deepsea stage run stage-name

where stage-name corresponds to the way Salt orchestration state files are referenced. For example, stage deploy, which corresponds to the directory located in /srv/salt/ceph/stage/deploy, is referenced as ceph.stage.deploy.

This command is an alternative to the Salt-based commands for running DeepSea stages (or any DeepSea orchestration state file).

The command deepsea stage run ceph.stage.0 is equivalent to salt-run state.orch ceph.stage.0.

For more information about the available command line options accepted by the deepsea stage run command, check its manual page:

root@master # man deepsea-stage run

In the following figure shows an example of the output of the DeepSea CLI when running Stage 2:

Figure 5.1: DeepSea CLI stage execution progress output #

root@master # deepsea salt-run state.orch stage-name

The /srv/pillar/ceph/proposals/policy.cfg configuration file is used to determine roles of individual cluster nodes. For example, which nodes act as Ceph OSDs or Ceph Monitors. Edit policy.cfg in order to reflect your desired cluster setup. The order of the sections is arbitrary, but the content of included lines overwrites matching keys from the content of previous lines.

Tip: Examples of policy.cfg

You can find several examples of complete policy files in the /usr/share/doc/packages/deepsea/examples/ directory.

cluster-ceph/cluster/*.sls

cluster-ceph/cluster/abc.domain.sls

cluster-ceph/cluster/mon*.sls

cluster-unassigned/cluster/client*.sls

5.5.1.2 Role Assignment #

This section provides you with details on assigning 'roles' to your cluster nodes. A 'role' in this context means the service you need to run on the node, such as Ceph Monitor, Object Gateway, or iSCSI Gateway. No role is assigned automatically, only roles added to policy.cfg will be deployed.

The assignment follows this pattern:

role-ROLE_NAME/PATH/FILES_TO_INCLUDE

Where the items have the following meaning and values:

• ROLE_NAME is any of the following: 'master', 'admin', 'mon', 'mgr', 'storage', 'mds', 'igw', 'rgw', 'ganesha', 'grafana', or 'prometheus'.

• PATH is a relative directory path to .sls or .yml files. In case of .sls files, it usually is cluster, while .yml files are located at stack/default/ceph/minions.

• FILES_TO_INCLUDE are the Salt state files or YAML configuration files. They normally consist of Salt minions' host names, for example ses5min2.yml. Shell globbing can be used for more specific matching.

An example for each role follows:

• master - the node has admin keyrings to all Ceph clusters. Currently, only a single Ceph cluster is supported. As the master role is mandatory, always add a similar line to the following:

  role-master/cluster/master*.sls

• admin - the minion will have an admin keyring. You define the role as follows:

  role-admin/cluster/abc*.sls

• mon - the minion will provide the monitor service to the Ceph cluster. This role requires addresses of the assigned minions. From SUSE Enterprise Storage 5, the public addresses are calculated dynamically and are no longer needed in the Salt pillar.

  role-mon/cluster/mon*.sls

  The example assigns the monitor role to a group of minions.

• mgr - the Ceph manager daemon which collects all the state information from the whole cluster. Deploy it on all minions where you plan to deploy the Ceph monitor role.

  role-mgr/cluster/mgr*.sls

• storage - use this role to specify storage nodes.

  role-storage/cluster/data*.sls

• mds - the minion will provide the metadata service to support CephFS.

  role-mds/cluster/mds*.sls

• igw - the minion will act as an iSCSI Gateway. This role requires addresses of the assigned minions, thus you need to also include the files from the stack directory:

  role-igw/cluster/*.sls

• rgw - the minion will act as an Object Gateway:

  role-rgw/cluster/rgw*.sls

• ganesha - the minion will act as an NFS Ganesha server. The 'ganesha' role requires either an 'rgw' or 'mds' role in cluster, otherwise the validation will fail in Stage 3.

  role-ganesha/cluster/ganesha*.sls

  To successfully install NFS Ganesha, additional configuration is required. If you want to use NFS Ganesha, read Chapter 12, Installation of NFS Ganesha before executing stages 2 and 4. However, it is possible to install NFS Ganesha later.

  In some cases it can be useful to define custom roles for NFS Ganesha nodes. For details, see Book “Administration Guide”, Chapter 30 “NFS Ganesha: Export Ceph Data via NFS”, Section 30.3 “Custom NFS Ganesha Roles”.

• grafana, prometheus - this node adds Grafana charts based on Prometheus alerting to the Ceph Dashboard. Refer to Book “Administration Guide” for its detailed description.

  role-grafana/cluster/grafana*.sls

  role-prometheus/cluster/prometheus*.sls

Note: Multiple Roles of Cluster Nodes

You can assign several roles to a single node. For example, you can assign the 'mds' roles to the monitor nodes:

role-mds/cluster/mon[1,2]*.sls

config/stack/default/global.yml
config/stack/default/ceph/cluster.yml

5.5.1.4 Item Filtering #

Sometimes it is not practical to include all files from a given directory with *.sls globbing. The policy.cfg file parser understands the following filters:

Warning: Advanced Techniques

This section describes filtering techniques for advanced users. When not used correctly, filtering can cause problems for example in case your node numbering changes.

slice=[start:end]

Use the slice filter to include only items start through end-1. Note that items in the given directory are sorted alphanumerically. The following line includes the third to fifth files from the role-mon/cluster/ subdirectory:

role-mon/cluster/*.sls slice[3:6]

re=regexp

Use the regular expression filter to include only items matching the given expressions. For example:

role-mon/cluster/mon*.sls re=.*1[135]\.subdomainX\.sls$

## Cluster Assignment
cluster-ceph/cluster/*.sls 1

## Roles
# ADMIN
role-master/cluster/examplesesadmin.sls 2
role-admin/cluster/sesclient*.sls 3

# MON
role-mon/cluster/ses-example-[123].sls 4

# MGR
role-mgr/cluster/ses-example-[123].sls 5

# STORAGE
role-storage/cluster/ses-example-[5678].sls 6

# MDS
role-mds/cluster/ses-example-4.sls 7

# IGW
role-igw/cluster/ses-example-4.sls 8

# RGW
role-rgw/cluster/ses-example-4.sls 9

# COMMON
config/stack/default/global.yml 10
config/stack/default/ceph/cluster.yml 11

cluster-unassigned/cluster/*.sls
cluster-ceph/cluster/ses-example-*.sls

DriveGroups specify the layouts of OSDs in the Ceph cluster. They are defined in a single file /srv/salt/ceph/configuration/files/drive_groups.yml.

An administrator should manually specify a group of OSDs that are interrelated (hybrid OSDs that are deployed on solid state and spinners) or share the same deployment options (identical, for example same object store, same encryption option, stand-alone OSDs). To avoid explicitly listing devices, DriveGroups use a list of filter items that correspond to a few selected fields of ceph-volume's inventory reports. In the simplest case this could be the 'rotational' flag (all solid-state drives are to be db_devices, all rotating ones data devices) or something more involved such as 'model' strings, or sizes. DeepSea will provide code that translates these DriveGroups into actual device lists for inspection by the user.

Note

Note that the filters use an OR gate to match against the drives.

Following is a simple procedure that demonstrates the basic workflow when configuring DriveGroups:

5.5.2.1 Specification #

/srv/salt/ceph/configuration/files/drive_groups.yml can take one of two basic forms, depending on whether BlueStore or FileStore is to be used. For BlueStore setups, drive_groups.yml can be as follows:

drive_group_default_name:
  target: *
  data_devices:
    drive_spec: DEVICE_SPECIFICATION
  db_devices:
    drive_spec: DEVICE_SPECIFICATION
  wal_devices:
    drive_spec: DEVICE_SPECIFICATION
  block_wal_size: '5G'  # (optional, unit suffixes permitted)
  block_db_size: '5G'   # (optional, unit suffixes permitted)
  osds_per_device: 1   # number of osd daemons per device
  format:              # 'bluestore' or 'filestore' (defaults to 'bluestore')
  encryption:           # 'True' or 'False' (defaults to 'False')

For FileStore setups, drive_groups.yml can be as follows:

drive_group_default_name:
  target: *
  data_devices:
    drive_spec: DEVICE_SPECIFICATION
  journal_devices:
    drive_spec: DEVICE_SPECIFICATION
  format: filestore
  encryption: True

Note

If you are unsure if your OSD is encrypted, see Book “Administration Guide”, Chapter 2 “Salt Cluster Administration”, Section 2.5 “Verify an Encrypted OSD”.

5.5.2.3 Filtering Devices by Size #

You can filter disk devices by their size—either by an exact size, or a size range. The size: parameter accepts arguments in the following form:

• '10G' - Includes disks of an exact size.

• '10G:40G' - Includes disks whose size is within the range.

• ':10G' - Includes disks less than or equal to 10 GB in size.

• '40G:' - Includes disks equal to or greater than 40 GB in size.

Example 5.1: Matching by Disk Size #

drive_group_default:
  target: '*'
  data_devices:
    size: '40TB:'
  db_devices:
    size: ':2TB'

Note: Quotes Required

When using the ':' delimiter, you need to enclose the size in quotes, otherwise the ':' sign will be interpreted as a new configuration hash.

Tip: Unit Shortcuts

Instead of (G)igabytes, you can specify the sizes in (M)egabytes or (T)erabytes as well.

drive_group_default:
  target: '*'
  data_devices:
    model: SSD-123-foo
  db_devices:
    model: MC-55-44-XZ

drive_group_default:
  target: '*'
  data_devices:
    rotational: 1
  db_devices:
    rotational: 0

drive_group_default:
  target: '*'
  data_devices:
    size: '2TB:'
  db_devices:
    size: ':2TB'

drive_group:
  target: '*'
  data_devices:
    rotational: 0
  db_devices:
    model: MC-55-44-XZ

drive_group_default:
  target: '*'
  data_devices:
    model: MC-55-44-XZ
  db_devices:
    vendor: samsung
    size: 256GB

drive_group_node_one_to_five:
  target: 'node[1-5]'
  data_devices:
    rotational: 1
  db_devices:
    rotational: 0

drive_group_the_rest:
  target: 'node[6-10]'
  data_devices:
    model: MC-55-44-XZ
  db_devices:
    model: SSD-123-foo

drive_group_default:
  target: '*'
  data_devices:
    model: MC-55-44-XZ
  db_devices:
    model: SSD-123-foo
  wal_devices:
    model: NVME-QQQQ-987

drive_group_hdd_nvme:
  target: '*'
  data_devices:
    rotational: 0
  db_devices:
    model: NVME-QQQQ-987

drive_group_hdd_ssd_nvme:
  target: '*'
  data_devices:
    rotational: 0
  db_devices:
    model: MC-55-44-XZ
  wal_devices:
    model: NVME-QQQQ-987

drive_group_ssd_nvme:
  target: '*'
  data_devices:
    model: SSD-123-foo
  db_devices:
    model: NVME-QQQQ-987

drive_group_ssd_standalone_encrypted:
  target: '*'
  data_devices:
    model: SSD-123-foo
  encryption: True

If you need to put custom settings into the ceph.conf configuration file, see Book “Administration Guide”, Chapter 2 “Salt Cluster Administration”, Section 2.14 “Adjusting ceph.conf with Custom Settings” for more details.

In the SES 6 release notes, you can find additional information on changes since the previous release of SUSE Enterprise Storage. Check the SES 6 release notes online to see whether:

You can find SES 6 release notes online at https://www.suse.com/releasenotes/.

Your password must be changed to meet SUSE Enterprise Storage 6 requirements. Ensure you change the username and password on all initiators as well. For more information on changing your password, see Section 10.4.4.3, “CHAP Authentication”.

In case you previously upgraded from version 4, verify that the upgrade to version 5 was completed successfully:

In case old RBD kernel clients (older than SUSE Linux Enterprise Server 12 SP3) are being used, refer to Book “Administration Guide”, Chapter 23 “RADOS Block Device”, Section 23.9 “Mapping RBD Using Old Kernel Clients”. We recommend upgrading old RBD kernel clients if possible.

If you used AppArmor in either 'complain' or 'enforce' mode, you need to set a Salt pillar variable before upgrading. Because SUSE Linux Enterprise Server 15 SP1 ships with AppArmor by default, AppArmor management was integrated into DeepSea stage 0. The default behavior in SUSE Enterprise Storage 6 is to remove AppArmor and related profiles. If you want to retain the behavior configured in SUSE Enterprise Storage 5.5, verify that one of the following lines is present in the /srv/pillar/ceph/stack/global.yml file before starting the upgrade:

apparmor_init: default-enforce

or

apparmor_init: default-complain

From SUSE Enterprise Storage 6, MDS names are no longer allowed to begin with a digit, and such names will cause MDS daemons to refuse to start. You can check whether your daemons have such names either by running the ceph fs status command, or by restarting an MDS and checking its logs for the following message:

deprecation warning: MDS id '1mon1' is invalid and will be forbidden in
a future version.  MDS names may not start with a numeric digit.

If you see the above message, the MDS names must be migrated before attempting to upgrade to SUSE Enterprise Storage 6. DeepSea provides an orchestration to automate such a migration. MDS names starting with a digit will be prepended with 'mds.':

root@master # salt-run state.orch ceph.mds.migrate-numerical-names

Tip: Custom Configuration Bound to MDS Names

If you have configuration settings that are bound to MDS names and your MDS daemons have names starting with a digit, verify that your configuration settings apply to the new names as well (with the 'mds.' prefix). Consider the following example section in the /etc/ceph/ceph.conf file:

[mds.123-my-mds] # config setting specific to MDS name with a name starting with a digit
mds cache memory limit = 1073741824
mds standby for name = 456-another-mds

The ceph.mds.migrate-numerical-names orchestrator will change the MDS daemon name '123-my-mds' to 'mds.123-my-mds'. You need to adjust the configuration to reflect the new name:

[mds.mds,123-my-mds] # config setting specific to MDS name with the new name
mds cache memory limit = 1073741824
mds standby for name = mds.456-another-mds

This will add MDS daemons with the new names before removing the old MDS daemons. The number of MDS daemons will double for a short time. Clients will be able to access CephFS only after a short pause for failover to happen. Therefore plan the migration for a time when you expect little or no CephFS load.

The osd_scrub_max_interval and osd_scrub_max_interval settings are used by both OSD and MON daemons. OSDs use these settings to decide when to run scrub, and MONs use them to decide if a warning about scrub not running in time (running too long) should be shown. Therefore, if non-default settings are used, they should be visible by both OSD and MON daemons (that is, defined either in both [osd] and [mon] sections, or in the [global] section), otherwise the monitor may give a false alarm.

In SES 5.5 the monitor warning are disabled by default and the issue may not be noticed if the settings are overridden in the [osd] section only. But when the monitors are upgraded to SES 6, it will start to complain, because in this version the warnings are enabled by default. So if you define non-default scrub settings in your configuration only in the [osd] section, it is desirable to move them to the [global] section before upgrading to SES 6 to avoid false alarms about scrub not running in time.

Although creating backups of a cluster's configuration and data is not mandatory, we strongly recommend backing up important configuration files and cluster data. Refer to Book “Administration Guide”, Chapter 3 “Backing Up Cluster Configuration and Data” for more details.

SUSE Linux Enterprise Server 15 SP1 no longer uses ntpd to synchronize the local host time. Instead, chronyd is used. You need to migrate the time synchronization daemon on each cluster node. You can migrate to chronyd either before migrating the cluster, or upgrade the cluster and migrate to chronyd afterward.

Warning

Before you continue, review your current ntpd settings and determine if you want to keep using the same time server. Keep in mind that the default behaviour will convert to using chronyd.

If you want to manually maintain the chronyd configuration, follow the instructions below and ensure you disable ntpd time configuration. See Procedure 7.1, “Disabling Time Synchronization” for more information.

Apply the latest patches to all cluster nodes prior to upgrade.

6.2.10.1 Required Software Repositories #

Check that required repositories are configured on each host of the cluster. To list all available repositories, run

root@minion > zypper lr

Important: Remove SUSE Enterprise Storage 5.5 LTSS Repositories

Upgrades will fail if LTSS repositories are configured in SUSE Enterprise Storage 5.5. Find their IDs and remove them from the system. For example:

root # zypper lr
[...]
12 | SUSE_Linux_Enterprise_Server_LTSS_12_SP3_x86_64:SLES12-SP3-LTSS-Debuginfo-Updates
13 | SUSE_Linux_Enterprise_Server_LTSS_12_SP3_x86_64:SLES12-SP3-LTSS-Updates
[...]
root # zypper rr 12 13

Tip: Upgrade Without Using SCC, SMT, or RMT

If your nodes are not subscribed to one of the supported software channel providers that handle automatic channel adjustment—such as SMT, RMT, or SCC—you may need to enable additional software modules and channels.

SUSE Enterprise Storage 5.5 requires:

• SLES12-SP3-Installer-Updates

• SLES12-SP3-Pool

• SLES12-SP3-Updates

• SUSE-Enterprise-Storage-5-Pool

• SUSE-Enterprise-Storage-5-Updates

NFS/SMB Gateway on SLE-HA on SUSE Linux Enterprise Server 12 SP3 requires:

• SLE-HA12-SP3-Pool

• SLE-HA12-SP3-Updates

The following commands provide details of the cluster state and specific configuration:

OSD BlueStore is a new back-end for the OSD daemons. It is the default option since SUSE Enterprise Storage 5. Compared to FileStore, which stores objects as files in an XFS file system, BlueStore can deliver increased performance because it stores objects directly on the underlying block device. BlueStore also enables other features, such as built-in compression and EC overwrites, that are unavailable with FileStore.

Specifically for BlueStore, an OSD has a 'wal' (Write Ahead Log) device and a 'db' (RocksDB database) device. The RocksDB database holds the metadata for a BlueStore OSD. These two devices will reside on the same device as an OSD by default, but either can be placed on different, for example faster, media.

In SUSE Enterprise Storage 5, both FileStore and BlueStore are supported and it is possible for FileStore and BlueStore OSDs to co-exist in a single cluster. During the SUSE Enterprise Storage upgrade procedure, FileStore OSDs are not automatically converted to BlueStore.

Warning

Migration to BlueStore needs to be completed on all OSD nodes before the cluster upgrade because FileStore OSDs are not supported in SES 6.

Before converting to BlueStore, the OSDs need to be running SUSE Enterprise Storage 5. The conversion is a slow process as all data gets re-written twice. Though the migration process can take a long time to complete, there is no cluster outage and all clients can continue accessing the cluster during this period. However, do expect lower performance for the duration of the migration. This is caused by rebalancing and backfilling of cluster data.

Use the following procedure to migrate FileStore OSDs to BlueStore:

Tip: Turn Off Safety Measures

Salt commands needed for running the migration are blocked by safety measures. In order to turn these precautions off, run the following command:

root@master # salt-run disengage.safety

Rebuild the nodes before continuing:

root@master #  salt-run rebuild.node TARGET

You can also choose to rebuild each node individually. For example:

root@master #  salt-run rebuild.node data1.ceph

The rebuild.node always removes and recreates all OSDs on the node.

Important

If one OSD fails to convert, re-running the rebuild destroys the already-converted BlueStore OSDs. Instead of re-running the rebuild, you can run:

root@master # salt-run disks.deploy target=NODE-ID

For example:

root@master # salt-run disks.deploy target=data1.ceph

After the migration to BlueStore, the object count will remain the same and disk usage will be nearly the same.

The Distribution Migration System (DMS) provides an upgrade path for an installed SUSE Linux Enterprise system from one major version to another. The following procedure utilizes DMS to upgrade SUSE Enterprise Storage 5.5 to version 6, including the underlying SUSE Linux Enterprise Server 12 SP3 to SUSE Linux Enterprise Server 15 SP1 migration.

Refer to https://documentation.suse.com/suse-distribution-migration-system/1.0/single-html/distribution-migration-system/ to find both general and detailed information about DMS.

6.5.2.1 Before You Begin #

Before the starting the upgrade process, check whether the sles-ltss-release or sles-ltss-release-POOL packages are installed on any node of the cluster:

root@minion > rpm -q sles-ltss-release
 root@minion > rpm -q sles-ltss-release-POOL

If either or both are installed, remove them:

root@minion > zypper rm -y sles-ltss-release sles-ltss-release-POOL

Important

This must be done on all nodes of the cluster before proceeding.

Note

Ensure you also follow the Section 6.2.12, “Check the Cluster's State” guidelines. The upgrade must not be started until all nodes are fully patched. See Section 6.2.10.3, “Patch the Whole Cluster to the Latest Patches” for more information.

For each host that has been upgraded — OSD, MON, MGR, MDS, and Gateway nodes, as well as client hosts — update your ceph.conf file so that it either specifies no monitor port (if you are running the monitors on the default ports) or references both the v2 and v1 addresses and ports explicitly.

Note

Things will still work if only the v1 IP and port are listed, but each CLI instantiation or daemon will need to reconnect after learning that the monitors also speak the v2 protocol. This slows things down and prevents a full transition to the v2 protocol.

Since Nautilus v14.2.20, a new health warning was introduced that informs you that insecure clients are allowed to join the cluster. This warning is on by default. The Ceph Dashboard will show the cluster in the HEALTH_WARN status and verifying the cluster status on the command line informs you as follows:

cephadm@adm > ceph status
cluster:
  id:     3fe8b35a-689f-4970-819d-0e6b11f6707c
  health: HEALTH_WARN
  mons are allowing insecure global_id reclaim
[...]

This warning means that the Ceph Monitors are still allowing old, unpatched clients to connect to the cluster. This ensures existing clients can still connect while the cluster is being upgraded, but warns you that there is a problem that needs to be addressed. When the cluster and all clients are upgraded to the latest version of Ceph, disallow unpatched clients by running the following command:

cephadm@adm > ceph config set mon auth_allow_insecure_global_id_reclaim false

Finally, consider enabling the Telemetry module to send anonymized usage statistics and crash information to the upstream Ceph developers. To see what would be reported (without actually sending any information to anyone):

cephadm@adm > ceph mgr module enable telemetry
cephadm@adm > ceph telemetry show

If you are comfortable with the high-level cluster metadata that will be reported, you can opt-in to automatically report it:

cephadm@adm > ceph telemetry on

To view information about the currently deployed OSDs, synchronize all Salt modules and run the disks.discover command:

root@master # salt '*' saltutil.sync_all
root@master # salt-run disks.discover

Alternatively, you can inspect the content of the files in the /srv/pillar/ceph/proposals/profile-*/ directories. They have a similar structure to the following:

ceph:
  storage:
    osds:
      /dev/disk/by-id/scsi-drive_name: format: bluestore
      /dev/disk/by-id/scsi-drive_name2: format: bluestore

Refer to Section 5.5.2.1, “Specification” for more details on DriveGroups specification.

The difference between a fresh deployment and upgrade scenario is that the drives to be migrated are already 'used'. Because

root@master # salt-run disks.list

looks for unused disks only, use

root@master # salt-run disks.list include_unavailable=True

Adjust DriveGroups until you match your current setup. For a more visual representation of what will be happening, use the following command. Note that it has no output if there are no free disks:

root@master # salt-run disks.report bypass_pillar=True

If you verified that your DriveGroups are properly configured and want to apply the new approach, remove the files from the /srv/pillar/ceph/proposals/profile-PROFILE_NAME/ directory, remove the corresponding profile-PROFILE_NAME/cluster/*.sls lines from the /srv/pillar/ceph/proposals/policy.cfg file, and run DeepSea stage 2 to refresh the Salt pillar.

root@master # salt-run state.orch ceph.stage.2

Verify the result by running the following commands:

root@master # salt target_node pillar.get ceph:storage
root@master # salt-run disks.report

Warning: Incorrect DriveGroups Configuration

If your DriveGroups are not properly configured and there are spare disks in your setup, they will be deployed in the way you specified them. We recommend running:

root@master # salt-run disks.report

As of the Ceph Mimic release (SES 5), the ceph-disk tool is deprecated, and as of the Ceph Nautilus release (SES 6) it is no longer shipped upstream.

ceph-disk is still supported in SUSE Enterprise Storage 6. Any pre-deployed ceph-disk OSDs will continue to function normally. However, when a disk breaks there is no migration path: a disk will need to be re-deployed.

For completeness, consider migrating OSDs on the whole node. There are two paths for SUSE Enterprise Storage 6 users:

Tip: Migrate to LVM Format

Whenever a single legacy OSD needs to be replaced on a node, all OSDs that share devices with it need to be migrated to the LVM-based format.

If you have a more sophisticated setup than just stand-alone OSDs, for example dedicated WAL/DBs or encrypted OSDs, the migration can only happen when all OSDs assigned to that WAL/DB device are removed. This is due to the ceph-volume command that creates Logical Volumes on disks before deployment. This prevents the user from mixing partition based deployments with LV based deployments. In such cases it is best to manually remove all OSDs that are assigned to a WAL/DB device and re-deploy them using the DriveGroups approach.

If you address a specific task outside of the DeepSea deployment process and therefore need to skip it, create a 'no-operation' file following this example:

If you need to replace the default behavior of a specific step with a custom one, create a custom sls file with replacement content.

By default /srv/salt/ceph/pool/default.sls creates an rbd image called 'demo'. In our example, we do not want this image to be created, but we need two images: 'archive1' and 'archive2'.

Note: Authorization

The creation of pools or images requires sufficient authorization. The admin.ceph minion has an admin keyring.

Tip: Alternative Way

Another option is to change the variable in /srv/pillar/ceph/stack/ceph/roles/master.yml instead. Using this file will reduce the clutter of pillar data for other minions.

Sometimes you may need a specific step to do some additional tasks. We do not recommend modifying the related state file as it may complicate a future upgrade. Instead, create a separate file to carry out the additional tasks identical to what was described in Section 7.1.2, “Replacing a Deployment Step”.

Name the new sls file descriptively. For example, if you need to create two rbd images in addition to the demo image, name the file archive.sls.

If you need to add a completely separate deployment step, create three new files—an sls file that performs the command, an orchestration file, and a custom file which aligns the new step with the original deployment steps.

For example, if you need to run logrotate on all minions as part of the preparation stage:

First create an sls file and include the logrotate command.

Because the orchestration file needs to run before all other preparation steps, add it to the Prep stage 0:

The last file is the custom one which includes the additional step with the original steps:

Note: Why global.yml?

The global.yml file is chosen over the cluster.yml because during the prep stage, no minion belongs to the Ceph cluster and has no access to any settings in cluster.yml.

During stage 0 (refer to DeepSea Stages Description for more information on DeepSea stages), the Salt master and Salt minions may optionally reboot because newly updated packages, for example kernel, require rebooting the system.

The default behavior is to install available new updates and not reboot the nodes even in case of kernel updates.

You can change the default update/reboot behavior of DeepSea stage 0 by adding/changing the stage_prep_master and stage_prep_minion options in the /srv/pillar/ceph/stack/global.yml file. stage_prep_master sets the behavior of the Salt master, and stage_prep_minion sets the behavior of all minions. All available parameters are:

For example, to prevent the cluster nodes from installing updates and rebooting, edit /srv/pillar/ceph/stack/global.yml and add the following lines:

stage_prep_master: default-no-update-no-reboot
stage_prep_minion: default-no-update-no-reboot

Tip: Values and Corresponding Files

The values of stage_prep_master correspond to file names located in /srv/salt/ceph/stage/0/master, while values of stage_prep_minion correspond to files in /srv/salt/ceph/stage/0/minion:

root@master # ls -l /srv/salt/ceph/stage/0/master
default-no-update-no-reboot.sls
default-no-update-reboot.sls
default-update-reboot.sls
[...]

root@master # ls -l /srv/salt/ceph/stage/0/minion
default-no-update-no-reboot.sls
default-no-update-reboot.sls
default-update-reboot.sls
[...]

Since IPv4 network addressing is prevalent, you need to enable IPv6 as a customization. DeepSea has no auto-discovery of IPv6 addressing.

To configure IPv6, set the public_network and cluster_network variables in the /srv/pillar/ceph/stack/global.yml file to valid IPv6 subnets. For example:

public_network: fd00:10::/64
cluster_network: fd00:11::/64

Then run DeepSea stage 2 and verify that the network information matches the setting. Stage 3 will generate the ceph.conf with the necessary flags.

Important: No Support for Dual Stack

Ceph does not support dual stack—running Ceph simultaneously on IPv4 and IPv6 is not possible. DeepSea validation will reject a mismatch between public_network and cluster_network or within either variable. The following example will fail the validation.

public_network: "192.168.10.0/24 fd00:10::/64"

Tip: Avoid Using fe80::/10 link-local Addresses

Avoid using fe80::/10 link-local addresses. All network interfaces have an assigned fe80 address and require an interface qualifier for proper routing. Either assign IPv6 addresses allocated to your site or consider using fd00::/8. These are part of ULA and not globally routable.

Several steps are required to configure an Object Gateway.

9.1.1.1 Basic Configuration #

Configuring a Ceph Object Gateway requires a running Ceph Storage Cluster. The Ceph Object Gateway is a client of the Ceph Storage Cluster. As a Ceph Storage Cluster client, it requires:

• A host name for the gateway instance, for example gateway.

• A storage cluster user name with appropriate permissions and a keyring.

• Pools to store its data.

• A data directory for the gateway instance.

• An instance entry in the Ceph configuration file.

Each instance must have a user name and key to communicate with a Ceph storage cluster. In the following steps, we use a monitor node to create a bootstrap keyring, then create the Object Gateway instance user keyring based on the bootstrap one. Then, we create a client user name and key. Next, we add the key to the Ceph Storage Cluster. Finally, we distribute the keyring to the node containing the gateway instance.

1. Create a keyring for the gateway:

   cephadm@adm > ceph-authtool --create-keyring /etc/ceph/ceph.client.rgw.keyring
   cephadm@adm > sudo chmod +r /etc/ceph/ceph.client.rgw.keyring

2. Generate a Ceph Object Gateway user name and key for each instance. As an example, we will use the name gateway after client.radosgw:

   cephadm@adm > ceph-authtool /etc/ceph/ceph.client.rgw.keyring \
     -n client.rgw.gateway --gen-key

3. Add capabilities to the key:

   cephadm@adm > ceph-authtool -n client.rgw.gateway --cap osd 'allow rwx' \
     --cap mon 'allow rwx' /etc/ceph/ceph.client.rgw.keyring

4. Once you have created a keyring and key to enable the Ceph Object Gateway with access to the Ceph Storage Cluster, add the key to your Ceph Storage Cluster. For example:

   cephadm@adm > ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.gateway \
     -i /etc/ceph/ceph.client.rgw.keyring

5. Distribute the keyring to the node with the gateway instance:

   cephadm@adm > scp /etc/ceph/ceph.client.rgw.keyring  ceph@HOST_NAME:/home/ceph
   cephadm@adm > ssh ceph@HOST_NAME
   cephadm@ogw > mv ceph.client.rgw.keyring /etc/ceph/ceph.client.rgw.keyring

Tip: Use Bootstrap Keyring

An alternative way is to create the Object Gateway bootstrap keyring, and then create the Object Gateway keyring from it:

1. Create an Object Gateway bootstrap keyring on one of the monitor nodes:

   cephadm@mon > ceph \
    auth get-or-create client.bootstrap-rgw mon 'allow profile bootstrap-rgw' \
    --connect-timeout=25 \
    --cluster=ceph \
    --name mon. \
    --keyring=/var/lib/ceph/mon/ceph-NODE_HOST/keyring \
    -o /var/lib/ceph/bootstrap-rgw/keyring

2. Create the /var/lib/ceph/radosgw/ceph-RGW_NAME directory for storing the bootstrap keyring:

   cephadm@mon > mkdir \
   /var/lib/ceph/radosgw/ceph-RGW_NAME

3. Create an Object Gateway keyring from the newly created bootstrap keyring:

   cephadm@mon > ceph \
    auth get-or-create client.rgw.RGW_NAME osd 'allow rwx' mon 'allow rw' \
    --connect-timeout=25 \
    --cluster=ceph \
    --name client.bootstrap-rgw \
    --keyring=/var/lib/ceph/bootstrap-rgw/keyring \
    -o /var/lib/ceph/radosgw/ceph-RGW_NAME/keyring

4. Copy the Object Gateway keyring to the Object Gateway host:

   cephadm@mon > scp \
   /var/lib/ceph/radosgw/ceph-RGW_NAME/keyring \
   RGW_HOST:/var/lib/ceph/radosgw/ceph-RGW_NAME/keyring

9.1.1.2 Create Pools (Optional) #

Ceph Object Gateways require Ceph Storage Cluster pools to store specific gateway data. If the user you created has proper permissions, the gateway will create the pools automatically. However, ensure that you have set an appropriate default number of placement groups per pool in the Ceph configuration file.

The pool names follow the ZONE_NAME.POOL_NAME syntax. When configuring a gateway with the default region and zone, the default zone name is 'default' as in our example:

.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.data

To create the pools manually, see Book “Administration Guide”, Chapter 22 “Managing Storage Pools”, Section 22.2.2 “Create a Pool”.

Important: Object Gateway and Erasure-Coded Pools

Only the default.rgw.buckets.data pool can be erasure coded. All other pools need to be replicated, otherwise the gateway is not accessible.

9.1.1.3 Adding Gateway Configuration to Ceph #

Add the Ceph Object Gateway configuration to the Ceph Configuration file. The Ceph Object Gateway configuration requires you to identify the Ceph Object Gateway instance. Then, specify the host name where you installed the Ceph Object Gateway daemon, a keyring (for use with cephx), and optionally a log file. For example:

[client.rgw.INSTANCE_NAME]
host = HOST_NAME
keyring = /etc/ceph/ceph.client.rgw.keyring

Tip: Object Gateway Log File

To override the default Object Gateway log file, include the following:

log file = /var/log/radosgw/client.rgw.INSTANCE_NAME.log

The [client.rgw.*] portion of the gateway instance identifies this portion of the Ceph configuration file as configuring a Ceph Storage Cluster client where the client type is a Ceph Object Gateway (radosgw). The instance name follows. For example:

[client.rgw.gateway]
host = ceph-gateway
keyring = /etc/ceph/ceph.client.rgw.keyring

Note

The HOST_NAME must be your machine host name, excluding the domain name.

Then turn off print continue. If you have it set to true, you may encounter problems with PUT operations:

rgw print continue = false

To use a Ceph Object Gateway with subdomain S3 calls (for example http://bucketname.hostname), you must add the Ceph Object Gateway DNS name under the [client.rgw.gateway] section of the Ceph configuration file:

[client.rgw.gateway]
...
rgw dns name = HOST_NAME

You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using the http://BUCKET_NAME.HOST_NAME syntax. The dnsmasq.conf file should include the following settings:

address=/HOST_NAME/HOST_IP_ADDRESS
listen-address=CLIENT_LOOPBACK_IP

Then, add the CLIENT_LOOPBACK_IP IP address as the first DNS server on the client machine(s).

root # mkdir -p /var/lib/ceph/radosgw/CLUSTER_ID

root # mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway

<ListAllMyBucketsResult>
      <Owner>
              <ID>anonymous</ID>
              <DisplayName/>
      </Owner>
      <Buckets/>
</ListAllMyBucketsResult>

The Linux kernel iSCSI target was originally named LIO for linux-iscsi.org, the project's original domain and Web site. For some time, no fewer than four competing iSCSI target implementations were available for the Linux platform, but LIO ultimately prevailed as the single iSCSI reference target. The mainline kernel code for LIO uses the simple, but somewhat ambiguous name "target", distinguishing between "target core" and a variety of front-end and back-end target modules.

The most commonly used front-end module is arguably iSCSI. However, LIO also supports Fibre Channel (FC), Fibre Channel over Ethernet (FCoE) and several other front-end protocols. At this time, only the iSCSI protocol is supported by SUSE Enterprise Storage.

The most frequently used target back-end module is one that is capable of simply re-exporting any available block device on the target host. This module is named iblock. However, LIO also has an RBD-specific back-end module supporting parallelized multipath I/O access to RBD images.

This section introduces brief information on iSCSI initiators used on Linux, Microsoft Windows, and VMware platforms.

You can deploy the iSCSI Gateway either during the Ceph cluster deployment process, or add it to an existing cluster using DeepSea.

To include the iSCSI Gateway during the cluster deployment process, refer to Section 5.5.1.2, “Role Assignment”.

To add the iSCSI Gateway to an existing cluster, refer to Book “Administration Guide”, Chapter 2 “Salt Cluster Administration”, Section 2.2 “Adding New Roles to Nodes”.

RBD images are created in the Ceph store and subsequently exported to iSCSI. We recommend that you use a dedicated RADOS pool for this purpose. You can create a volume from any host that is able to connect to your storage cluster using the Ceph rbd command line utility. This requires the client to have at least a minimal ceph.conf configuration file, and appropriate CephX authentication credentials.

To create a new volume for subsequent export via iSCSI, use the rbd create command, specifying the volume size in megabytes. For example, in order to create a 100 GB volume named 'testvol' in the pool named 'iscsi-images', run:

cephadm@adm > rbd --pool iscsi-images create --size=102400 'testvol'

To export RBD images via iSCSI, you can use either Ceph Dashboard Web interface or the ceph-iscsi gwcli utility. In this section we will focus on gwcli only, demonstrating how to create an iSCSI target that exports an RBD image using the command line.

Note

Only the following RBD image features are supported: layering, striping (v2), exclusive-lock, fast-diff, and data-pool. RBD images with any other feature enabled cannot be exported.

As root, start the iSCSI gateway command line interface:

root # gwcli

Go to iscsi-targets and create a target with the name iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol:

gwcli >  /> cd /iscsi-targets
gwcli >  /iscsi-targets> create iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol

Create the iSCSI gateways by specifying the gateway name and ip address:

gwcli >  /iscsi-targets> cd iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/gateways
gwcli >  /iscsi-target...tvol/gateways> create iscsi1 192.168.124.104
gwcli >  /iscsi-target...tvol/gateways> create iscsi2 192.168.124.105

Tip

Use the help command to show the list of available commands in the current configuration node.

Add the RBD image with the name 'testvol' in the pool 'iscsi-images':

gwcli >  /iscsi-target...tvol/gateways> cd /disks
gwcli >  /disks> attach iscsi-images/testvol

Map the RBD image to the target:

gwcli >  /disks> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/disks
gwcli >  /iscsi-target...testvol/disks> add iscsi-images/testvol

Note

You can use lower level tools, such as targetcli, to query the local configuration, but not to modify it.

Tip

You can use the ls command to review the configuration. Some configuration nodes also support the info command, which can be used to display more detailed information.

Note that, by default, ACL authentication is enabled so this target is not accessible yet. Check Section 10.4.4, “Authentication and Access Control” for more information about authentication and access control.

iSCSI authentication is flexible and covers many authentication possibilities.

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts
gwcli >  /iscsi-target...testvol/hosts> auth disable_acl

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts
gwcli >  /iscsi-target...testvol/hosts> create iqn.1996-04.de.suse:01:e6ca28cc9f20

gwcli >  /iscsi-target...:e6ca28cc9f20> disk add rbd/testvol

10.4.4.3 CHAP Authentication #

In addition to the ACL, you can enable the CHAP authentication by specifying a user name and password for each initiator:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts/iqn.1996-04.de.suse:01:e6ca28cc9f20
gwcli >  /iscsi-target...:e6ca28cc9f20> auth username=common12 password=pass12345678

Note

User names must have a length of 8 to 64 characters and can contain alphanumeric characters, '.', '@', '-', '_' or ':'.

Passwords must have a length of 12 to 16 characters and can contain alphanumeric characters, '@', '-', '_' or '/'.

Optionally, you can also enable the CHAP mutual authentication by specifying the mutual_username and mutual_password parameters in the auth command.

10.4.4.4 Discovery and Mutual Authentication #

Discovery authentication is independent of the previous authentication methods. It requires credentials for browsing, it is optional, and can be configured by:

gwcli >  /> cd /iscsi-targets
gwcli >  /iscsi-targets> discovery_auth username=du123456 password=dp1234567890

Note

User-names must have a length of 8 to 64 characters and can only contain letters, '.', '@', '-', '_' or ':'.

Passwords must have a length of 12 to 16 characters and can only contain letters, '@', '-', '_' or '/'.

Optionally, you can also specify the mutual_username and mutual_password parameters in the discovery_auth command.

Discovery authentication can be disabled by using the following command:

gwcli >  /iscsi-targets> discovery_auth nochap

ceph-iscsi can be configured with advanced parameters which are subsequently passed on to the LIO I/O target. The parameters are divided up into 'target' and 'disk' parameters.

Warning

Unless otherwise noted, changing these parameters from the default setting is not recommended.

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol
gwcli >  /iscsi-target...i.SYSTEM-ARCH:testvol> info

gwcli >  /iscsi-target...i.SYSTEM-ARCH:testvol> reconfigure login_timeout 20

gwcli >  /> cd /disks/rbd/testvol
gwcli >  /disks/rbd/testvol> info

gwcli >  /disks/rbd/testvol> reconfigure rbd/testvol emulate_pr 0

You can deploy MDS either during the initial cluster deployment process as described in Section 5.3, “Cluster Deployment”, or add it to an already deployed cluster as described in Book “Administration Guide”, Chapter 2 “Salt Cluster Administration”, Section 2.1 “Adding New Cluster Nodes”.

After you deploy your MDS, allow the Ceph OSD/MDS service in the firewall setting of the server where MDS is deployed: Start yast, navigate to Security and Users / Firewall / Allowed Services and in the Service to Allow drop–down menu select Ceph OSD/MDS. If the Ceph MDS node is not allowed full traffic, mounting of a file system fails, even though other operations may work properly.

You can remove a metadata server in your cluster as described in .

You can fine-tune the MDS behavior by inserting relevant options in the ceph.conf configuration file.

A CephFS requires at least two RADOS pools: one for data and one for metadata. When configuring these pools, you might consider:

When assigning a role-mds in the policy.cfg, the required pools are automatically created. You can manually create the pools cephfs_data and cephfs_metadata for manual performance tuning before setting up the Metadata Server. DeepSea will not create these pools if they already exist.

For more information on managing pools, see Book “Administration Guide”, Chapter 22 “Managing Storage Pools”.

To create the two required pools—for example, 'cephfs_data' and 'cephfs_metadata'—with default settings for use with CephFS, run the following commands:

cephadm@adm > ceph osd pool create cephfs_data pg_num
cephadm@adm > ceph osd pool create cephfs_metadata pg_num

It is possible to use EC pools instead of replicated pools. We recommend to only use EC pools for low performance requirements and infrequent random access, for example cold storage, backups, archiving. CephFS on EC pools requires BlueStore to be enabled and the pool must have the allow_ec_overwrite option set. This option can be set by running ceph osd pool set ec_pool allow_ec_overwrites true.

Erasure coding adds significant overhead to file system operations, especially small updates. This overhead is inherent to using erasure coding as a fault tolerance mechanism. This penalty is the trade off for significantly reduced storage space overhead.

When the pools are created, you may enable the file system with the ceph fs new command:

cephadm@adm > ceph fs new fs_name metadata_pool_name data_pool_name

For example:

cephadm@adm > ceph fs new cephfs cephfs_metadata cephfs_data

You can check that the file system was created by listing all available CephFSs:

cephadm@adm > ceph fs ls
 name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

When the file system has been created, your MDS will be able to enter an active state. For example, in a single MDS system:

cephadm@adm > ceph mds stat
e5: 1/1/1 up

Tip: More Topics

You can find more information of specific tasks—for example mounting, unmounting, and advanced CephFS setup—in Book “Administration Guide”, Chapter 28 “Clustered File System”.

A CephFS instance can be served by multiple active MDS daemons. All active MDS daemons that are assigned to a CephFS instance will distribute the file system's directory tree between themselves, and thus spread the load of concurrent clients. In order to add an active MDS daemon to a CephFS instance, a spare standby is needed. Either start an additional daemon or use an existing standby instance.

The following command will display the current number of active and passive MDS daemons.

cephadm@adm > ceph mds stat

The following command sets the number of active MDSs to two in a file system instance.

cephadm@adm > ceph fs set fs_name max_mds 2

In order to shrink the MDS cluster prior to an update, set the max_mds option so that only one instance remains:

cephadm@adm > ceph fs set fs_name max_mds 1

We recommend at least one MDS is left as a standby daemon.

During Ceph updates, the feature flags on a file system instance may change (usually by adding new features). Incompatible daemons (such as the older versions) are not able to function with an incompatible feature set and will refuse to start. This means that updating and restarting one daemon can cause all other not yet updated daemons to stop and refuse to start. For this reason, we recommend shrinking the active MDS cluster to size one and stopping all standby daemons before updating Ceph. The manual steps for this update procedure are as follows:

If you use DeepSea, it will follow this procedure in case the ceph package was updated during stages 0 and 4. It is possible to perform this procedure while clients have the CephFS instance mounted and I/O is ongoing. Note however that there will be a very brief I/O pause while the active MDS restarts. Clients will recover automatically.

It is good practice to reduce the I/O load as much as possible before updating an MDS cluster. An idle MDS cluster will go through this update procedure quicker. Conversely, on a heavily loaded cluster with multiple MDS daemons it is essential to reduce the load in advance to prevent a single MDS daemon from being overwhelmed by ongoing I/O.

The layout of a file controls how its contents are mapped to Ceph RADOS objects. You can read and write a file’s layout using virtual extended attributes or xattrs for shortly.

The name of the layout xattrs depends on whether a file is a regular file or a directory. Regular files’ layout xattrs are called ceph.file.layout, while directories’ layout xattrs are called ceph.dir.layout. Where examples refer to ceph.file.layout, substitute the .dir. part as appropriate when dealing with directories.

11.3.4.2 Reading Layout with getfattr #

Use the getfattr command to read the layout information of an example file file as a single string:

root # touch file
root # getfattr -n ceph.file.layout file
# file: file
ceph.file.layout="stripe_unit=4194304 stripe_count=1 object_size=419430

Read individual layout fields:

root # getfattr -n ceph.file.layout.pool file
# file: file
ceph.file.layout.pool="cephfs_data"
root # getfattr -n ceph.file.layout.stripe_unit file
# file: file
ceph.file.layout.stripe_unit="4194304"

Tip: Pool ID or Name

When reading layouts, the pool will usually be indicated by name. However, in rare cases when pools have only just been created, the ID may be output instead.

Directories do not have an explicit layout until it is customized. Attempts to read the layout will fail if it has never been modified: this indicates that the layout of the next ancestor directory with an explicit layout will be used.

root # mkdir dir
root # getfattr -n ceph.dir.layout dir
dir: ceph.dir.layout: No such attribute
root # setfattr -n ceph.dir.layout.stripe_count -v 2 dir
root # getfattr -n ceph.dir.layout dir
# file: dir
ceph.dir.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 pool=cephfs_data"

11.3.4.3 Writing Layouts with setfattr #

Use the setfattr command to modify the layout fields of an example file file:

cephadm@adm > ceph osd lspools
0 rbd
1 cephfs_data
2 cephfs_metadata
root # setfattr -n ceph.file.layout.stripe_unit -v 1048576 file
root # setfattr -n ceph.file.layout.stripe_count -v 8 file
# Setting pool by ID:
root # setfattr -n ceph.file.layout.pool -v 1 file
# Setting pool by name:
root # setfattr -n ceph.file.layout.pool -v cephfs_data file

Note: Empty File

When the layout fields of a file are modified using setfattr, this file needs to be empty otherwise an error will occur.

root # setfattr -x ceph.dir.layout mydir

# Create a directory and set a namespace on it
root # mkdir mydir
root # setfattr -n ceph.dir.layout.pool_namespace -v foons mydir
root # getfattr -n ceph.dir.layout mydir
ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 \
 pool=cephfs_data_a pool_namespace=foons"

# Clear the namespace from the directory's layout
root # setfattr -x ceph.dir.layout.pool_namespace mydir
root # getfattr -n ceph.dir.layout mydir
ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 \
 pool=cephfs_data_a"

root # getfattr -n ceph.dir.layout dir
# file: dir
ceph.dir.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 \
 pool=cephfs_data"

# file1 inherits its parent's layout
root # touch dir/file1
root # getfattr -n ceph.file.layout dir/file1
# file: dir/file1
ceph.file.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 \
 pool=cephfs_data"

# update the layout of the directory before creating a second file
root # setfattr -n ceph.dir.layout.stripe_count -v 4 dir
root # touch dir/file2

# file1's layout is unchanged
root # getfattr -n ceph.file.layout dir/file1
# file: dir/file1
ceph.file.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 \
 pool=cephfs_data"

# ...while file2 has the parent directory's new layout
root # getfattr -n ceph.file.layout dir/file2
# file: dir/file2
ceph.file.layout="stripe_unit=4194304 stripe_count=4 object_size=4194304 \
 pool=cephfs_data"

root # getfattr -n ceph.dir.layout dir
# file: dir
ceph.dir.layout="stripe_unit=4194304 stripe_count=4 object_size=4194304 \
 pool=cephfs_data"
root # mkdir dir/childdir
root # getfattr -n ceph.dir.layout dir/childdir
dir/childdir: ceph.dir.layout: No such attribute
root # touch dir/childdir/grandchild
root # getfattr -n ceph.file.layout dir/childdir/grandchild
# file: dir/childdir/grandchild
ceph.file.layout="stripe_unit=4194304 stripe_count=4 object_size=4194304 \
 pool=cephfs_data"

11.3.4.6 Adding a Data Pool to the Metadata Server #

Before you can use a pool with CephFS, you need to add it to the Metadata Server:

cephadm@adm > ceph fs add_data_pool cephfs cephfs_data_ssd
cephadm@adm > ceph fs ls  # Pool should now show up
.... data pools: [cephfs_data cephfs_data_ssd ]

Tip: cephx Keys

Make sure that your cephx keys allow the client to access this new pool.

You can then update the layout on a directory in CephFS to use the pool you added:

root # mkdir /mnt/cephfs/myssddir
root # setfattr -n ceph.dir.layout.pool -v cephfs_data_ssd /mnt/cephfs/myssddir

All new files created within that directory will now inherit its layout and place their data in your newly added pool. You may notice that the number of objects in your primary data pool continues to increase, even if files are being created in the pool you newly added. This is normal: the file data is stored in the pool specified by the layout, but a small amount of metadata is kept in the primary data pool for all files.

To successfully deploy NFS Ganesha, you need to add a role-ganesha to your /srv/pillar/ceph/proposals/policy.cfg. For details, see Section 5.5.1, “The policy.cfg File”. NFS Ganesha also needs either a role-rgw or a role-mds present in the policy.cfg.

Although it is possible to install and run the NFS Ganesha server on an already existing Ceph node, we recommend running it on a dedicated host with access to the Ceph cluster. The client hosts are typically not part of the cluster, but they need to have network access to the NFS Ganesha server.

To enable the NFS Ganesha server at any point after the initial installation, add the role-ganesha to the policy.cfg and re-run at least DeepSea stages 2 and 4. For details, see Section 5.3, “Cluster Deployment”.

NFS Ganesha is configured via the file /etc/ganesha/ganesha.conf that exists on the NFS Ganesha node. However, this file is overwritten each time DeepSea stage 4 is executed. Therefore we recommend to edit the template used by Salt, which is the file /srv/salt/ceph/ganesha/files/ganesha.conf.j2 on the Salt master. For details about the configuration file, see Book “Administration Guide”, Chapter 30 “NFS Ganesha: Export Ceph Data via NFS”, Section 30.2 “Configuration”.

The following requirements need to be met before DeepSea stages 2 and 4 can be executed to install NFS Ganesha:

For our example configuration, you need the following:

After you deploy the NFS Ganesha nodes, verify that the cluster is operational and the default CephFS pools are there:

cephadm@adm > rados lspools
cephfs_data
cephfs_metadata

Check that both NFS Ganesha nodes have the file /etc/ganesha/ganesha.conf installed. Add the following blocks, if they do not exist yet, to the configuration file in order to enable RADOS as the recovery backend of NFS Ganesha.

NFS_CORE_PARAM
{
    Enable_NLM = false;
    Enable_RQUOTA = false;
    Protocols = 4;
}
NFSv4
{
    RecoveryBackend = rados_cluster;
    Minor_Versions = 1,2;
}
CACHEINODE {
    Dir_Chunk = 0;
    NParts = 1;
    Cache_Size = 1;
}
RADOS_KV
{
    pool = "rados_pool";
    namespace = "pool_namespace";
    nodeid = "fqdn"
    UserId = "cephx_user_id";
    Ceph_Conf = "path_to_ceph.conf"
}

You can find out the values for rados_pool and pool_namespace by checking the already existing line in the configuration of the form:

%url rados://rados_pool/pool_namespace/...

The value for nodeid option corresponds to the FQDN of the machine, and UserId and Ceph_Conf options value can be found in the already existing RADOS_URLS block.

Because legacy versions of NFS prevent us from lifting the grace period early and therefore prolong a server restart, we disable options for NFS prior to version 4.2. We also disable most of the NFS Ganesha caching as Ceph libraries do aggressive caching already.

The 'rados_cluster' recovery back-end stores its info in RADOS objects. Although it is not a lot of data, we want it highly available. We use the CephFS metadata pool for this purpose, and declare a new 'ganesha' namespace in it to keep it distinct from CephFS objects.

Note: Cluster Node IDs

Most of the configuration is identical between the two hosts, however the nodeid option in the 'RADOS_KV' block needs to be a unique string for each node. By default, NFS Ganesha sets nodeid to the host name of the node.

If you need to use different fixed values other than host names, you can for example set nodeid = 'a' on one node and nodeid = 'b' on the other one.

We need to verify that all of the nodes in the cluster know about each other. This done via a RADOS object that is shared between the hosts. NFS Ganesha uses this object to communicate the current state with regard to a grace period.

The nfs-ganesha-rados-grace package contains a command line tool for querying and manipulating this database. If the package is not installed on at least one of the nodes, install it with

root # zypper install nfs-ganesha-rados-grace

We will use the command to create the DB and add both nodeids. In our example, the two NFS Ganesha nodes are named ses6min1.example.com and ses6min2.example.com On one of the NFS Ganesha hosts, run

cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha add ses6min1.example.com
cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha add ses6min2.example.com
cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha
cur=1 rec=0
======================================================
ses6min1.example.com     E
ses6min2.example.com     E

This creates the grace database and adds both 'ses6min1.example.com' and 'ses6min2.example.com' to it. The last command dumps the current state. Newly added hosts are always considered to be enforcing the grace period so they both have the 'E' flag set. The 'cur' and 'rec' values show the current and recovery epochs, which is how we keep track of what hosts are allowed to perform recovery and when.

On both NFS Ganesha nodes, restart the related services:

root # systemctl restart nfs-ganesha.service

After the services are restarted, check the grace database:

cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha
cur=3 rec=0
======================================================
ses6min1.example.com
ses6min2.example.com

Note: Cleared the 'E' Flag

Note that both nodes have cleared their 'E' flags, indicating that they are no longer enforcing the grace period and are now in normal operation mode.

After you complete all the preceding steps, you can mount the exported NFS from either of the two NFS Ganesha servers, and perform normal NFS operations against them.

Our example configuration assumes that if one of the two NFS Ganesha servers goes down, you will restart it manually within 5 minutes. After 5 minutes, the Metadata Server may cancel the session that the NFS Ganesha client held and all of the state associated with it. If the session’s capabilities get cancelled before the rest of the cluster goes into the grace period, the server’s clients may not be able to recover all of their state.

Install the Rook-Ceph common components, CSI roles, and the Rook-Ceph operator by executing the following command on the SUSE CaaS Platform master node:

root # kubectl apply -f common.yaml -f operator.yaml

common.yaml will create the 'rook-ceph' namespace, Ceph Custom Resource Definitions (CRDs) (see https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to make Kubernetes aware of Ceph Objects (for example, 'CephCluster'), and the RBAC roles and Pod Security Policies (see https://kubernetes.io/docs/concepts/policy/pod-security-policy/) which are necessary for allowing Rook to manage the cluster-specific resources.

Tip: hostNetwork and hostPorts Usage

Allowing the usage of hostNetwork is required when using hostNetwork: true in the Cluster Resource Definition. Allowing the usage of hostPorts in the PodSecurityPolicy is also required.

Verify the installation by running kubectl get pods -n rook-ceph on SUSE CaaS Platform master node, for example:

root # kubectl get pods -n rook-ceph
NAME                                     READY   STATUS      RESTARTS   AGE
rook-ceph-agent-57c9j                    1/1     Running     0          22h
rook-ceph-agent-b9j4x                    1/1     Running     0          22h
rook-ceph-operator-cf6fb96-lhbj7         1/1     Running     0          22h
rook-discover-mb8gv                      1/1     Running     0          22h
rook-discover-tztz4                      1/1     Running     0          22h

After you modify cluster.yaml according to your needs, you can create the Ceph cluster. Run the following command on the SUSE CaaS Platform master node:

root # kubectl apply -f cluster.yaml

Watch the 'rook-ceph' namespace to see the Ceph cluster being created. You will see as many Ceph Monitors as configured in the cluster.yaml manifest (default is 3), one Ceph Manager, and as many Ceph OSDs as you have free disks.

Tip: Temporary OSD Pods

While bootstrapping the Ceph cluster, you will see some pods with the name rook-ceph-osd-prepare-NODE-NAME run for a while and then terminate with the status 'Completed'. As their name implies, these pods provision Ceph OSDs. They are left without being deleted so that you can inspect their logs after their termination. For example:

root # kubectl get pods --namespace rook-ceph
NAME                                         READY  STATUS     RESTARTS  AGE
rook-ceph-agent-57c9j                        1/1    Running    0         22h
rook-ceph-agent-b9j4x                        1/1    Running    0         22h
rook-ceph-mgr-a-6d48564b84-k7dft             1/1    Running    0         22h
rook-ceph-mon-a-cc44b479-5qvdb               1/1    Running    0         22h
rook-ceph-mon-b-6c6565ff48-gm9wz             1/1    Running    0         22h
rook-ceph-operator-cf6fb96-lhbj7             1/1    Running    0         22h
rook-ceph-osd-0-57bf997cbd-4wspg             1/1    Running    0         22h
rook-ceph-osd-1-54cf468bf8-z8jhp             1/1    Running    0         22h
rook-ceph-osd-prepare-caasp4-worker-0-f2tmw  0/2    Completed  0         9m35s
rook-ceph-osd-prepare-caasp4-worker-1-qsfhz  0/2    Completed  0         9m33s
rook-ceph-tools-76c7d559b6-64rkw             1/1    Running    0         22h
rook-discover-mb8gv                          1/1    Running    0         22h
rook-discover-tztz4                          1/1    Running    0         22h