The basic component of Ceph is called RADOS (Reliable Autonomic Distributed Object Store). It is responsible for managing the data stored in the cluster. Data in Ceph is usually stored as objects. Each object consists of an identifier and the data.

RADOS provides the following access methods to the stored objects that cover many use cases:

librados is used by Object Gateway and RBD while CephFS directly interfaces with RADOS Figure 1.1, “Interfaces to the Ceph Object Store”.

Figure 1.1: Interfaces to the Ceph Object Store #

At the core of a Ceph cluster is the CRUSH algorithm. CRUSH is the acronym for Controlled Replication Under Scalable Hashing. CRUSH is a function that handles the storage allocation and needs comparably few parameters. That means only a small amount of information is necessary to calculate the storage position of an object. The parameters are a current map of the cluster including the health state, some administrator-defined placement rules and the name of the object that needs to be stored or retrieved. With this information, all nodes in the Ceph cluster are able to calculate where an object and its replicas are stored. This makes writing or reading data very efficient. CRUSH tries to evenly distribute data over all nodes in the cluster.

The CRUSH map contains all storage nodes and administrator-defined placement rules for storing objects in the cluster. It defines a hierarchical structure that usually corresponds to the physical structure of the cluster. For example, the data-containing disks are in hosts, hosts are in racks, racks in rows and rows in data centers. This structure can be used to define failure domains. Ceph then ensures that replications are stored on different branches of a specific failure domain.

If the failure domain is set to rack, replications of objects are distributed over different racks. This can mitigate outages caused by a failed switch in a rack. If one power distribution unit supplies a row of racks, the failure domain can be set to row. When the power distribution unit fails, the replicated data is still available on other rows.

In Ceph, nodes are servers working for the cluster. They can run several different types of daemons. We recommend to run only one type of daemon on each node, except for Ceph Manager daemons which can be collocated with Ceph Monitors. Each cluster requires at least Ceph Monitor, Ceph Manager, and Ceph OSD daemons:

To use CephFS, Object Gateway, NFS Ganesha, or iSCSI Gateway, additional nodes are required:

Objects that are stored in a Ceph cluster are put into pools. Pools represent logical partitions of the cluster to the outside world. For each pool a set of rules can be defined, for example, how many replications of each object must exist. The standard configuration of pools is called replicated pool.

Pools usually contain objects but can also be configured to act similar to a RAID 5. In this configuration, objects are stored in chunks along with additional coding chunks. The coding chunks contain the redundant information. The number of data and coding chunks can be defined by the administrator. In this configuration, pools are referred to as erasure coded pools.

Placement Groups (PGs) are used for the distribution of data within a pool. When creating a pool, a certain number of placement groups is set. The placement groups are used internally to group objects and are an important factor for the performance of a Ceph cluster. The PG for an object is determined by the object's name.

This section provides a simplified example of how Ceph manages data (see Figure 1.2, “Small Scale Ceph Example”). This example does not represent a recommended configuration for a Ceph cluster. The hardware setup consists of three storage nodes or Ceph OSDs (Host 1, Host 2, Host 3). Each node has three hard disks which are used as OSDs (osd.1 to osd.9). The Ceph Monitor nodes are neglected in this example.

Note: Difference between Ceph OSD and OSD

While Ceph OSD or Ceph OSD daemon refers to a daemon that is run on a node, the word OSD refers to the logical disk that the daemon interacts with.

The cluster has two pools, Pool A and Pool B. While Pool A replicates objects only two times, resilience for Pool B is more important and it has three replications for each object.

When an application puts an object into a pool, for example via the REST API, a Placement Group (PG1 to PG4) is selected based on the pool and the object name. The CRUSH algorithm then calculates on which OSDs the object is stored, based on the Placement Group that contains the object.

In this example the failure domain is set to host. This ensures that replications of objects are stored on different hosts. Depending on the replication level set for a pool, the object is stored on two or three OSDs that are used by the Placement Group.

An application that writes an object only interacts with one Ceph OSD, the primary Ceph OSD. The primary Ceph OSD takes care of replication and confirms the completion of the write process after all other OSDs have stored the object.

If osd.5 fails, all object in PG1 are still available on osd.1. As soon as the cluster recognizes that an OSD has failed, another OSD takes over. In this example osd.4 is used as a replacement for osd.5. The objects stored on osd.1 are then replicated to osd.4 to restore the replication level.

Figure 1.2: Small Scale Ceph Example #

If a new node with new OSDs is added to the cluster, the cluster map is going to change. The CRUSH function then returns different locations for objects. Objects that receive new locations will be relocated. This process results in a balanced usage of all OSDs.

For the Ceph network environment, we recommend two bonded 25 GbE (or faster) network interfaces bonded using 802.3ad (LACP). The use of two network interfaces provides aggregation and fault-tolerance. The bond should then be used to provide two VLAN interfaces, one for the public network, and the second for the cluster network. Details on bonding the interfaces can be found in https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-network.html#sec-network-iface-bonding.

Fault tolerance can be enhanced through isolating the components into failure domains. To improve fault tolerance of the network, bonding one interface from two separate Network Interface Cards (NIC) offers protection against failure of a single NIC. Similarly, creating a bond across two switches protects against failure of a switch. We recommend consulting with the network equipment vendor in order to architect the level of fault tolerance required.

Important: Administration Network not Supported

Additional administration network setup—that enables for example separating SSH, Salt, or DNS networking—is neither tested nor supported.

Tip: Nodes Configured via DHCP

If your storage nodes are configured via DHCP, the default timeouts may not be sufficient for the network to be configured correctly before the various Ceph daemons start. If this happens, the Ceph MONs and OSDs will not start correctly (running systemctl status ceph\* will result in "unable to bind" errors). To avoid this issue, we recommend increasing the DHCP client timeout to at least 30 seconds on each node in your storage cluster. This can be done by changing the following settings on each node:

In /etc/sysconfig/network/dhcp, set

DHCLIENT_WAIT_AT_BOOT="30"

In /etc/sysconfig/network/config, set

WAIT_FOR_INTERFACES="60"

[global]
 [...]
 mon host = 192.168.123.12, 1.2.3.4, 242.12.33.12
 mon initial members = MON1, MON2, MON3
 [...]

[mon.MON1]
 public network = 192.168.123.0/24

 [mon.MON2]
 public network = 1.2.3.0/24

 [mon.MON3]
 public network = 242.12.33.12/0

A minimal product cluster configuration consists of:

A detailed configuration is:

Figure 2.2: Minimum Cluster Configuration #

Once you grow your cluster, we recommend to relocate monitors, Metadata Server, and gateways on separate nodes to ensure better fault tolerance.

There are two types of disk space needed to run on OSD: the space for the disk journal (for FileStore) or WAL/DB device (for BlueStore), and the primary space for the stored data. The minimum (and default) value for the journal/WAL/DB is 6 GB. The minimum space for data is 5 GB, as partitions smaller than 5 GB are automatically assigned the weight of 0.

So although the minimum disk space for an OSD is 11 GB, we do not recommend a disk smaller than 20 GB, even for testing purposes.

Solid-state drives (SSD) have no moving parts. This reduces random access time and read latency while accelerating data throughput. Because their price per 1MB is significantly higher than the price of spinning hard disks, SSDs are only suitable for smaller storage.

OSDs may see a significant performance improvement by storing their journal on an SSD and the object data on a separate hard disk.

Tip: Sharing an SSD for Multiple Journals

As journal data occupies relatively little space, you can mount several journal directories to a single SSD disk. Keep in mind that with each shared journal, the performance of the SSD disk degrades. We do not recommend sharing more than six journals on the same SSD disk and 12 on NVMe disks.

You can have as many disks in one server as it allows. There are a few things to consider when planning the number of disks per server:

SUSE Manager and SUSE Enterprise Storage are not integrated, therefore SUSE Manager cannot currently manage a SUSE Enterprise Storage cluster.

Salt has several standard locations and several naming conventions used on your master node:

DeepSea commands are executed via the Salt infrastructure. When using the salt command, you need to specify a set of Salt minions that the command will affect. We describe the set of the minions as a target for the salt command. The following sections describe possible methods to target the minions.

5.2.2.1 Matching the Minion Name #Edit source

You can target a minion or a group of minions by matching their names. A minion's name is usually the short host name of the node where the minion runs. This is a general Salt targeting method, not related to DeepSea. You can use globbing, regular expressions, or lists to limit the range of minion names. The general syntax follows:

root@master # salt target example.module

Tip: Ceph-only Cluster

If all Salt minions in your environment belong to your Ceph cluster, you can safely substitute target with '*' to include all registered minions.

Match all minions in the example.net domain (assuming the minion names are identical to their "full" host names):

root@master # salt '*.example.net' test.ping

Match the 'web1' to 'web5' minions:

root@master # salt 'web[1-5]' test.ping

Match both 'web1-prod' and 'web1-devel' minions using a regular expression:

root@master # salt -E 'web1-(prod|devel)' test.ping

Match a simple list of minions:

root@master # salt -L 'web1,web2,web3' test.ping

Match all minions in the cluster:

root@master # salt '*' test.ping

root@master # salt target grains.append deepsea default

root@master # salt target grains.delval deepsea destructive=True

root@master # salt -G 'deepsea:*' test.ping

root@master # salt -C 'G@deepsea:*' test.ping

5.2.2.3 Set the deepsea_minions Option #Edit source

Setting the deepsea_minions option's target is a requirement for DeepSea deployments. DeepSea uses it to instruct minions during the execution of stages (refer to DeepSea Stages Description for details.

To set or change the deepsea_minions option, edit the /srv/pillar/ceph/deepsea_minions.sls file on the Salt master and add or replace the following line:

deepsea_minions: target

Tip: deepsea_minions Target

As the target for the deepsea_minions option, you can use any targeting method: both Matching the Minion Name and Targeting with a DeepSea Grain.

Match all Salt minions in the cluster:

deepsea_minions: '*'

Match all minions with the 'deepsea' grain:

deepsea_minions: 'G@deepsea:*'

The progress monitor provides a detailed, real-time visualization of what is happening during execution of stages using salt-run state.orch commands in other terminal sessions.

Tip: Start Monitor in a New Terminal Session

You need to start the monitor in a new terminal window before running any salt-run state.orch so that the monitor can detect the start of the stage's execution.

If you start the monitor after issuing the salt-run state.orch command, then no execution progress will be shown.

You can start the monitor mode by running the following command:

root@master # deepsea monitor

For more information about the available command line options of the deepsea monitor command, check its manual page:

root@master # man deepsea-monitor

In the stand-alone mode, DeepSea CLI can be used to run a DeepSea stage, showing its execution in real-time.

The command to run a DeepSea stage from the DeepSea CLI has the following form:

root@master # deepsea stage run stage-name

where stage-name corresponds to the way Salt orchestration state files are referenced. For example, stage deploy, which corresponds to the directory located in /srv/salt/ceph/stage/deploy, is referenced as ceph.stage.deploy.

This command is an alternative to the Salt-based commands for running DeepSea stages (or any DeepSea orchestration state file).

The command deepsea stage run ceph.stage.0 is equivalent to salt-run state.orch ceph.stage.0.

For more information about the available command line options accepted by the deepsea stage run command, check its manual page:

root@master # man deepsea-stage run

In the following figure shows an example of the output of the DeepSea CLI when running Stage 2:

Figure 5.1: DeepSea CLI stage execution progress output #

root@master # deepsea salt-run state.orch stage-name

The /srv/pillar/ceph/proposals/policy.cfg configuration file is used to determine roles of individual cluster nodes. For example, which nodes act as Ceph OSDs or Ceph Monitors. Edit policy.cfg in order to reflect your desired cluster setup. The order of the sections is arbitrary, but the content of included lines overwrites matching keys from the content of previous lines.

Tip: Examples of policy.cfg

You can find several examples of complete policy files in the /usr/share/doc/packages/deepsea/examples/ directory.

cluster-ceph/cluster/*.sls

cluster-ceph/cluster/abc.domain.sls

cluster-ceph/cluster/mon*.sls

cluster-unassigned/cluster/client*.sls

5.5.1.2 Role Assignment #Edit source

This section provides you with details on assigning 'roles' to your cluster nodes. A 'role' in this context means the service you need to run on the node, such as Ceph Monitor, Object Gateway, or iSCSI Gateway. No role is assigned automatically, only roles added to policy.cfg will be deployed.

The assignment follows this pattern:

role-ROLE_NAME/PATH/FILES_TO_INCLUDE

Where the items have the following meaning and values:

• ROLE_NAME is any of the following: 'master', 'admin', 'mon', 'mgr', 'storage', 'mds', 'igw', 'rgw', 'ganesha', 'grafana', or 'prometheus'.

• PATH is a relative directory path to .sls or .yml files. In case of .sls files, it usually is cluster, while .yml files are located at stack/default/ceph/minions.

• FILES_TO_INCLUDE are the Salt state files or YAML configuration files. They normally consist of Salt minions' host names, for example ses5min2.yml. Shell globbing can be used for more specific matching.

An example for each role follows:

• master - the node has admin keyrings to all Ceph clusters. Currently, only a single Ceph cluster is supported. As the master role is mandatory, always add a similar line to the following:

  role-master/cluster/master*.sls

• admin - the minion will have an admin keyring. You define the role as follows:

  role-admin/cluster/abc*.sls

• mon - the minion will provide the monitor service to the Ceph cluster. This role requires addresses of the assigned minions. From SUSE Enterprise Storage 5, the public addresses are calculated dynamically and are no longer needed in the Salt pillar.

  role-mon/cluster/mon*.sls

  The example assigns the monitor role to a group of minions.

• mgr - the Ceph manager daemon which collects all the state information from the whole cluster. Deploy it on all minions where you plan to deploy the Ceph monitor role.

  role-mgr/cluster/mgr*.sls

• storage - use this role to specify storage nodes.

  role-storage/cluster/data*.sls

• mds - the minion will provide the metadata service to support CephFS.

  role-mds/cluster/mds*.sls

• igw - the minion will act as an iSCSI Gateway. This role requires addresses of the assigned minions, thus you need to also include the files from the stack directory:

  role-igw/cluster/*.sls

• rgw - the minion will act as an Object Gateway:

  role-rgw/cluster/rgw*.sls

• ganesha - the minion will act as an NFS Ganesha server. The 'ganesha' role requires either an 'rgw' or 'mds' role in cluster, otherwise the validation will fail in Stage 3.

  role-ganesha/cluster/ganesha*.sls

  To successfully install NFS Ganesha, additional configuration is required. If you want to use NFS Ganesha, read Chapter 12, Installation of NFS Ganesha before executing stages 2 and 4. However, it is possible to install NFS Ganesha later.

  In some cases it can be useful to define custom roles for NFS Ganesha nodes. For details, see Book “Administration Guide”, Chapter 19 “NFS Ganesha: Export Ceph Data via NFS”, Section 19.3 “Custom NFS Ganesha Roles”.

• grafana, prometheus - this node adds Grafana charts based on Prometheus alerting to the Ceph Dashboard. Refer to Book “Administration Guide”, Chapter 20 “Ceph Dashboard” for its detailed description.

  role-grafana/cluster/grafana*.sls

  role-prometheus/cluster/prometheus*.sls

Note: Multiple Roles of Cluster Nodes

You can assign several roles to a single node. For example, you can assign the 'mds' roles to the monitor nodes:

role-mds/cluster/mon[1,2]*.sls

config/stack/default/global.yml
config/stack/default/ceph/cluster.yml

5.5.1.4 Item Filtering #Edit source

Sometimes it is not practical to include all files from a given directory with *.sls globbing. The policy.cfg file parser understands the following filters:

Warning: Advanced Techniques

This section describes filtering techniques for advanced users. When not used correctly, filtering can cause problems for example in case your node numbering changes.

slice=[start:end]

Use the slice filter to include only items start through end-1. Note that items in the given directory are sorted alphanumerically. The following line includes the third to fifth files from the role-mon/cluster/ subdirectory:

role-mon/cluster/*.sls slice[3:6]

re=regexp

Use the regular expression filter to include only items matching the given expressions. For example:

role-mon/cluster/mon*.sls re=.*1[135]\.subdomainX\.sls$

## Cluster Assignment
cluster-ceph/cluster/*.sls 1

## Roles
# ADMIN
role-master/cluster/examplesesadmin.sls 2
role-admin/cluster/sesclient*.sls 3

# MON
role-mon/cluster/ses-example-[123].sls 4

# MGR
role-mgr/cluster/ses-example-[123].sls 5

# STORAGE
role-storage/cluster/ses-example-[5,6,7,8].sls 6

# MDS
role-mds/cluster/ses-example-4.sls 7

# IGW
role-igw/cluster/ses-example-4.sls 8

# RGW
role-rgw/cluster/ses-example-4.sls 9

# COMMON
config/stack/default/global.yml 10
config/stack/default/ceph/cluster.yml 11

cluster-unassigned/cluster/*.sls
cluster-ceph/cluster/ses-example-*.sls

DriveGroups specify the layouts of OSDs in the Ceph cluster. They are defined in a single file /srv/salt/ceph/configuration/files/drive_groups.yml.

An administrator should manually specify a group of OSDs that are interrelated (hybrid OSDs that are deployed on solid state and spinners) or share the same deployment options (identical, for example same object store, same encryption option, stand-alone OSDs). To avoid explicitly listing devices, DriveGroups use a list of filter items that correspond to a few selected fields of ceph-volume's inventory reports. In the simplest case this could be the 'rotational' flag (all solid-state drives are to be db_devices, all rotating ones data devices) or something more involved such as 'model' strings, or sizes. DeepSea will provide code that translates these DriveGroups into actual device lists for inspection by the user.

Following is a simple procedure that demonstrates the basic workflow when configuring DriveGroups:

5.5.2.1 Specification #Edit source

/srv/salt/ceph/configuration/files/drive_groups.yml can take one of two basic forms, depending on whether BlueStore or FileStore is to be used. For BlueStore setups, drive_groups.yml can be as follows:

drive_group_default_name:
  target: *
  data_devices:
    drive_spec: DEVICE_SPECIFICATION
  db_devices:
    drive_spec: DEVICE_SPECIFICATION
  wal_devices:
    drive_spec: DEVICE_SPECIFICATION
  block_wal_size: '5G'  # (optional, unit suffixes permitted)
  block_db_size: '5G'   # (optional, unit suffixes permitted)
  osds_per_device: 1   # number of osd daemons per device
  format:              # 'bluestore' or 'filestore' (defaults to 'bluestore')
  encryption:           # 'True' or 'False' (defaults to 'False')

For FileStore setups, drive_groups.yml can be as follows:

drive_group_default_name:
  target: *
  data_devices:
    drive_spec: DEVICE_SPECIFICATION
  journal_devices:
    drive_spec: DEVICE_SPECIFICATION
  format: filestore
  encryption: True

Note

If you are unsure if your OSD is encrypted, see Book “Administration Guide”, Chapter 2 “Salt Cluster Administration”, Section 2.5 “Verify an Encrypted OSD”.

5.5.2.3 Filtering Devices by Size #Edit source

You can filter disk devices by their size—either by an exact size, or a size range. The size: parameter accepts arguments in the following form:

• '10G' - Includes disks of an exact size.

• '10G:40G' - Includes disks whose size is within the range.

• ':10G' - Includes disks less than or equal to 10 GB in size.

• '40G:' - Includes disks equal to or greater than 40 GB in size.

Example 5.1: Matching by Disk Size #

drive_group_default:
  target: '*'
  data_devices:
    size: '40TB:'
  db_devices:
    size: ':2TB'

Note: Quotes Required

When using the ':' delimiter, you need to enclose the size in quotes, otherwise the ':' sign will be interpreted as a new configuration hash.

Tip: Unit Shortcuts

Instead of (G)igabytes, you can specify the sizes in (M)egabytes or (T)erabytes as well.

drive_group_default:
  target: '*'
  data_devices:
    model: SSD-123-foo
  db_devices:
    model: MC-55-44-XZ

drive_group_default:
  target: '*'
  data_devices:
    rotational: 1
  db_devices:
    rotational: 0

drive_group_default:
  target: '*'
  data_devices:
    size: '2TB:'
  db_devices:
    size: ':2TB'

drive_group:
  target: '*'
  data_devices:
    rotational: 0
  db_devices:
    model: MC-55-44-XZ

drive_group_default:
  target: '*'
  data_devices:
    model: MC-55-44-XZ
  db_devices:
    vendor: samsung
    size: 256GB

drive_group_node_one_to_five:
  target: 'node[1-5]'
  data_devices:
    rotational: 1
  db_devices:
    rotational: 0

drive_group_the_rest:
  target: 'node[6-10]'
  data_devices:
    model: MC-55-44-XZ
  db_devices:
    model: SSD-123-foo

drive_group_default:
  target: '*'
  data_devices:
    model: MC-55-44-XZ
  db_devices:
    model: SSD-123-foo
  wal_devices:
    model: NVME-QQQQ-987

drive_group_hdd_nvme:
  target: '*'
  data_devices:
    rotational: 0
  db_devices:
    model: NVME-QQQQ-987

drive_group_hdd_ssd_nvme:
  target: '*'
  data_devices:
    rotational: 0
  db_devices:
    model: MC-55-44-XZ
  wal_devices:
    model: NVME-QQQQ-987

drive_group_ssd_nvme:
  target: '*'
  data_devices:
    model: SSD-123-foo
  db_devices:
    model: NVME-QQQQ-987

drive_group_ssd_standalone_encrypted:
  target: '*'
  data_devices:
    model: SSD-123-foo
  encryption: True

If you need to put custom settings into the ceph.conf configuration file, see Book “Administration Guide”, Chapter 2 “Salt Cluster Administration”, Section 2.14 “Adjusting ceph.conf with Custom Settings” for more details.

Check that required repositories are configured on each cluster's node. To list all available repositories, run

Tip: Upgrade Without Using SCC, SMT, or RMT

If your nodes are not subscribed to one of the supported software channel providers that handle automatic channel adjustment—such as SMT, RMT, or SCC—you may need to enable additional software modules and channels.

root@minion > zypper lr

SUSE Enterprise Storage 5.5 requires:

NFS/SMB Gateway on SLE-HA on SUSE Linux Enterprise Server 12 SP3 requires:

If you are using one of the repository staging systems—SMT, or RMT—create a new frozen patch level for the current and the new SUSE Enterprise Storage version.

Find more information in:

The Distribution Migration System (DMS) provides an upgrade path for an installed SUSE Linux Enterprise system from one major version to another. The following procedure utilizes DMS to upgrade SUSE Enterprise Storage 5.5 to version 6, including the underlying SUSE Linux Enterprise Server 12 SP3 to SUSE Linux Enterprise Server 15 SP1 migration.

Refer to https://documentation.suse.com/suse-distribution-migration-system/1.0/single-html/distribution-migration-system/ to find both general and detailed information about DMS.

To view information about the currently deployed OSDs, use the following command:

root@master # salt-run disks.discover

Alternatively, you can inspect the content of the files in the /srv/pillar/ceph/proposals/profile-*/ directories. They have a similar structure to the following:

ceph:
  storage:
    osds:
      /dev/disk/by-id/scsi-drive_name: format: bluestore
      /dev/disk/by-id/scsi-drive_name2: format: bluestore

Refer to Section 5.5.2.1, “Specification” for more details on DriveGroups specification.

The difference between a fresh deployment and upgrade scenario is that the drives to be migrated are already 'used'. Because

root@master # salt-run disks.list

looks for unused disks only, use

root@master # salt-run disks.list include_unavailable=True

Adjust DriveGroups until you match your current setup. For a more visual representation of what will be happening, use the following command. Note that it has no output if there are no free disks:

root@master # salt-run disks.report bypass_pillar=True

If you verified that your DriveGroups are properly configured and want to apply the new approach, remove the files from the /srv/pillar/ceph/proposals/profile-PROFILE_NAME/ directory, remove the corresponding profile-PROFILE_NAME/cluster/*.sls lines from the /srv/pillar/ceph/proposals/policy.cfg file, and run DeepSea stage 2 to refresh the Salt pillar.

root@master # salt-run state.orch ceph.stage.2

Verify the result by running the following commands:

root@master # salt target_node pillar.get ceph:storage
root@master # salt-run disks.report

Warning: Incorrect DriveGroups Configuration

If your DriveGroups are not properly configured and there are spare disks in your setup, they will be deployed in the way you specified them. We recommend running:

root@master # salt-run disks.report

As of the Mimic release, the ceph-disk tool is deprecated, and as of the Nautilus release it is no longer shipped upstream.

ceph-disk is still supported in SUSE Enterprise Storage 6. Any pre-deployed ceph-disk OSDs will continue to function normally. However, when a disk breaks there is no migration path: a disk will need to be re-deployed.

For completeness, consider migrating OSDs on the whole node. There are two paths for SUSE Enterprise Storage 6 users:

Tip: Migrate to LVM Format

Whenever a single legacy OSD needs to be replaced on a node, all OSDs that share devices with it need to be migrated to the LVM-based format.

If you have a more sophisticated setup than just stand-alone OSDs, for example dedicated WAL/DBs or encrypted OSDs, the migration can only happen when all OSDs assigned to that WAL/DB device are removed. This is due to the ceph-volume command that creates Logical Volumes on disks before deployment. This prevents the user from mixing partition based deployments with LV based deployments. In such cases it is best to manually remove all OSDs that are assigned to a WAL/DB device and re-deploy them using the DriveGroups approach.

If you address a specific task outside of the DeepSea deployment process and therefore need to skip it, create a 'no-operation' file following this example:

If you need to replace the default behavior of a specific step with a custom one, create a custom sls file with replacement content.

By default /srv/salt/ceph/pool/default.sls creates an rbd image called 'demo'. In our example, we do not want this image to be created, but we need two images: 'archive1' and 'archive2'.

Note: Authorization

The creation of pools or images requires sufficient authorization. The admin.ceph minion has an admin keyring.

Tip: Alternative Way

Another option is to change the variable in /srv/pillar/ceph/stack/ceph/roles/master.yml instead. Using this file will reduce the clutter of pillar data for other minions.

Sometimes you may need a specific step to do some additional tasks. We do not recommend modifying the related state file as it may complicate a future upgrade. Instead, create a separate file to carry out the additional tasks identical to what was described in Section 7.1.2, “Replacing a Deployment Step”.

Name the new sls file descriptively. For example, if you need to create two rbd images in addition to the demo image, name the file archive.sls.

If you need to add a completely separate deployment step, create three new files—an sls file that performs the command, an orchestration file, and a custom file which aligns the new step with the original deployment steps.

For example, if you need to run logrotate on all minions as part of the preparation stage:

First create an sls file and include the logrotate command.

Because the orchestration file needs to run before all other preparation steps, add it to the Prep stage 0:

The last file is the custom one which includes the additional step with the original steps:

Note: Why global.yml?

The global.yml file is chosen over the cluster.yml because during the prep stage, no minion belongs to the Ceph cluster and has no access to any settings in cluster.yml.

During stage 0 (refer to DeepSea Stages Description for more information on DeepSea stages), the Salt master and Salt minions may optionally reboot because newly updated packages, for example kernel, require rebooting the system.

The default behavior is to install available new updates and not reboot the nodes even in case of kernel updates.

You can change the default update/reboot behavior of DeepSea stage 0 by adding/changing the stage_prep_master and stage_prep_minion options in the /srv/pillar/ceph/stack/global.yml file. stage_prep_master sets the behavior of the Salt master, and stage_prep_minion sets the behavior of all minions. All available parameters are:

For example, to prevent the cluster nodes from installing updates and rebooting, edit /srv/pillar/ceph/stack/global.yml and add the following lines:

stage_prep_master: default-no-update-no-reboot
stage_prep_minion: default-no-update-no-reboot

Tip: Values and Corresponding Files

The values of stage_prep_master correspond to file names located in /srv/salt/ceph/stage/0/master, while values of stage_prep_minion correspond to files in /srv/salt/ceph/stage/0/minion:

root@master # ls -l /srv/salt/ceph/stage/0/master
default-no-update-no-reboot.sls
default-no-update-reboot.sls
default-update-reboot.sls
[...]

root@master # ls -l /srv/salt/ceph/stage/0/minion
default-no-update-no-reboot.sls
default-no-update-reboot.sls
default-update-reboot.sls
[...]

Since IPv4 network addressing is prevalent, you need to enable IPv6 as a customization. DeepSea has no auto-discovery of IPv6 addressing.

To configure IPv6, set the public_network and cluster_network variables in the /srv/pillar/ceph/stack/global.yml file to valid IPv6 subnets. For example:

public_network: fd00:10::/64
cluster_network: fd00:11::/64

Then run DeepSea stage 2 and verify that the network information matches the setting. Stage 3 will generate the ceph.conf with the necessary flags.

Important: No Support for Dual Stack

Ceph does not support dual stack—running Ceph simultaneously on IPv4 and IPv6 is not possible. DeepSea validation will reject a mismatch between public_network and cluster_network or within either variable. The following example will fail the validation.

public_network: "192.168.10.0/24 fd00:10::/64"

Tip: Avoid Using fe80::/10 link-local Addresses

Avoid using fe80::/10 link-local addresses. All network interfaces have an assigned fe80 address and require an interface qualifier for proper routing. Either assign IPv6 addresses allocated to your site or consider using fd00::/8. These are part of ULA and not globally routable.

Several steps are required to configure an Object Gateway.

9.1.1.1 Basic Configuration #Edit source

Configuring a Ceph Object Gateway requires a running Ceph Storage Cluster. The Ceph Object Gateway is a client of the Ceph Storage Cluster. As a Ceph Storage Cluster client, it requires:

• A host name for the gateway instance, for example gateway.

• A storage cluster user name with appropriate permissions and a keyring.

• Pools to store its data.

• A data directory for the gateway instance.

• An instance entry in the Ceph configuration file.

Each instance must have a user name and key to communicate with a Ceph storage cluster. In the following steps, we use a monitor node to create a bootstrap keyring, then create the Object Gateway instance user keyring based on the bootstrap one. Then, we create a client user name and key. Next, we add the key to the Ceph Storage Cluster. Finally, we distribute the keyring to the node containing the gateway instance.

1. Create a keyring for the gateway:

   cephadm@adm > ceph-authtool --create-keyring /etc/ceph/ceph.client.rgw.keyring
   cephadm@adm > sudo chmod +r /etc/ceph/ceph.client.rgw.keyring

2. Generate a Ceph Object Gateway user name and key for each instance. As an example, we will use the name gateway after client.radosgw:

   cephadm@adm > ceph-authtool /etc/ceph/ceph.client.rgw.keyring \
     -n client.rgw.gateway --gen-key

3. Add capabilities to the key:

   cephadm@adm > ceph-authtool -n client.rgw.gateway --cap osd 'allow rwx' \
     --cap mon 'allow rwx' /etc/ceph/ceph.client.rgw.keyring

4. Once you have created a keyring and key to enable the Ceph Object Gateway with access to the Ceph Storage Cluster, add the key to your Ceph Storage Cluster. For example:

   cephadm@adm > ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.gateway \
     -i /etc/ceph/ceph.client.rgw.keyring

5. Distribute the keyring to the node with the gateway instance:

   cephadm@adm > scp /etc/ceph/ceph.client.rgw.keyring  ceph@HOST_NAME:/home/ceph
   cephadm@adm > ssh ceph@HOST_NAME
   cephadm@ogw > mv ceph.client.rgw.keyring /etc/ceph/ceph.client.rgw.keyring

Tip: Use Bootstrap Keyring

An alternative way is to create the Object Gateway bootstrap keyring, and then create the Object Gateway keyring from it:

1. Create an Object Gateway bootstrap keyring on one of the monitor nodes:

   cephadm@mon > ceph \
    auth get-or-create client.bootstrap-rgw mon 'allow profile bootstrap-rgw' \
    --connect-timeout=25 \
    --cluster=ceph \
    --name mon. \
    --keyring=/var/lib/ceph/mon/ceph-NODE_HOST/keyring \
    -o /var/lib/ceph/bootstrap-rgw/keyring

2. Create the /var/lib/ceph/radosgw/ceph-RGW_NAME directory for storing the bootstrap keyring:

   cephadm@mon > mkdir \
   /var/lib/ceph/radosgw/ceph-RGW_NAME

3. Create an Object Gateway keyring from the newly created bootstrap keyring:

   cephadm@mon > ceph \
    auth get-or-create client.rgw.RGW_NAME osd 'allow rwx' mon 'allow rw' \
    --connect-timeout=25 \
    --cluster=ceph \
    --name client.bootstrap-rgw \
    --keyring=/var/lib/ceph/bootstrap-rgw/keyring \
    -o /var/lib/ceph/radosgw/ceph-RGW_NAME/keyring

4. Copy the Object Gateway keyring to the Object Gateway host:

   cephadm@mon > scp \
   /var/lib/ceph/radosgw/ceph-RGW_NAME/keyring \
   RGW_HOST:/var/lib/ceph/radosgw/ceph-RGW_NAME/keyring

9.1.1.2 Create Pools (Optional) #Edit source

Ceph Object Gateways require Ceph Storage Cluster pools to store specific gateway data. If the user you created has proper permissions, the gateway will create the pools automatically. However, ensure that you have set an appropriate default number of placement groups per pool in the Ceph configuration file.

The pool names follow the ZONE_NAME.POOL_NAME syntax. When configuring a gateway with the default region and zone, the default zone name is 'default' as in our example:

.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.data

To create the pools manually, see Book “Administration Guide”, Chapter 11 “Managing Storage Pools”, Section 11.2.2 “Create a Pool”.

Important: Object Gateway and Erasure-Coded Pools

Only the default.rgw.buckets.data pool can be erasure coded. All other pools need to be replicated, otherwise the gateway is not accessible.

9.1.1.3 Adding Gateway Configuration to Ceph #Edit source

Add the Ceph Object Gateway configuration to the Ceph Configuration file. The Ceph Object Gateway configuration requires you to identify the Ceph Object Gateway instance. Then, specify the host name where you installed the Ceph Object Gateway daemon, a keyring (for use with cephx), and optionally a log file. For example:

[client.rgw.INSTANCE_NAME]
host = HOST_NAME
keyring = /etc/ceph/ceph.client.rgw.keyring

Tip: Object Gateway Log File

To override the default Object Gateway log file, include the following:

log file = /var/log/radosgw/client.rgw.INSTANCE_NAME.log

The [client.rgw.*] portion of the gateway instance identifies this portion of the Ceph configuration file as configuring a Ceph Storage Cluster client where the client type is a Ceph Object Gateway (radosgw). The instance name follows. For example:

[client.rgw.gateway]
host = ceph-gateway
keyring = /etc/ceph/ceph.client.rgw.keyring

Note

The HOST_NAME must be your machine host name, excluding the domain name.

Then turn off print continue. If you have it set to true, you may encounter problems with PUT operations:

rgw print continue = false

To use a Ceph Object Gateway with subdomain S3 calls (for example http://bucketname.hostname), you must add the Ceph Object Gateway DNS name under the [client.rgw.gateway] section of the Ceph configuration file:

[client.rgw.gateway]
...
rgw dns name = HOST_NAME

You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using the http://BUCKET_NAME.HOST_NAME syntax. The dnsmasq.conf file should include the following settings:

address=/HOST_NAME/HOST_IP_ADDRESS
listen-address=CLIENT_LOOPBACK_IP

Then, add the CLIENT_LOOPBACK_IP IP address as the first DNS server on the client machine(s).

root # mkdir -p /var/lib/ceph/radosgw/CLUSTER_ID

root # mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway

<ListAllMyBucketsResult>
      <Owner>
              <ID>anonymous</ID>
              <DisplayName/>
      </Owner>
      <Buckets/>
</ListAllMyBucketsResult>

The Linux kernel iSCSI target was originally named LIO for linux-iscsi.org, the project's original domain and Web site. For some time, no fewer than four competing iSCSI target implementations were available for the Linux platform, but LIO ultimately prevailed as the single iSCSI reference target. The mainline kernel code for LIO uses the simple, but somewhat ambiguous name "target", distinguishing between "target core" and a variety of front-end and back-end target modules.

The most commonly used front-end module is arguably iSCSI. However, LIO also supports Fibre Channel (FC), Fibre Channel over Ethernet (FCoE) and several other front-end protocols. At this time, only the iSCSI protocol is supported by SUSE Enterprise Storage.

The most frequently used target back-end module is one that is capable of simply re-exporting any available block device on the target host. This module is named iblock. However, LIO also has an RBD-specific back-end module supporting parallelized multipath I/O access to RBD images.

This section introduces brief information on iSCSI initiators used on Linux, Microsoft Windows, and VMware platforms.

You can deploy the iSCSI Gateway either during the Ceph cluster deployment process, or add it to an existing cluster using DeepSea.

To include the iSCSI Gateway during the cluster deployment process, refer to Section 5.5.1.2, “Role Assignment”.

To add the iSCSI Gateway to an existing cluster, refer to Book “Administration Guide”, Chapter 2 “Salt Cluster Administration”, Section 2.2 “Adding New Roles to Nodes”.

RBD images are created in the Ceph store and subsequently exported to iSCSI. We recommend that you use a dedicated RADOS pool for this purpose. You can create a volume from any host that is able to connect to your storage cluster using the Ceph rbd command line utility. This requires the client to have at least a minimal ceph.conf configuration file, and appropriate CephX authentication credentials.

To create a new volume for subsequent export via iSCSI, use the rbd create command, specifying the volume size in megabytes. For example, in order to create a 100 GB volume named 'testvol' in the pool named 'iscsi-images', run:

cephadm@adm > rbd --pool iscsi-images create --size=102400 'testvol'

To export RBD images via iSCSI, you can use either Ceph Dashboard Web interface or the ceph-iscsi gwcli utility. In this section we will focus on gwcli only, demonstrating how to create an iSCSI target that exports an RBD image using the command line.

Note

Only the following RBD image features are supported: layering, striping (v2), exclusive-lock, fast-diff, and data-pool. RBD images with any other feature enabled cannot be exported.

As root, start the iSCSI gateway command line interface:

root # gwcli

Go to iscsi-targets and create a target with the name iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol:

gwcli >  /> cd /iscsi-targets
gwcli >  /iscsi-targets> create iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol

Create the iSCSI gateways by specifying the gateway name and ip address:

gwcli >  /iscsi-targets> cd iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/gateways
gwcli >  /iscsi-target...tvol/gateways> create iscsi1 192.168.124.104
gwcli >  /iscsi-target...tvol/gateways> create iscsi2 192.168.124.105

Tip

Use the help command to show the list of available commands in the current configuration node.

Add the RBD image with the name 'testvol' in the pool 'iscsi-images':

gwcli >  /iscsi-target...tvol/gateways> cd /disks
gwcli >  /disks> attach iscsi-images/testvol

Map the RBD image to the target:

gwcli >  /disks> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/disks
gwcli >  /iscsi-target...testvol/disks> add iscsi-images/testvol

Note

You can use lower level tools, such as targetcli, to query the local configuration, but not to modify it.

Tip

You can use the ls command to review the configuration. Some configuration nodes also support the info command, which can be used to display more detailed information.

Note that, by default, ACL authentication is enabled so this target is not accessible yet. Check Section 10.4.4, “Authentication and Access Control” for more information about authentication and access control.

iSCSI authentication is flexible and covers many authentication possibilities.

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts
gwcli >  /iscsi-target...testvol/hosts> auth disable_acl

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts
gwcli >  /iscsi-target...testvol/hosts> create iqn.1996-04.de.suse:01:e6ca28cc9f20

gwcli >  /iscsi-target...:e6ca28cc9f20> disk add rbd/testvol

10.4.4.3 CHAP Authentication #Edit source

In addition to the ACL, you can enable the CHAP authentication by specifying a user name and password for each initiator:

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol/hosts/iqn.1996-04.de.suse:01:e6ca28cc9f20
gwcli >  /iscsi-target...:e6ca28cc9f20> auth username=common12 password=pass12345678

Note

User names must have a length of 8 to 64 characters and can contain alphanumeric characters, '.', '@', '-', '_' or ':'.

Passwords must have a length of 12 to 16 characters and can contain alphanumeric characters, '@', '-', '_' or '/'.

Optionally, you can also enable the CHAP mutual authentication by specifying the mutual_username and mutual_password parameters in the auth command.

10.4.4.4 Discovery and Mutual Authentication #Edit source

Discovery authentication is independent of the previous authentication methods. It requires credentials for browsing, it is optional, and can be configured by:

gwcli >  /> cd /iscsi-targets
gwcli >  /iscsi-targets> discovery_auth username=du123456 password=dp1234567890

Note

User-names must have a length of 8 to 64 characters and can only contain letters, '.', '@', '-', '_' or ':'.

Passwords must have a length of 12 to 16 characters and can only contain letters, '@', '-', '_' or '/'.

Optionally, you can also specify the mutual_username and mutual_password parameters in the discovery_auth command.

Discovery authentication can be disabled by using the following command:

gwcli >  /iscsi-targets> discovery_auth nochap

ceph-iscsi can be configured with advanced parameters which are subsequently passed on to the LIO I/O target. The parameters are divided up into 'target' and 'disk' parameters.

Warning

Unless otherwise noted, changing these parameters from the default setting is not recommended.

gwcli >  /> cd /iscsi-targets/iqn.2003-01.org.linux-iscsi.iscsi.SYSTEM-ARCH:testvol
gwcli >  /iscsi-target...i.SYSTEM-ARCH:testvol> info

gwcli >  /iscsi-target...i.SYSTEM-ARCH:testvol> reconfigure login_timeout 20

gwcli >  /> cd /disks/rbd/testvol
gwcli >  /disks/rbd/testvol> info

gwcli >  /disks/rbd/testvol> reconfigure rbd/testvol emulate_pr 0

You can deploy MDS either during the initial cluster deployment process as described in Section 5.3, “Cluster Deployment”, or add it to an already deployed cluster as described in Book “Administration Guide”, Chapter 2 “Salt Cluster Administration”, Section 2.1 “Adding New Cluster Nodes”.

After you deploy your MDS, allow the Ceph OSD/MDS service in the firewall setting of the server where MDS is deployed: Start yast, navigate to Security and Users › Firewall › Allowed Services and in the Service to Allow drop–down menu select Ceph OSD/MDS. If the Ceph MDS node is not allowed full traffic, mounting of a file system fails, even though other operations may work properly.

You can remove a metadata server in your cluster as described in .

You can fine-tune the MDS behavior by inserting relevant options in the ceph.conf configuration file.

A CephFS requires at least two RADOS pools: one for data and one for metadata. When configuring these pools, you might consider:

When assigning a role-mds in the policy.cfg, the required pools are automatically created. You can manually create the pools cephfs_data and cephfs_metadata for manual performance tuning before setting up the Metadata Server. DeepSea will not create these pools if they already exist.

For more information on managing pools, see Book “Administration Guide”, Chapter 11 “Managing Storage Pools”.

To create the two required pools—for example, 'cephfs_data' and 'cephfs_metadata'—with default settings for use with CephFS, run the following commands:

cephadm@adm > ceph osd pool create cephfs_data pg_num
cephadm@adm > ceph osd pool create cephfs_metadata pg_num

It is possible to use EC pools instead of replicated pools. We recommend to only use EC pools for low performance requirements and infrequent random access, for example cold storage, backups, archiving. CephFS on EC pools requires BlueStore to be enabled and the pool must have the allow_ec_overwrite option set. This option can be set by running ceph osd pool set ec_pool allow_ec_overwrites true.

Erasure coding adds significant overhead to file system operations, especially small updates. This overhead is inherent to using erasure coding as a fault tolerance mechanism. This penalty is the trade off for significantly reduced storage space overhead.

When the pools are created, you may enable the file system with the ceph fs new command:

cephadm@adm > ceph fs new fs_name metadata_pool_name data_pool_name

For example:

cephadm@adm > ceph fs new cephfs cephfs_metadata cephfs_data

You can check that the file system was created by listing all available CephFSs:

cephadm@adm > ceph fs ls
 name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

When the file system has been created, your MDS will be able to enter an active state. For example, in a single MDS system:

cephadm@adm > ceph mds stat
e5: 1/1/1 up

Tip: More Topics

You can find more information of specific tasks—for example mounting, unmounting, and advanced CephFS setup—in Book “Administration Guide”, Chapter 17 “Clustered File System”.

A CephFS instance can be served by multiple active MDS daemons. All active MDS daemons that are assigned to a CephFS instance will distribute the file system's directory tree between themselves, and thus spread the load of concurrent clients. In order to add an active MDS daemon to a CephFS instance, a spare standby is needed. Either start an additional daemon or use an existing standby instance.

The following command will display the current number of active and passive MDS daemons.

cephadm@adm > ceph mds stat

The following command sets the number of active MDSs to two in a file system instance.

cephadm@adm > ceph fs set fs_name max_mds 2

In order to shrink the MDS cluster prior to an update, two steps are necessary. First, set max_mds so that only one instance remains:

cephadm@adm > ceph fs set fs_name max_mds 1

and after that, explicitly deactivate the other active MDS daemons:

cephadm@adm > ceph mds deactivate fs_name:rank

where rank is the number of an active MDS daemon of a file system instance, ranging from 0 to max_mds-1.

We recommend at least one MDS is left as a standby daemon.

During Ceph updates, the feature flags on a file system instance may change (usually by adding new features). Incompatible daemons (such as the older versions) are not able to function with an incompatible feature set and will refuse to start. This means that updating and restarting one daemon can cause all other not yet updated daemons to stop and refuse to start. For this reason, we recommend shrinking the active MDS cluster to size one and stopping all standby daemons before updating Ceph. The manual steps for this update procedure are as follows:

If you use DeepSea, it will follow this procedure in case the ceph package was updated during stages 0 and 4. It is possible to perform this procedure while clients have the CephFS instance mounted and I/O is ongoing. Note however that there will be a very brief I/O pause while the active MDS restarts. Clients will recover automatically.

It is good practice to reduce the I/O load as much as possible before updating an MDS cluster. An idle MDS cluster will go through this update procedure quicker. Conversely, on a heavily loaded cluster with multiple MDS daemons it is essential to reduce the load in advance to prevent a single MDS daemon from being overwhelmed by ongoing I/O.

The layout of a file controls how its contents are mapped to Ceph RADOS objects. You can read and write a file’s layout using virtual extended attributes or xattrs for shortly.

The name of the layout xattrs depends on whether a file is a regular file or a directory. Regular files’ layout xattrs are called ceph.file.layout, while directories’ layout xattrs are called ceph.dir.layout. Where examples refer to ceph.file.layout, substitute the .dir. part as appropriate when dealing with directories.

11.3.4.2 Reading Layout with getfattr #Edit source

Use the getfattr command to read the layout information of an example file file as a single string:

root # touch file
root # getfattr -n ceph.file.layout file
# file: file
ceph.file.layout="stripe_unit=4194304 stripe_count=1 object_size=419430

Read individual layout fields:

root # getfattr -n ceph.file.layout.pool file
# file: file
ceph.file.layout.pool="cephfs_data"
root # getfattr -n ceph.file.layout.stripe_unit file
# file: file
ceph.file.layout.stripe_unit="4194304"

Tip: Pool ID or Name

When reading layouts, the pool will usually be indicated by name. However, in rare cases when pools have only just been created, the ID may be output instead.

Directories do not have an explicit layout until it is customized. Attempts to read the layout will fail if it has never been modified: this indicates that the layout of the next ancestor directory with an explicit layout will be used.

root # mkdir dir
root # getfattr -n ceph.dir.layout dir
dir: ceph.dir.layout: No such attribute
root # setfattr -n ceph.dir.layout.stripe_count -v 2 dir
root # getfattr -n ceph.dir.layout dir
# file: dir
ceph.dir.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 pool=cephfs_data"

11.3.4.3 Writing Layouts with setfattr #Edit source

Use the setfattr command to modify the layout fields of an example file file:

cephadm@adm > ceph osd lspools
0 rbd
1 cephfs_data
2 cephfs_metadata
root # setfattr -n ceph.file.layout.stripe_unit -v 1048576 file
root # setfattr -n ceph.file.layout.stripe_count -v 8 file
# Setting pool by ID:
root # setfattr -n ceph.file.layout.pool -v 1 file
# Setting pool by name:
root # setfattr -n ceph.file.layout.pool -v cephfs_data file

Note: Empty File

When the layout fields of a file are modified using setfattr, this file needs to be empty otherwise an error will occur.

root # setfattr -x ceph.dir.layout mydir

# Create a directory and set a namespace on it
root # mkdir mydir
root # setfattr -n ceph.dir.layout.pool_namespace -v foons mydir
root # getfattr -n ceph.dir.layout mydir
ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 \
 pool=cephfs_data_a pool_namespace=foons"

# Clear the namespace from the directory's layout
root # setfattr -x ceph.dir.layout.pool_namespace mydir
root # getfattr -n ceph.dir.layout mydir
ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 \
 pool=cephfs_data_a"

root # getfattr -n ceph.dir.layout dir
# file: dir
ceph.dir.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 \
 pool=cephfs_data"

# file1 inherits its parent's layout
root # touch dir/file1
root # getfattr -n ceph.file.layout dir/file1
# file: dir/file1
ceph.file.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 \
 pool=cephfs_data"

# update the layout of the directory before creating a second file
root # setfattr -n ceph.dir.layout.stripe_count -v 4 dir
root # touch dir/file2

# file1's layout is unchanged
root # getfattr -n ceph.file.layout dir/file1
# file: dir/file1
ceph.file.layout="stripe_unit=4194304 stripe_count=2 object_size=4194304 \
 pool=cephfs_data"

# ...while file2 has the parent directory's new layout
root # getfattr -n ceph.file.layout dir/file2
# file: dir/file2
ceph.file.layout="stripe_unit=4194304 stripe_count=4 object_size=4194304 \
 pool=cephfs_data"

root # getfattr -n ceph.dir.layout dir
# file: dir
ceph.dir.layout="stripe_unit=4194304 stripe_count=4 object_size=4194304 \
 pool=cephfs_data"
root # mkdir dir/childdir
root # getfattr -n ceph.dir.layout dir/childdir
dir/childdir: ceph.dir.layout: No such attribute
root # touch dir/childdir/grandchild
root # getfattr -n ceph.file.layout dir/childdir/grandchild
# file: dir/childdir/grandchild
ceph.file.layout="stripe_unit=4194304 stripe_count=4 object_size=4194304 \
 pool=cephfs_data"

11.3.4.6 Adding a Data Pool to the Metadata Server #Edit source

Before you can use a pool with CephFS, you need to add it to the Metadata Server:

cephadm@adm > ceph fs add_data_pool cephfs cephfs_data_ssd
cephadm@adm > ceph fs ls  # Pool should now show up
.... data pools: [cephfs_data cephfs_data_ssd ]

Tip: cephx Keys

Make sure that your cephx keys allow the client to access this new pool.

You can then update the layout on a directory in CephFS to use the pool you added:

root # mkdir /mnt/cephfs/myssddir
root # setfattr -n ceph.dir.layout.pool -v cephfs_data_ssd /mnt/cephfs/myssddir

All new files created within that directory will now inherit its layout and place their data in your newly added pool. You may notice that the number of objects in your primary data pool continues to increase, even if files are being created in the pool you newly added. This is normal: the file data is stored in the pool specified by the layout, but a small amount of metadata is kept in the primary data pool for all files.

To successfully deploy NFS Ganesha, you need to add a role-ganesha to your /srv/pillar/ceph/proposals/policy.cfg. For details, see Section 5.5.1, “The policy.cfg File”. NFS Ganesha also needs either a role-rgw or a role-mds present in the policy.cfg.

Although it is possible to install and run the NFS Ganesha server on an already existing Ceph node, we recommend running it on a dedicated host with access to the Ceph cluster. The client hosts are typically not part of the cluster, but they need to have network access to the NFS Ganesha server.

To enable the NFS Ganesha server at any point after the initial installation, add the role-ganesha to the policy.cfg and re-run at least DeepSea stages 2 and 4. For details, see Section 5.3, “Cluster Deployment”.

NFS Ganesha is configured via the file /etc/ganesha/ganesha.conf that exists on the NFS Ganesha node. However, this file is overwritten each time DeepSea stage 4 is executed. Therefore we recommend to edit the template used by Salt, which is the file /srv/salt/ceph/ganesha/files/ganesha.conf.j2 on the Salt master. For details about the configuration file, see Book “Administration Guide”, Chapter 19 “NFS Ganesha: Export Ceph Data via NFS”, Section 19.2 “Configuration”.

The following requirements need to be met before DeepSea stages 2 and 4 can be executed to install NFS Ganesha:

In this setup earth has the IP address 192.168.1.1 and mars has the address 192.168.1.2.

Additionally, two floating virtual IP addresses are used, allowing clients to connect to the service independent of which physical node it is running on. 192.168.1.10 is used for cluster administration with Hawk2 and 192.168.2.1 is used exclusively for the NFS exports. This makes it easier to apply security restrictions later.

The following procedure describes the example installation. More details can be found at https://documentation.suse.com/sle-ha/15-SP1/single-html/SLE-HA-install-quick/.

In the event of an NFS Ganesha failure at one of the node, for example earth, fix the issue and clean up the resource. Only after the resource is cleaned up can the resource fail back to earth in case NFS Ganesha fails at mars.

To clean up the resource:

root@earth # crm resource cleanup nfs-ganesha-clone earth
root@earth # crm resource cleanup ganesha-ip earth

It may happen that the server is unable to reach the client because of a network issue. A ping resource can detect and mitigate this problem. Configuring this resource is optional.

When a service goes down, the TCP connection that is in use by NFS Ganesha is required to be closed otherwise it continues to run until a system-specific timeout occurs. This timeout can take upwards of 3 minutes.

To shorten the timeout time, the TCP connection needs to be reset. We recommend configuring portblock to reset stale TCP connections.

You can choose to use portblock with or without the tickle_dir parameters that could unblock and reconnect clients to the new service faster. We recommend to have tickle_dir as the shared CephFS mount between two HA nodes (where NFS Ganesha services are running).

Note

Configuring the following resource is optional.

1. On earth, run the crm shell to execute the following commands to configure the resource for NFS Ganesha daemons:

   root@earth # crm configure

2. Configure the block action for portblock and omit the tickle_dir option if you have not configured a shared directory:

   crm(live)configure#  primitive nfs-ganesha-block ocf:portblock \
   protocol=tcp portno=2049 action=block ip=192.168.2.1 op monitor depth="0" timeout="10" interval="10" tickle_dir="/tmp/ganesha/tickle/"

3. Configure the unblock action for portblock and omit the reset_local_on_unblock_stop option if you have not configured a shared directory:

   crm(live)configure#  primitive nfs-ganesha-unblock ocf:portblock \
   protocol=tcp portno=2049 action=unblock ip=192.168.2.1 op monitor depth="0" timeout="10" interval="10" reset_local_on_unblock_stop=true tickle_dir="/tmp/ganesha/tickle/"

4. Configure the IPAddr2 resource with portblock:

   crm(live)configure#  colocation ganesha-portblock inf: ganesha-ip nfs-ganesha-block nfs-ganesha-unblock
   crm(live)configure#  edit ganesha-ip-after-nfs-ganesha-server
   order ganesha-ip-after-nfs-ganesha-server Mandatory: nfs-ganesha-block nfs-ganesha-clone ganesha-ip nfs-ganesha-unblock

5. Save your changes:

   crm(live)configure#  commit

6. Your configuration should look like this:

   crm(live)configure#  show

   "
   node 1084782956: nfs1
   node 1084783048: nfs2
   primitive ganesha-ip IPaddr2 \
           params ip=192.168.2.1 cidr_netmask=24 nic=eth0 \
           op monitor interval=10 timeout=20
   primitive nfs-ganesha-block portblock \
           params protocol=tcp portno=2049 action=block ip=192.168.2.1 \
           tickle_dir="/tmp/ganesha/tickle/" op monitor timeout=10 interval=10 depth=0
   primitive nfs-ganesha-server systemd:nfs-ganesha \
           op monitor interval=30s
   primitive nfs-ganesha-unblock portblock \
           params protocol=tcp portno=2049 action=unblock ip=192.168.2.1 \
           reset_local_on_unblock_stop=true tickle_dir="/tmp/ganesha/tickle/" \
           op monitor timeout=10 interval=10 depth=0
   clone nfs-ganesha-clone nfs-ganesha-server \
           meta interleave=true
   location cli-prefer-ganesha-ip ganesha-ip role=Started inf: nfs1
   order ganesha-ip-after-nfs-ganesha-server Mandatory: nfs-ganesha-block nfs-ganesha-clone ganesha-ip nfs-ganesha-unblock
   colocation ganesha-ip-with-nfs-ganesha-server inf: ganesha-ip nfs-ganesha-clone
   colocation ganesha-portblock inf: ganesha-ip nfs-ganesha-block nfs-ganesha-unblock
   property cib-bootstrap-options: \
           have-watchdog=false \
           dc-version=1.1.16-6.5.1-77ea74d \
           cluster-infrastructure=corosync \
           cluster-name=hacluster \
           stonith-enabled=false \
           placement-strategy=balanced \
           last-lrm-refresh=1544793779
   rsc_defaults rsc-options: \
           resource-stickiness=1 \
           migration-threshold=3
   op_defaults op-options: \
           timeout=600 \
           record-pending=true
   "

   In this example /tmp/ganesha/ is the CephFS mount on both nodes (nfs1 and nfs2):

   172.16.1.11:6789:/ganesha on /tmp/ganesha type ceph (rw,relatime,name=admin,secret=...hidden...,acl,wsize=16777216)

   The tickle directory has been initially created.

DeepSea does not support configuring NFS Ganesha HA. To prevent DeepSea from failing after NFS Ganesha HA is configured, exclude starting and stopping the NFS Ganesha service from DeepSea stage 4:

To prevent DeepSea from restarting NFS Ganesha service on stage 4:

For our example configuration, you need the following:

After you deploy the NFS Ganesha nodes, verify that the cluster is operational and the default CephFS pools are there:

cephadm@adm > rados lspools
cephfs_data
cephfs_metadata

Check that both NFS Ganesha nodes have the file /etc/ganesha/ganesha.conf installed. Add the following blocks, if they do not exist yet, to the configuration file in order to enable RADOS as the recovery backend of NFS Ganesha.

NFS_CORE_PARAM
{
    Enable_NLM = false;
    Enable_RQUOTA = false;
    Protocols = 4;
}
NFSv4
{
    RecoveryBackend = rados_cluster;
    Minor_Versions = 1,2;
}
CACHEINODE {
    Dir_Chunk = 0;
    NParts = 1;
    Cache_Size = 1;
}
RADOS_KV
{
    pool = "rados_pool";
    namespace = "pool_namespace";
    nodeid = "fqdn"
    UserId = "cephx_user_id";
    Ceph_Conf = "path_to_ceph.conf"
}

You can find out the values for rados_pool and pool_namespace by checking the already existing line in the configuration of the form:

%url rados://rados_pool/pool_namespace/...

The value for nodeid option corresponds to the FQDN of the machine, and UserId and Ceph_Conf options value can be found in the already existing RADOS_URLS block.

Because legacy versions of NFS prevent us from lifting the grace period early and therefore prolong a server restart, we disable options for NFS prior to version 4.2. We also disable most of the NFS Ganesha caching as Ceph libraries do aggressive caching already.

The 'rados_cluster' recovery back-end stores its info in RADOS objects. Although it is not a lot of data, we want it highly available. We use the CephFS metadata pool for this purpose, and declare a new 'ganesha' namespace in it to keep it distinct from CephFS objects.

Note: Cluster Node IDs

Most of the configuration is identical between the two hosts, however the nodeid option in the 'RADOS_KV' block needs to be a unique string for each node. By default, NFS Ganesha sets nodeid to the host name of the node.

If you need to use different fixed values other than host names, you can for example set nodeid = 'a' on one node and nodeid = 'b' on the other one.

We need to verify that all of the nodes in the cluster know about each other. This done via a RADOS object that is shared between the hosts. NFS Ganesha uses this object to communicate the current state with regard to a grace period.

The nfs-ganesha-rados-grace package contains a command line tool for querying and manipulating this database. If the package is not installed on at least one of the nodes, install it with

root # zypper install nfs-ganesha-rados-grace

We will use the command to create the DB and add both nodeids. In our example, the two NFS Ganesha nodes are named ses6min1.example.com and ses6min2.example.com On one of the NFS Ganesha hosts, run

cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha add ses6min1.example.com
cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha add ses6min2.example.com
cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha
cur=1 rec=0
======================================================
ses6min1.example.com     E
ses6min2.example.com     E

This creates the grace database and adds both 'ses6min1.example.com' and 'ses6min2.example.com' to it. The last command dumps the current state. Newly added hosts are always considered to be enforcing the grace period so they both have the 'E' flag set. The 'cur' and 'rec' values show the current and recovery epochs, which is how we keep track of what hosts are allowed to perform recovery and when.

On both NFS Ganesha nodes, restart the related services:

root # systemctl restart nfs-ganesha.service

After the services are restarted, check the grace database:

cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha
cur=3 rec=0
======================================================
ses6min1.example.com
ses6min2.example.com

Note: Cleared the 'E' Flag

Note that both nodes have cleared their 'E' flags, indicating that they are no longer enforcing the grace period and are now in normal operation mode.

After you complete all the preceding steps, you can mount the exported NFS from either of the two NFS Ganesha servers, and perform normal NFS operations against them.

Our example configuration assumes that if one of the two NFS Ganesha servers goes down, you will restart it manually within 5 minutes. After 5 minutes, the Metadata Server may cancel the session that the NFS Ganesha client held and all of the state associated with it. If the session’s capabilities get cancelled before the rest of the cluster goes into the grace period, the server’s clients may not be able to recover all of their state.