The basic component of Ceph is called RADOS (Reliable Autonomic Distributed Object Store). It is responsible for managing the data stored in the cluster. Data in Ceph is usually stored as objects. Each object consists of an identifier and the data.

RADOS provides the following access methods to the stored objects that cover many use cases:

librados is used by Object Gateway and RBD while CephFS directly interfaces with RADOS Figure 1.1, “Interfaces to the Ceph Object Store”.

Figure 1.1: Interfaces to the Ceph Object Store #

At the core of a Ceph cluster is the CRUSH algorithm. CRUSH is the acronym for Controlled Replication Under Scalable Hashing. CRUSH is a function that handles the storage allocation and needs comparably few parameters. That means only a small amount of information is necessary to calculate the storage position of an object. The parameters are a current map of the cluster including the health state, some administrator-defined placement rules and the name of the object that needs to be stored or retrieved. With this information, all nodes in the Ceph cluster are able to calculate where an object and its replicas are stored. This makes writing or reading data very efficient. CRUSH tries to evenly distribute data over all nodes in the cluster.

The CRUSH map contains all storage nodes and administrator-defined placement rules for storing objects in the cluster. It defines a hierarchical structure that usually corresponds to the physical structure of the cluster. For example, the data-containing disks are in hosts, hosts are in racks, racks in rows and rows in data centers. This structure can be used to define failure domains. Ceph then ensures that replications are stored on different branches of a specific failure domain.

If the failure domain is set to rack, replications of objects are distributed over different racks. This can mitigate outages caused by a failed switch in a rack. If one power distribution unit supplies a row of racks, the failure domain can be set to row. When the power distribution unit fails, the replicated data is still available on other rows.

In Ceph, nodes are servers working for the cluster. They can run several different types of daemons. It is recommended to run only one type of daemon on each node, except for MGR daemons which can be collocated with MONs. Each cluster requires at least MON, MGR and OSD daemons:

To use CephFS, Object Gateway, NFS Ganesha, or iSCSI Gateway, additional nodes are required:

These commands help you to deploy or upgrade the Ceph cluster, run commands on several (or all) cluster nodes at the same time, or assist you when adding or removing cluster nodes. The most frequently used are salt, salt-run, and deepsea. You need to run Salt commands on the Salt master node (refer to Section 4.2, “Introduction to DeepSea” for details) as root. These commands are introduced with the following prompt:

root@master # 

For example:

root@master # salt '*.example.net' test.ping

These are lower level commands to configure and fine tune all aspects of the cluster and its gateways on the command line. ceph, rbd, radosgw-admin, or crushtool to name some of them.

To run Ceph related commands, you need to have read access to a Ceph key. The key's capabilities then define your privileges within the Ceph environment. One option is to run Ceph commands as root (or via sudo) and use the unrestricted default keyring 'ceph.client.admin.key'.

Safer and recommended option is to create a more restrictive individual key for each administrator user and put it in a directory where the users can read it, for example:

~/.ceph/ceph.client.USERNAME.keyring

Tip: Path to Ceph Keys

To use a custom admin user and keyring, you need to specify the user name and path to the key each time you run the ceph command using the -n client.USER_NAME and --keyring PATH/TO/KEYRING options.

To avoid this, include these options in the CEPH_ARGS variable in the individual users' ~/.bashrc files.

Although you can run Ceph related commands on any cluster node, we recommend running them on the node with the 'admin' role (see Section 4.5.1.2, “Role Assignment” for details). This documentation uses the cephadm user to run the commands, therefore they are introduced with the following prompt:

cephadm > 

For example:

cephadm > ceph auth list

Linux commands not related to Ceph or DeepSea, such as mount, cat, or openssl, are introduced either with the cephadm > or root # prompts, depending on which privileges the related command requires.

For more information on Ceph key management, refer to Book “Administration Guide”, Chapter 6 “Authentication with cephx”, Section 6.2 “Key Management”.

Objects that are stored in a Ceph cluster are put into pools. Pools represent logical partitions of the cluster to the outside world. For each pool a set of rules can be defined, for example, how many replications of each object must exist. The standard configuration of pools is called replicated pool.

Pools usually contain objects but can also be configured to act similar to a RAID 5. In this configuration, objects are stored in chunks along with additional coding chunks. The coding chunks contain the redundant information. The number of data and coding chunks can be defined by the administrator. In this configuration, pools are referred to as erasure coded pools.

Placement Groups (PGs) are used for the distribution of data within a pool. When creating a pool, a certain number of placement groups is set. The placement groups are used internally to group objects and are an important factor for the performance of a Ceph cluster. The PG for an object is determined by the object's name.

This section provides a simplified example of how Ceph manages data (see Figure 1.2, “Small Scale Ceph Example”). This example does not represent a recommended configuration for a Ceph cluster. The hardware setup consists of three storage nodes or Ceph OSDs (Host 1, Host 2, Host 3). Each node has three hard disks which are used as OSDs (osd.1 to osd.9). The Ceph Monitor nodes are neglected in this example.

Note: Difference between Ceph OSD and OSD

While Ceph OSD or Ceph OSD daemon refers to a daemon that is run on a node, the word OSD refers to the logical disk that the daemon interacts with.

The cluster has two pools, Pool A and Pool B. While Pool A replicates objects only two times, resilience for Pool B is more important and it has three replications for each object.

When an application puts an object into a pool, for example via the REST API, a Placement Group (PG1 to PG4) is selected based on the pool and the object name. The CRUSH algorithm then calculates on which OSDs the object is stored, based on the Placement Group that contains the object.

In this example the failure domain is set to host. This ensures that replications of objects are stored on different hosts. Depending on the replication level set for a pool, the object is stored on two or three OSDs that are used by the Placement Group.

An application that writes an object only interacts with one Ceph OSD, the primary Ceph OSD. The primary Ceph OSD takes care of replication and confirms the completion of the write process after all other OSDs have stored the object.

If osd.5 fails, all object in PG1 are still available on osd.1. As soon as the cluster recognizes that an OSD has failed, another OSD takes over. In this example osd.4 is used as a replacement for osd.5. The objects stored on osd.1 are then replicated to osd.4 to restore the replication level.

Figure 1.2: Small Scale Ceph Example #

If a new node with new OSDs is added to the cluster, the cluster map is going to change. The CRUSH function then returns different locations for objects. Objects that receive new locations will be relocated. This process results in a balanced usage of all OSDs.

There are two types of disk space needed to run on OSD: the space for the disk journal (for FileStore) or WAL/DB device (for BlueStore), and the primary space for the stored data. The minimum (and default) value for the journal/WAL/DB is 6 GB. The minimum space for data is 5 GB, as partitions smaller than 5 GB are automatically assigned the weight of 0.

So although the minimum disk space for an OSD is 11 GB, we do not recommend a disk smaller than 20 GB, even for testing purposes.

Tip: More Information

Refer to Section 1.5, “BlueStore” for more information on BlueStore.

Following are several rules for WAL/DB device sizing. When using DeepSea to deploy OSDs with BlueStore, it applies the recommended rules automatically and notifies the administrator about the fact.

Solid-state drives (SSD) have no moving parts. This reduces random access time and read latency while accelerating data throughput. Because their price per 1MB is significantly higher than the price of spinning hard disks, SSDs are only suitable for smaller storage.

OSDs may see a significant performance improvement by storing their journal on an SSD and the object data on a separate hard disk.

Tip: Sharing an SSD for Multiple Journals

As journal data occupies relatively little space, you can mount several journal directories to a single SSD disk. Keep in mind that with each shared journal, the performance of the SSD disk degrades. We do not recommend sharing more than six journals on the same SSD disk and 12 on NVMe disks.

You can have as many disks in one server as it allows. There are a few things to consider when planning the number of disks per server:

If you do not specify a cluster network during Ceph deployment, it assumes a single public network environment. While Ceph operates fine with a public network, its performance and security improves when you set a second private cluster network. To support two networks, each Ceph node needs to have at least two network cards.

You need to apply the following changes to each Ceph node. It is relatively quick to do for a small cluster, but can be very time consuming if you have a cluster consisting of hundreds or thousands of nodes.

If the monitor nodes are on multiple subnets, for example they are located in different rooms and served by different switches, you need to adjust the ceph.conf file accordingly. For example if the nodes have IP addresses 192.168.123.12, 1.2.3.4, and 242.12.33.12, add the following lines to its global section:

[global]
[...]
mon host = 192.168.123.12, 1.2.3.4, 242.12.33.12
mon initial members = MON1, MON2, MON3
[...]

Additionally, if you need to specify a per-monitor public address or network, you need to add a [mon.X] section per each monitor:

[mon.MON1]
public network = 192.168.123.0/24

[mon.MON2]
public network = 1.2.3.0/24

[mon.MON3]
public network = 242.12.33.12/0

SUSE Manager and SUSE Enterprise Storage are not integrated, therefore SUSE Manager cannot currently manage a SUSE Enterprise Storage 5.5 cluster.

Salt has several standard locations and several naming conventions used on your master node:

DeepSea commands are executed via the Salt infrastructure. When using the salt command, you need to specify a set of Salt minions that the command will affect. We describe the set of the minions as a target for the salt command. The following sections describe possible methods to target the minions.

4.2.2.1 Matching the Minion Name #Edit source

You can target a minion or a group of minions by matching their names. A minion's name is usually the short host name of the node where the minion runs. This is a general Salt targeting method, not related to DeepSea. You can use globbing, regular expressions, or lists to limit the range of minion names. The general syntax follows:

root@master # salt target example.module

Tip: Ceph-only Cluster

If all Salt minions in your environment belong to your Ceph cluster, you can safely substitute target with '*' to include all registered minions.

Match all minions in the example.net domain (assuming the minion names are identical to their "full" host names):

root@master # salt '*.example.net' test.ping

Match the 'web1' to 'web5' minions:

root@master # salt 'web[1-5]' test.ping

Match both 'web1-prod' and 'web1-devel' minions using a regular expression:

root@master # salt -E 'web1-(prod|devel)' test.ping

Match a simple list of minions:

root@master # salt -L 'web1,web2,web3' test.ping

Match all minions in the cluster:

root@master # salt '*' test.ping

root@master # salt target grains.append deepsea default

root@master # salt target grains.delval deepsea destructive=True

root@master # salt -G 'deepsea:*' test.ping

root@master # salt -C 'G@deepsea:*' test.ping

4.2.2.3 Set the deepsea_minions Option #Edit source

Setting the deepsea_minions option's target is a requirement for DeepSea deployments. DeepSea uses it to instruct minions during stages execution (refer to DeepSea Stages Description for details.

To set or change the deepsea_minions option, edit the /srv/pillar/ceph/deepsea_minions.sls file on the Salt master and add or replace the following line:

deepsea_minions: target

Tip: deepsea_minions Target

As the target for the deepsea_minions option, you can use any targeting method: both Matching the Minion Name and Targeting with a 'deepsea' Grain.

Match all Salt minions in the cluster:

deepsea_minions: '*'

Match all minions with the 'deepsea' grain:

deepsea_minions: 'G@deepsea:*'

The progress monitor provides a detailed, real-time visualization of what is happening during execution of stages using salt-run state.orch commands in other terminal sessions.

Tip: Start Monitor in a New Terminal Session

You need to start the monitor in a new terminal window before running any salt-run state.orch so that the monitor can detect the start of the stage's execution.

If you start the monitor after issuing the salt-run state.orch command, then no execution progress will be shown.

You can start the monitor mode by running the following command:

root@master # deepsea monitor

For more information about the available command line options of the deepsea monitor command check its manual page:

cephadm > man deepsea-monitor

In the stand-alone mode, DeepSea CLI can be used to run a DeepSea stage, showing its execution in real-time.

The command to run a DeepSea stage from the DeepSea CLI has the following form:

root@master # deepsea stage run stage-name

where stage-name corresponds to the way Salt orchestration state files are referenced. For example, stage deploy, which corresponds to the directory located in /srv/salt/ceph/stage/deploy, is referenced as ceph.stage.deploy.

This command is an alternative to the Salt-based commands for running DeepSea stages (or any DeepSea orchestration state file).

The command deepsea stage run ceph.stage.0 is equivalent to salt-run state.orch ceph.stage.0.

For more information about the available command line options accepted by the deepsea stage run command check its manual page:

root@master # man deepsea-stage run

In the following figure shows an example of the output of the DeepSea CLI when running Stage 2:

Figure 4.1: DeepSea CLI stage execution progress output #

root@master # deepsea salt-run state.orch stage-name

The /srv/pillar/ceph/proposals/policy.cfg configuration file is used to determine roles of individual cluster nodes. For example, which node acts as an OSD or which as a monitor node. Edit policy.cfg in order to reflect your desired cluster setup. The order of the sections is arbitrary, but the content of included lines overwrites matching keys from the content of previous lines.

Tip: Examples of policy.cfg

You can find several examples of complete policy files in the /usr/share/doc/packages/deepsea/examples/ directory.

cluster-ceph/cluster/*.sls

cluster-ceph/cluster/abc.domain.sls

cluster-ceph/cluster/mon*.sls

cluster-unassigned/cluster/client*.sls

4.5.1.2 Role Assignment #Edit source

This section provides you with details on assigning 'roles' to your cluster nodes. A 'role' in this context means the service you need to run on the node, such as Ceph Monitor, Object Gateway, iSCSI Gateway, or openATTIC. No role is assigned automatically, only roles added to policy.cfg will be deployed.

The assignment follows this pattern:

role-ROLE_NAME/PATH/FILES_TO_INCLUDE

Where the items have the following meaning and values:

• ROLE_NAME is any of the following: 'master', 'admin', 'mon', 'mgr', 'mds', 'igw', 'rgw', 'ganesha', or 'openattic'.

• PATH is a relative directory path to .sls or .yml files. In case of .sls files, it usually is cluster, while .yml files are located at stack/default/ceph/minions.

• FILES_TO_INCLUDE are the Salt state files or YAML configuration files. They normally consist of Salt minions host names, for example ses5min2.yml. Shell globbing can be used for more specific matching.

An example for each role follows:

• master - the node has admin keyrings to all Ceph clusters. Currently, only a single Ceph cluster is supported. As the master role is mandatory, always add a similar line to the following:

  role-master/cluster/master*.sls

• admin - the minion will have an admin keyring. You define the role as follows:

  role-admin/cluster/abc*.sls

• mon - the minion will provide the monitoring service to the Ceph cluster. This role requires addresses of the assigned minions. As of SUSE Enterprise Storage 5.5, the public address are calculated dynamically and are no longer needed in the Salt pillar.

  role-mon/cluster/mon*.sls

  The example assigns the monitoring role to a group of minions.

• mgr - the Ceph manager daemon which collects all the state information from the whole cluster. Deploy it on all minions where you plan to deploy the Ceph monitor role.

  role-mgr/cluster/mgr*.sls

• mds - the minion will provide the metadata service to support CephFS.

  role-mds/cluster/mds*.sls

• igw - the minion will act as an iSCSI Gateway. This role requires addresses of the assigned minions, thus you need to also include the files from the stack directory:

  role-igw/cluster/*.sls

• rgw - the minion will act as an Object Gateway:

  role-rgw/cluster/rgw*.sls

• openattic - the minion will act as an openATTIC server:

  role-openattic/cluster/openattic*.sls

  For more information, see Book “Administration Guide”, Chapter 17 “openATTIC”.

• ganesha - the minion will act as an NFS Ganesha server. The 'ganesha' role requires either an 'rgw' or 'mds' role in cluster, otherwise the validation will fail in Stage 3.

  To successfully install NFS Ganesha, additional configuration is required. If you want to use NFS Ganesha, read Chapter 12, Installation of NFS Ganesha before executing stages 2 and 4. However, it is possible to install NFS Ganesha later.

  In some cases it can be useful to define custom roles for NFS Ganesha nodes. For details, see Book “Administration Guide”, Chapter 16 “NFS Ganesha: Export Ceph Data via NFS”, Section 16.3 “Custom NFS Ganesha Roles”.

Note: Multiple Roles of Cluster Nodes

You can assign several roles to a single node. For example, you can assign the mds roles to the monitor nodes:

role-mds/cluster/mon[1,2]*.sls

config/stack/default/global.yml
config/stack/default/ceph/cluster.yml

profile-default/cluster/*.sls
profile-default/stack/default/ceph/minions/*.yml

4.5.1.6 Deploying Encrypted OSDs #Edit source

Since SUSE Enterprise Storage 5.5, OSDs are by default deployed using BlueStore instead of FileStore. Although BlueStore supports encryption, Ceph OSDs are deployed unencrypted by default. The following procedure describes steps to encrypt OSDs during the upgrade process. Let us assume that both data and WAL/DB disks to be used for OSD deployment are clean with no partitions. If the disk were previously used, wipe them following the procedure described in Step 12.

To use encrypted OSDs for your new deployment, first wipe the disks following the procedure described in Step 12, then use the proposal.populate runner with the encryption=dmcrypt argument:

root@master # salt-run proposal.populate encryption=dmcrypt

Important: Slow Boots

Encrypted OSDs require longer boot and activation times compared to the default unencrypted ones.

1. Determine the bluestore block db size and bluestore block wal size values for your deployment and add them to the /srv/salt/ceph/configuration/files/ceph.conf.d/global.conf file on the Salt master. The values need to be specified in bytes.

   [global]
   bluestore block db size = 48318382080
   bluestore block wal size = 2147483648

   For more information on customizing the ceph.conf file, refer to Book “Administration Guide”, Chapter 1 “Salt Cluster Administration”, Section 1.12 “Adjusting ceph.conf with Custom Settings”.

2. Run DeepSea Stage 3 to distribute the changes:

   root@master # salt-run state.orch ceph.stage.3

3. Verify that the ceph.conf file is updated on the relevant OSD nodes:

   root@minion > cat /etc/ceph/ceph.conf

4. Edit the *.yml files in the /srv/pillar/ceph/proposals/profile-default/stack/default/ceph/minions directory that are relevant to the OSDs you are encrypting. Double check their path with the one defined in the /srv/pillar/ceph/proposals/policy.cfg file to ensure that you modify the correct *.yml files.

   

   Important: Long Disk Identifiers

   When identifying OSD disks in the /srv/pillar/ceph/proposals/profile-default/stack/default/ceph/minions/*.yml files, use long disk identifiers.

   An example of an OSD configuration follows. Note that because we need encryption, the db_size and wal_size options are removed:

   ceph:
    storage:
      osds:
        /dev/disk/by-id/scsi-SDELL_PERC_H730_Mini_007027b1065faa972100d34d7aa06d86:
          format: bluestore
          encryption: dmcrypt
          db: /dev/disk/by-id/nvme-INTEL_SSDPEDMD020T4D_HHHL_NVMe_2000GB_PHFT642400HV2P0EGN
          wal: /dev/disk/by-id/nvme-INTEL_SSDPEDMD020T4D_HHHL_NVMe_2000GB_PHFT642400HV2P0EGN
        /dev/disk/by-id/scsi-SDELL_PERC_H730_Mini_00d146b1065faa972100d34d7aa06d86:
          format: bluestore
          encryption: dmcrypt
          db: /dev/disk/by-id/nvme-INTEL_SSDPEDMD020T4D_HHHL_NVMe_2000GB_PHFT642400HV2P0EGN
          wal: /dev/disk/by-id/nvme-INTEL_SSDPEDMD020T4D_HHHL_NVMe_2000GB_PHFT642400HV2P0EGN

5. Deploy the new Block Storage OSDs with encryption by running DeepSea Stages 2 and 3:

   root@master # salt-run state.orch ceph.stage.2
   root@master # salt-run state.orch ceph.stage.3

   You can watch the progress with ceph -s or ceph osd tree. It is critical that you let the cluster rebalance before repeating the process on the next OSD node.

4.5.1.7 Item Filtering #Edit source

Sometimes it is not practical to include all files from a given directory with *.sls globbing. The policy.cfg file parser understands the following filters:

Warning: Advanced Techniques

This section describes filtering techniques for advanced users. When not used correctly, filtering can cause problems for example in case your node numbering changes.

slice=[start:end]

Use the slice filter to include only items start through end-1. Note that items in the given directory are sorted alphanumerically. The following line includes the third to fifth files from the role-mon/cluster/ subdirectory:

role-mon/cluster/*.sls slice[3:6]

re=regexp

Use the regular expression filter to include only items matching the given expressions. For example:

role-mon/cluster/mon*.sls re=.*1[135]\.subdomainX\.sls$

## Cluster Assignment
cluster-ceph/cluster/*.sls 1

## Roles
# ADMIN
role-master/cluster/examplesesadmin.sls 2
role-admin/cluster/sesclient*.sls 3

# MON
role-mon/cluster/ses-example-[123].sls 4

# MGR
role-mgr/cluster/ses-example-[123].sls 5

# MDS
role-mds/cluster/ses-example-4.sls 6

# IGW
role-igw/cluster/ses-example-4.sls 7

# RGW
role-rgw/cluster/ses-example-4.sls 8

# openATTIC
role-openattic/cluster/openattic*.sls 9

# COMMON
config/stack/default/global.yml 10
config/stack/default/ceph/cluster.yml 11

## Profiles
profile-default/cluster/*.sls 12
profile-default/stack/default/ceph/minions/*.yml 13

cluster-unassigned/cluster/*.sls
cluster-ceph/cluster/ses-example-*.sls

If you need to put custom settings into the ceph.conf configuration file, see Book “Administration Guide”, Chapter 1 “Salt Cluster Administration”, Section 1.12 “Adjusting ceph.conf with Custom Settings” for more details.

OSD BlueStore is a new back end for the OSD daemons. It is the default option since SUSE Enterprise Storage 5.5. Compared to FileStore, which stores objects as files in an XFS file system, BlueStore can deliver increased performance because it stores objects directly on the underlying block device. BlueStore also enables other features, such as built-in compression and EC overwrites, that are unavailable with FileStore.

Specifically for BlueStore, an OSD has a 'wal' (Write Ahead Log) device and a 'db' (RocksDB database) device. The RocksDB database holds the metadata for a BlueStore OSD. These two devices will reside on the same device as an OSD by default, but either can be placed on faster/different media.

In SES5, both FileStore and BlueStore are supported and it is possible for FileStore and BlueStore OSDs to co-exist in a single cluster. During the SUSE Enterprise Storage upgrade procedure, FileStore OSDs are not automatically converted to BlueStore. Be aware that the BlueStore-specific features will not be available on OSDs that have not been migrated to BlueStore.

Before converting to BlueStore, the OSDs need to be running SUSE Enterprise Storage 5.5. The conversion is a slow process as all data gets re-written twice. Though the migration process can take a long time to complete, there is no cluster outage and all clients can continue accessing the cluster during this period. However, do expect lower performance for the duration of the migration. This is caused by rebalancing and backfilling of cluster data.

Use the following procedure to migrate FileStore OSDs to BlueStore:

Tip: Turn Off Safety Measures

Salt commands needed for running the migration are blocked by safety measures. In order to turn these precautions off, run the following command:

root@master # salt-run disengage.safety

After the migration to BlueStore, the object count will remain the same and disk usage will be nearly the same.

During an upgrade via salt targetceph.maintenance.upgrade, DeepSea applies all available updates/patches on all servers in the cluster in parallel without rebooting them. After these updates/patches are applied, the actual upgrade begins:

In case this process is interrupted by an accident or intentionally by the administrator, never reboot the nodes manually because after rebooting the first OSD node and OSD daemon, it will not be able to join the cluster anymore.

If you address a specific task outside of the DeepSea deployment process and therefore need to skip it, create a 'no-operation' file following this example:

If you need to replace the default behavior of a specific step with a custom one, create a custom sls file with replacement content.

By default /srv/salt/ceph/pool/default.sls creates an rbd image called 'demo'. In our example, we do not want this image to be created, but we need two images: 'archive1' and 'archive2'.

Note: Authorization

The creation of pools or images requires sufficient authorization. The admin.ceph minion has an admin keyring.

Tip: Alternative Way

Another option is to change the variable in /srv/pillar/ceph/stack/ceph/roles/master.yml instead. Using this file will reduce the clutter of pillar data for other minions.

Sometimes you may need a specific step to do some additional tasks. We do not recommend modifying the related state file as it may complicate a future upgrade. Instead, create a separate file to carry out the additional tasks identical to what was described in Section 7.1.2, “Replacing a Deployment Step”.

Name the new sls file descriptively. For example, if you need to create two rbd images in addition to the demo image, name the file archive.sls.

If you need to add a completely separate deployment step, create three new files—an sls file that performs the command, an orchestration file, and a custom file which aligns the new step with the original deployment steps.

For example, if you need to run logrotate on all minions as part of the preparation stage:

First create an sls file and include the logrotate command.

Because the orchestration file needs to run before all other preparation steps, add it to the Prep stage 0:

The last file is the custom one which includes the additional step with the original steps:

Note: Why global.yml?

The global.yml file is chosen over the cluster.yml because during the prep stage, no minion belongs to the Ceph cluster and has no access to any settings in cluster.yml.

During Stage 0 (refer to DeepSea Stages Description for more information on DeepSea stages), the Salt master and Salt minions may optionally reboot because newly updated packages, for example kernel, require rebooting the system.

The default behavior is to install available new updates and not reboot the nodes even in case of kernel updates.

You can change the default update/reboot behavior of DeepSea Stage 0 by adding/changing the stage_prep_master and stage_prep_minion options in the /srv/pillar/ceph/stack/global.yml file. stage_prep_master sets the behavior of the Salt master, and stage_prep_minion sets the behavior of all minions. All available parameters are:

For example, to prevent the cluster nodes from installing updates and rebooting, edit /srv/pillar/ceph/stack/global.yml and add the following lines:

stage_prep_master: default-no-update-no-reboot
stage_prep_minion: default-no-update-no-reboot

Tip: Values and Corresponding Files

The values of stage_prep_master correspond to file names located in /srv/salt/ceph/stage/0/master, while values of stage_prep_minion correspond to files in /srv/salt/ceph/stage/0/minion:

cephadm > ls -l /srv/salt/ceph/stage/0/master
default-no-update-no-reboot.sls
default-no-update-reboot.sls
default-update-reboot.sls
[...]

cephadm > ls -l /srv/salt/ceph/stage/0/minion
default-no-update-no-reboot.sls
default-no-update-reboot.sls
default-update-reboot.sls
[...]

Several steps are required to configure an Object Gateway.

9.1.1.1 Basic Configuration #Edit source

Configuring a Ceph Object Gateway requires a running Ceph Storage Cluster. The Ceph Object Gateway is a client of the Ceph Storage Cluster. As a Ceph Storage Cluster client, it requires:

• A host name for the gateway instance, for example gateway.

• A storage cluster user name with appropriate permissions and a keyring.

• Pools to store its data.

• A data directory for the gateway instance.

• An instance entry in the Ceph configuration file.

Each instance must have a user name and key to communicate with a Ceph storage cluster. In the following steps, we use a monitor node to create a bootstrap keyring, then create the Object Gateway instance user keyring based on the bootstrap one. Then, we create a client user name and key. Next, we add the key to the Ceph Storage Cluster. Finally, we distribute the keyring to the node containing the gateway instance.

1. Create a keyring for the gateway:

   root # ceph-authtool --create-keyring /etc/ceph/ceph.client.rgw.keyring
   root # chmod +r /etc/ceph/ceph.client.rgw.keyring

2. Generate a Ceph Object Gateway user name and key for each instance. As an example, we will use the name gateway after client.radosgw:

   root # ceph-authtool /etc/ceph/ceph.client.rgw.keyring \
     -n client.rgw.gateway --gen-key

3. Add capabilities to the key:

   root # ceph-authtool -n client.rgw.gateway --cap osd 'allow rwx' \
     --cap mon 'allow rwx' /etc/ceph/ceph.client.rgw.keyring

4. Once you have created a keyring and key to enable the Ceph Object Gateway with access to the Ceph Storage Cluster, add the key to your Ceph Storage Cluster. For example:

   root # ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.rgw.gateway \
     -i /etc/ceph/ceph.client.rgw.keyring

5. Distribute the keyring to the node with the gateway instance:

   root # scp /etc/ceph/ceph.client.rgw.keyring  ceph@hostname:/home/ceph
   cephadm > ssh hostname
   root # mv ceph.client.rgw.keyring /etc/ceph/ceph.client.rgw.keyring

Tip: Use Bootstrap Keyring

An alternative way is to create the Object Gateway bootstrap keyring, and then create the Object Gateway keyring from it:

1. Create an Object Gateway bootstrap keyring on one of the monitor nodes:

   root # ceph \
    auth get-or-create client.bootstrap-rgw mon 'allow profile bootstrap-rgw' \
    --connect-timeout=25 \
    --cluster=ceph \
    --name mon. \
    --keyring=/var/lib/ceph/mon/ceph-node_host/keyring \
    -o /var/lib/ceph/bootstrap-rgw/keyring

2. Create the /var/lib/ceph/radosgw/ceph-rgw_name directory for storing the bootstrap keyring:

   root # mkdir \
   /var/lib/ceph/radosgw/ceph-rgw_name

3. Create an Object Gateway keyring from the newly created bootstrap keyring:

   root # ceph \
    auth get-or-create client.rgw.rgw_name osd 'allow rwx' mon 'allow rw' \
    --connect-timeout=25 \
    --cluster=ceph \
    --name client.bootstrap-rgw \
    --keyring=/var/lib/ceph/bootstrap-rgw/keyring \
    -o /var/lib/ceph/radosgw/ceph-rgw_name/keyring

4. Copy the Object Gateway keyring to the Object Gateway host:

   root # scp \
   /var/lib/ceph/radosgw/ceph-rgw_name/keyring \
   rgw_host:/var/lib/ceph/radosgw/ceph-rgw_name/keyring

9.1.1.2 Create Pools (Optional) #Edit source

Ceph Object Gateways require Ceph Storage Cluster pools to store specific gateway data. If the user you created has proper permissions, the gateway will create the pools automatically. However, ensure that you have set an appropriate default number of placement groups per pool in the Ceph configuration file.

The pool names follow the ZONE_NAME.POOL_NAME syntax. When configuring a gateway with the default region and zone, the default zone name is 'default' as in our example:

.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.data

To create the pools manually, see Book “Administration Guide”, Chapter 8 “Managing Storage Pools”, Section 8.2.2 “Create a Pool”.

Important: Object Gateway and Erasure-coded Pools

Only the default.rgw.buckets.data pool can be erasure-coded. All other pools need to be replicated, otherwise the gateway is not accessible.

9.1.1.3 Adding Gateway Configuration to Ceph #Edit source

Add the Ceph Object Gateway configuration to the Ceph Configuration file. The Ceph Object Gateway configuration requires you to identify the Ceph Object Gateway instance. Then, specify the host name where you installed the Ceph Object Gateway daemon, a keyring (for use with cephx), and optionally a log file. For example:

[client.rgw.instance-name]
host = hostname
keyring = /etc/ceph/ceph.client.rgw.keyring

Tip: Object Gateway Log File

To override the default Object Gateway log file, include the following:

log file = /var/log/radosgw/client.rgw.instance-name.log

The [client.rgw.*] portion of the gateway instance identifies this portion of the Ceph configuration file as configuring a Ceph Storage Cluster client where the client type is a Ceph Object Gateway (radosgw). The instance name follows. For example:

[client.rgw.gateway]
host = ceph-gateway
keyring = /etc/ceph/ceph.client.rgw.keyring

Note

The host must be your machine host name, excluding the domain name.

Then turn off print continue. If you have it set to true, you may encounter problems with PUT operations:

rgw print continue = false

To use a Ceph Object Gateway with subdomain S3 calls (for example http://bucketname.hostname), you must add the Ceph Object Gateway DNS name under the [client.rgw.gateway] section of the Ceph configuration file:

[client.rgw.gateway]
...
rgw dns name = hostname

You should also consider installing a DNS server such as Dnsmasq on your client machine(s) when using the http://bucketname.hostname syntax. The dnsmasq.conf file should include the following settings:

address=/hostname/host-ip-address
listen-address=client-loopback-ip

Then, add the client-loopback-ip IP address as the first DNS server on the client machine(s).

root # mkdir -p /var/lib/ceph/radosgw/cluster-id

root # mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.gateway

<ListAllMyBucketsResult>
      <Owner>
              <ID>anonymous</ID>
              <DisplayName/>
      </Owner>
      <Buckets/>
</ListAllMyBucketsResult>

The Linux kernel iSCSI target was originally named LIO for linux-iscsi.org, the project's original domain and Web site. For some time, no fewer than four competing iSCSI target implementations were available for the Linux platform, but LIO ultimately prevailed as the single iSCSI reference target. The mainline kernel code for LIO uses the simple, but somewhat ambiguous name "target", distinguishing between "target core" and a variety of front-end and back-end target modules.

The most commonly used front-end module is arguably iSCSI. However, LIO also supports Fibre Channel (FC), Fibre Channel over Ethernet (FCoE) and several other front-end protocols. At this time, only the iSCSI protocol is supported by SUSE Enterprise Storage.

The most frequently used target back-end module is one that is capable of simply re-exporting any available block device on the target host. This module is named iblock. However, LIO also has an RBD-specific back-end module supporting parallelized multipath I/O access to RBD images.

This section introduces brief information on iSCSI initiators used on Linux, Microsoft Windows, and VMware platforms.

You can deploy the iSCSI Gateway either during Ceph cluster deployment process, or add it to an existing cluster using DeepSea.

To include the iSCSI Gateway during the cluster deployment process, refer to Section 4.5.1.2, “Role Assignment”.

To add the iSCSI Gateway to an existing cluster, refer to Book “Administration Guide”, Chapter 1 “Salt Cluster Administration”, Section 1.2 “Adding New Roles to Nodes”.

RBD images are created in the Ceph store and subsequently exported to iSCSI. We recommend that you use a dedicated RADOS pool for this purpose. You can create a volume from any host that is able to connect to your storage cluster using the Ceph rbd command line utility. This requires the client to have at least a minimal ceph.conf configuration file, and appropriate CephX authentication credentials.

To create a new volume for subsequent export via iSCSI, use the rbd create command, specifying the volume size in megabytes. For example, in order to create a 100 GB volume named testvol in the pool named iscsi, run:

cephadm > rbd --pool iscsi create --size=102400 testvol

The above command creates an RBD volume in the default format 2.

Note

Since SUSE Enterprise Storage 3, the default volume format is 2, and format 1 is deprecated. However, you can still create the deprecated format 1 volumes with the --image-format 1 option.

To export RBD images via iSCSI, use the lrbd utility. lrbd allows you to create, review, and modify the iSCSI target configuration, which uses a JSON format.

Tip: Import Changes into openATTIC

Any changes made to the iSCSI Gateway configuration using the lrbd command are not visible to DeepSea and openATTIC. To import your manual changes, you need to export the iSCSI Gateway configuration to a file:

root@minion > lrbd -o /tmp/lrbd.conf

Then copy it to the Salt master so that DeepSea and openATTIC can see it:

root@minion > scp /tmp/lrbd.conf ses5master:/srv/salt/ceph/igw/cache/lrbd.conf

Finally, edit /srv/pillar/ceph/stack/global.yml and set:

igw_config: default-ui

In order to edit the configuration, use lrbd -e or lrbd --edit. This command will invoke the default editor, as defined by the EDITOR environment variable. You may override this behavior by setting the -E option in addition to -e.

Below is an example configuration for

{
    "auth": [
        {
            "target": "iqn.2003-01.org.linux-iscsi.iscsi.x86:testvol",
            "authentication": "none"
        }
    ],
    "targets": [
        {
            "target": "iqn.2003-01.org.linux-iscsi.iscsi.x86:testvol",
            "hosts": [
                {
                    "host": "iscsi1.example.com",
                    "portal": "east"
                },
                {
                    "host": "iscsi2.example.com",
                    "portal": "west"
                }
            ]
        }
    ],
    "portals": [
        {
            "name": "east",
            "addresses": [
                "192.168.124.104"
            ]
        },
        {
            "name": "west",
            "addresses": [
                "192.168.124.105"
            ]
        }
    ],
    "pools": [
        {
            "pool": "rbd",
            "gateways": [
                {
                    "target": "iqn.2003-01.org.linux-iscsi.iscsi.x86:testvol",
                    "tpg": [
                        {
                            "image": "testvol"
                        }
                    ]
                }
            ]
        }
    ]
    }

Note that whenever you refer to a host name in the configuration, this host name must match the iSCSI gateway's uname -n command output.

The edited JSON is stored in the extended attributes (xattrs) of a single RADOS object per pool. This object is available to the gateway hosts where the JSON is edited, as well as to all gateway hosts connected to the same Ceph cluster. No configuration information is stored locally on the lrbd gateway.

To activate the configuration, store it in the Ceph cluster, and do one of the following things (as root):

or

The lrbd "service" does not operate any background daemon. Instead, it simply invokes the lrbd command. This type of service is known as a "one-shot" service.

You should also enable lrbd to auto-configure on system start-up. To do so, run the systemctl enable lrbd command.

The configuration above reflects a simple, one-gateway setup. lrbd configuration can be much more complex and powerful. The lrbd RPM package comes with an extensive set of configuration examples, which you may refer to by checking the content of the /usr/share/doc/packages/lrbd/samples directory after installation. The samples are also available from https://github.com/SUSE/lrbd/tree/master/samples.

iSCSI authentication is flexible and covers many possibilities. The five possible top level settings are none, tpg, acls, tpg+identified and identified.

{
    "host": "igw1",
    "authentication": none
}

{
  "target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant",
  "authentication": tpg,
  "tpg": {
      "userid": "common1",
      "password": "pass1"
  }
},
{
    "host": "igw1",
    "authentication": tpg,
    "tpg": {
        "userid": "common1",
        "password": "pass1"
    }
}

{
    "host": "igw1",
    "authentication": acls,
    "acls": [
        {
            "initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
            "userid": "initiator1",
            "password": "pass1",
        }
    ]
},

{
  "target": "iqn.2003-01.org.linux-iscsi.igw.x86:sn.redundant",
  "authentication": tpg+identified,
  "tpg": {
      "userid": "common1",
      "password": "pass1"
  }
},
{
    "host": "igw1",
    "authentication": tpg+identified,
    "tpg": {
        "userid": "common1",
        "password": "pass1"
    }
}

{
    "target": "iqn.2003-01.org.linux-iscsi:igw.x86:sn.redundant",
    "authentication": "identified",
},
{
    "host": "igw1",
    "authentication": "identified",
}

The following settings may be useful for some environments. For images, there are uuid, lun, retries, sleep, and retry_errors attributes. The first two—uuid and lun—allow hardcoding of the 'uuid' or 'lun' for a specific image. You can specify either of them for an image. The retries, sleep and retry_errors affect attempts to map an rbd image.

Tip

If a site needs statically assigned LUNs, then assign numbers to each LUN.

"pools": [
    {
        "pool": "rbd",
        "gateways": [
        {
        "host": "igw1",
        "tpg": [
                    {
                        "image": "archive",
                        "uuid": "12345678-abcd-9012-efab-345678901234",
                        "lun": "2",
                        "retries": "3",
                        "sleep": "4",
                        "retry_errors": [ 95 ],
                        [...]
                    }
                ]
            }
        ]
    }
]

lrbd can be configured with advanced parameters which are subsequently passed on to the LIO I/O target. The parameters are divided up into iSCSI and backing store components, which can then be specified in the "targets" and "tpg" sections, respectively, of the lrbd configuration.

Warning

Unless otherwise noted, changing these parameters from the default setting is not recommended.

"targets": [
    {
        [...]
        "tpg_default_cmdsn_depth": "64",
        "tpg_default_erl": "0",
        "tpg_login_timeout": "10",
        "tpg_netif_timeout": "2",
        "tpg_prod_mode_write_protect": "0",
    }
]

A description of the options follows:

"pools": [
    {
        "pool": "rbd",
        "gateways": [
        {
        "host": "igw1",
        "tpg": [
                    {
                        "image": "archive",
                        "backstore_block_size": "512",
                        "backstore_emulate_3pc": "1",
                        "backstore_emulate_caw": "1",
                        "backstore_emulate_dpo": "0",
                        "backstore_emulate_fua_read": "0",
                        "backstore_emulate_fua_write": "1",
                        "backstore_emulate_model_alias": "0",
                        "backstore_emulate_pr": "1",
                        "backstore_emulate_rest_reord": "0",
                        "backstore_emulate_tas": "1",
                        "backstore_emulate_tpu": "0",
                        "backstore_emulate_tpws": "0",
                        "backstore_emulate_ua_intlck_ctrl": "0",
                        "backstore_emulate_write_cache": "0",
                        "backstore_enforce_pr_isids": "1",
                        "backstore_fabric_max_sectors": "8192",
                        "backstore_hw_block_size": "512",
                        "backstore_hw_max_sectors": "8192",
                        "backstore_hw_pi_prot_type": "0",
                        "backstore_hw_queue_depth": "128",
                        "backstore_is_nonrot": "1",
                        "backstore_max_unmap_block_desc_count": "1",
                        "backstore_max_unmap_lba_count": "8192",
                        "backstore_max_write_same_len": "65535",
                        "backstore_optimal_sectors": "8192",
                        "backstore_pi_prot_format": "0",
                        "backstore_pi_prot_type": "0",
                        "backstore_queue_depth": "128",
                        "backstore_unmap_granularity": "8192",
                        "backstore_unmap_granularity_alignment": "4194304"
                    }
                ]
            }
        ]
    }
]

A description of the options follows:

For targets, the tpg attributes allow tuning of kernel parameters. Use with caution.

"targets": [
{
    "host": "igw1",
    "target": "iqn.2003-01.org.linux-iscsi.generic.x86:sn.abcdefghijk",
    "tpg_default_cmdsn_depth": "64",
    "tpg_default_erl": "0",
    "tpg_login_timeout": "10",
    "tpg_netif_timeout": "2",
    "tpg_prod_mode_write_protect": "0",
    "tpg_t10_pi": "0"
}

For initiators, the attrib and param settings allow the tuning of kernel parameters. Use with caution. These are set in the authentication section. If the authentication is tpg+identified or identified, then the subsection is identified.

"auth": [
  {
      "authentication": "tpg+identified",
      "identified": [
        {
          "initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
          "attrib_dataout_timeout": "3",
          "attrib_dataout_timeout_retries": "5",
          "attrib_default_erl": "0",
          "attrib_nopin_response_timeout": "30",
          "attrib_nopin_timeout": "15",
          "attrib_random_datain_pdu_offsets": "0",
          "attrib_random_datain_seq_offsets": "0",
          "attrib_random_r2t_offsets": "0",
          "param_DataPDUInOrder": "1",
          "param_DataSequenceInOrder": "1",
          "param_DefaultTime2Retain": "0",
          "param_DefaultTime2Wait": "2",
          "param_ErrorRecoveryLevel": "0",
          "param_FirstBurstLength": "65536",
          "param_ImmediateData": "1",
          "param_InitialR2T": "1",
          "param_MaxBurstLength": "262144",
          "param_MaxConnections": "1",
          "param_MaxOutstandingR2T": "1"
        }
      ]
  }
]

If the authentication is acls, then the settings are included in the acls subsection. One caveat is that settings are only applied for active initiators. If an initiator is absent from the pools section, the acl entry is not created and settings cannot be applied.

On your iSCSI Gateway node, install the tcmu-runner-handler-rbd package from the SUSE Enterprise Storage 5 media, together with the libtcmu1 and tcmu-runner package dependencies. Install the targetcli-fb package for configuration purposes. Note that the targetcli-fb package is incompatible with the 'non-fb' version of the targetcli package.

Confirm that the tcmu-runner systemd service is running:

root # systemctl enable tcmu-runner
tcmu-gw:~ # systemctl status tcmu-runner
● tcmu-runner.service - LIO Userspace-passthrough daemon
  Loaded: loaded (/usr/lib/systemd/system/tcmu-runner.service; static; vendor
  preset: disabled)
    Active: active (running) since ...

Create a RADOS Block Device image on your existing Ceph cluster. In the following example, we will use a 10G image called 'tcmu-lu' located in the 'rbd' pool.

Following RADOS Block Device image creation, run targetcli, and ensure that the tcmu-runner RBD handler (plug-in) is available:

root # targetcli
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ................................... [...]
  o- backstores ........................ [...]
...
  | o- user:rbd ......... [Storage Objects: 0]

Create a backstore configuration entry for the RBD image:

/> cd backstores/user:rbd
/backstores/user:rbd> create tcmu-lu 10G /rbd/tcmu-lu
Created user-backed storage object tcmu-lu size 10737418240.

Create an iSCSI transport configuration entry. In the following example, the target IQN "iqn.2003-01.org.linux-iscsi.tcmu-gw.x8664:sn.cb3d2a3a" is automatically generated by targetcli for use as a unique iSCSI target identifier:

/backstores/user:rbd> cd /iscsi
/iscsi> create
Created target iqn.2003-01.org.linux-iscsi.tcmu-gw.x8664:sn.cb3d2a3a.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.

Create an ACL entry for the iSCSI initiator(s) that you want to connect to the target. In the following example, an initiator IQN of "iqn.1998-01.com.vmware:esxi-872c4888" is used:

/iscsi> cd
iqn.2003-01.org.linux-iscsi.tcmu-gw.x8664:sn.cb3d2a3a/tpg1/acls/
/iscsi/iqn.20...a3a/tpg1/acls> create iqn.1998-01.com.vmware:esxi-872c4888

Finally, link the previously created RBD backstore configuration to the iSCSI target:

/iscsi/iqn.20...a3a/tpg1/acls> cd ../luns
/iscsi/iqn.20...a3a/tpg1/luns> create /backstores/user:rbd/tcmu-lu
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.1998-01.com.vmware:esxi-872c4888

Exit the shell to save the existing configuration:

/iscsi/iqn.20...a3a/tpg1/luns> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json

From your iSCSI initiator (client) node, connect to your newly provisioned iSCSI target using the IQN and host name configured above.

"auth": [
    {
        "host": "igw1",
        "authentication": "acls",
        "acls": [
            {
                "initiator": "iqn.1996-04.de.suse:01:e6ca28cc9f20",
                "userid": "initiator1",
                "password": "pass1",
                "attrib_dataout_timeout": "3",
                "attrib_dataout_timeout_retries": "5",
                "attrib_default_erl": "0",
                "attrib_nopin_response_timeout": "30",
                "attrib_nopin_timeout": "15",
                "attrib_random_datain_pdu_offsets": "0",
                "attrib_random_datain_seq_offsets": "0",
                "attrib_random_r2t_offsets": "0",
                "param_DataPDUInOrder": "1",
                "param_DataSequenceInOrder": "1",
                "param_DefaultTime2Retain": "0",
                "param_DefaultTime2Wait": "2",
                "param_ErrorRecoveryLevel": "0",
                "param_FirstBurstLength": "65536",
                "param_ImmediateData": "1",
                "param_InitialR2T": "1",
                "param_MaxBurstLength": "262144",
                "param_MaxConnections": "1",
                "param_MaxOutstandingR2T": "1"
            }
        ]
    },
]

You can deploy MDS either during the initial cluster deployment process as described in Section 4.3, “Cluster Deployment”, or add it to an already deployed cluster as described in Book “Administration Guide”, Chapter 1 “Salt Cluster Administration”, Section 1.1 “Adding New Cluster Nodes”.

After you deploy your MDS, allow the Ceph OSD/MDS service in the firewall setting of the server where MDS is deployed: Start yast, navigate to Security and Users › Firewall › Allowed Services and in the Service to Allow drop–down menu select Ceph OSD/MDS. If the Ceph MDS node is not allowed full traffic, mounting of a file system fails, even though other operations may work properly.

You can fine-tune the MDS behavior by inserting relevant options in the ceph.conf configuration file.

For a detailed list of MDS related configuration options, see http://docs.ceph.com/docs/master/cephfs/mds-config-ref/.

For a detailed list of MDS journaler configuration options, see http://docs.ceph.com/docs/master/cephfs/journaler/.

A CephFS requires at least two RADOS pools: one for data and one for metadata. When configuring these pools, you might consider:

When assigning a role-mds in the policy.cfg, the required pools are automatically created. You can manually create the pools cephfs_data and cephfs_metadata for manual performance tuning before setting up the Metadata Server. DeepSea will not create these pools if they already exist.

For more information on managing pools, see Book “Administration Guide”, Chapter 8 “Managing Storage Pools”.

To create the two required pools—for example, 'cephfs_data' and 'cephfs_metadata'—with default settings for use with CephFS, run the following commands:

cephadm > ceph osd pool create cephfs_data pg_num
cephadm > ceph osd pool create cephfs_metadata pg_num

It is possible to use EC pools instead of replicated pools. We recommend to only use EC pools for low performance requirements and infrequent random access, for example cold storage, backups, archiving. CephFS on EC pools requires BlueStore to be enabled and the pool must have the allow_ec_overwrite option set. This option can be set by running ceph osd pool set ec_pool allow_ec_overwrites true.

Erasure coding adds significant overhead to file system operations, especially small updates. This overhead is inherent to using erasure coding as a fault tolerance mechanism. This penalty is the trade off for significantly reduced storage space overhead.

When the pools are created, you may enable the file system with the ceph fs new command:

cephadm > ceph fs new fs_name metadata_pool_name data_pool_name

For example:

cephadm > ceph fs new cephfs cephfs_metadata cephfs_data

You can check that the file system was created by listing all available CephFSs:

cephadm > ceph fs ls
 name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

When the file system has been created, your MDS will be able to enter an active state. For example, in a single MDS system:

cephadm > ceph mds stat
e5: 1/1/1 up

Tip: More Topics

You can find more information of specific tasks—for example mounting, unmounting, and advanced CephFS setup—in Book “Administration Guide”, Chapter 15 “Clustered File System”.

A CephFS instance can be served by multiple active MDS daemons. All active MDS daemons that are assigned to a CephFS instance will distribute the file system's directory tree between themselves, and thus spread the load of concurrent clients. In order to add an active MDS daemon to a CephFS instance, a spare standby is needed. Either start an additional daemon or use an existing standby instance.

The following command will display the current number of active and passive MDS daemons.

cephadm > ceph mds stat

The following command sets the number of active MDS's to two in a file system instance.

cephadm > ceph fs set fs_name max_mds 2

In order to shrink the MDS cluster prior to an update, two steps are necessary. First set max_mds so that only one instance remains:

cephadm > ceph fs set fs_name max_mds 1

and after that explicitly deactivate the other active MDS daemons:

cephadm > ceph mds deactivate fs_name:rank

where rank is the number of an active MDS daemon of a file system instance, ranging from 0 to max_mds-1. See http://docs.ceph.com/docs/luminous/cephfs/multimds/ for additional information.

During Ceph updates, the feature flags on a file system instance may change (usually by adding new features). Incompatible daemons (such as the older versions) are not able to function with an incompatible feature set and will refuse to start. This means that updating and restarting one daemon can cause all other not yet updated daemons to stop and refuse to start. For this reason we, recommend shrinking the active MDS cluster to size one and stopping all standby daemons before updating Ceph. The manual steps for this update procedure are as follows:

If you use DeepSea, it will follow this procedure in case the ceph package was updated during Stages 0 and 4. It is possible to perform this procedure while clients have the CephFS instance mounted and I/O is ongoing. Note however that there will be a very brief I/O pause while the active MDS restarts. Clients will recover automatically.

It is good practice to reduce the I/O load as much as possible before updating an MDS cluster. An idle MDS cluster will go through this update procedure quicker. Conversely, on a heavily loaded cluster with multiple MDS daemons it is essential to reduce the load in advance to prevent a single MDS daemon from being overwhelmed by ongoing I/O.

To successfully deploy NFS Ganesha, you need to add a role-ganesha to your /srv/pillar/ceph/proposals/policy.cfg. For details, see Section 4.5.1, “The policy.cfg File”. NFS Ganesha also needs either a role-rgw or a role-mds present in the policy.cfg.

Although it is possible to install and run the NFS Ganesha server on an already existing Ceph node, we recommend running it on a dedicated host with access to the Ceph cluster. The client hosts are typically not part of the cluster, but they need to have network access to the NFS Ganesha server.

To enable the NFS Ganesha server at any point after the initial installation, add the role-ganesha to the policy.cfg and re-run at least DeepSea stages 2 and 4. For details, see Section 4.3, “Cluster Deployment”.

NFS Ganesha is configured via the file /etc/ganesha/ganesha.conf that exists on the NFS Ganesha node. However, this file is overwritten each time DeepSea stage 4 is executed. Therefore we recommend to edit the template used by Salt, which is the file /srv/salt/ceph/ganesha/files/ganesha.conf.j2 on the Salt master. For details about the configuration file, see Book “Administration Guide”, Chapter 16 “NFS Ganesha: Export Ceph Data via NFS”, Section 16.2 “Configuration”.

The following requirements need to be met before DeepSea stages 2 and 4 can be executed to install NFS Ganesha:

In this setup earth has the IP address 192.168.1.1 and mars has the address 192.168.1.2.

Additionally, two floating virtual IP addresses are used, allowing clients to connect to the service independent of which physical node it is running on. 192.168.1.10 is used for cluster administration with Hawk2 and 192.168.2.1 is used exclusively for the NFS exports. This makes it easier to apply security restrictions later.

The following procedure describes the example installation. More details can be found at https://documentation.suse.com/sle-ha/12-SP5/single-html/SLE-HA-install-quick/.

In the event of an NFS Ganesha failure at one of the node, for example earth, fix the issue and clean up the resource. Only after the resource is cleaned up can the resource fail back to earth in case NFS Ganesha fails at mars.

To clean up the resource:

root@earth # crm resource cleanup nfs-ganesha-clone earth
root@earth # crm resource cleanup ganesha-ip earth

It may happen that the server is unable to reach the client because of a network issue. A ping resource can detect and mitigate this problem. Configuring this resource is optional.

When a service goes down, the TCP connection that is in use by NFS Ganesha is required to be closed otherwise it continues to run until a system-specific timeout occurs. This timeout can take upwards of 3 minutes.

To shorten the timeout time, the TCP connection needs to be reset. We recommend configuring portblock to reset stale TCP connections.

You can choose to use portblock with or without the tickle_dir parameters that could unblock and reconnect clients to the new service faster. We recommend to have tickle_dir as the shared CephFS mount between two HA nodes (where NFS Ganesha services are running).

Note

Configuring the following resource is optional.

1. On earth, run the crm shell to execute the following commands to configure the resource for NFS Ganesha daemons:

   root@earth # crm configure

2. Configure the block action for portblock and omit the tickle_dir option if you have not configured a shared directory:

   crm(live)configure#  primitive nfs-ganesha-block ocf:portblock \
   protocol=tcp portno=2049 action=block ip=192.168.2.1 op monitor depth="0" timeout="10" interval="10" tickle_dir="/tmp/ganesha/tickle/"

3. Configure the unblock action for portblock and omit the reset_local_on_unblock_stop option if you have not configured a shared directory:

   crm(live)configure#  primitive nfs-ganesha-unblock ocf:portblock \
   protocol=tcp portno=2049 action=unblock ip=192.168.2.1 op monitor depth="0" timeout="10" interval="10" reset_local_on_unblock_stop=true tickle_dir="/tmp/ganesha/tickle/"

4. Configure the IPAddr2 resource with portblock:

   crm(live)configure#  colocation ganesha-portblock inf: ganesha-ip nfs-ganesha-block nfs-ganesha-unblock
   crm(live)configure#  edit ganesha-ip-after-nfs-ganesha-server
   order ganesha-ip-after-nfs-ganesha-server Mandatory: nfs-ganesha-block nfs-ganesha-clone ganesha-ip nfs-ganesha-unblock

5. Save your changes:

   crm(live)configure#  commit

6. Your configuration should look like this:

   crm(live)configure#  show

   "
   node 1084782956: nfs1
   node 1084783048: nfs2
   primitive ganesha-ip IPaddr2 \
           params ip=192.168.2.1 cidr_netmask=24 nic=eth0 \
           op monitor interval=10 timeout=20
   primitive nfs-ganesha-block portblock \
           params protocol=tcp portno=2049 action=block ip=192.168.2.1 \
           tickle_dir="/tmp/ganesha/tickle/" op monitor timeout=10 interval=10 depth=0
   primitive nfs-ganesha-server systemd:nfs-ganesha \
           op monitor interval=30s
   primitive nfs-ganesha-unblock portblock \
           params protocol=tcp portno=2049 action=unblock ip=192.168.2.1 \
           reset_local_on_unblock_stop=true tickle_dir="/tmp/ganesha/tickle/" \
           op monitor timeout=10 interval=10 depth=0
   clone nfs-ganesha-clone nfs-ganesha-server \
           meta interleave=true
   location cli-prefer-ganesha-ip ganesha-ip role=Started inf: nfs1
   order ganesha-ip-after-nfs-ganesha-server Mandatory: nfs-ganesha-block nfs-ganesha-clone ganesha-ip nfs-ganesha-unblock
   colocation ganesha-ip-with-nfs-ganesha-server inf: ganesha-ip nfs-ganesha-clone
   colocation ganesha-portblock inf: ganesha-ip nfs-ganesha-block nfs-ganesha-unblock
   property cib-bootstrap-options: \
           have-watchdog=false \
           dc-version=1.1.16-6.5.1-77ea74d \
           cluster-infrastructure=corosync \
           cluster-name=hacluster \
           stonith-enabled=false \
           placement-strategy=balanced \
           last-lrm-refresh=1544793779
   rsc_defaults rsc-options: \
           resource-stickiness=1 \
           migration-threshold=3
   op_defaults op-options: \
           timeout=600 \
           record-pending=true
   "

   In this example /tmp/ganesha/ is the CephFS mount on both nodes (nfs1 and nfs2):

   172.16.1.11:6789:/ganesha on /tmp/ganesha type ceph (rw,relatime,name=admin,secret=...hidden...,acl,wsize=16777216)

   The tickle directory has been initially created.

DeepSea does not support configuring NFS Ganesha HA. To prevent DeepSea from failing after NFS Ganesha HA was configured, exclude starting and stopping the NFS Ganesha service from DeepSea Stage 4:

To prevent DeepSea from restarting NFS Ganesha service on stage 4:

To configure and export a Samba share, the following packages need to be installed: samba-ceph and samba-winbind. If these packages are not installed, install them:

cephadm@smb > zypper install samba-ceph samba-winbind

In preparation for exporting a Samba share, choose an appropriate node to act as a Samba Gateway. The node needs to have access to the Ceph client network, as well as sufficient CPU, memory, and networking resources.

Failover functionality can be provided with CTDB and the SUSE Linux Enterprise High Availability Extension. Refer to Section 13.1.3, “High Availability Configuration” for more information on HA setup.

Important: Transparent Failover Not Supported

Although a multi-node Samba + CTDB deployment is more highly available compared to the single node (see Chapter 13, Exporting Ceph Data via Samba), client-side transparent failover is not supported. Applications will likely experience a short outage on Samba Gateway node failure.

This section provides an example of how to set up a two-node high availability configuration of Samba servers. The setup requires the SUSE Linux Enterprise High Availability Extension. The two nodes are called earth (192.168.1.1) and mars (192.168.1.2).

For details about SUSE Linux Enterprise High Availability Extension, see https://documentation.suse.com/sle-ha/12-SP5/.

Additionally, two floating virtual IP addresses allow clients to connect to the service no matter which physical node it is running on. 192.168.1.10 is used for cluster administration with Hawk2 and 192.168.2.1 is used exclusively for the CIFS exports. This makes it easier to apply security restrictions later.

The following procedure describes the example installation. More details can be found at https://documentation.suse.com/sle-ha/12-SP5/single-html/SLE-HA-install-quick/.