The procedure of adding new nodes to the cluster is almost identical to the initial cluster node deployment described in Book “Deployment Guide”, Chapter 4 “Deploying with DeepSea/Salt”:

Tip: Prevent Rebalancing

When adding an OSD to the existing cluster, bear in mind that the cluster will be rebalancing for some time afterward. To minimize the rebalancing periods, add all OSDs you intend to add at the same time.

Additional way is to set the osd crush initial weight = 0 option in the ceph.conf file before adding the OSDs:

1. Add osd crush initial weight = 0 to /srv/salt/ceph/configuration/files/ceph.conf.d/global.conf.

2. Create the new configuration:

   root@master # salt MASTER state.apply ceph.configuration.create

   Or:

   root@master # salt-call state.apply ceph.configuration.create

3. Apply the new configuration:

   root@master # salt TARGET state.apply ceph.configuration

   

   Note

   If this is not a new node, but you want to proceed as if it were, ensure you remove the /etc/ceph/destroyedOSDs.yml file from the node. Otherwise, any devices from the first attempt will be restored with their previous OSD ID and reweight.

   Run the following commands:

   root@master # salt-run state.orch ceph.stage.1
   root@master # salt-run state.orch ceph.stage.2
   root@master # salt 'node*' state.apply ceph.osd

4. After the new OSDs are added, adjust their weights as required with the ceph osd reweight command in small increments. This allows the cluster to rebalance and become healthy between increasing increments so it does not overwhelm the cluster and clients accessing the cluster.

You can deploy all types of supported roles with DeepSea. See Book “Deployment Guide”, Chapter 4 “Deploying with DeepSea/Salt”, Section 4.5.1.2 “Role Assignment” for more information on supported role types and examples of matching them.

To add a new service to an existing node, follow these steps:

Tip: Removing a Cluster Node Temporarily

The Salt master expects all minions to be present in the cluster and responsive. If a minion breaks and is not responsive any more, it causes problems to the Salt infrastructure, mainly to DeepSea and openATTIC.

Before you fix the minion, delete its key from the Salt master temporarily:

root@master # salt-key -d MINION_HOST_NAME

After the minions is fixed, add its key to the Salt master again:

root@master # salt-key -a MINION_HOST_NAME

To remove a role from a cluster, edit /srv/pillar/ceph/proposals/policy.cfg and remove the corresponding line(s). Then run Stages 2 and 5 as described in Book “Deployment Guide”, Chapter 4 “Deploying with DeepSea/Salt”, Section 4.3 “Cluster Deployment”.

Note: Removing OSDs from Cluster

In case you need to remove a particular OSD node from your cluster, ensure that your cluster has more free disk space than the disk you intend to remove. Bear in mind that removing an OSD results in rebalancing of the whole cluster.

Before running stage.5 to do the actual removal, always check which OSD's are going to be removed by DeepSea:

root@master # salt-run rescinded.ids

When a role is removed from a minion, the objective is to undo all changes related to that role. For most of the roles, the task is simple, but there may be problems with package dependencies. If a package is uninstalled, its dependencies are not.

Removed OSDs appear as blank drives. The related tasks overwrite the beginning of the file systems and remove backup partitions in addition to wiping the partition tables.

Note: Preserving Partitions Created by Other Methods

Disk drives previously configured by other methods, such as ceph-deploy, may still contain partitions. DeepSea will not automatically destroy these. The administrator must reclaim these drives manually.

[...]
# Hardware Profile
profile-default/cluster/data*.sls
profile-default/stack/default/ceph/minions/data*.yml
[...]

[...]
# Hardware Profile
profile-default/cluster/data[1,3-6]*.sls
profile-default/stack/default/ceph/minions/data[1,3-6]*.yml
[...]

root@master # salt-run state.orch ceph.stage.2
root@master # salt-run rescinded.ids
root@master # salt-run state.orch ceph.stage.5

# Hardware Profile
profile-default/cluster/data[1-7]*.sls
profile-default/stack/default/ceph/minions/data[1-7]*.sls

# Roles
role-rgw/cluster/data8*.sls

# Hardware Profile
profile-default/cluster/data[1-8]*.sls
profile-default/stack/default/ceph/minions/data[1-8]*.sls

# Roles
role-rgw/cluster/rgw1*.sls

root@master # salt-run state.orch ceph.stage.2
root@master # salt-run state.orch ceph.stage.3
root@master # salt-run state.orch ceph.stage.4
root@master # salt-run rescinded.ids
root@master # salt-run state.orch ceph.stage.5

When one or more of your monitor nodes fail and are not responding, you need to remove the failed monitors from the cluster and possibly then re-add them back in the cluster.

Important: The Minimum is Three Monitor Nodes

The number of monitor nodes must not be less than three. If a monitor node fails, and as a result your cluster has only two monitor nodes only, you need to temporarily assign the monitor role to other cluster nodes before you redeploy the failed monitor nodes. After you redeploy the failed monitor nodes, you can uninstall the temporary monitor roles.

For more information on adding new nodes/roles to the Ceph cluster, see Section 1.1, “Adding New Cluster Nodes” and Section 1.2, “Adding New Roles to Nodes”.

For more information on removing cluster nodes, refer to Section 1.3, “Removing and Reinstalling Cluster Nodes”.

There are two basic degrees of a Ceph node failure:

To add a disk to an existing OSD node, verify that any partition on the disk was removed and wiped. Refer to Step 12 in Book “Deployment Guide”, Chapter 4 “Deploying with DeepSea/Salt”, Section 4.3 “Cluster Deployment” for more details. After the disk is empty, add the disk to the YAML file of the node. The path to the file is /srv/pillar/ceph/proposals/profile-default/stack/default/ceph/minions/node_name.yml. After saving the file, run DeepSea stages 2 and 3:

root@master # salt-run state.orch ceph.stage.2
root@master # salt-run state.orch ceph.stage.3

Tip: Updated Profiles Automatically

Instead of manually editing the YAML file, DeepSea can create new profiles. To let DeepSea create new profiles, the existing profiles need to be moved:

root@master # old /srv/pillar/ceph/proposals/profile-default/
root@master # salt-run state.orch ceph.stage.1
root@master # salt-run state.orch ceph.stage.2
root@master # salt-run state.orch ceph.stage.3

We recommend verifying the suggested proposals before deploying the changes. Refer to Book “Deployment Guide”, Chapter 4 “Deploying with DeepSea/Salt”, Section 4.5.1.4 “Profile Assignment” for more details on viewing proposals.

You can remove an Ceph OSD from the cluster by running the following command:

root@master # salt-run disengage.safety
root@master # salt-run remove.osd OSD_ID

OSD_ID needs to be a number of the OSD without the osd. prefix. For example, from osd.3 only use the digit 3.

1.6.1 Removing Multiple OSDs #

Use the same procedure as mentioned in Section 1.6, “Removing an OSD” but simply supply multiple OSD IDs:

root@master # salt-run disengage.safety
safety is now disabled for cluster ceph

root@master # salt-run remove.osd 1 13 20
Removing osds 1, 13, 20 from minions
Press Ctrl-C to abort
Removing osd 1 from minion data4.ceph
Removing osd 13 from minion data4.ceph
Removing osd 20 from minion data4.ceph
Removing osd 1 from Ceph
Removing osd 13 from Ceph
Removing osd 20 from Ceph

Important: Removed OSD ID Still Present in grains

After the remove.osd command finishes, the ID of the removed OSD is still part of Salt grains and you can see it after running salt target osd.list. The reason is that if the remove.osd command partially fails on removing the data disk, the only reference to related partitions on the shared devices is in the grains. If we updated the grains immediately, then those partitions would be orphaned.

To update the grains manually, run salt target osd.retain. It is part of DeepSea Stage 3, therefore if you are going to run Stage 3 after the OSD removal, the grains get updated automatically.

Tip: Automatic Retries

You can append the timeout parameter (in seconds) after which Salt retries the OSD removal:

root@master # salt-run remove.osd 20 timeout=6
Removing osd 20 from minion data4.ceph
  Timeout expired - OSD 20 has 22 PGs remaining
Retrying...
Removing osd 20 from Ceph

1.6.2 Removing Broken OSDs Forcefully #

There are cases when removing an OSD gracefully (see Section 1.6, “Removing an OSD”) fails. This may happen for example if the OSD or its journal, Wall or DB are broken, when it suffers from hanging I/O operations, or when the OSD disk fails to unmount. In such case, you need to force the OSD removal. The following command removes both the data partition, and the journal or WAL/DB partitions:

root@master # salt target osd.remove OSD_ID force=True

Tip: Hanging Mounts

If a partition is still mounted on the disk being removed, the command will exit with the 'Unmount failed - check for processes on DEVICE' message. You can then list all processes that access the file system with the fuser -m DEVICE. If fuser returns nothing, try manual unmount DEVICE and watch the output of dmesg or journalctl commands.

There are several reasons why you may need to replace an OSD disk, for example:

The replacement procedure is the same for both cases. It is also valid for both default and customized CRUSH Maps.

Warning: The Number of Free Disks

When doing an automated OSDs replacement, the number of free disks needs to be the same as the number of disks you need to replace. If there are more free disks available in the system, it is impossible to guess which free disks to replace. Therefore the automated replacement will not be performed.

1.7.2 Automated Configuration #

While the default profile for Stage 1 may work for the simplest setups, this stage can be optionally customized:

1. Set the stage_discovery: CUSTOM_STAGE_NAME option in /srv/pillar/ceph/stack/global.yml.

2. Create the corresponding file /srv/salt/ceph/stage/1/CUSTOM_STAGE_NAME.sls and customize it to reflect your specific requirements for Stage 1. See Appendix A, DeepSea Stage 1 Custom Example for an example.

   

   Tip: Inspect init.sls

   Inspect the /srv/salt/ceph/stage/1/init.sls file to see what variables you can use in your custom Stage 1 .sls file.

3. Refresh the Pillar:

   root@master # salt '*' saltutil.pillar_refresh

4. Run Stage 1 to generate the new configuration file:

   root@master # salt-run state.orch ceph.stage.1

Tip: Custom Options

To list all available options, inspect the output of the salt-run proposal.help command.

If you customized the cluster deployment with a specific command

salt-run proposal.populate OPTION=VALUE

use the same configuration when doing the automated configuration.

If the operating system breaks and is not recoverable on one of your OSD nodes, follow these steps to recover it and redeploy its OSD role with cluster data untouched:

The installation can be automated by using the Salt reactor. For virtual environments or consistent hardware environments, this configuration will allow the creation of a Ceph cluster with the specified behavior.

Warning

Salt cannot perform dependency checks based on reactor events. There is a real risk of putting your Salt master into a death spiral.

The automated installation requires the following:

The default reactor configuration will only run Stages 0 and 1. This allows testing of the reactor without waiting for subsequent stages to complete.

When the first salt-minion starts, Stage 0 will begin. A lock prevents multiple instances. When all minions complete Stage 0, Stage 1 will begin.

If the operation is performed properly, edit the file

/etc/salt/master.d/reactor.conf

and replace the following line

- /srv/salt/ceph/reactor/discovery.sls

with

- /srv/salt/ceph/reactor/all_stages.sls

Verify that the line is not commented out.

Keep the Ceph cluster nodes up-to-date by applying rolling updates regularly.

Important: Access to Software Repositories

Before patching the cluster with latest software packages, verify that all its nodes have access to SUSE Linux Enterprise Server repositories that match your version of SUSE Enterprise Storage. For SUSE Enterprise Storage 5.5, the following repositories are required:

root # zypper lr -E
#  | Alias   | Name                              | Enabled | GPG Check | Refresh
---+---------+-----------------------------------+---------+-----------+--------
 4 | [...]   | SUSE-Enterprise-Storage-5-Pool    | Yes     | (r ) Yes  | No
 6 | [...]   | SUSE-Enterprise-Storage-5-Updates | Yes     | (r ) Yes  | Yes
 9 | [...]   | SLES12-SP3-Pool                   | Yes     | (r ) Yes  | No
11 | [...]   | SLES12-SP3-Updates                | Yes     | (r ) Yes  | Yes

Tip: Repository Staging

If you use a staging tool—for example, SUSE Manager, Subscription Management Tool, or Repository Mirroring Tool—that serves software repositories to the cluster nodes, verify that stages for both 'Updates' repositories for SUSE Linux Enterprise Server and SUSE Enterprise Storage are created at the same point in time.

We strongly recommend to use a staging tool to apply patches which have frozen or staged patch levels. This ensures that new nodes joining the cluster have the same patch level as the nodes already running in the cluster. This way you avoid the need to apply the latest patches to all the cluster's nodes before new nodes can join the cluster.

To update the software packages on all cluster nodes to the latest version, follow these steps:

Note: Possible Downtime of Ceph Services

When applying updates to Ceph cluster nodes, Ceph services may be restarted. If there is a single point of failure for services such as Object Gateway, NFS Ganesha, or iSCSI, the client machines may be temporarily disconnected from related services.

If DeepSea detects a running Ceph cluster, it applies available updates, restarts running Ceph services, and optionally restarts nodes sequentially if a kernel update was installed. DeepSea follows Ceph's official recommendation of first updating the monitors, then the OSDs, and lastly additional services, such as Metadata Server, Object Gateway, iSCSI Gateway, or NFS Ganesha. DeepSea stops the update process if it detects an issue in the cluster. A trigger for that can be:

Making these arrangements ensures that even with corrupted or failing updates, the Ceph cluster is still operational.

DeepSea Stage 0 updates the system via zypper update and optionally reboots the system if the kernel is updated. If you want to eliminate the possibility of a forced reboot of potentially all nodes, either make sure that the latest kernel is installed and running before initiating DeepSea Stage 0, or disable automatic node reboots as described in Book “Deployment Guide”, Chapter 7 “Customizing the Default Configuration”, Section 7.1.5 “Updates and Reboots during Stage 0”.

Tip: zypper patch

If you prefer to update the system using the zypper patch command, edit /srv/pillar/ceph/stack/global.yml and add the following line:

update_method_init: zypper-patch

You can change the default update/reboot behavior of DeepSea Stage 0 by adding/changing the stage_prep_master and stage_prep_minion options. For more information, see Book “Deployment Guide”, Chapter 7 “Customizing the Default Configuration”, Section 7.1.5 “Updates and Reboots during Stage 0”.

In some cases it may be necessary to halt or reboot the whole cluster. We recommended carefully checking for dependencies of running services. The following steps provide an outline for stopping and starting the cluster:

If you need to put custom settings into the ceph.conf file, you can do so by modifying the configuration files in the /srv/salt/ceph/configuration/files/ceph.conf.d directory:

Note: Unique rgw.conf

The Object Gateway offers a lot flexibility and is unique compared to the other ceph.conf sections. All other Ceph components have static headers such as [mon] or [osd]. The Object Gateway has unique headers such as [client.rgw.rgw1]. This means that the rgw.conf file needs a header entry. For examples, see

/srv/salt/ceph/configuration/files/rgw.conf

or

/srv/salt/ceph/configuration/files/rgw-ssl.conf

Important: Run Stage 3

After you make custom changes to the above mentioned configuration files, run Stages 3 and 4 to apply these changes to the cluster nodes:

root@master # salt-run state.orch ceph.stage.3
root@master # salt-run state.orch ceph.stage.4

These files are included from the /srv/salt/ceph/configuration/files/ceph.conf.j2 template file, and correspond to the different sections that the Ceph configuration file accepts. Putting a configuration snippet in the correct file enables DeepSea to place it into the correct section. You do not need to add any of the section headers.

Tip

To apply any configuration options only to specific instances of a daemon, add a header such as [osd.1]. The following configuration options will only be applied to the OSD daemon with the ID 1.

auth cluster required = none
auth service required = none
auth client required = none

root@master # salt-run state.orch ceph.configuration.create

1.12.2 Including Configuration Files #

If you need to apply a lot of custom configurations, use the following include statements within the custom configuration files to make file management easier. Following is an example of the osd.conf file:

[osd.1]
{% include "ceph/configuration/files/ceph.conf.d/osd1.conf" ignore missing %}
[osd.2]
{% include "ceph/configuration/files/ceph.conf.d/osd2.conf" ignore missing %}
[osd.3]
{% include "ceph/configuration/files/ceph.conf.d/osd3.conf" ignore missing %}
[osd.4]
{% include "ceph/configuration/files/ceph.conf.d/osd4.conf" ignore missing %}

In the previous example, the osd1.conf, osd2.conf, osd3.conf, and osd4.conf files contain the configuration options specific to the related OSD.

Tip: Runtime Configuration

Changes made to Ceph configuration files take effect after the related Ceph daemons restart. See Section 12.1, “Runtime Configuration” for more information on changing the Ceph runtime configuration.

AppArmor is a security solution that confines programs by a specific profile. For more details, refer to https://documentation.suse.com/sles/12-SP5/single-html/SLES-security/#part-apparmor.

DeepSea provides three states for AppArmor profiles: 'enforce', 'complain', and 'disable'. To activate a particular AppArmor state, run:

salt -I "deepsea_minions:*" state.apply ceph.apparmor.default-STATE

To put the AppArmor profiles in an 'enforce' state:

root@master # salt -I "deepsea_minions:*" state.apply ceph.apparmor.default-enforce

To put the AppArmor profiles in a 'complain' status:

root@master # salt -I "deepsea_minions:*" state.apply ceph.apparmor.default-complain

To disable the AppArmor profiles:

root@master # salt -I "deepsea_minions:*" state.apply ceph.apparmor.default-disable

Tip: Enabling the AppArmor Service

Each of these three calls verifies if AppArmor is installed and installs it if not, and starts and enables the related systemd service. DeepSea will warn you if AppArmor was installed and started/enabled in another way and therefore runs without DeepSea profiles.

Use the systemctl command to operate all Ceph related services. The operation takes place on the node you are currently logged in to. You need to have root privileges to be able to operate on Ceph services.

cephadm > ls /usr/lib/systemd/system/ceph*.target
ceph.target
ceph-osd.target
ceph-mon.target
ceph-mgr.target
ceph-mds.target
ceph-radosgw.target
ceph-rbd-mirror.target

root # systemctl start ceph.target
root # systemctl stop ceph.target
root # systemctl restart ceph.target

root # systemctl start ceph-osd.target
root # systemctl stop ceph-osd.target
root # systemctl restart ceph-osd.target

ceph-osd@.service
ceph-mon@.service
ceph-mds@.service
ceph-mgr@.service
ceph-radosgw@.service
ceph-rbd-mirror@.service

root # systemctl start ceph-osd@1.service
root # systemctl stop ceph-osd@1.service
root # systemctl restart ceph-osd@1.service

root # systemctl list-units --all --type=service ceph* lrbd*

root # systemctl list-units --all --state=inactive --type=service ceph* lrbd*

root@master # salt TARGET cmd.shell \
 "systemctl list-units --all --type=service ceph* lrbd* | sed -e '/^$/,$ d'"

root@master # salt -I 'roles:storage' cmd.shell \
 'systemctl list-units --all --type=service ceph* lrbd*'

root # systemctl status ceph-osd@1.service
root # systemctl status ceph-mon@HOSTNAME.service

After applying updates to the cluster nodes, the affected Ceph related services need to be restarted. Normally, restarts are performed automatically by DeepSea. This section describes how to restart the services manually.

Tip: Watching the Restart

The process of restarting the cluster may take some time. You can watch the events by using the Salt event bus by running:

root@master # salt-run state.event pretty=True

Another command to monitor active jobs is

root@master # salt-run jobs.active

3.2.1 Restarting All Services #

Warning: Interruption of Services

If Ceph related services—specifically iSCSI or NFS Ganesha—are configured as single points of access with no High Availability setup, restarting then will result in their temporary outage as viewed from the client side.

Tip: Samba not Managed by DeepSea

Because DeepSea and openATTIC do not currently support Samba deployments, you need to manage Samba related services manually. For more details, see Book “Deployment Guide”, Chapter 13 “Exporting Ceph Data via Samba”.

To restart all services on the cluster, run the following command:

root@master # salt-run state.orch ceph.restart

All roles you have configured restart in the following order: Ceph Monitor, Ceph Manager, Ceph OSD, Metadata Server, Object Gateway, iSCSI Gateway, NFS Ganesha. To keep the downtime low and to find potential issues as early as possible, nodes are restarted sequentially. For example, only one monitoring node is restarted at a time.

The command waits for the cluster to recover if the cluster is in a degraded, unhealthy state.

root@master # salt-run state.orch ceph.restart.service_name

root@master # salt-run state.orch ceph.restart.rgw

root@master # salt-run state.orch ceph.restart.mon

root@master # salt-run state.orch ceph.restart.mgr

root@master # salt-run state.orch ceph.restart.osd

root@master # salt-run state.orch ceph.restart.mds

root@master # salt-run state.orch ceph.restart.rgw

root@master # salt-run state.orch ceph.restart.igw

root@master # salt-run state.orch ceph.restart.ganesha

Shutting down and restarting the cluster may be necessary in the case of a planned power outage. To stop all Ceph related services and restart without issue, follow the steps below.

To check a cluster's status, execute the following:

cephadm > ceph status

or

cephadm > ceph -s

In interactive mode, type status and press Enter.

ceph> status

Ceph will print the cluster status. For example, a tiny Ceph cluster consisting of one monitor and two OSDs may print the following:

cluster b370a29d-9287-4ca3-ab57-3d824f65e339
 health HEALTH_OK
 monmap e1: 1 mons at {ceph1=10.0.0.8:6789/0}, election epoch 2, quorum 0 ceph1
 osdmap e63: 2 osds: 2 up, 2 in
  pgmap v41332: 952 pgs, 20 pools, 17130 MB data, 2199 objects
        115 GB used, 167 GB / 297 GB avail
               1 active+clean+scrubbing+deep
             951 active+clean

After you start your cluster and before you start reading and/or writing data, check your cluster's health:

cephadm > ceph health
HEALTH_WARN 10 pgs degraded; 100 pgs stuck unclean; 1 mons down, quorum 0,2 \
node-1,node-2,node-3

Tip

If you specified non-default locations for your configuration or keyring, you may specify their locations:

cephadm > ceph -c /path/to/conf -k /path/to/keyring health

The Ceph cluster returns one of the following health codes:

You can find the immediate state of the cluster using ceph -s. For example, a tiny Ceph cluster consisting of one monitor, and two OSDs may print the following when a workload is running:

cephadm > ceph -s
cluster:
  id:     ea4cf6ce-80c6-3583-bb5e-95fa303c893f
  health: HEALTH_WARN
          too many PGs per OSD (408 > max 300)

services:
  mon: 3 daemons, quorum ses5min1,ses5min3,ses5min2
  mgr: ses5min1(active), standbys: ses5min3, ses5min2
  mds: cephfs-1/1/1 up  {0=ses5min3=up:active}
  osd: 4 osds: 4 up, 4 in
  rgw: 1 daemon active

data:
  pools:   8 pools, 544 pgs
  objects: 253 objects, 3821 bytes
  usage:   6252 MB used, 13823 MB / 20075 MB avail
  pgs:     544 active+clean

The output provides the following information:

Tip: How Ceph Calculates Data Usage

The used value reflects the actual amount of raw storage used. The xxx GB / xxx GB value means the amount available (the lesser number) of the overall storage capacity of the cluster. The notional number reflects the size of the stored data before it is replicated, cloned or snapshot. Therefore, the amount of data actually stored typically exceeds the notional amount stored, because Ceph creates replicas of the data and may also use storage capacity for cloning and snapshotting.

Other commands that display immediate status information are:

To get the information updated in real time, put any of these commands (including ceph -s) as an argument of the watch command:

root # watch -n 10 'ceph -s'

Press Ctrl–C when you are tired of watching.

To check a cluster’s data usage and distribution among pools, use the ceph df command. To get more details, use ceph df detail.

cephadm > ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    65886G     45826G        7731M            16
POOLS:
    NAME         ID     USED      %USED     MAX AVAIL     OBJECTS
    data         1      1726M        10        17676G        1629
    rbd          4      5897M        27        22365G        3547
    ecpool       6        69M       0.2        35352G          31
[...]

The GLOBAL section of the output provides an overview of the amount of storage your cluster uses for your data.

The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section does not reflect replicas, clones or snapshots. For example, if you store an object with 1MB of data, the notional usage will be 1MB, but the actual usage may be 2MB or more depending on the number of replicas, clones and snapshots.

Note

The numbers in the POOLS section are notional. They are not inclusive of the number of replicas, snapshots or clones. As a result, the sum of the USED and %USED amounts will not add up to the RAW USED and %RAW USED amounts in the %GLOBAL section of the output.

You can check OSDs to ensure they are up and on by executing:

cephadm > ceph osd stat

or

cephadm > ceph osd dump

You can also view OSDs according to their position in the CRUSH map.

cephadm > ceph osd tree

Ceph will print a CRUSH tree with a host, its OSDs, whether they are up and their weight.

# id    weight  type name       up/down reweight
-1      3       pool default
-3      3               rack mainrack
-2      3                       host osd-host
0       1                               osd.0   up      1
1       1                               osd.1   up      1
2       1                               osd.2   up      1

Ceph prevents you from writing to a full OSD so that you do not lose data. In an operational cluster, you should receive a warning when your cluster is getting near its full ratio. The mon osd full ratio defaults to 0.95, or 95% of capacity before it stops clients from writing data. The mon osd nearfull ratio defaults to 0.85, or 85% of capacity, when it generates a health warning.

Full OSD nodes will be reported by ceph health:

cephadm > ceph health
  HEALTH_WARN 1 nearfull osds
  osd.2 is near full at 85%

or

cephadm > ceph health
  HEALTH_ERR 1 nearfull osds, 1 full osds
  osd.2 is near full at 85%
  osd.3 is full at 97%

The best way to deal with a full cluster is to add new OSD hosts/disks allowing the cluster to redistribute data to the newly available storage.

Tip: Preventing Full OSDs

After an OSD becomes full—it uses 100% of its disk space—it will normally crash quickly without warning. Following are a few tips to remember when administering OSD nodes.

• Each OSD's disk space (usually mounted under /var/lib/ceph/osd/osd-{1,2..}) needs to be placed on a dedicated underlying disk or partition.

• Check the Ceph configuration files and make sure that Ceph does not store its log file to the disks/partitions dedicated for use by OSDs.

• Make sure that no other process writes to the disks/partitions dedicated for use by OSDs.

After you start the cluster and before first reading and/or writing data, check the Ceph Monitors quorum status. When the cluster is already serving requests, check the Ceph Monitors status periodically to ensure that they are running.

To display the monitor map, execute the following:

cephadm > ceph mon stat

or

cephadm > ceph mon dump

To check the quorum status for the monitor cluster, execute the following:

cephadm > ceph quorum_status

Ceph will return the quorum status. For example, a Ceph cluster consisting of three monitors may return the following:

{ "election_epoch": 10,
  "quorum": [
        0,
        1,
        2],
  "monmap": { "epoch": 1,
      "fsid": "444b489c-4f16-4b75-83f0-cb8097468898",
      "modified": "2011-12-12 13:28:27.505520",
      "created": "2011-12-12 13:28:27.505520",
      "mons": [
            { "rank": 0,
              "name": "a",
              "addr": "192.168.1.10:6789\/0"},
            { "rank": 1,
              "name": "b",
              "addr": "192.168.1.11:6789\/0"},
            { "rank": 2,
              "name": "c",
              "addr": "192.168.1.12:6789\/0"}
           ]
    }
}

Placement groups map objects to OSDs. When you monitor your placement groups, you will want them to be active and clean. For a detailed discussion, refer to Monitoring OSDs and Placement Groups.

The Ceph admin socket allows you to query a daemon via a socket interface. By default, Ceph sockets reside under /var/run/ceph. To access a daemon via the admin socket, log in to the host running the daemon and use the following command:

cephadm > ceph --admin-daemon /var/run/ceph/socket-name

To view the available admin socket commands, execute the following command:

cephadm > ceph --admin-daemon /var/run/ceph/socket-name help

The admin socket command enables you to show and set your configuration at runtime. Refer to Viewing a Configuration at Runtimefor details.

Additionally, you can set configuration values at runtime directly (the admin socket bypasses the monitor, unlike ceph tell daemon-type.id injectargs, which relies on the monitor but does not require you to log in directly to the host in question).

Alertmanager's configuration is different for each deployment. Therefore, DeepSea does not ship any related defaults. You need to provide your own alertmanager.yml configuration file. The alertmanager package by default installs a configuration file /etc/prometheus/alertmanager.yml which can serve as an example configuration. If you prefer to have your Alertmanager configuration managed by DeepSea, add the following key to your pillar, for example to the /srv/pillar/ceph/stack/ceph/minions/YOUR_SALT_MASTER_MINION_ID.sls file:

For a complete example of Alertmanager's configuration file, see Appendix B, Default Alerts for SUSE Enterprise Storage.

monitoring:
 alertmanager_config:
   /path/to/your/alertmanager/config.yml

Alertmanager's configuration file is written in the YAML format. It follows the scheme described below. Parameters in brackets are optional. For non-list parameters the default value is used. The following generic placeholders are used in the scheme:

global:
# the time after which an alert is declared resolved if it has not been updated
[ resolve_timeout: DURATION | default = 5m ]

# The default SMTP From header field.
[ smtp_from: TMPL_STRING ]
# The default SMTP smarthost used for sending emails, including port number.
# Port number usually is 25, or 587 for SMTP over TLS
# (sometimes referred to as STARTTLS).
# Example: smtp.example.org:587
[ smtp_smarthost: STRING ]
# The default host name to identify to the SMTP server.
[ smtp_hello: STRING | default = "localhost" ]
[ smtp_auth_username: STRING ]
# SMTP Auth using LOGIN and PLAIN.
[ smtp_auth_password: SECRET ]
# SMTP Auth using PLAIN.
[ smtp_auth_identity: STRING ]
# SMTP Auth using CRAM-MD5.
[ smtp_auth_secret: SECRET ]
# The default SMTP TLS requirement.
[ smtp_require_tls: BOOL | default = true ]

# The API URL to use for Slack notifications.
[ slack_api_url: STRING ]
[ victorops_api_key: STRING ]
[ victorops_api_url: STRING | default = "https://victorops.example.com/integrations/alert/" ]
[ pagerduty_url: STRING | default = "https://pagerduty.example.com/v2/enqueue" ]
[ opsgenie_api_key: STRING ]
[ opsgenie_api_url: STRING | default = "https://opsgenie.example.com/" ]
[ hipchat_api_url: STRING | default = "https://hipchat.example.com/" ]
[ hipchat_auth_token: SECRET ]
[ wechat_api_url: STRING | default = "https://wechat.example.com/cgi-bin/" ]
[ wechat_api_secret: SECRET ]
[ wechat_api_corp_id: STRING ]

# The default HTTP client configuration
[ http_config: HTTP_CONFIG ]

# Files from which custom notification template definitions are read.
# The last component may use a wildcard matcher, e.g. 'templates/*.tmpl'.
templates:
[ - FILEPATH ... ]

# The root node of the routing tree.
route: ROUTE

# A list of notification receivers.
receivers:
- RECEIVER ...

# A list of inhibition rules.
inhibit_rules:
[ - INHIBIT_RULE ... ]

[ receiver: STRING ]
[ group_by: '[' LABELNAME, ... ']' ]

# If an alert should continue matching subsequent sibling nodes.
[ continue: BOOLEAN | default = false ]

# A set of equality matchers an alert has to fulfill to match a node.
match:
 [ LABELNAME: LABELVALUE, ... ]

# A set of regex-matchers an alert has to fulfill to match a node.
match_re:
 [ LABELNAME: REGEX, ... ]

# Time to wait before sending a notification for a group of alerts.
[ group_wait: DURATION | default = 30s ]

# Time to wait before sending a notification about new alerts
# added to a group of alerts for which an initial notification has
# already been sent.
[ group_interval: DURATION | default = 5m ]

# Time to wait before re-sending a notification
[ repeat_interval: DURATION | default = 4h ]

# Possible child routes.
routes:
 [ - ROUTE ... ]

# Matchers that need to be fulfilled for the alerts to be muted.
target_match:
 [ LABELNAME: LABELVALUE, ... ]
target_match_re:
 [ LABELNAME: REGEX, ... ]

# Matchers for which at least one alert needs to exist so that the
# inhibition occurs.
source_match:
 [ LABELNAME: LABELVALUE, ... ]
source_match_re:
 [ LABELNAME: REGEX, ... ]

# Labels with an equal value in the source and target
# alert for the inhibition to take effect.
[ equal: '[' LABELNAME, ... ']' ]

# Sets the 'Authorization' header with the user name and password.
basic_auth:
[ username: STRING ]
[ password: SECRET ]

# Sets the 'Authorization' header with the bearer token.
[ bearer_token: SECRET ]

# Sets the 'Authorization' header with the bearer token read from a file.
[ bearer_token_file: FILEPATH ]

# TLS settings.
tls_config:
# CA certificate to validate the server certificate with.
[ ca_file: FILEPATH ]
# Certificate and key files for client cert authentication to the server.
[ cert_file: FILEPATH ]
[ key_file: FILEPATH ]
# ServerName extension to indicate the name of the server.
# http://tools.ietf.org/html/rfc4366#section-3.1
[ server_name: STRING ]
# Disable validation of the server certificate.
[ insecure_skip_verify: BOOLEAN | default = false]

# Optional proxy URL.
[ proxy_url: STRING ]

# The unique name of the receiver.
name: STRING

# Configurations for several notification integrations.
email_configs:
[ - EMAIL_CONFIG, ... ]
hipchat_configs:
[ - HIPCHAT_CONFIG, ... ]
pagerduty_configs:
[ - PAGERDUTY_CONFIG, ... ]
pushover_configs:
[ - PUSHOVER_CONFIG, ... ]
slack_configs:
[ - SLACK_CONFIG, ... ]
opsgenie_configs:
[ - OPSGENIE_CONFIG, ... ]
webhook_configs:
[ - WEBHOOK_CONFIG, ... ]
victorops_configs:
[ - VICTOROPS_CONFIG, ... ]
wechat_configs:
[ - WECHAT_CONFIG, ... ]

# Whether to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = false ]

# The email address to send notifications to.
to: TMPL_STRING

# The sender address.
[ from: TMPL_STRING | default = global.smtp_from ]

# The SMTP host through which emails are sent.
[ smarthost: STRING | default = global.smtp_smarthost ]

# The host name to identify to the SMTP server.
[ hello: STRING | default = global.smtp_hello ]

# SMTP authentication details.
[ auth_username: STRING | default = global.smtp_auth_username ]
[ auth_password: SECRET | default = global.smtp_auth_password ]
[ auth_secret: SECRET | default = global.smtp_auth_secret ]
[ auth_identity: STRING | default = global.smtp_auth_identity ]

# The SMTP TLS requirement.
[ require_tls: BOOL | default = global.smtp_require_tls ]

# The HTML body of the email notification.
[ html: TMPL_STRING | default = '{{ template "email.default.html" . }}' ]
# The text body of the email notification.
[ text: TMPL_STRING ]

# Further headers email header key/value pairs. Overrides any headers
# previously set by the notification implementation.
[ headers: { STRING: TMPL_STRING, ... } ]

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = false ]

# The HipChat Room ID.
room_id: TMPL_STRING
# The authentication token.
[ auth_token: SECRET | default = global.hipchat_auth_token ]
# The URL to send API requests to.
[ api_url: STRING | default = global.hipchat_api_url ]

# A label to be shown in addition to the sender's name.
[ from:  TMPL_STRING | default = '{{ template "hipchat.default.from" . }}' ]
# The message body.
[ message:  TMPL_STRING | default = '{{ template "hipchat.default.message" . }}' ]
# Whether this message will trigger a user notification.
[ notify:  BOOLEAN | default = false ]
# Determines how the message is treated by the alertmanager and rendered inside HipChat. Valid values are 'text' and 'html'.
[ message_format:  STRING | default = 'text' ]
# Background color for message.
[ color:  TMPL_STRING | default = '{{ if eq .Status "firing" }}red{{ else }}green{{ end }}' ]

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The PagerDuty integration key (when using 'Events API v2').
routing_key: TMPL_SECRET
# The PagerDuty integration key (when using 'Prometheus').
service_key: TMPL_SECRET

# The URL to send API requests to.
[ url: STRING | default = global.pagerduty_url ]

# The client identification of the Alertmanager.
[ client:  TMPL_STRING | default = '{{ template "pagerduty.default.client" . }}' ]
# A backlink to the notification sender.
[ client_url:  TMPL_STRING | default = '{{ template "pagerduty.default.clientURL" . }}' ]

# The incident description.
[ description: TMPL_STRING | default = '{{ template "pagerduty.default.description" .}}' ]

# Severity of the incident.
[ severity: TMPL_STRING | default = 'error' ]

# A set of arbitrary key/value pairs that provide further details.
[ details: { STRING: TMPL_STRING, ... } | default = {
 firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
 resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
 num_firing:   '{{ .Alerts.Firing | len }}'
 num_resolved: '{{ .Alerts.Resolved | len }}'
} ]

# The HTTP client's configuration.
[ http_config: HTTP_CONFIG | default = global.http_config ]

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The recipient user key.
user_key: SECRET

# Registered application’s API token.
token: SECRET

# Notification title.
[ title: TMPL_STRING | default = '{{ template "pushover.default.title" . }}' ]

# Notification message.
[ message: TMPL_STRING | default = '{{ template "pushover.default.message" . }}' ]

# A supplementary URL displayed together with the message.
[ url: TMPL_STRING | default = '{{ template "pushover.default.url" . }}' ]

# Priority.
[ priority: TMPL_STRING | default = '{{ if eq .Status "firing" }}2{{ else }}0{{ end }}' ]

# How often the Pushover servers will send the same notification (at least 30 seconds).
[ retry: DURATION | default = 1m ]

# How long your notification will continue to be retried (unless the user
# acknowledges the notification).
[ expire: DURATION | default = 1h ]

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = false ]

# The Slack webhook URL.
[ api_url: SECRET | default = global.slack_api_url ]

# The channel or user to send notifications to.
channel: TMPL_STRING

# API request data as defined by the Slack webhook API.
[ icon_emoji: TMPL_STRING ]
[ icon_url: TMPL_STRING ]
[ link_names: BOOLEAN | default = false ]
[ username: TMPL_STRING | default = '{{ template "slack.default.username" . }}' ]
# The following parameters define the attachment.
actions:
[ ACTION_CONFIG ... ]
[ color: TMPL_STRING | default = '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' ]
[ fallback: TMPL_STRING | default = '{{ template "slack.default.fallback" . }}' ]
fields:
[ FIELD_CONFIG ... ]
[ footer: TMPL_STRING | default = '{{ template "slack.default.footer" . }}' ]
[ pretext: TMPL_STRING | default = '{{ template "slack.default.pretext" . }}' ]
[ short_fields: BOOLEAN | default = false ]
[ text: TMPL_STRING | default = '{{ template "slack.default.text" . }}' ]
[ title: TMPL_STRING | default = '{{ template "slack.default.title" . }}' ]
[ title_link: TMPL_STRING | default = '{{ template "slack.default.titlelink" . }}' ]
[ image_url: TMPL_STRING ]
[ thumb_url: TMPL_STRING ]

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]

# Provide a button to tell Slack you want to render a button.
type: TMPL_STRING
# Label for the button.
text: TMPL_STRING
# http or https URL to deliver users to. If you specify invalid URLs, the message will be posted with no button.
url: TMPL_STRING
#  If set to 'primary', the button will be green, indicating the best forward action to take
#  'danger' turns the button red, indicating a destructive action.
[ style: TMPL_STRING [ default = '' ]

# A bold heading without markup above the value text.
title: TMPL_STRING
# The text of the field. It can span across several lines.
value: TMPL_STRING
# A flag indicating if value is short enough to be displayed together with other values.
[ short: BOOLEAN | default = slack_config.short_fields ]

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The API key to use with the OpsGenie API.
[ api_key: SECRET | default = global.opsgenie_api_key ]

# The host to send OpsGenie API requests to.
[ api_url: STRING | default = global.opsgenie_api_url ]

# Alert text (maximum is 130 characters).
[ message: TMPL_STRING ]

# A description of the incident.
[ description: TMPL_STRING | default = '{{ template "opsgenie.default.description" . }}' ]

# A backlink to the sender.
[ source: TMPL_STRING | default = '{{ template "opsgenie.default.source" . }}' ]

# A set of arbitrary key/value pairs that provide further detail.
[ details: { STRING: TMPL_STRING, ... } ]

# Comma separated list of team responsible for notifications.
[ teams: TMPL_STRING ]

# Comma separated list of tags attached to the notifications.
[ tags: TMPL_STRING ]

# Additional alert note.
[ note: TMPL_STRING ]

# Priority level of alert, one of P1, P2, P3, P4, and P5.
[ priority: TMPL_STRING ]

# Configuration of the HTTP.
[ http_config: HTTP_CONFIG | default = global.http_config ]

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The API key for talking to the VictorOps API.
[ api_key: SECRET | default = global.victorops_api_key ]

# The VictorOps API URL.
[ api_url: STRING | default = global.victorops_api_url ]

# A key used to map the alert to a team.
routing_key: TMPL_STRING

# Describes the behavior of the alert (one of 'CRITICAL', 'WARNING', 'INFO').
[ message_type: TMPL_STRING | default = 'CRITICAL' ]

# Summary of the alerted problem.
[ entity_display_name: TMPL_STRING | default = '{{ template "victorops.default.entity_display_name" . }}' ]

# Long explanation of the alerted problem.
[ state_message: TMPL_STRING | default = '{{ template "victorops.default.state_message" . }}' ]

# The monitoring tool the state message is from.
[ monitoring_tool: TMPL_STRING | default = '{{ template "victorops.default.monitoring_tool" . }}' ]

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The endpoint for sending HTTP POST requests.
url: STRING

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]

{
 "version": "4",
 "groupKey": STRING, // identifycation of the group of alerts (to deduplicate)
 "status": "<resolved|firing>",
 "receiver": STRING,
 "groupLabels": OBJECT,
 "commonLabels": OBJECT,
 "commonAnnotations": OBJECT,
 "externalURL": STRING, // backlink to Alertmanager.
 "alerts": [
   {
     "status": "<resolved|firing>",
     "labels": OBJECT,
     "annotations": OBJECT,
     "startsAt": "<rfc3339>",
     "endsAt": "<rfc3339>",
     "generatorURL": STRING // identifies the entity that caused the alert
   },
   ...
 ]
}

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = false ]

# The API key to use for the WeChat API.
[ api_secret: SECRET | default = global.wechat_api_secret ]

# The WeChat API URL.
[ api_url: STRING | default = global.wechat_api_url ]

# The corp id used to authenticate.
[ corp_id: STRING | default = global.wechat_api_corp_id ]

# API request data as defined by the WeChat API.
[ message: TMPL_STRING | default = '{{ template "wechat.default.message" . }}' ]
[ agent_id: STRING | default = '{{ template "wechat.default.agent_id" . }}' ]
[ to_user: STRING | default = '{{ template "wechat.default.to_user" . }}' ]
[ to_party: STRING | default = '{{ template "wechat.default.to_party" . }}' ]
[ to_tag: STRING | default = '{{ template "wechat.default.to_tag" . }}' ]

You can define your custom alert conditions to send notifications to an external service. Prometheus uses its own expression language for defining custom alerts. Following is an example of a rule with an alert:

groups:
- name: example
 rules:
  # alert on high deviation from average PG count
  - alert: high pg count deviation
   expr: abs(((ceph_osd_pgs > 0) - on (job) group_left avg(ceph_osd_pgs > 0) by (job)) / on (job) group_left avg(ceph_osd_pgs > 0) by (job)) > 0.35
   for: 5m
   labels:
    severity: warning
    type: ses_default
   annotations:
   description: >
    OSD {{ $labels.osd }} deviates by more then 30% from average PG count

The optional for clause specifies the time Prometheus waits between first encountering a new expression output vector element and counting an alert as firing. In this case, Prometheus checks that the alert continues to be active for 5 minutes before firing the alert. Elements in a pending state are active, but not firing yet.

The labels clause specifies a set of additional labels attached to the alert. Conflicting labels will be overwritten. Labels can be templated (see Section 5.2.1, “Templates” for more details on templating).

The annotations clause specifies informational labels. You can use them to store additional information, for example alert descriptions or runbook links. Annotations can be templated (see Section 5.2.1, “Templates” for more details on templating).

To add your custom alerts to SUSE Enterprise Storage, either...

or

{{ $labels.LABELNAME }}
{{ $value }}

cephx uses shared secret keys for authentication, meaning both the client and Ceph Monitors have a copy of the client’s secret key. The authentication protocol enables both parties to prove to each other that they have a copy of the key without actually revealing it. This provides mutual authentication, which means the cluster is sure the user possesses the secret key, and the user is sure that the cluster has a copy of the secret key as well.

A key scalability feature of Ceph is to avoid a centralized interface to the Ceph object store. This means that Ceph clients can interact with OSDs directly. To protect data, Ceph provides its cephx authentication system, which authenticates Ceph clients.

Each monitor can authenticate clients and distribute keys, so there is no single point of failure or bottleneck when using cephx. The monitor returns an authentication data structure that contains a session key for use in obtaining Ceph services. This session key is itself encrypted with the client’s permanent secret key, so that only the client can request services from the Ceph monitors. The client then uses the session key to request its desired services from the monitor, and the monitor provides the client with a ticket that will authenticate the client to the OSDs that actually handle data. Ceph monitors and OSDs share a secret, so the client can use the ticket provided by the monitor with any OSD or metadata server in the cluster. cephx tickets expire, so an attacker cannot use an expired ticket or session key obtained wrongfully.

To use cephx, an administrator must setup clients/users first. In the following diagram, the client.admin user invokes ceph auth get-or-create-key from the command line to generate a user name and secret key. Ceph’s auth subsystem generates the user name and key, stores a copy with the monitor(s) and transmits the user’s secret back to the client.admin user. This means that the client and the monitor share a secret key.

Figure 6.1: Basic cephx Authentication #

To authenticate with the monitor, the client passes the user name to the monitor. The monitor generates a session key and encrypts it with the secret key associated with the user name and transmits the encrypted ticket back to the client. The client then decrypts the data with the shared secret key to retrieve the session key. The session key identifies the user for the current session. The client then requests a ticket related to the user, which is signed by the session key. The monitor generates a ticket, encrypts it with the user’s secret key and transmits it back to the client. The client decrypts the ticket and uses it to sign requests to OSDs and metadata servers throughout the cluster.

Figure 6.2: cephx Authentication #

The cephx protocol authenticates ongoing communications between the client machine and the Ceph servers. Each message sent between a client and a server after the initial authentication is signed using a ticket that the monitors, OSDs, and metadata servers can verify with their shared secret.

Figure 6.3: cephx Authentication - MDS and OSD #

Important

The protection offered by this authentication is between the Ceph client and the Ceph cluster hosts. The authentication is not extended beyond the Ceph client. If the user accesses the Ceph client from a remote host, Ceph authentication is not applied to the connection between the user’s host and the client host.

This section describes Ceph client users and their authentication and authorization with the Ceph storage cluster. Users are either individuals or system actors such as applications, which use Ceph clients to interact with the Ceph storage cluster daemons.

When Ceph runs with authentication and authorization enabled (enabled by default), you must specify a user name and a keyring containing the secret key of the specified user (usually via the command line). If you do not specify a user name, Ceph will use client.admin as the default user name. If you do not specify a keyring, Ceph will look for a keyring via the keyring setting in the Ceph configuration file. For example, if you execute the ceph health command without specifying a user name or keyring, Ceph interprets the command like this:

cephadm > ceph -n client.admin --keyring=/etc/ceph/ceph.client.admin.keyring health

Alternatively, you may use the CEPH_ARGS environment variable to avoid re-entering the user name and secret.

6.2.1 Background Information #

Regardless of the type of Ceph client (for example, block device, object storage, file system, native API), Ceph stores all data as objects within pools. Ceph users need to have access to pools in order to read and write data. Additionally, Ceph users must have execute permissions to use Ceph's administrative commands. The following concepts will help you understand Ceph user management.

6.2.1.1 User #

A user is either an individual or a system actor such as an application. Creating users allows you to control who (or what) can access your Ceph storage cluster, its pools, and the data within pools.

Ceph uses types of users. For the purposes of user management, the type will always be client. Ceph identifies users in period (.) delimited form, consisting of the user type and the user ID. For example, TYPE.ID, client.admin, or client.user1. The reason for user typing is that Ceph monitors, OSDs, and metadata servers also use the cephx protocol, but they are not clients. Distinguishing the user type helps to distinguish between client users and other users, streamlining access control, user monitoring, and traceability.

Note

A Ceph storage cluster user is not the same as a Ceph object storage user or a Ceph file system user. The Ceph Object Gateway uses a Ceph storage cluster user to communicate between the gateway daemon and the storage cluster, but the gateway has its own user management functionality for end users. The Ceph file system uses POSIX semantics. The user space associated with it is not the same as a Ceph storage cluster user.

6.2.1.2 Authorization and Capabilities #

Ceph uses the term 'capabilities' (caps) to describe authorizing an authenticated user to exercise the functionality of the monitors, OSDs, and metadata servers. Capabilities can also restrict access to data within a pool or pool namespace. A Ceph administrative user sets a user's capabilities when creating or updating a user.

Capability syntax follows the form:

daemon-type 'allow capability' [...]

Following is a list of capabilities for each service type:

Monitor capabilities

include r, w, x and allow profile cap.

mon 'allow rwx'
mon 'allow profile osd'

OSD capabilities

include r, w, x, class-read, class-write and profile osd. Additionally, OSD capabilities also allow for pool and namespace settings.

osd 'allow capability' [pool=poolname] [namespace=namespace-name]

MDS capability

simply requires allow, or blank.

mds 'allow'

The following entries describe each capability:

allow

Precedes access settings for a daemon. Implies rw for MDS only.

r

Gives the user read access. Required with monitors to retrieve the CRUSH map.

w

Gives the user write access to objects.

x

Gives the user the capability to call class methods (both read and write) and to conduct auth operations on monitors.

class-read

Gives the user the capability to call class read methods. Subset of x.

class-write

Gives the user the capability to call class write methods. Subset of x.

*

Gives the user read, write, and execute permissions for a particular daemon/pool, and the ability to execute admin commands.

profile osd

Gives a user permissions to connect as an OSD to other OSDs or monitors. Conferred on OSDs to enable OSDs to handle replication heartbeat traffic and status reporting.

profile mds

Gives a user permissions to connect as an MDS to other MDSs or monitors.

profile bootstrap-osd

Gives a user permissions to bootstrap an OSD. Delegated to deployment tools so that they have permissions to add keys when bootstrapping an OSD.

profile bootstrap-mds

Gives a user permissions to bootstrap a metadata server. Delegated to deployment tools so they have permissions to add keys when bootstrapping a metadata server.

6.2.2 Managing Users #

User management functionality provides Ceph cluster administrators with the ability to create, update, and delete users directly in the Ceph cluster.

When you create or delete users in the Ceph cluster, you may need to distribute keys to clients so that they can be added to keyrings. See Section 6.2.3, “Keyring Management” for details.

6.2.2.1 Listing Users #

To list the users in your cluster, execute the following:

cephadm > ceph auth list

Ceph will list all users in your cluster. For example, in a cluster with two nodes, ceph auth list output looks similar to this:

installed auth entries:

osd.0
        key: AQCvCbtToC6MDhAATtuT70Sl+DymPCfDSsyV4w==
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.1
        key: AQC4CbtTCFJBChAAVq5spj0ff4eHZICxIOVZeA==
        caps: [mon] allow profile osd
        caps: [osd] allow *
client.admin
        key: AQBHCbtT6APDHhAA5W00cBchwkQjh3dkKsyPjw==
        caps: [mds] allow
        caps: [mon] allow *
        caps: [osd] allow *
client.bootstrap-mds
        key: AQBICbtTOK9uGBAAdbe5zcIGHZL3T/u2g6EBww==
        caps: [mon] allow profile bootstrap-mds
client.bootstrap-osd
        key: AQBHCbtT4GxqORAADE5u7RkpCN/oo4e5W0uBtw==
        caps: [mon] allow profile bootstrap-osd

Note: TYPE.ID Notation

Note that the TYPE.ID notation for users applies such that osd.0 specifies a user of type osd and its ID is 0. client.admin is a user of type client and its ID is admin. Note also that each entry has a key: value entry, and one or more caps: entries.

You may use the -o filename option with ceph auth list to save the output to a file.

6.2.2.2 Getting Information about Users #

To retrieve a specific user, key, and capabilities, execute the following:

cephadm > ceph auth get TYPE.ID

For example:

cephadm > ceph auth get client.admin
exported keyring for client.admin
[client.admin]
	key = AQA19uZUqIwkHxAAFuUwvq0eJD4S173oFRxe0g==
	caps mds = "allow"
	caps mon = "allow *"
 caps osd = "allow *"

Developers may also execute the following:

cephadm > ceph auth export TYPE.ID

The auth export command is identical to auth get, but also prints the internal authentication ID.

6.2.2.3 Adding Users #

Adding a user creates a user name (TYPE.ID), a secret key, and any capabilities included in the command you use to create the user.

A user's key enables the user to authenticate with the Ceph storage cluster. The user's capabilities authorize the user to read, write, or execute on Ceph monitors (mon), Ceph OSDs (osd), or Ceph metadata servers (mds).

There are a few commands available to add a user:

ceph auth add

This command is the canonical way to add a user. It will create the user, generate a key, and add any specified capabilities.

ceph auth get-or-create

This command is often the most convenient way to create a user, because it returns a keyfile format with the user name (in brackets) and the key. If the user already exists, this command simply returns the user name and key in the keyfile format. You may use the -o filename option to save the output to a file.

ceph auth get-or-create-key

This command is a convenient way to create a user and return the user's key (only). This is useful for clients that need the key only (for example libvirt). If the user already exists, this command simply returns the key. You may use the -o filename option to save the output to a file.

When creating client users, you may create a user with no capabilities. A user with no capabilities can authenticate but nothing more. Such client cannot retrieve the cluster map from the monitor. However, you can create a user with no capabilities if you want to defer adding capabilities later using the ceph auth caps command.

A typical user has at least read capabilities on the Ceph monitor and read and write capabilities on Ceph OSDs. Additionally, a user's OSD permissions are often restricted to accessing a particular pool.

cephadm > ceph auth add client.john mon 'allow r' osd \
 'allow rw pool=liverpool'
cephadm > ceph auth get-or-create client.paul mon 'allow r' osd \
 'allow rw pool=liverpool'
cephadm > ceph auth get-or-create client.george mon 'allow r' osd \
 'allow rw pool=liverpool' -o george.keyring
cephadm > ceph auth get-or-create-key client.ringo mon 'allow r' osd \
 'allow rw pool=liverpool' -o ringo.key

Important

If you provide a user with capabilities to OSDs, but you do not restrict access to particular pools, the user will have access to all pools in the cluster.

6.2.2.4 Modifying User Capabilities #

The ceph auth caps command allows you to specify a user and change the user's capabilities. Setting new capabilities will overwrite current ones. To view current capabilities run ceph auth get USERTYPE.USERID. To add capabilities, you also need to specify the existing capabilities when using the following form:

cephadm > ceph auth caps USERTYPE.USERID daemon 'allow [r|w|x|*|...] \
     [pool=pool-name] [namespace=namespace-name]' [daemon 'allow [r|w|x|*|...] \
     [pool=pool-name] [namespace=namespace-name]']

For example:

cephadm > ceph auth get client.john
cephadm > ceph auth caps client.john mon 'allow r' osd 'allow rw pool=prague'
cephadm > ceph auth caps client.paul mon 'allow rw' osd 'allow r pool=prague'
cephadm > ceph auth caps client.brian-manager mon 'allow *' osd 'allow *'

To remove a capability, you may reset the capability. If you want the user to have no access to a particular daemon that was previously set, specify an empty string:

cephadm > ceph auth caps client.ringo mon ' ' osd ' '

6.2.2.5 Deleting Users #

To delete a user, use ceph auth del:

cephadm > ceph auth del TYPE.ID

where TYPE is one of client, osd, mon, or mds, and ID is the user name or ID of the daemon.

If you created users with permissions strictly for a pool that no longer exists, you should consider deleting those users too.

6.2.2.6 Printing a User's Key #

To print a user’s authentication key to standard output, execute the following:

cephadm > ceph auth print-key TYPE.ID

where TYPE is one of client, osd, mon, or mds, and ID is the user name or ID of the daemon.

Printing a user's key is useful when you need to populate client software with a user's key (such as libvirt), as in the following example:

root # mount -t ceph host:/ mount_point \
-o name=client.user,secret=`ceph auth print-key client.user`

6.2.2.7 Importing Users #

To import one or more users, use ceph auth import and specify a keyring:

cephadm > ceph auth import -i /etc/ceph/ceph.keyring

Note

The Ceph storage cluster will add new users, their keys and their capabilities and will update existing users, their keys and their capabilities.

6.2.3 Keyring Management #

When you access Ceph via a Ceph client, the client will look for a local keyring. Ceph presets the keyring setting with the following four keyring names by default so you do not need to set them in your Ceph configuration file unless you want to override the defaults:

/etc/ceph/cluster.name.keyring
/etc/ceph/cluster.keyring
/etc/ceph/keyring
/etc/ceph/keyring.bin

The cluster metavariable is your Ceph cluster name as defined by the name of the Ceph configuration file. ceph.conf means that the cluster name is ceph, thus ceph.keyring. The name metavariable is the user type and user ID, for example client.admin, thus ceph.client.admin.keyring.

After you create a user (for example client.ringo), you must get the key and add it to a keyring on a Ceph client so that the user can access the Ceph storage cluster.

Section 6.2, “Key Management” details how to list, get, add, modify and delete users directly in the Ceph storage cluster. However, Ceph also provides the ceph-authtool utility to allow you to manage keyrings from a Ceph client.

6.2.3.1 Creating a Keyring #

When you use the procedures in Section 6.2, “Key Management” to create users, you need to provide user keys to the Ceph client(s) so that the client can retrieve the key for the specified user and authenticate with the Ceph storage cluster. Ceph clients access keyrings to look up a user name and retrieve the user's key:

cephadm > ceph-authtool --create-keyring /path/to/keyring

When creating a keyring with multiple users, we recommend using the cluster name (for example cluster.keyring) for the keyring file name and saving it in the /etc/ceph directory so that the keyring configuration default setting will pick up the file name without requiring you to specify it in the local copy of your Ceph configuration file. For example, create ceph.keyring by executing the following:

cephadm > ceph-authtool -C /etc/ceph/ceph.keyring

When creating a keyring with a single user, we recommend using the cluster name, the user type and the user name and saving it in the /etc/ceph directory. For example, ceph.client.admin.keyring for the client.admin user.

6.2.3.2 Adding a User to a Keyring #

When you add a user to the Ceph storage cluster (see Section 6.2.2.3, “Adding Users”), you can retrieve the user, key and capabilities, and save the user to a keyring.

If you only want to use one user per keyring, the ceph auth get command with the -o option will save the output in the keyring file format. For example, to create a keyring for the client.admin user, execute the following:

cephadm > ceph auth get client.admin -o /etc/ceph/ceph.client.admin.keyring

When you want to import users to a keyring, you can use ceph-authtool to specify the destination keyring and the source keyring:

cephadm > ceph-authtool /etc/ceph/ceph.keyring \
  --import-keyring /etc/ceph/ceph.client.admin.keyring

Important

If your keyring is compromised, delete your key from the /etc/ceph directory and recreate a new key using the same instructions from Section 6.2.3.1, “Creating a Keyring”.

6.2.3.3 Creating a User #

Ceph provides the ceph auth add command to create a user directly in the Ceph storage cluster. However, you can also create a user, keys and capabilities directly on a Ceph client keyring. Then, you can import the user to the Ceph storage cluster:

cephadm > ceph-authtool -n client.ringo --cap osd 'allow rwx' \
  --cap mon 'allow rwx' /etc/ceph/ceph.keyring

You can also create a keyring and add a new user to the keyring simultaneously:

cephadm > ceph-authtool -C /etc/ceph/ceph.keyring -n client.ringo \
  --cap osd 'allow rwx' --cap mon 'allow rwx' --gen-key

In the previous scenarios, the new user client.ringo is only in the keyring. To add the new user to the Ceph storage cluster, you must still add the new user to the cluster:

cephadm > ceph auth add client.ringo -i /etc/ceph/ceph.keyring

6.2.3.4 Modifying Users #

To modify the capabilities of a user record in a keyring, specify the keyring and the user followed by the capabilities:

cephadm > ceph-authtool /etc/ceph/ceph.keyring -n client.ringo \
  --cap osd 'allow rwx' --cap mon 'allow rwx'

To update the modified user within the Ceph cluster environment, you must import the changes from the keyring to the user entry in the Ceph cluster:

cephadm > ceph auth import -i /etc/ceph/ceph.keyring

See Section 6.2.2.7, “Importing Users” for details on updating a Ceph storage cluster user from a keyring.

To map placement groups to OSDs, a CRUSH Map requires a list of OSD devices (the name of the OSD daemon). The list of devices appears first in the CRUSH Map.

#devices
device NUM osd.OSD_NAME class CLASS_NAME

For example:

#devices
device 0 osd.0 class hdd
device 1 osd.1 class ssd
device 2 osd.2 class nvme
device 3 osd.3class ssd

As a general rule, an OSD daemon maps to a single disk.

7.1.1 Device Classes #

The flexibility of the CRUSH Map in controlling data placement is one of the Ceph's strengths. It is also one of the most difficult parts of the cluster to manage. Device classes automate one of the most common reasons why CRUSH Maps are directly manually edited.

7.1.1.1 The CRUSH Management Problem #

Ceph clusters are frequently built with multiple types of storage devices: HDDs, SSDs, NVMe’s, or even mixed classes of the above. We call these different types of storage devices device classes to avoid confusion between the type property of CRUSH buckets (e.g., host, rack, row, see Section 7.2, “Buckets” for more details). Ceph OSDs backed by SSDs are much faster than those backed by spinning disks, making them better suited for certain workloads. Ceph makes it easy to create RADOS pools for different data sets or workloads and to assign different CRUSH rules to control data placement for those pools.

Figure 7.1: OSDs with Mixed Device Classes #

However, setting up the CRUSH rules to place data only on a certain class of device is tedious. Rules work in terms of the CRUSH hierarchy, but if the devices are mixed into the same hosts or racks (as in the sample hierarchy above), they will (by default) be mixed together and appear in the same subtrees of the hierarchy. Manually separating them out into separate trees involved creating multiple versions of each intermediate node for each device class in previous versions of SUSE Enterprise Storage.

7.1.1.2 Device Classes #

An elegant solution that Ceph offers is to add a property called device class to each OSD. By default, OSDs will automatically set their device classes to either 'hdd', 'ssd', or 'nvme' based on the hardware properties exposed by the Linux kernel. These device classes are reported in a new column of the ceph osd tree command output:

cephadm > ceph osd tree
 ID CLASS WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
 -1       83.17899 root default
 -4       23.86200     host cpach
 2   hdd  1.81898         osd.2      up  1.00000 1.00000
 3   hdd  1.81898         osd.3      up  1.00000 1.00000
 4   hdd  1.81898         osd.4      up  1.00000 1.00000
 5   hdd  1.81898         osd.5      up  1.00000 1.00000
 6   hdd  1.81898         osd.6      up  1.00000 1.00000
 7   hdd  1.81898         osd.7      up  1.00000 1.00000
 8   hdd  1.81898         osd.8      up  1.00000 1.00000
 15  hdd  1.81898         osd.15     up  1.00000 1.00000
 10  nvme 0.93100         osd.10     up  1.00000 1.00000
 0   ssd  0.93100         osd.0      up  1.00000 1.00000
 9   ssd  0.93100         osd.9      up  1.00000 1.00000

If the automatic device class detection fails for example because the device driver is not properly exposing information about the device via /sys/block, you can adjust device classes from the command line:

cephadm > ceph osd crush rm-device-class osd.2 osd.3
done removing class of osd(s): 2,3
cephadm > ceph osd crush set-device-class ssd osd.2 osd.3
set osd(s) 2,3 to class 'ssd'

7.1.1.3 CRUSH Placement Rules #

CRUSH rules can restrict placement to a specific device class. For example, you can create a 'fast' replicated pool that distributes data only over SSD disks by running the following command:

cephadm > ceph osd crush rule create-replicated RULE_NAME ROOT FAILURE_DOMAIN_TYPE DEVICE_CLASS

For example:

cephadm > ceph osd crush rule create-replicated fast default host ssd

Create a pool named 'fast_pool' and assign it to the 'fast' rule:

cephadm > ceph osd pool create fast_pool 128 128 replicated fast

The process for creating erasure code rules is slightly different. First, you create an erasure code profile that includes a property for your desired device class. Then use that profile when creating the erasure coded pool:

cephadm > ceph osd erasure-code-profile set myprofile \
 k=4 m=2 crush-device-class=ssd crush-failure-domain=host
cephadm > ceph osd pool create mypool 64 erasure myprofile

If you need to manually edit the CRUSH Map to customize your rule, the syntax has been extended to allow the device class to be specified. For example, the CRUSH rule generated by the above commands looks as follows:

rule ecpool {
  id 2
  type erasure
  min_size 3
  max_size 6
  step set_chooseleaf_tries 5
  step set_choose_tries 100
  step take default class ssd
  step chooseleaf indep 0 type host
  step emit
}

The important difference there is that the 'take' command includes the additional 'class CLASS_NAME' suffix.

7.1.1.4 Additional Commands #

To list device classes used in a CRUSH Map, run:

cephadm > ceph osd crush class ls
[
  "hdd",
  "ssd"
]

To list existing CRUSH rules, run:

cephadm > ceph osd crush rule ls
replicated_rule
fast

To view details of the CRUSH rule named 'fast', run:

cephadm > ceph osd crush rule dump fast
{
		"rule_id": 1,
		"rule_name": "fast",
		"ruleset": 1,
		"type": 1,
		"min_size": 1,
		"max_size": 10,
		"steps": [
						{
										"op": "take",
										"item": -21,
										"item_name": "default~ssd"
						},
						{
										"op": "chooseleaf_firstn",
										"num": 0,
										"type": "host"
						},
						{
										"op": "emit"
						}
		]
}

To list OSDs that belong to a 'ssd' class, run:

cephadm > ceph osd crush class ls-osd ssd
0
1

7.1.1.5 Migrating from a Legacy SSD Rule to Device Classes #

In SUSE Enterprise Storage prior to version 5, you needed to manually edit the CRUSH Map and maintain a parallel hierarchy for each specialized device type (such as SSD) in order to write rules that apply to these devices. Since SUSE Enterprise Storage 5, the device class feature has enabled this transparently.

You can transform a legacy rule and hierarchy to the new class-based rules by using the crushtool command. There are several types of transformation possible:

crushtool --reclassify-root ROOT_NAME DEVICE_CLASS

This command takes everything in the hierarchy beneath ROOT_NAME and adjusts any rules that reference that root via

take ROOT_NAME

to instead

take ROOT_NAME class DEVICE_CLASS

It renumbers the buckets so that the old IDs are used for the specified class’s 'shadow tree'. As a consequence, no data movement occurs.

Example 7.1: crushtool --reclassify-root #

Consider the following existing rule:

rule replicated_ruleset {
   id 0
   type replicated
   min_size 1
   max_size 10
   step take default
   step chooseleaf firstn 0 type rack
   step emit
}

If you reclassify the root 'default' as class 'hdd', the rule will become

rule replicated_ruleset {
   id 0
   type replicated
   min_size 1
   max_size 10
   step take default class hdd
   step chooseleaf firstn 0 type rack
   step emit
}

crushtool --set-subtree-class BUCKET_NAME DEVICE_CLASS

This method marks every device in the subtree rooted at BUCKET_NAME with the specified device class.

--set-subtree-class is normally used in conjunction with the --reclassify-root option to ensure that all devices in that root are labeled with the correct class. However some of those devices may intentionally have a different class, and therefore you do not want to relabel them. In such cases, exclude the --set-subtree-class option. Keep in mind that such remapping will not be perfect, because the previous rule is distributed across devices of multiple classes but the adjusted rules will only map to devices of the specified device class.

crushtool --reclassify-bucket MATCH_PATTERN DEVICE_CLASS DEFAULT_PATTERN

This method allows merging a parallel type-specific hierarchy with the normal hierarchy. For example, many users have CRUSH Maps similar to the following one:

Example 7.2: crushtool --reclassify-bucket #

host node1 {
   id -2           # do not change unnecessarily
   # weight 109.152
   alg straw
   hash 0  # rjenkins1
   item osd.0 weight 9.096
   item osd.1 weight 9.096
   item osd.2 weight 9.096
   item osd.3 weight 9.096
   item osd.4 weight 9.096
   item osd.5 weight 9.096
   [...]
}

host node1-ssd {
   id -10          # do not change unnecessarily
   # weight 2.000
   alg straw
   hash 0  # rjenkins1
   item osd.80 weight 2.000
   [...]
}

root default {
   id -1           # do not change unnecessarily
   alg straw
   hash 0  # rjenkins1
   item node1 weight 110.967
   [...]
}

root ssd {
   id -18          # do not change unnecessarily
   # weight 16.000
   alg straw
   hash 0  # rjenkins1
   item node1-ssd weight 2.000
   [...]
}

This function reclassifies each bucket that matches a given pattern. The pattern can look like %suffix or prefix%. In the above example, you would use the pattern %-ssd. For each matched bucket, the remaining portion of the name that matches the '%' wild card specifies the base bucket. All devices in the matched bucket are labeled with the specified device class and then moved to the base bucket. If the base bucket does not exist (for example if 'node12-ssd' exists but 'node12' does not), then it is created and linked underneath the specified default parent bucket. The old bucket IDs are preserved for the new shadow buckets to prevent data movement. Rules with the take steps that reference the old buckets are adjusted.

crushtool --reclassify-bucket BUCKET_NAME DEVICE_CLASS BASE_BUCKET

You can use the --reclassify-bucket option without a wild card to map a single bucket. For example, in the previous example, we want the 'ssd' bucket to be mapped to the default bucket.

The final command to convert the map comprised of the above fragments would be as follows:

cephadm > ceph osd getcrushmap -o original
cephadm > crushtool -i original --reclassify \
  --set-subtree-class default hdd \
  --reclassify-root default hdd \
  --reclassify-bucket %-ssd ssd default \
  --reclassify-bucket ssd ssd default \
  -o adjusted

In order to verify that the conversion is correct, there is a --compare option that tests a large sample of inputs to the CRUSH Map and compares if the same result comes back out. These inputs are controlled by the same options that apply to the --test. For the above example the command would be as follows:

cephadm > crushtool -i original --compare adjusted
rule 0 had 0/10240 mismatched mappings (0)
rule 1 had 0/10240 mismatched mappings (0)
maps appear equivalent

Tip

If there were differences, you would see what ratio of inputs are remapped in the parentheses.

If you are satisfied with the adjusted CRUSH Map, you can apply it to the cluster:

cephadm > ceph osd setcrushmap -i adjusted

7.1.1.6 For More Information #

Find more details on CRUSH Maps in Section 7.4, “CRUSH Map Manipulation”.

Find more details on Ceph pools in general in Chapter 8, Managing Storage Pools.

Find more details about erasure coded pools in Chapter 10, Erasure Coded Pools.

CRUSH maps contain a list of OSDs, which can be organized into 'buckets' for aggregating the devices into physical locations.

Tip

You can modify the existing types and create your own bucket types.

Ceph’s deployment tools generate a CRUSH Map that contains a bucket for each host, and a root named 'default', which is useful for the default rbd pool. The remaining bucket types provide a means for storing information about the physical location of nodes/buckets, which makes cluster administration much easier when OSDs, hosts, or network hardware malfunction and the administrator needs access to physical hardware.

A bucket has a type, a unique name (string), a unique ID expressed as a negative integer, a weight relative to the total capacity/capability of its item(s), the bucket algorithm ( straw2 by default), and the hash (0 by default, reflecting CRUSH Hash rjenkins1). A bucket may have one or more items. The items may consist of other buckets or OSDs. Items may have a weight that reflects the relative weight of the item.

[bucket-type] [bucket-name] {
  id [a unique negative numeric ID]
  weight [the relative capacity/capability of the item(s)]
  alg [the bucket type: uniform | list | tree | straw2 | straw ]
  hash [the hash type: 0 by default]
  item [item-name] weight [weight]
}

The following example illustrates how you can use buckets to aggregate a pool and physical locations like a data center, a room, a rack and a row.

host ceph-osd-server-1 {
        id -17
        alg straw2
        hash 0
        item osd.0 weight 0.546
        item osd.1 weight 0.546
}

row rack-1-row-1 {
        id -16
        alg straw2
        hash 0
        item ceph-osd-server-1 weight 2.00
}

rack rack-3 {
        id -15
        alg straw2
        hash 0
        item rack-3-row-1 weight 2.00
        item rack-3-row-2 weight 2.00
        item rack-3-row-3 weight 2.00
        item rack-3-row-4 weight 2.00
        item rack-3-row-5 weight 2.00
}

rack rack-2 {
        id -14
        alg straw2
        hash 0
        item rack-2-row-1 weight 2.00
        item rack-2-row-2 weight 2.00
        item rack-2-row-3 weight 2.00
        item rack-2-row-4 weight 2.00
        item rack-2-row-5 weight 2.00
}

rack rack-1 {
        id -13
        alg straw2
        hash 0
        item rack-1-row-1 weight 2.00
        item rack-1-row-2 weight 2.00
        item rack-1-row-3 weight 2.00
        item rack-1-row-4 weight 2.00
        item rack-1-row-5 weight 2.00
}

room server-room-1 {
        id -12
        alg straw2
        hash 0
        item rack-1 weight 10.00
        item rack-2 weight 10.00
        item rack-3 weight 10.00
}

datacenter dc-1 {
        id -11
        alg straw2
        hash 0
        item server-room-1 weight 30.00
        item server-room-2 weight 30.00
}

root data {
        id -10
        alg straw2
        hash 0
        item dc-1 weight 60.00
        item dc-2 weight 60.00
}

CRUSH maps support the notion of 'CRUSH rules', which are the rules that determine data placement for a pool. For large clusters, you will likely create many pools where each pool may have its own CRUSH ruleset and rules. The default CRUSH Map has a rule for the default root. If you want more roots and more rules, you need to create them later or they will be created automatically when new pools are created.

Note

In most cases, you will not need to modify the default rules. When you create a new pool, its default ruleset is 0.

A rule takes the following form:

rule rulename {

        ruleset ruleset
        type type
        min_size min-size
        max_size max-size
        step step

}

7.3.1 Iterating Through the Node Tree #

The structure defined with the buckets can be viewed as a node tree. Buckets are nodes and OSDs are leafs in this tree.

Rules in the CRUSH Map define how OSDs are selected from this tree. A rule starts with a node and then iterates down the tree to return a set of OSDs. It is not possible to define which branch needs to be selected. Instead the CRUSH algorithm assures that the set of OSDs fulfills the replication requirements and evenly distributes the data.

With step take bucket the iteration through the node tree begins at the given bucket (not bucket type). If OSDs from all branches in the tree are to be returned, the bucket must be the root bucket. Otherwise the following steps are only iterating through a sub-tree.

After step take one or more step choose entries follow in the rule definition. Each step choose chooses a defined number of nodes (or branches) from the previously selected upper node.

In the end the selected OSDs are returned with step emit.

step chooseleaf is a convenience function that directly selects OSDs from branches of the given bucket.

Figure 7.2, “Example Tree” provides an example of how step is used to iterate through a tree. The orange arrows and numbers correspond to example1a and example1b, while blue corresponds to example2 in the following rule definitions.

Figure 7.2: Example Tree #

# orange arrows
rule example1a {
        ruleset 0
        type replicated
        min_size 2
        max_size 10
        # orange (1)
        step take rack1
        # orange (2)
        step choose firstn 0 host
        # orange (3)
        step choose firstn 1 osd
        step emit
}

rule example1b {
        ruleset 0
        type replicated
        min_size 2
        max_size 10
        # orange (1)
        step take rack1
        # orange (2) + (3)
        step chooseleaf firstn 0 host
        step emit
}

# blue arrows
rule example2 {
        ruleset 0
        type replicated
        min_size 2
        max_size 10
        # blue (1)
        step take room1
        # blue (2)
        step chooseleaf firstn 0 rack
        step emit
}

7.3.2 firstn and indep #

A CRUSH rule defines replacements for failed nodes or OSDs (see Section 7.3, “Rule Sets”). The keyword step requires either firstn or indep as parameter. Figure Figure 7.3, “Node Replacement Methods” provides an example.

firstn adds replacement nodes to the end of the list of active nodes. In case of a failed node, the following healthy nodes are shifted to the left to fill the gap of the failed node. This is the default and desired method for replicated pools, because a secondary node already has all data and therefore can take over the duties of the primary node immediately.

indep selects fixed replacement nodes for each active node. The replacement of a failed node does not change the order of the remaining nodes. This is desired for erasure coded pools. In erasure coded pools the data stored on a node depends on its position in the node selection. When the order of nodes changes, all data on affected nodes needs to be relocated.

Figure 7.3: Node Replacement Methods #

This section introduces ways to basic CRUSH Map manipulation, such as editing a CRUSH Map, changing CRUSH Map parameters, and adding/moving/removing an OSD.

7.4.1 Editing a CRUSH Map #

To edit an existing CRUSH map, do the following:

1. Get a CRUSH Map. To get the CRUSH Map for your cluster, execute the following:

   cephadm > ceph osd getcrushmap -o compiled-crushmap-filename

   Ceph will output (-o) a compiled CRUSH Map to the file name you specified. Since the CRUSH Map is in a compiled form, you must decompile it first before you can edit it.

2. Decompile a CRUSH Map. To decompile a CRUSH Map, execute the following:

   cephadm > crushtool -d compiled-crushmap-filename \
    -o decompiled-crushmap-filename

   Ceph will decompile (-d) the compiled CRUSH Mapand output (-o) it to the file name you specified.

3. Edit at least one of Devices, Buckets and Rules parameters.

4. Compile a CRUSH Map. To compile a CRUSH Map, execute the following:

   cephadm > crushtool -c decompiled-crush-map-filename \
    -o compiled-crush-map-filename

   Ceph will store a compiled CRUSH Mapto the file name you specified.

5. Set a CRUSH Map. To set the CRUSH Map for your cluster, execute the following:

   cephadm > ceph osd setcrushmap -i compiled-crushmap-filename

   Ceph will input the compiled CRUSH Map of the file name you specified as the CRUSH Map for the cluster.

Tip: Use Versioning System

Use a versioning system—such as git or svn—for the exported and modified CRUSH Map files. It makes a possible rollback simple.

Tip: Test the New CRUSH Map

Test the new adjusted CRUSH Map using the crushtool --test command, and compare to the state before applying the new CRUSH Map. You may find the following command switches useful: --show-statistics, --show-mappings, --show-bad-mappings, --show-utilization, --show-utilization-all, --show-choose-tries

cephadm > ceph osd crush set id_or_name weight root=pool-name
bucket-type=bucket-name ...

cephadm > ceph osd crush set osd.0 1.0 root=data datacenter=dc1 room=room1 \
row=foo rack=bar host=foo-bar-1

7.4.3 Difference between ceph osd reweight and ceph osd crush reweight #

There are two similar commands that change the 'weight' of an Ceph OSD. Context of their usage is different and may cause confusion.

7.4.3.1 ceph osd reweight #

Usage:

cephadm > ceph osd reweight OSD_NAME NEW_WEIGHT

ceph osd reweight sets an override weight on the Ceph OSD. This value is in the range 0 to 1, and forces CRUSH to re-place of the data that would otherwise live on this drive. It does not change the weights assigned to the buckets above the OSD, and is a corrective measure in case the normal CRUSH distribution is not working out quite right. For example, if one of your OSDs is at 90% and the others are at 40%, you could reduce this weight to try and compensate for it.

Note: OSD Weight is Temporary

Note that ceph osd reweight is not a persistent setting. When an OSD gets marked out, its weight will be set to 0 and when it gets marked in again, the weight will be changed to 1.

7.4.3.2 ceph osd crush reweight #

Usage:

cephadm > ceph osd crush reweight OSD_NAME NEW_WEIGHT

ceph osd crush reweight sets the CRUSH weight of the OSD. This weight is an arbitrary value—generally the size of the disk in TB—and controls how much data the system tries to allocate to the OSD.

cephadm > ceph osd crush remove OSD_NAME

cephadm > ceph osd crush add-bucket BUCKET_NAME BUCKET_TYPE

cephadm > ceph osd crush move BUCKET_NAME BUCKET_TYPE=BUCKET_NAME [...]

cephadm > ceph osd crush move bucket1 datacenter=dc1 room=room1 row=foo rack=bar host=foo-bar-1

7.4.7 Remove a Bucket #

To remove a bucket from the CRUSH Map hierarchy, execute the following:

cephadm > ceph osd crush remove BUCKET_NAME

Note: Empty Bucket Only

A bucket must be empty before removing it from the CRUSH hierarchy.

In addition to making multiple copies of objects, Ceph insures data integrity by scrubbing placement groups (find more information about placement groups in Book “Deployment Guide”, Chapter 1 “SUSE Enterprise Storage 5.5 and Ceph”, Section 1.4.2 “Placement Group”). Ceph scrubbing is analogous to running fsck on the object storage layer. For each placement group, Ceph generates a catalog of all objects and compares each primary object and its replicas to ensure that no objects are missing or mismatched. Daily light scrubbing checks the object size and attributes, while weekly deep scrubbing reads the data and uses checksums to ensure data integrity.

Scrubbing is important for maintaining data integrity, but it can reduce performance. You can adjust the following settings to increase or decrease scrubbing operations:

Before using pools, you need to associate them with an application. Pools that will be used with CephFS, or pools that are automatically created by Object Gateway are automatically associated.

For other cases, you can manually associate a free-form application name with a pool:

cephadm > ceph osd pool application enable pool_name application_name

Tip: Default Application Names

CephFS uses the application name cephfs, RADOS Block Device uses rbd, and Object Gateway uses rgw.

A pool can be associated with multiple applications, and each application can have its own metadata. You can display the application metadata for a given pool using the following command:

cephadm > ceph osd pool application get pool_name

This section introduces practical information to perform basic tasks with pools. You can find out how to list, create, and delete pools, as well as show pool statistics or manage snapshots of a pool.

cephadm > ceph osd pool ls

cephadm > ceph osd pool create pool_name pg_num pgp_num replicated crush_ruleset_name \
expected_num_objects

cephadm > ceph osd pool create pool_name pg_num pgp_num erasure erasure_code_profile \
 crush_ruleset_name expected_num_objects

cephadm > ceph osd pool set-quota pool-name max_objects obj-count max_bytes bytes

cephadm > ceph osd pool set-quota data max_objects 10000

8.2.4 Delete a Pool #

Warning: Pool Deletion is Not Reversible

Pools may contain important data. Deleting a pool causes all data in the pool to disappear, and there is no way to recover it.

Because inadvertent pool deletion is a real danger, Ceph implements two mechanisms that prevent pools from being deleted. Both mechanisms must be disabled before a pool can be deleted.

The first mechanism is the NODELETE flag. Each pool has this flag, and its default value is 'false'. To find out the value of this flag on a pool, run the following command:

cephadm > ceph osd pool get pool_name nodelete

If it outputs nodelete: true, it is not possible to delete the pool until you change the flag using the following command:

cephadm > ceph osd pool set pool_name nodelete false

The second mechanism is the cluster-wide configuration parameter mon allow pool delete, which defaults to 'false'. This means that, by default, it is not possible to delete a pool. The error message displayed is:

Error EPERM: pool deletion is disabled; you must first set the
mon_allow_pool_delete config option to true before you can destroy a pool

To delete the pool in spite of this safety setting, you can temporarily set mon allow pool delete to 'true', delete the pool, and then return the parameter to 'false':

cephadm > ceph tell mon.* injectargs --mon-allow-pool-delete=true
cephadm > ceph osd pool delete pool_name pool_name --yes-i-really-really-mean-it
cephadm > ceph tell mon.* injectargs --mon-allow-pool-delete=false

The injectargs command displays the following message:

injectargs:mon_allow_pool_delete = 'true' (not observed, change may require restart)

This is merely confirming that the command was executed successfully. It is not an error.

If you created your own rulesets and rules for a pool you created, you should consider removing them when you no longer need your pool.

cephadm > ceph osd pool rename current-pool-name new-pool-name

cephadm > rados df
pool name  category  KB  objects   lones  degraded  unfound  rd  rd KB  wr  wr KB
cold-storage    -   228   1         0      0          0       0   0      1   228
data            -    1    4         0      0          0       0   0      4    4
hot-storage     -    1    2         0      0          0       15  10     5   231
metadata        -    0    0         0      0          0       0   0      0    0
pool1           -    0    0         0      0          0       0   0      0    0
rbd             -    0    0         0      0          0       0   0      0    0
total used          266268          7
total avail       27966296
total space       28232564

8.2.7 Get Pool Values #

To get a value from a pool, execute:

cephadm > ceph osd pool get pool-name key

You can get values for keys listed in Section 8.2.8, “Set Pool Values” plus the following keys:

pg_num

The number of placement groups for the pool.

pgp_num

The effective number of placement groups to use when calculating data placement. Valid range is equal to or less than pg_num.

Tip: All Pool's Values

To list all values related to a specific pool, run:

cephadm > ceph osd pool get POOL_NAME all

cephadm > ceph osd pool set pool-name key value

8.2.9 Set the Number of Object Replicas #

To set the number of object replicas on a replicated pool, execute the following:

cephadm > ceph osd pool set poolname size num-replicas

The num-replicas includes the object itself. For example if you want the object and two copies of the object for a total of three instances of the object, specify 3.

Warning: Do not Set Less than 3 Replicas

If you set the num-replicas to 2, there will be only one copy of your data. If you lose one object instance, you need to trust that the other copy has not been corrupted for example since the last scrubbing during recovery (refer to Section 7.5, “Scrubbing” for details).

Setting a pool to one replica means that there is exactly one instance of the data object in the pool. If the OSD fails, you lose the data. A possible usage for a pool with one replica is storing temporary data for a short time.

Tip: Setting More than 3 Replicas

Setting 4 replicas for a pool increases the reliability by 25%.

In case of two data centers, you need to set at least 4 replicas for a pool to have two copies in each data center so that in case one data center is lost, still two copies exist and you can still lose one disk without loosing data.

Note

An object might accept I/Os in degraded mode with fewer than pool size replicas. To set a minimum number of required replicas for I/O, you should use the min_size setting. For example:

cephadm > ceph osd pool set data min_size 2

This ensures that no object in the data pool will receive I/O with fewer than min_size replicas.

Tip: Get the Number of Object Replicas

To get the number of object replicas, execute the following:

cephadm > ceph osd dump | grep 'replicated size'

Ceph will list the pools, with the replicated size attribute highlighted. By default, Ceph creates two replicas of an object (a total of three copies, or a size of 3).

8.2.10 Increasing the Number of Placement Groups #

When creating a new pool, you specify the number of placement groups for the pool (see Section 8.2.2, “Create a Pool” ). After adding more OSDs to the cluster, you usually need to increase the number of placement groups as well for performance and data durability reasons. For each placement group, OSD and monitor nodes need memory, network and CPU at all times and even more during recovery. From which follows that minimizing the number of placement groups saves significant amounts of resources. On the other hand - too small number of placement groups causes unequal data distribution among OSDs.

Warning: Too High Value of pg_num

When changing the pg_num value for a pool, it may happen that the new number of placement groups exceeds the allowed limit. For example

cephadm > ceph osd pool set rbd pg_num 4096
 Error E2BIG: specified pg_num 3500 is too large (creating 4096 new PGs \
 on ~64 OSDs exceeds per-OSD max of 32)

The limit prevents extreme placement group splitting, and is derived from the mon_osd_max_split_count value.

Important: Reducing the Number of Placement Groups not Possible

While increasing the number of placement groups on a pool is possible at any time, reducing is not possible.

To determine the right new number of placement groups for a resized cluster is a complex task. One approach is to continuously grow the number of placement groups up to the state when the cluster performance is satisfactory. To determine the new incremented number of placement groups, you need to get the value of the mon_osd_max_split_count parameter (default is 32), and add it to the current number of placement groups. To give you a basic idea, take a look at the following script:

cephadm > max_inc=`ceph daemon mon.a config get mon_osd_max_split_count 2>&1 \
  | tr -d '\n ' | sed 's/.*"\([[:digit:]]\+\)".*/\1/'`
cephadm > pg_num=`ceph osd pool get POOL_NAME pg_num | cut -f2 -d: | tr -d ' '`
cephadm > echo "current pg_num value: $pg_num, max increment: $max_inc"
cephadm > next_pg_num="$(($pg_num+$max_inc))"
cephadm > echo "allowed increment of pg_num: $next_pg_num"

After finding out the next number of placement groups, increase it with

cephadm > ceph osd pool set POOL_NAME pg_num NEXT_PG_NUM

When creating a pool (see Section 8.2.2, “Create a Pool”) you need to specify its initial parameters, such as the pool type or the number of placement groups. If you later decide to change any of these parameters—for example when converting a replicated pool into an erasure coded one, or decreasing the number of placement groups—, you need to migrate the pool data to another one whose parameters suit your deployment.

This section describes two migration methods—a cache tier method for general pool data migration, and a method using rbd migrate sub-commands to migrate RBD images to a new pool. Each method has its specifics and limitations.

Pool snapshots are snapshots of the state of the whole Ceph pool. With pool snapshots, you can retain the history of the pool's state. Creating pool snapshots consumes storage space proportional to the pool size. Always check the related storage for enough disk space before creating a snapshot of a pool.

cephadm > ceph osd pool mksnap POOL-NAME SNAP-NAME

cephadm > ceph osd pool mksnap pool1 snap1
created pool pool1 snap snap1

cephadm > rados lssnap -p POOL_NAME

cephadm > rados lssnap -p pool1
1	snap1	2018.12.13 09:36:20
2	snap2	2018.12.13 09:46:03
2 snaps

cephadm > ceph osd pool rmsnap POOL-NAME SNAP-NAME

BlueStore (find more details in Book “Deployment Guide”, Chapter 1 “SUSE Enterprise Storage 5.5 and Ceph”, Section 1.5 “BlueStore”) provides on-the-fly data compression to save disk space. The compression ratio depends on the data stored in the system. Note that compression / de-compression requires additional CPU power.

You can configure data compression globally (see Section 8.5.3, “Global Compression Options”), and then override specific compression settings for each individual pool.

You can enable or disable pool data compression, or change the compression algorithm and mode at any time, regardless of whether the pool contains data or not.

No compression will be applied to existing data after enabling the pool compression.

After disabling the compression of a pool, all its data will be decompressed.

8.5.1 Enable Compression #

To enable data compression for a pool named POOL_NAME, run the following command:

cephadm > ceph osd pool set POOL_NAME compression_algorithm COMPRESSION_ALGORITHM
cephadm > ceph osd pool set POOL_NAME compression_mode COMPRESSION_MODE

Tip: Disabling Pool Compression

To disable data compression for a pool, use 'none' as the compression algorithm:

cephadm > ceph osd pool set POOL_NAME compression_algorithm none

The rbd command enables you to create, list, introspect, and remove block device images. You can also use it, for example, to clone images, create snapshots, rollback an image to a snapshot, or view a snapshot.

9.1.1 Creating a Block Device Image in a Replicated Pool #

Before you can add a block device to a client, you need to create a related image in an existing pool (see Chapter 8, Managing Storage Pools):

cephadm > rbd create --size MEGABYTES POOL-NAME/IMAGE-NAME

For example, to create a 1GB image named 'myimage' that stores information in a pool named 'mypool', execute the following:

cephadm > rbd create --size 1024 mypool/myimage

Tip: Image Size Units

If you omit a size unit shortcut ('G' or 'T'), the image's size is in megabytes. Use 'G' or 'T' after the size number to specify gigabytes or terabytes.

cephadm > ceph osd pool create POOL_NAME 12 12 erasure
cephadm > ceph osd pool set POOL_NAME allow_ec_overwrites true

#Metadata will reside in pool "OTHER_POOL", and data in pool "POOL_NAME"
cephadm > rbd create IMAGE_NAME --size=1G --data-pool POOL_NAME --pool=OTHER_POOL

cephadm > rbd ls mypool

cephadm > rbd info mypool/myimage

cephadm > rbd resize --size 2048 POOL_NAME/IMAGE_NAME # to increase
cephadm > rbd resize --size 2048 POOL_NAME/IMAGE_NAME --allow-shrink # to decrease

cephadm > rbd rm mypool/myimage

After you create a RADOS Block Device, you can use it as any other disk device: format it, mount it to be able to exchange files, and unmount it when done.

Tip: Manual (Un)mounting

Since manually mapping and mounting RBD images after boot and unmounting and unmapping them before shutdown can be tedious, an rbdmap script and systemd unit is provided. Refer to Section 9.2.1, “rbdmap: Map RBD Devices at Boot Time”.

image_specification rbd_options

An RBD snapshot is a snapshot of a RADOS Block Device image. With snapshots, you retain a history of the image's state. Ceph also supports snapshot layering, which allows you to clone VM images quickly and easily. Ceph supports block device snapshots using the rbd command and many higher-level interfaces, including QEMU, libvirt, OpenStack, and CloudStack.

Note

Stop input and output operations and flush all pending writes before snapshotting an image. If the image contains a file system, the file system must be in a consistent state at the time of snapshotting.

9.3.1 Cephx Notes #

When cephx is enabled (see http://ceph.com/docs/master/rados/configuration/auth-config-ref/ for more information), you must specify a user name or ID and a path to the keyring containing the corresponding key for the user. See User Management for more details. You may also add the CEPH_ARGS environment variable to avoid re-entry of the following parameters.

cephadm > rbd --id user-ID --keyring=/path/to/secret commands
cephadm > rbd --name username --keyring=/path/to/secret commands

For example:

cephadm > rbd --id admin --keyring=/etc/ceph/ceph.keyring commands
cephadm > rbd --name client.admin --keyring=/etc/ceph/ceph.keyring commands

Tip

Add the user and secret to the CEPH_ARGS environment variable so that you do not need to enter them each time.

9.3.2 Snapshot Basics #

The following procedures demonstrate how to create, list, and remove snapshots using the rbd command on the command line.

9.3.2.1 Create Snapshot #

To create a snapshot with rbd, specify the snap create option, the pool name, and the image name.

cephadm > rbd --pool pool-name snap create --snap snap-name image-name
cephadm > rbd snap create pool-name/image-name@snap-name

For example:

cephadm > rbd --pool rbd snap create --snap snapshot1 image1
cephadm > rbd snap create rbd/image1@snapshot1

9.3.2.2 List Snapshots #

To list snapshots of an image, specify the pool name and the image name.

cephadm > rbd --pool pool-name snap ls image-name
cephadm > rbd snap ls pool-name/image-name

For example:

cephadm > rbd --pool rbd snap ls image1
cephadm > rbd snap ls rbd/image1

9.3.2.3 Rollback Snapshot #

To rollback to a snapshot with rbd, specify the snap rollback option, the pool name, the image name, and the snapshot name.

cephadm > rbd --pool pool-name snap rollback --snap snap-name image-name
cephadm > rbd snap rollback pool-name/image-name@snap-name

For example:

cephadm > rbd --pool pool1 snap rollback --snap snapshot1 image1
cephadm > rbd snap rollback pool1/image1@snapshot1

Note

Rolling back an image to a snapshot means overwriting the current version of the image with data from a snapshot. The time it takes to execute a rollback increases with the size of the image. It is faster to clone from a snapshot than to rollback an image to a snapshot, and it is the preferred method of returning to a pre-existing state.

9.3.2.4 Delete a Snapshot #

To delete a snapshot with rbd, specify the snap rm option, the pool name, the image name, and the user name.

cephadm > rbd --pool pool-name snap rm --snap snap-name image-name
cephadm > rbd snap rm pool-name/image-name@snap-name

For example:

cephadm > rbd --pool pool1 snap rm --snap snapshot1 image1
cephadm > rbd snap rm pool1/image1@snapshot1

Note

Ceph OSDs delete data asynchronously, so deleting a snapshot does not free up the disk space immediately.

9.3.2.5 Purge Snapshots #

To delete all snapshots for an image with rbd, specify the snap purge option and the image name.

cephadm > rbd --pool pool-name snap purge image-name
cephadm > rbd snap purge pool-name/image-name

For example:

cephadm > rbd --pool pool1 snap purge image1
cephadm > rbd snap purge pool1/image1

9.3.3 Layering #

Ceph supports the ability to create multiple copy-on-write (COW) clones of a block device snapshot. Snapshot layering enables Ceph block device clients to create images very quickly. For example, you might create a block device image with a Linux VM written to it, then, snapshot the image, protect the snapshot, and create as many copy-on-write clones as you like. A snapshot is read-only, so cloning a snapshot simplifies semantics—making it possible to create clones rapidly.

Note

The terms 'parent' and 'child' mentioned in the command line examples below mean a Ceph block device snapshot (parent) and the corresponding image cloned from the snapshot (child).

Each cloned image (child) stores a reference to its parent image, which enables the cloned image to open the parent snapshot and read it.

A COW clone of a snapshot behaves exactly like any other Ceph block device image. You can read to, write from, clone, and resize cloned images. There are no special restrictions with cloned images. However, the copy-on-write clone of a snapshot refers to the snapshot, so you must protect the snapshot before you clone it.

Note: --image-format 1 not Supported

You cannot create snapshots of images created with the deprecated rbd create --image-format 1 option. Ceph only supports cloning of the default format 2 images.

9.3.3.1 Getting Started with Layering #

Ceph block device layering is a simple process. You must have an image. You must create a snapshot of the image. You must protect the snapshot. After you have performed these steps, you can begin cloning the snapshot.

The cloned image has a reference to the parent snapshot, and includes the pool ID, image ID, and snapshot ID. The inclusion of the pool ID means that you may clone snapshots from one pool to images in another pool.

• Image Template: A common use case for block device layering is to create a master image and a snapshot that serves as a template for clones. For example, a user may create an image for a Linux distribution (for example, SUSE Linux Enterprise Server), and create a snapshot for it. Periodically, the user may update the image and create a new snapshot (for example, zypper ref && zypper patch followed by rbd snap create). As the image matures, the user can clone any one of the snapshots.

• Extended Template: A more advanced use case includes extending a template image that provides more information than a base image. For example, a user may clone an image (a VM template) and install other software (for example, a database, a content management system, or an analytics system), and then snapshot the extended image, which itself may be updated in the same way as the base image.

• Template Pool: One way to use block device layering is to create a pool that contains master images that act as templates, and snapshots of those templates. You may then extend read-only privileges to users so that they may clone the snapshots without the ability to write or execute within the pool.

• Image Migration/Recovery: One way to use block device layering is to migrate or recover data from one pool into another pool.

9.3.3.2 Protecting a Snapshot #

Clones access the parent snapshots. All clones would break if a user inadvertently deleted the parent snapshot. To prevent data loss, you need to protect the snapshot before you can clone it.

cephadm > rbd --pool pool-name snap protect \
 --image image-name --snap snapshot-name
cephadm > rbd snap protect pool-name/image-name@snapshot-name

For example:

cephadm > rbd --pool pool1 snap protect --image image1 --snap snapshot1
cephadm > rbd snap protect pool1/image1@snapshot1

Note

You cannot delete a protected snapshot.

9.3.3.3 Cloning a Snapshot #

To clone a snapshot, you need to specify the parent pool, image, snapshot, the child pool, and the image name. You need to protect the snapshot before you can clone it.

cephadm > rbd clone --pool pool-name --image parent-image \
 --snap snap-name --dest-pool pool-name \
 --dest child-image
cephadm > rbd clone pool-name/parent-image@snap-name \
pool-name/child-image-name

For example:

cephadm > rbd clone pool1/image1@snapshot1 pool1/image2

Note

You may clone a snapshot from one pool to an image in another pool. For example, you may maintain read-only images and snapshots as templates in one pool, and writable clones in another pool.

9.3.3.4 Unprotecting a Snapshot #

Before you can delete a snapshot, you must unprotect it first. Additionally, you may not delete snapshots that have references from clones. You need to flatten each clone of a snapshot before you can delete the snapshot.

cephadm > rbd --pool pool-name snap unprotect --image image-name \
 --snap snapshot-name
cephadm > rbd snap unprotect pool-name/image-name@snapshot-name

For example:

cephadm > rbd --pool pool1 snap unprotect --image image1 --snap snapshot1
cephadm > rbd snap unprotect pool1/image1@snapshot1

9.3.3.5 Listing Children of a Snapshot #

To list the children of a snapshot, execute the following:

cephadm > rbd --pool pool-name children --image image-name --snap snap-name
cephadm > rbd children pool-name/image-name@snapshot-name

For example:

cephadm > rbd --pool pool1 children --image image1 --snap snapshot1
cephadm > rbd children pool1/image1@snapshot1

9.3.3.6 Flattening a Cloned Image #

Cloned images retain a reference to the parent snapshot. When you remove the reference from the child clone to the parent snapshot, you effectively 'flatten' the image by copying the information from the snapshot to the clone. The time it takes to flatten a clone increases with the size of the snapshot. To delete a snapshot, you must flatten the child images first.

cephadm > rbd --pool pool-name flatten --image image-name
cephadm > rbd flatten pool-name/image-name

For example:

cephadm > rbd --pool pool1 flatten --image image1
cephadm > rbd flatten pool1/image1

Note

Since a flattened image contains all the information from the snapshot, a flattened image will take up more storage space than a layered clone.

RBD images can be asynchronously mirrored between two Ceph clusters. This capability uses the RBD journaling image feature to ensure crash-consistent replication between clusters. Mirroring is configured on a per-pool basis within peer clusters and can be configured to automatically mirror all images within a pool or only a specific subset of images. Mirroring is configured using the rbd command. The rbd-mirror daemon is responsible for pulling image updates from the remote peer cluster and applying them to the image within the local cluster.

Note: rbd-mirror Daemon

To use RBD mirroring, you need to have two Ceph clusters, each running the rbd-mirror daemon.

Important: RADOS Block Devices Exported via iSCSI

You cannot mirror RBD devices that are exported via iSCSI using lrbd.

Refer to Chapter 14, Ceph iSCSI Gateway for more details on iSCSI.

9.4.1 rbd-mirror Daemon #

The two rbd-mirror daemons are responsible for watching image journals on the remote, peer cluster and replaying the journal events against the local cluster. The RBD image journaling feature records all modifications to the image in the order they occur. This ensures that a crash-consistent mirror of the remote image is available locally.

The rbd-mirror daemon is available in the rbd-mirror package. You can install the package on OSD nodes, gateway nodes, or even on dedicated nodes. We do not recommend installing the rbd-mirror on the Salt master/admin node. Install, enable, and start rbd-mirror:

root # zypper install rbd-mirror
root # systemctl enable ceph-rbd-mirror@server_name.service
root # systemctl start ceph-rbd-mirror@server_name.service

Important

Each rbd-mirror daemon requires the ability to connect to both clusters simultaneously.

9.4.2 Pool Configuration #

The following procedures demonstrate how to perform the basic administrative tasks to configure mirroring using the rbd command. Mirroring is configured on a per-pool basis within the Ceph clusters.

You need to perform the pool configuration steps on both peer clusters. These procedures assume two clusters, named 'local' and 'remote', are accessible from a single host for clarity.

See the rbd manual page (man 8 rbd) for additional details on how to connect to different Ceph clusters.

Tip: Multiple Clusters

The cluster name in the following examples corresponds to a Ceph configuration file of the same name /etc/ceph/remote.conf. See the ceph-conf documentation for how to configure multiple clusters.

9.4.2.1 Enable Mirroring on a Pool #

To enable mirroring on a pool, specify the mirror pool enable subcommand, the pool name, and the mirroring mode. The mirroring mode can either be pool or image:

pool

All images in the pool with the journaling feature enabled are mirrored.

image

Mirroring needs to be explicitly enabled on each image. See Section 9.4.3.2, “Enable Image Mirroring” for more information.

For example:

cephadm > rbd --cluster local mirror pool enable POOL_NAME pool
cephadm > rbd --cluster remote mirror pool enable POOL_NAME pool

9.4.2.2 Disable Mirroring #

To disable mirroring on a pool, specify the mirror pool disable subcommand and the pool name. When mirroring is disabled on a pool in this way, mirroring will also be disabled on any images (within the pool) for which mirroring was enabled explicitly.

cephadm > rbd --cluster local mirror pool disable POOL_NAME
cephadm > rbd --cluster remote mirror pool disable POOL_NAME

9.4.2.3 Add Cluster Peer #

In order for the rbd-mirror daemon to discover its peer cluster, the peer needs to be registered to the pool. To add a mirroring peer cluster, specify the mirror pool peer add subcommand, the pool name, and a cluster specification:

cephadm > rbd --cluster local mirror pool peer add POOL_NAME client.remote@remote
cephadm > rbd --cluster remote mirror pool peer add POOL_NAME client.local@local

9.4.2.4 Remove Cluster Peer #

To remove a mirroring peer cluster, specify the mirror pool peer remove subcommand, the pool name, and the peer UUID (available from the rbd mirror pool info command):

cephadm > rbd --cluster local mirror pool peer remove POOL_NAME \
 55672766-c02b-4729-8567-f13a66893445
cephadm > rbd --cluster remote mirror pool peer remove POOL_NAME \
 60c0e299-b38f-4234-91f6-eed0a367be08

9.4.3 Image Configuration #

Unlike pool configuration, image configuration only needs to be performed against a single mirroring peer Ceph cluster.

Mirrored RBD images are designated as either primary or non-primary. This is a property of the image and not the pool. Images that are designated as non-primary cannot be modified.

Images are automatically promoted to primary when mirroring is first enabled on an image (either implicitly if the pool mirror mode was 'pool' and the image has the journaling image feature enabled, or explicitly (see Section 9.4.3.2, “Enable Image Mirroring”) by the rbd command).

9.4.3.1 Image Journaling Support #

RBD mirroring uses the RBD journaling feature to ensure that the replicated image always remains crash-consistent. Before an image can be mirrored to a peer cluster, the journaling feature must be enabled. The feature can be enabled at the time of image creation by providing the --image-feature exclusive-lock,journaling option to the rbd command.

Alternatively, the journaling feature can be dynamically enabled on pre-existing RBD images. To enable journaling, specify the feature enable subcommand, the pool and image name, and the feature name:

cephadm > rbd --cluster local feature enable POOL_NAME/IMAGE_NAME journaling

Note: Option Dependency

The journaling feature is dependent on the exclusive-lock feature. If the exclusive-lock feature is not already enabled, you need to enable it prior to enabling the journaling feature.

Warning: Journaling on All New Images

You can enable journaling on all new images by default by appending the journaling value to the rbd default features option in the Ceph configuration file. For example:

rbd default features = layering,exclusive-lock,object-map,deep-flatten,journaling

Before applying such change, carefully consider if enabling journaling on all new images is good for your deployment because it can have negative performance impact.

9.4.3.2 Enable Image Mirroring #

If mirroring is configured in the 'image' mode, then it is necessary to explicitly enable mirroring for each image within the pool. To enable mirroring for a specific image, specify the mirror image enable subcommand along with the pool and image name:

cephadm > rbd --cluster local mirror image enable POOL_NAME/IMAGE_NAME

9.4.3.3 Disable Image Mirroring #

To disable mirroring for a specific image, specify the mirror image disable subcommand along with the pool and image name:

cephadm > rbd --cluster local mirror image disable POOL_NAME/IMAGE_NAME

9.4.3.4 Image Promotion and Demotion #

In a failover scenario where the primary designation needs to be moved to the image in the peer cluster, you need to stop access to the primary image, demote the current primary image, promote the new primary image, and resume access to the image on the alternate cluster.

Note: Forced Promotion

Promotion can be forced using the --force option. Forced promotion is needed when the demotion cannot be propagated to the peer cluster (for example, in case of cluster failure or communication outage). This will result in a split-brain scenario between the two peers, and the image will no longer be synchronized until a resyncsubcommand is issued.

To demote a specific image to non-primary, specify the mirror image demote subcommand along with the pool and image name:

cephadm > rbd --cluster local mirror image demote POOL_NAME/IMAGE_NAME

To demote all primary images within a pool to non-primary, specify the mirror pool demote subcommand along with the pool name:

cephadm > rbd --cluster local mirror pool demote POOL_NAME

To promote a specific image to primary, specify the mirror image promote subcommand along with the pool and image name:

cephadm > rbd --cluster remote mirror image promote POOL_NAME/IMAGE_NAME

To promote all non-primary images within a pool to primary, specify the mirror pool promote subcommand along with the pool name:

cephadm > rbd --cluster local mirror pool promote POOL_NAME

Tip: Split I/O Load

Since the primary or non-primary status is per-image, it is possible to have two clusters split the IO load and stage failover or failback.

9.4.3.5 Force Image Resync #

If a split-brain event is detected by the rbd-mirror daemon, it will not attempt to mirror the affected image until corrected. To resume mirroring for an image, first demote the image determined to be out of date and then request a resync to the primary image. To request an image resync, specify the mirror image resync subcommand along with the pool and image name:

cephadm > rbd mirror image resync POOL_NAME/IMAGE_NAME

9.4.4 Mirror Status #

The peer cluster replication status is stored for every primary mirrored image. This status can be retrieved using the mirror image status and mirror pool status subcommands:

To request the mirror image status, specify the mirror image status subcommand along with the pool and image name:

cephadm > rbd mirror image status POOL_NAME/IMAGE_NAME

To request the mirror pool summary status, specify the mirror pool status subcommand along with the pool name:

cephadm > rbd mirror pool status POOL_NAME

Tip:

Adding the --verbose option to the mirror pool status subcommand will additionally output status details for every mirroring image in the pool.

RADOS Block Device supports advanced features that enhance the functionality of RBD images. You can specify the features either on the command line when creating an RBD image, or in the Ceph configuration file by using the rbd_default_features option.

You can specify the values of the rbd_default_features option in two ways:

Note: Features not Supported by iSCSI

RBD images with the following features will not be supported by iSCSI: deep-flatten, striping, exclusive-lock, object-map, journaling, fast-diff

List of advanced RBD features follows:

To make use of erasure coding, you need to:

Keep in mind that changing the profile and the details in the profile will not be possible after the pool was created and has data.

Ensure that the CRUSH rules for erasure pools use indep for step. For details see Section 7.3.2, “firstn and indep”.

The simplest erasure coded pool is equivalent to RAID5 and requires at least three hosts. This procedure describes how to create a pool for testing purposes.