13 Operational tasks #

To identify used and unused disk devices on all cluster nodes, list them by running the following command:

cephuser@adm > ceph orch device ls
HOST       PATH      TYPE SIZE  DEVICE  AVAIL REJECT REASONS
ses-admin  /dev/vda  hdd  42.0G         False locked
ses-node1  /dev/vda  hdd  42.0G         False locked
ses-node1  /dev/vdb  hdd  8192M  387836 False locked, LVM detected, Insufficient space (<5GB) on vgs
ses-node2  /dev/vdc  hdd  8192M  450575 True

To re-use a disk device, you need to erase (or zap) it first:

ceph orch device zap HOST_NAME DISK_DEVICE

For example:

cephuser@adm > ceph orch device zap ses-node2 /dev/vdc

Note

If you previously deployed OSDs by using DriveGroups or the --all-available-devices option while the unmanaged flag was not set, cephadm will deploy these OSDs automatically after you erase them.

DriveGroups specify the layouts of OSDs in the Ceph cluster. They are defined in a single YAML file. In this section, we will use drive_groups.yml as an example.

An administrator should manually specify a group of OSDs that are interrelated (hybrid OSDs that are deployed on a mixture of HDDs and SDDs) or share identical deployment options (for example, the same object store, same encryption option, stand-alone OSDs). To avoid explicitly listing devices, DriveGroups use a list of filter items that correspond to a few selected fields of ceph-volume's inventory reports. cephadm will provide code that translates these DriveGroups into actual device lists for inspection by the user.

The command to apply the OSD specification to the cluster is:

cephuser@adm > ceph orch apply osd -i drive_groups.yml

To see a preview of actions and test your application, you can use the --dry-run option together with the ceph orch apply osd command. For example:

cephuser@adm > ceph orch apply osd -i drive_groups.yml --dry-run
...
+---------+------+------+----------+----+-----+
|SERVICE  |NAME  |HOST  |DATA      |DB  |WAL  |
+---------+------+------+----------+----+-----+
|osd      |test  |mgr0  |/dev/sda  |-   |-    |
|osd      |test  |mgr0  |/dev/sdb  |-   |-    |
+---------+------+------+----------+----+-----+

If the --dry-run output matches your expectations, then simply re-run the command without the --dry-run option.

13.4.3.1 Unmanaged OSDs #

  

All available clean disk devices that match the DriveGroups specification will be used as OSDs automatically after you add them to the cluster. This behavior is called a managed mode.

To disable the managed mode, add the unmanaged: true line to the relevant specifications, for example:

service_type: osd
service_id: example_drvgrp_name
placement:
 hosts:
 - ses-node2
 - ses-node3
encrypted: true
unmanaged: true

Tip

To change already deployed OSDs from the managed to unmanaged mode, add the unmanaged: true lines where applicable during the procedure described in Section 13.1, “Modifying the cluster configuration”.

13.4.3.2 DriveGroups specification #

  

Following is an example DriveGroups specification file:

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  drive_spec: DEVICE_SPECIFICATION
db_devices:
  drive_spec: DEVICE_SPECIFICATION
wal_devices:
  drive_spec: DEVICE_SPECIFICATION
block_wal_size: '5G'  # (optional, unit suffixes permitted)
block_db_size: '5G'   # (optional, unit suffixes permitted)
encrypted: true       # 'True' or 'False' (defaults to 'False')

Note

The option previously called "encryption" in DeepSea has been renamed to "encrypted". When applying DriveGroups in SUSE Enterprise Storage 7, ensure you use this new terminology in your service specification, otherwise the ceph orch apply operation will fail.

13.4.3.4 Filtering devices by size #

  

You can filter disk devices by their size—either by an exact size, or a size range. The size: parameter accepts arguments in the following form:

• '10G' - Includes disks of an exact size.

• '10G:40G' - Includes disks whose size is within the range.

• ':10G' - Includes disks less than or equal to 10 GB in size.

• '40G:' - Includes disks equal to or greater than 40 GB in size.

Example 13.1: Matching by disk size #

  

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  size: '40TB:'
db_devices:
  size: ':2TB'

Note: Quotes required

When using the ':' delimiter, you need to enclose the size in quotes, otherwise the ':' sign will be interpreted as a new configuration hash.

Tip: Unit shortcuts

Instead of Gigabytes (G), you can specify the sizes in Megabytes (M) or Terabytes (T).

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  model: SSD-123-foo
db_devices:
  model: MC-55-44-XZ

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  rotational: 1
db_devices:
  rotational: 0

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  size: '2TB:'
db_devices:
  size: ':2TB'

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  rotational: 0
db_devices:
  model: MC-55-44-XZ

service_type: osd
service_id: example_drvgrp_name2
placement:
  host_pattern: '*'
data_devices:
  model: MC-55-44-XZ
db_devices:
  vendor: samsung
  size: 256GB

service_type: osd
service_id: example_drvgrp_one2five
placement:
  host_pattern: 'node[1-5]'
data_devices:
  rotational: 1
db_devices:
  rotational: 0

service_type: osd
service_id: example_drvgrp_rest
placement:
  host_pattern: 'node[6-10]'
data_devices:
  model: MC-55-44-XZ
db_devices:
  model: SSD-123-foo

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  model: MC-55-44-XZ
db_devices:
  model: SSD-123-foo
wal_devices:
  model: NVME-QQQQ-987

service_type: osd
service_id: example_drvgrp_hdd_nvme
placement:
  host_pattern: '*'
data_devices:
  rotational: 0
db_devices:
  model: NVME-QQQQ-987

service_type: osd
service_id: example_drvgrp_hdd_ssd_nvme
placement:
  host_pattern: '*'
data_devices:
  rotational: 0
db_devices:
  model: MC-55-44-XZ
wal_devices:
  model: NVME-QQQQ-987

service_type: osd
service_id: example_drvgrp_ssd_nvme
placement:
  host_pattern: '*'
data_devices:
  model: SSD-123-foo
db_devices:
  model: NVME-QQQQ-987

service_type: osd
service_id: example_drvgrp_standalone_encrypted
placement:
  host_pattern: '*'
data_devices:
  model: SSD-123-foo
encrypted: True

Before removing an OSD node from the cluster, verify that the cluster has more free disk space than the OSD disk you are going to remove. Be aware that removing an OSD results in rebalancing of the whole cluster.

cephuser@adm > ceph orch osd rm stop OSD_SERVICE_ID

There are several reasons why you may need to replace an OSD disk. For example:

To replace an OSD while preserving its ID, run:

cephuser@adm > ceph orch osd rm OSD_SERVICE_ID --replace

For example:

cephuser@adm > ceph orch osd rm 4 --replace

Replacing an OSD is identical to removing an OSD (see Section 13.4.4, “Removing OSDs” for more details) with the exception that the OSD is not permanently removed from the CRUSH hierarchy and is assigned a destroyed flag instead.

The destroyed flag is used to determined OSD IDs that will be reused during the next OSD deployment. Newly added disks that match the DriveGroups specification (see Section 13.4.3, “Adding OSDs using DriveGroups specification” for more details) will be assigned OSD IDs of their replaced counterpart.

Note

In the case of replacing an OSD after a failure, we highly recommend triggering a deep scrub of the placement groups. See Section 17.6, “Scrubbing placement groups” for more details.

Run the following command to initiate a deep scrub:

cephuser@adm > ceph osd deep-scrub osd.OSD_NUMBER

Important: Shared device failure

If a shared device for DB/WAL fails you will need to perform the replacement procedure for all OSDs that share the failed device.

DB device belongs to an OSD and stores its metadata (see Section 1.4, “BlueStore” for more details). There are several reasons why you may want to migrate an existing DB device to a new one—for example, when OSDs have different DB sizes and you need to align them.

Tip: ceph-volume naming convention

Some clusters may have old volume group (VG) or logical volume (LV) names prefixed with ceph-block-dbs and osd-block-db, for example:

ceph-block-dbs-c3dc9227-ca3e-49bc-992c-00602cb3eec7/osd-block-db-b346b9ff-dbbe-40db-a95e-2419ccd31f2c

The current naming convention is as follows:

ceph-c3dc9227-ca3e-49bc-992c-00602cb3eec7/osd-db-b346b9ff-dbbe-40db-a95e-2419ccd31f2c

Before patching the cluster with the latest software packages, verify that all the cluster's nodes have access to the relevant repositories. Refer to Section 10.1.5.1, “Software repositories” for a complete list of the required repositories.

If you use a staging tool—for example, SUSE Manager, Subscription Management Tool, or RMT—that serves software repositories to the cluster nodes, verify that stages for both 'Updates' repositories for SUSE Linux Enterprise Server and SUSE Enterprise Storage are created at the same point in time.

We strongly recommend to use a staging tool to apply patches which have frozen or staged patch levels. This ensures that new nodes joining the cluster have the same patch level as the nodes already running in the cluster. This way you avoid the need to apply the latest patches to all the cluster's nodes before new nodes can join the cluster.

Depending on the configuration, cluster nodes may be rebooted during the update. If there is a single point of failure for services such as Object Gateway, Samba Gateway, NFS Ganesha, or iSCSI, the client machines may be temporarily disconnected from services whose nodes are being rebooted.

To update the software packages on all cluster nodes to the latest version, run the following command:

root@master # ceph-salt update

Before you start the update, verify that all nodes are currently online and your cluster is healthy:

cephuser@adm > cephadm shell -- ceph -s

To update to a specific Ceph release:

cephuser@adm > ceph orch upgrade start --image REGISTRY_URL

For example:

cephuser@adm > ceph orch upgrade start --image registry.suse.com/ses/7.1/ceph/ceph:latest

Upgrade packages on the hosts:

cephuser@adm > ceph-salt update

Run the following command to determine whether an update is in progress:

cephuser@adm > ceph orch upgrade status

While the update is in progress, you will see a progress bar in the Ceph status output:

cephuser@adm > ceph -s
[...]
  progress:
    Upgrade to registry.suse.com/ses/7.1/ceph/ceph:latest (00h 20m 12s)
      [=======.....................] (time remaining: 01h 43m 31s)

You can also watch the cephadm log:

cephuser@adm > ceph -W cephadm

You can stop the update process at any time:

cephuser@adm > ceph orch upgrade stop

The procedure of refreshing the Ceph Dashboard SSL certificate is detailed in Section 10.1, “Configuring TLS/SSL support”:

Renewing the Grafana SSL certificate is almost identical to initial Renewing the Grafana SSL certificate is almost identical to the initial SSL certificate setup mentioned in