Operation of Ceph services | Administration and Operations Guide

Applies to SUSE Enterprise Storage 7.1

14 Operation of Ceph services #

You can operate Ceph services on a daemon, node, or cluster level. Depending on which approach you need, use cephadm or the systemctl command.

14.1 Operating individual services #

If you need to operate an individual service, identify it first:

cephuser@adm > ceph orch ps
NAME                                HOST        STATUS         REFRESHED  [...]
mds.my_cephfs.ses-min1.oterul       ses-min1    running (5d)   8m ago
mgr.ses-min1.gpijpm                 ses-min1    running (5d)   8m ago
mgr.ses-min2.oopvyh                 ses-min2    running (5d)   8m ago
mon.ses-min1                        ses-min1    running (5d)   8m ago
mon.ses-min2                        ses-min2    running (5d)   8m ago
mon.ses-min4                        ses-min4    running (5d)   7m ago
osd.0                               ses-min2    running (61m)  8m ago
osd.1                               ses-min3    running (61m)  7m ago
osd.2                               ses-min4    running (61m)  7m ago
rgw.myrealm.myzone.ses-min1.kwwazo  ses-min1    running (5d)   8m ago
rgw.myrealm.myzone.ses-min2.jngabw  ses-min2    error          8m ago

To identify a service on a specific node, run:

ceph orch ps NODE_HOST_NAME

For example:

cephuser@adm > ceph orch ps ses-min2
NAME                                HOST      STATUS         REFRESHED
mgr.ses-min2.oopvyh                 ses-min2  running (5d)   3m ago
mon.ses-min2                        ses-min2  running (5d)   3m ago
osd.0                               ses-min2  running (67m)  3m ago

Tip

The ceph orch ps command supports several output formats. To change it, append the --format FORMAT option where FORMAT is one of json, json-pretty, or yaml. For example:

cephuser@adm > ceph orch ps --format yaml

Once you know the name of the service you can start, restart, or stop it:

ceph orch daemon COMMAND SERVICE_NAME

For example, to restart the OSD service with ID 0, run:

cephuser@adm > ceph orch daemon restart osd.0

14.2 Operating service types #

If you need to operate a specific type of service across the whole Ceph cluster, use the following command:

ceph orch COMMAND SERVICE_TYPE

Replace COMMAND with either start, stop, or restart.

For example, the following command restarts all MONs in the cluster, regardless of which nodes they actually run on:

cephuser@adm > ceph orch restart mon

14.3 Operating services on a single node #

By using the systemctl command, you can operate Ceph related systemd services and targets on a single node.

14.3.1 Identifying services and targets #

Before operating Ceph related systemd services and targets, you need to identify the file names of their unit files. File names of the services have the following pattern:

ceph-FSID@SERVICE_TYPE.ID.service

For example:

ceph-b4b30c6e-9681-11ea-ac39-525400d7702d@mon.doc-ses-min1.service

ceph-b4b30c6e-9681-11ea-ac39-525400d7702d@rgw.myrealm.myzone.doc-ses-min1.kwwazo.service

FSID: Unique ID of the Ceph cluster. You can find it in the output of the ceph fsid command.
SERVICE_TYPE: Type of the service, for example osd, mon, or rgw.
ID: Identification string of the service. For OSDs, it is the ID number of the service. For other services, it can be either a host name of the node, or additional strings relevant for the service type.

Tip

The SERVICE_TYPE.ID part is identical to the content of the NAME column in the output of the ceph orch ps command.

14.3.2 Operating all services on a node #

By using Ceph's systemd targets, you can simultaneously operate either all services on a node, or all services that belong to a cluster identified by its FSID.

For example, to stop all Ceph services on a node regardless to which cluster the services belong to, run:

root@minion > systemctl stop ceph.target

To restart all services that belong to a Ceph cluster with ID b4b30c6e-9681-11ea-ac39-525400d7702d, run:

root@minion > systemctl restart ceph-b4b30c6e-9681-11ea-ac39-525400d7702d.target

14.3.3 Operating an individual service on a node #

After you have identified the name of a specific service, operate it the following way:

systemctl COMMAND SERVICE_NAME

For example, to restart a single OSD service with ID 1 on a cluster with ID b4b30c6e-9681-11ea-ac39-525400d7702d, run:

# systemctl restart ceph-b4b30c6e-9681-11ea-ac39-525400d7702d@osd.1.service

14.3.4 Querying the service status #

You can query systemd for the status of services. For example:

# systemctl status ceph-b4b30c6e-9681-11ea-ac39-525400d7702d@osd.0.service

14.4 Shutting down and restarting the whole Ceph cluster #

Shutting down and restarting the cluster may be necessary in the case of a planned power outage. To stop all Ceph related services and restart without issue, follow the steps below.

Procedure 14.1: Shutting down the whole Ceph cluster #

Shut down or disconnect any clients accessing the cluster.
To prevent CRUSH from automatically rebalancing the cluster, set the cluster to noout:
```
cephuser@adm > ceph osd set noout
```
Stop all Ceph services on all cluster nodes:
```
root@master # ceph-salt stop
```

Power off all cluster nodes:

root@master # salt -G 'ceph-salt:member' cmd.run "shutdown -h"

Procedure 14.2: Starting the whole Ceph cluster #

Power on the Admin Node.
Power on the Ceph Monitor nodes.
Power on the Ceph OSD nodes.
Unset the previously set noout flag:
```
root@master # ceph osd unset noout
```
Power on all configured gateways.
Power on or connect cluster clients.