If you would like to see the configuration settings at runtime, you must log in to a host with a running daemon and execute the following:

cephuser@adm > cephadm enter --name osd.ID -- ceph daemon osd.ID config show | less

For example:

cephuser@adm > cephadm enter --name osd.0 -- ceph daemon osd.0 config show | less

To activate Ceph’s debugging output (i.e., dout()) at runtime, use the ceph tell command to inject arguments into the runtime configuration:

cephuser@adm > ceph tell {daemon-type}.{daemon id or *} config set {name} {value}

Replace {daemon-type} with one of osd, mon or mds. You may apply the runtime setting to all daemons of a particular type with *, or specify a specific daemon’s ID. For example, to increase debug logging for a ceph-osd daemon named osd.0, execute the following:

cephuser@adm > ceph tell osd.0 config set debug_osd 0/5

The ceph tell command goes through the monitors. If you cannot bind to the monitor, you can make the change by logging into the daemon host using ceph daemon. For example:

cephuser@adm > cephadm enter --name osd.0 -- ceph daemon osd.0 config set debug_osd 0/5

See Section 2.5, “Enable system, log, and debug settings” for details on available settings.

To activate Ceph’s debugging output (i.e., dout()) at boot time, you must add settings to your Ceph configuration file. Subsystems common to each daemon may be set under [global] in your configuration file. Subsystems for particular daemons are set under the daemon section in your configuration file (e.g., [mon], [osd], [mds]). For example:

  [global]
          debug ms = 1/5

  [mon]
          debug mon = 20
          debug paxos = 1/5
          debug auth = 2

  [osd]
          debug osd = 1/5
          debug filestore = 1/5
          debug journal = 1
          debug monc = 5/20

  [mds]
          debug mds = 1
          debug mds balancer = 1

See Section 2.5, “Enable system, log, and debug settings” for details.

If your OS disk is relatively full, you can accelerate log rotation by modifying the Ceph log rotation file at etc/logrotate.d/ceph. Add a size setting after the rotation frequency to accelerate log rotation (via cronjob) if your logs exceed the size setting. For example, the default setting looks like this:

  rotate 7
  weekly
  compress
  sharedscripts

Modify it by adding a size setting:

  rotate 7
  weekly
  size 500M
  compress
  sharedscripts

Then, start the crontab editor for your user space:

crontab -e

Finally, add an entry to check the etc/logrotate.d/ceph file:

30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2><&1

The preceding example checks the etc/logrotate.d/ceph file every 30 minutes.

To monitor the memory utilization of the Ceph processes, pay attention to the top command's VIRT or the output of ps -u ceph -l in the VSZ column. While some fluctuation of memory utilization over time is normal, it should not constantly increase and this will help you pinpoint the problematic process if facing low- or out-of-memory concerns.

In most cases, you will enable debug logging output via subsystems.

debug {subsystem} = {log-level}/{memory-level}
#for example
debug mds balancer = 1/20

The RBD and CephFS kernel clients provide means to enable runtime logging that allow low-level debugging. However, the performance penalty may be high, which is why dynamic debugging is turned off by default. Also, analysing these logs requires comparing them with the kernel source code.

For example, to see the debug messages related with CephFS snapshot handling, you could try the following:

# echo "file fs/ceph/snap.c +pf" > /sys/kernel/debug/dynamic_debug/control

The kernel logs would now include any debug messages in file fs/ceph/snap.c. In order to disable dynamic debugging:

# echo "file fs/ceph/snap.c -p" > /sys/kernel/debug/dynamic_debug/control

In order to simplify debugging failed daemon deployments, cephadm stores events on a per-service and per-daemon basis. To view the list of events for a specific service, run the following example command:

cephuser@adm > ceph orch ls --service_name=alertmanager --format yaml
  service_type: alertmanager
  service_name: alertmanager
  placement:
    hosts:
    - unknown_host
[...]
  events:
  - 2021-02-01T08:58:02.741162 service:alertmanager [INFO] "service was created"
  - 2021-02-01T12:09:25.264584 service:alertmanager [ERROR] "Failed to apply: \
   Cannot place <AlertManagerSpec for service_name=alertmanager> on \
   unknown_host: Unknown hosts"'

As opposed to a specific service, to view the list of events for a specific daemon, for example, run the following command:

cephuser@adm > ceph orch ceph --service-type mds \
 --daemon-id=hostname.ppdhsz --format yaml
  daemon_type: mds
  daemon_id: cephfs.hostname.ppdhsz
  hostname: hostname
[...]
  events:
  - 2021-02-01T08:59:43.845866 daemon:mds.cephfs.hostname.ppdhsz [INFO] \
   "Reconfigured mds.cephfs.hostname.ppdhsz on host 'hostname'"

When a process is crashing, it is useful to find out the reason why it is crashing. If the systemd-coredump service is active, a dump of the memory of the crashing process—also referred to as core dump—is created. It is then possible to attach the gdb debugger to the dump and inspect the cause of the crash.

The following procedure illustrates attaching the gdb debugger to a crashing OSD process. Run the commands in this example on the host where the crashing process is running.

If something goes wrong and cephadm is behaving in a way you do not like, you can pause most background activity by executing the following:

cephuser@adm > ceph orch pause

This will stop any changes, but cephadm will still periodically check hosts to refresh its inventory of daemons and devices. You can disable cephadm completely with:

cephuser@adm > ceph orch set backend ''
cephuser@adm > ceph mgr module disable cephadm

This disables all of the ceph orch ... CLI commands but the previously deployed daemon containers will continue to exist and start as they did before.

You can monitor the cephadm log in real time:

cephuser@adm > ceph -W cephadm

To view the last few messages, execute:

cephuser@adm > ceph log last cephadm

SUSE Enterprise Storage 7.1 supports Ceph logging via systemd-journald. To access the logs of Ceph daemons in SUSE Enterprise Storage 7.1, follow the instructions below.

To print the state of a systemd unit, run:

systemctl status "ceph-$(cephadm shell ceph fsid)@service name.service";

To fetch all state of all daemons of a given host, run:

  fsid="$(cephadm shell ceph fsid)"
  for name in $(cephadm ls | jq -r '.[].name') ; do
    systemctl status "ceph-$fsid@$name.service" > $name;
  done

cephadm has a set of container images configured by default.

To get the value of the configured default container:

cephuser@adm > ceph config get mgr mgr/cephadm/OPTION_NAME

Where OPTION_NAME is any of the following names:

For example:

cephuser@adm > ceph config get mgr mgr/cephadm/container_image_base \
 registry.suse.com/ses/7.1/ceph/ceph

Note

mgr/cephadm/container_image_base is the default Ceph container image used by all services except the monitoring stack and the ingress stack.

To list all container images that are downloaded on a host:

Note

Image may also be called ImageID.

  podman ps -a --format json | jq '.[].Image'
  "docker.io/library/centos:8"
  "registry.opensuse.org/opensuse/leap:15.2"

cephadm writes small wrappers that run a containers. Refer to /var/lib/ceph/cluster-fsid/service-name/unit.run for the container execution command.

xxxxxx.gateway_bootstrap.HostNotFound: -F /tmp/cephadm-conf-kbqvkrkw root@10.10.1.2
raise OrchestratorError('Failed to connect to %s (%s).  Check that the host is reachable and accepts  connections using the cephadm SSH key' % (host, addr)) from
orchestrator._interface.OrchestratorError: Failed to connect to 10.10.1.2 (10.10.1.2).  Check that the host is reachable and accepts connections using the cephadm SSH key

root@master # cephadm shell -- ceph config-key get mgr/cephadm/ssh_identity_pub > key.pub
root@master # grep "`cat key.pub`"  /root/.ssh/authorized_keys

If you see one of the following errors:

ERROR: Failed to infer CIDR network for mon ip ***; pass --skip-mon-network to configure it later

or

Must set public_network config option or specify a CIDR network, ceph addrvec, or plain IP

You need to specify a subnet for Ceph Monitors:

cephuser@adm > ceph config set mon public_network MON_NETWORK

Each Ceph daemon provides an admin socket that bypasses the MONs. To access the admin socket, enter the daemon container on the host:

root@master # cephadm enter --name daemon-name
root@master # ceph --admin-daemon /var/run/ceph/ceph-daemon-name.asok config show

Use the following steps to deploy a new Ceph Manager on a host.

Occasionally, in order to fix a given issue, SUSE will provide a set of packages known as a Program Temporary Fix (PTF). Such a PTF is fully supported by SUSE until the Maintenance Update (MU) containing a permanent fix has been released via the regular update repositories. Customers running PTF fixes will be notified through the related Service Request when a permanent patch for a PTF has been released.

PTFs are published on the registry.suse.com server through a path similar to:

registry.suse.com/ptf/PTF_NUMBER/ses/7.1/ceph/ceph:PTF-PTF_NUMBER

The following steps describe how to deploy PTF images on an existing Ceph cluster:

Important

If you plan to run the ceph-salt update command after applying a PTF (as described in Book “Administration and Operations Guide”, Chapter 13 “Operational tasks”, Section 13.6.4 “Running the update”), it will not affect the manual changes made by the temporary fix.

Conversely, running the ceph orch upgrade command (see Book “Administration and Operations Guide”, Chapter 13 “Operational tasks”, Section 13.7.1 “Starting the update”) will upgrade daemons manually deployed by the PTF.

When cephadm fails to add a new cluster host—no matter whether by means of ceph-salt (see Book “Deployment Guide”, Chapter 7 “Deploying the bootstrap cluster using ceph-salt”, Section 7.2.2 “Adding Salt Minions”) or manually using the ceph orch host add command—you can diagnose the reason for failure. The failure message has the following pattern:

Failed to connect to HOSTNAME (FQDN).

For example:

cephuser@adm > ceph orch host add node5.example.com
Failed to connect to node5 (node5.example.com).

If the message includes a fully qualified domain name (FQDN, node5.example com in our example) as the HOSTNAME, try adding the host using its FQDN:

cephuser@adm > ceph orch host add node5.example.com

If the message includes a short host name (node5 in our example) as the HOSTNAME, try adding the host using both its short name and FQDN:

cephuser@adm > ceph orch host add node5 node5.example.com

If either of the previous two commands succeed, ceph-salt is probably causing the problem. Otherwise, try adding the host using its short host name and IP address:

cephuser@adm > ceph orch host add node5 192.168.2.102

If this succeeds, the cluster does not have proper name resolution. Refer to Book “Deployment Guide”, Chapter 5 “Installing and configuring SUSE Linux Enterprise Server” for more details.

Tip

For more information about resolving full and bare host names via DNS, refer to https://docs.ceph.com/en/octopus/cephadm/concepts/.

By default, Ceph daemons are automatically deployed to new hosts after you add these hosts to the placement specification and apply it (refer to Book “Deployment Guide”, Chapter 8 “Deploying the remaining core services using cephadm” for more details).

If you need to disable the automated deployment of Ceph daemons, add unmanaged: true to its specification file and apply it, for example:

cat mgr.yaml
service_type: mgr
unmanaged: true
placement:
  label: mgr
[...]

cephuser@adm > ceph orch apply -i mgr.yaml

After applying this specification, cephadm will no longer deploy any new daemons on hosts that match the placement specification.

To manually deploy a daemon on a new host, run:

ceph orch daemon add DAEMON_TYPE --placement=PLACEMENT_SPEC

For example:

cephuser@adm > ceph orch daemon add mgr --placement=ses-node3

To manually remove a daemon, run:

ceph orch daemon rm DAEMON_NAME [--force]

For example:

cephuser@adm > ceph orch daemon rm mgr.ses-node3.xietsy

To begin troubleshooting the OSDs, obtain any information including the information collected from Book “Administration and Operations Guide”, Chapter 12 “Determine the cluster state”, Section 12.9 “Monitoring OSDs and placement groups”.

cephuser@adm > ls /var/log/ceph

cephuser@adm > ls /var/run/ceph

cephuser@adm > ceph daemon DAEMON-NAME help

cephuser@adm > ceph daemon SOCKET-FILE help

cephuser@adm > df -h

cephuser@adm > iostat -x

cephuser@adm > dmesg | grep scsi

Periodically you may need to perform maintenance on a subset of your cluster or resolve a problem that affects a failure domain. If you do not want CRUSH to automatically rebalance the cluster as you stop OSDs for maintenance, set the cluster to noout first:

cephuser@adm > ceph osd set noout

Once the cluster is set to noout, you can begin stopping the OSDs within the failure domain that requires maintenance work:

cephuser@adm > ceph orch daemon stop osd.ID

Note

Placement groups within the OSDs you stop will become degraded while you are addressing issues with within the failure domain.

Once you have completed your maintenance, restart the OSDs:

cephuser@adm > ceph orch daemon start osd.ID

Finally, unset the cluster from noout:

cephuser@adm > ceph osd unset noout

Under normal circumstances, restarting the ceph-osd daemon will allow it to rejoin the cluster and recover. If this is not true, continue on to the next section.

cephuser@adm > ceph health
HEALTH_WARN 1/3 in osds are down

cephuser@adm > ceph health detail
HEALTH_WARN 1/3 in osds are down
osd.0 is down since epoch 23, last address 192.168.106.220:6800/11080

cephuser@adm > ceph -W cephadm

cephuser@adm > ceph log last cephadm

4.3.3 Preventing write action to OSD #

  

Ceph prevents writing to a full OSD so that you do not lose data. In an operational cluster, you should receive a warning when your cluster is getting near its full ratio. The mon osd full ratio defaults to 0.95, or 95% of capacity before it stops clients from writing data. The mon osd backfillfull ratio defaults to 0.90, or 90% of capacity when it blocks backfills from starting. The OSD nearfull ratio defaults to 0.85, or 85% of capacity when it generates a health warning. Change this using the following command:

cephuser@adm > ceph osd set-nearfull-ratio <float[0.0-1.0]>

Full cluster issues usually arise when testing how Ceph handles an OSD failure on a small cluster. When one node has a high percentage of the cluster’s data, the cluster can easily eclipse its nearfull and full ratio immediately. If you are testing how Ceph reacts to OSD failures on a small cluster, you should leave ample free disk space and consider temporarily lowering the OSD full ratio, OSD backfillfull ratio and OSD nearfull ratio using these commands:

cephuser@adm > ceph osd set-nearfull-ratio <float[0.0-1.0]>
cephuser@adm > ceph osd set-full-ratio <float[0.0-1.0]>
cephuser@adm > ceph osd set-backfillfull-ratio <float[0.0-1.0]>

Full ceph-osds will be reported by ceph health:

cephuser@adm > ceph health
  HEALTH_WARN 1 nearfull osd(s)

Or:

cephuser@adm > ceph health detail
  HEALTH_ERR 1 full osd(s); 1 backfillfull osd(s); 1 nearfull osd(s)
  osd.3 is full at 97%
  osd.4 is backfill full at 91%
  osd.2 is near full at 87%

We recommend adding new ceph-osds to deal with a full cluster, allowing the cluster to redistribute data to the newly available storage. If you cannot start an OSD because it is full, you may delete some data by deleting some placement group directories in the full OSD.

Important

If you choose to delete a placement group directory on a full OSD, do not delete the same placement group directory on another full OSD, or you may loose data. You must maintain at least one copy of your data on at least one OSD.

A commonly recurring issue involves slow or unresponsive OSDs. Ensure that you have eliminated other troubleshooting possibilities before delving into OSD performance issues. For example, ensure that your network(s) is working properly and your OSDs are running. Check to see if OSDs are throttling recovery traffic.

osd.0 192.168.106.220:6800/18813 312 : [WRN] old request \
  osd_op(client.5099.0:790 fatty_26485_object789 [write 0~4096] 2.5e54f643) \
  v4 received at 2012-03-06 15:42:56.054801 currently waiting for sub ops

{date} {osd.num} [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.005692 secs
{date} {osd.num}  [WRN] slow request 30.005692 seconds old, received at \
{date-time}: osd_op(client.4240.0:8 benchmark_data_ceph-1_39426_object7 \
[write 0~4194304] 0.69848840) v4 currently waiting for subops from [610]

When OSD starts, it is assigned a weight. The higher the weight, the bigger the chance that the cluster writes data to the OSD. The weight is either specified in a cluster CRUSH Map, or calculated by the OSDs' start-up script.

OSD daemon is either running, or stopped/down. There are 3 general reasons why an OSD is down:

You can see the detailed status of OSDs by running

cephuser@adm > ceph osd tree
# id  weight  type name up/down reweight
 -1    0.02998  root default
 -2    0.009995   host doc-ceph1
 0     0.009995      osd.0 up  1
 -3    0.009995   host doc-ceph2
 1     0.009995      osd.1 up  1
 -4    0.009995   host doc-ceph3
 2     0.009995      osd.2 down

The example listing shows that the osd.2 is down. Then you may check if the disk where the OSD is located is mounted:

# lsblk -f
NAME                                                                                                  FSTYPE      LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINT
vda
├─vda1
├─vda2                                                                                                vfat        EFI   7BDD-40C3                                18.8M     6% /boot/efi
└─vda3                                                                                                ext4        ROOT  144d3eec-c193-4793-b6a9-1ac295259f4b     36.7G     5% /
vdb                                                                                                   LVM2_member       Fj8nlY-Dnmm-8Y0w-tJrO-8rAe-SUTH-lA2Ux2
└─ceph--6dc6f71f--ff02--4316--b145--c7804d5fb2f4-osd--block--2cb16f65--c87d--405d--a913--4e4ea6037ca1
vdc                                                                                                   LVM2_member       0101m1-mc83-K8ce-j4Dv-yxUO-5KPj-6FpuSu
└─ceph--1082eac0--4478--48cf--b8ab--4d3a62ca1c27-osd--data--185c9dd6--ce28--4b98--a9b6--9d5b1f444a4f
vdd                                                                                                   LVM2_member       wqE8sC-w5mx-J9t1-FoM1-vIRC-0xxq-5FPyuL
└─ceph--9e71d81a--e394--49b7--bef9--b639083be779-osd--data--eb94f8e6--af6d--4ef6--911b--7f44f7484c85

You can track the reason why the OSD is down by inspecting its log file. See Section 4.3.2, “Failing OSD” for instructions on how to source the log files. After you find and fix the reason why the OSD is not running, start it with the following command. To identify the unique FSID of the cluster, run ceph fsid. To identify the Object Gateway daemon name, run ceph orch ps ---hostname HOSTNAME.

# systemctl start ceph-FSID@osd.2

Do not forget to replace 2 with the actual number of your stopped OSD.

When tuning the cluster performance, it is very important to identify slow storage/OSDs within the cluster. The reason is that if the data is written to the slow(est) disk, the complete write operation slows down as it always waits until it is finished on all the related disks.

It is not trivial to locate the storage bottleneck. You need to examine each and every OSD to find out the ones slowing down the write process. To do a benchmark on a single OSD, run:

ceph tell osd.OSD_ID_NUMBER bench

For example:

cephuser@adm > ceph tell osd.0 bench
 { "bytes_written": 1073741824,
   "blocksize": 4194304,
   "bytes_per_sec": "19377779.000000"}

Then you need to run this command on each OSD and compare the bytes_per_sec value to get the slow(est) OSDs.

We recommend using both a public (front-end) network and a cluster (back-end) network so that you can better meet the capacity requirements of object replication. Another advantage is that you can run a cluster network such that it is not connected to the internet, thereby preventing some denial of service attacks. When OSDs peer and check heartbeats, they use the cluster (back-end) network when it’s available.

However, if the cluster (back-end) network fails or develops significant latency while the public (front-end) network operates optimally, OSDs currently do not handle this situation well. What happens is that OSDs mark each other down on the monitor, while marking themselves up. This is called flapping.

If something is causing OSDs to flap (repeatedly getting marked down and then up again), you can force the monitors to stop the flapping with:

cephuser@adm > ceph osd set noup      # prevent OSDs from getting marked up
cephuser@adm > ceph osd set nodown    # prevent OSDs from getting marked down

These flags are recorded in the osdmap structure:

cephuser@adm > ceph osd dump | grep flags
flags no-up,no-down

You can clear the flags with:

cephuser@adm > ceph osd unset noup
cephuser@adm > ceph osd unset nodown

Two other flags are supported, noin and noout, which prevent booting OSDs from being marked in (allocated data) or protect OSDs from eventually being marked out (regardless of what the current value for mon osd down out interval is).

Note

noup, noout, and nodown are temporary in the sense that once the flags are cleared, the action they were blocking should occur shortly after. The noin flag, on the other hand, prevents OSDs from being marked in on boot, and any daemons that started while the flag was set will remain that way.

As previously noted, a placement group is not necessarily problematic because its state is not active+clean. Generally, Ceph's ability to self-repair may not be working when placement groups get stuck. The stuck states include:

To identify stuck placement groups, run the following:

cephuser@adm > ceph pg dump_stuck [unclean|inactive|stale|undersized|degraded]

When you create a cluster and your cluster remains in active, active+remapped, or active+degraded status and never achieves an active+clean status, you likely have a problem with the configuration. As a general rule, you should run your cluster with more than one OSD and a pool size greater than 1 object replica.

5.2.1 Experimenting with a one node cluster #

  

Ceph no longer provides documentation for operating on a single node. Mounting client kernel modules on a single node containing a Ceph daemon can cause a deadlock due to issues with the Linux kernel itself (unless you use VMs for the clients). However, we recommend experimenting with Ceph in a 1-node configuration regardless of the limitations.

If you are trying to create a cluster on a single node, change the default of the osd crush chooseleaf type setting from 1 (meaning host or node) to 0 (meaning osd) in your Ceph configuration file before you create your monitors and OSDs. This tells Ceph that an OSD can peer with another OSD on the same host. If you are trying to set up a 1-node cluster and osd crush chooseleaf type is greater than 0, Ceph tries to pair the PGs of one OSD with the PGs of another OSD on another node, chassis, rack, row, or even datacenter depending on the setting.

Note

Do not mount kernel clients directly on the same node as your Ceph Storage Cluster, because kernel conflicts can arise. However, you can mount kernel clients within virtual machines (VMs) on a single node.

If you are creating OSDs using a single disk, you must create directories for the data manually first. For example:

cephuser@adm > ceph-deploy osd create --data {disk} {host}

5.2.2 Fewer OSDs than replicas #

  

If you have brought up two OSDs to an up and in state, but you still do not see active+clean placement groups, you may have an osd pool default size set to greater than 2. There are a few ways to address this situation. If you want to operate your cluster in an active+degraded state with two replicas, you can set the osd pool default min size to 2 so that you can write objects in an active+degraded state. You may also set the osd pool default size setting to 2 so that you only have two stored replicas (the original and one replica), in which case the cluster should achieve an active+clean state.

Note

You can make the changes at runtime. If you make the changes in your Ceph configuration file, you may need to restart your cluster.

cephuser@adm > ceph osd force-create-pg <pgid>

It is normal for placement groups to enter states such as degraded or peering following a failure. These states indicate the normal progression through the failure recovery process. However, if a placement group stays in one of these states for a long time this may be an indication of a larger problem. For this reason, the monitor will warn when placement groups get stuck in a non-optimal state. Specifically, check for:

You can explicitly list stuck placement groups with one of:

cephuser@adm > ceph pg dump_stuck stale
cephuser@adm > ceph pg dump_stuck inactive
cephuser@adm > ceph pg dump_stuck unclean

For stuck stale placement groups, ensure you have the right ceph-osd daemons running again. For stuck inactive placement groups, it is can be a peering problem. For stuck unclean placement groups, there can be something preventing recovery from completing, like unfound objects.

In certain cases, the ceph-osd peering process can run into problems, preventing a PG from becoming active and usable. For example, ceph health may report:

cephuser@adm > ceph health detail
  HEALTH_ERR 7 pgs degraded; 12 pgs down; 12 pgs peering; 1 pgs recovering;  \
  6 pgs stuck unclean; 114/3300 degraded (3.455%); 1/3 in osds are down
  ...
  pg 0.5 is down+peering
  pg 1.4 is down+peering
  ...
  osd.1 is down since epoch 69, last address 192.168.106.220:6801/8651

Query the cluster to determine exactly why the PG is marked down by executing the following:

cephuser@adm > ceph pg 0.5 query
  { "state": "down+peering",
    ...
    "recovery_state": [
         { "name": "Started\/Primary\/Peering\/GetInfo",
           "enter_time": "2012-03-06 14:40:16.169679",
           "requested_info_from": []},
         { "name": "Started\/Primary\/Peering",
           "enter_time": "2012-03-06 14:40:16.169659",
           "probing_osds": [
                 0,
                 1],
           "blocked": "peering is blocked due to down osds",
           "down_osds_we_would_probe": [
                 1],
           "peering_blocked_by": [
                 { "osd": 1,
                   "current_lost_at": 0,
                   "comment": "starting or marking this osd lost may let us proceed"}]},
         { "name": "Started",
           "enter_time": "2012-03-06 14:40:16.169513"}
     ]
  }

recovery_state section shows that peering is blocked due to down ceph-osd daemons, specifically osd.1. In this case, restart the ceph-osd to recover. Alternatively, if there is a catastrophic failure of osd.1 such as a disk failure, tell the cluster that it is lost and to cope as best it can.

Important

The cluster cannot guarantee that the other copies of the data are consistent and up to date.

To instruct Ceph to continue anyway:

cephuser@adm > ceph osd lost 1

Recovery will proceed.

Under certain combinations of failures Ceph may complain about unfound objects:

cephuser@adm > ceph health detail
  HEALTH_WARN 1 pgs degraded; 78/3778 unfound (2.065%)
  pg 2.4 is active+degraded, 78 unfound

This means that the storage cluster knows that some objects (or newer copies of existing objects) exist, but it has not found copies of them. One example of how this might come about for a PG whose data is on ceph-osds 1 and 2:

In this example, 1 is aware that these object exist, but there is no live ceph-osd who has a copy. In this case, I/O to those objects blocks, and the cluster hopes that the failed node comes back soon. This is assumed to be preferable to returning an I/O error to the user.

Identify which objects are unfound by executing the following:

cephuser@adm > ceph pg 2.4 list_unfound [starting offset, in json]
  { "offset": { "oid": "",
       "key": "",
       "snapid": 0,
       "hash": 0,
       "max": 0},
   "num_missing": 0,
   "num_unfound": 0,
   "objects": [
      { "oid": "object 1",
        "key": "",
        "hash": 0,
        "max": 0 },
      ...
   ],
   "more": 0}

If there are too many objects to list in a single result, the more field is true and you can query for more.

Identify which OSDs have been probed or might contain data:

cephuser@adm > ceph pg 2.4 query
  "recovery_state": [
       { "name": "Started\/Primary\/Active",
         "enter_time": "2012-03-06 15:15:46.713212",
         "might_have_unfound": [
               { "osd": 1,
                 "status": "osd is down"}]},

In this case, for example, the cluster knows that osd.1 might have data, but it is down. The full range of possible states include:

Sometimes it takes some time for the cluster to query possible locations.

It is possible that there are other locations where the object can exist that are not listed. For example, if a ceph-osd is stopped and taken out of the cluster, the cluster fully recovers, and due to some future set of failures ends up with an unfound object, it will not consider the long-departed ceph-osd as a potential location to consider.

If all possible locations have been queried and objects are still lost, you may have to give up on the lost objects. This, again, is possible given unusual combinations of failures that allow the cluster to learn about writes that were performed before the writes themselves are recovered. To mark the unfound objects as lost:

cephuser@adm > ceph pg 2.5 mark_unfound_lost revert|delete

This the final argument specifies how the cluster should deal with lost objects. The delete option forgets about them entirely. The revert option (not available for erasure coded pools) either rolls back to a previous version of the object or (if it was a new object) forgets about it entirely. Use this with caution, as it may confuse applications that expected the object to exist.

It is possible for all OSDs that had copies of a given placement groups to fail. If that is the case, that subset of the object store is unavailable, and the monitor receives no status updates for those placement groups. To detect this situation, the monitor marks any placement group whose primary OSD has failed as stale. For example:

cephuser@adm > ceph health
  HEALTH_WARN 24 pgs stale; 3/300 in osds are down

Identify which placement groups are stale, and what were the last OSDs to store them by executing the following:

cephuser@adm > ceph health detail
  HEALTH_WARN 24 pgs stale; 3/300 in osds are down
  ...
  pg 2.5 is stuck stale+active+remapped, last acting [2,0]
  ...
  osd.10 is down since epoch 23, last address 192.168.106.220:6800/11080
  osd.11 is down since epoch 13, last address 192.168.106.220:6803/11539
  osd.12 is down since epoch 24, last address 192.168.106.220:6806/11861

For example, to get placement group 2.5 back online, this output shows that it was last managed by osd.0 and osd.2. Restarting the ceph-osd daemons allows the cluster to recover that placement group.

If you have many nodes in your cluster and only a few of them receive data, check the number of placement groups in your pool. See Book “Administration and Operations Guide”, Chapter 12 “Determine the cluster state”, Section 12.7 “Checking placement group states” for more information. Since placement groups get mapped to OSDs, a small number of placement groups will not distribute across the cluster. Create a pool with a placement group count that is a multiple of the number of OSDs. See Book “Administration and Operations Guide”, Chapter 17 “Stored data management”, Section 17.4 “Placement groups” for details.

If your cluster is up but some OSDs are down and you cannot write data, check to ensure that you have the minimum number of OSDs running for the placement group. If you do not have the minimum number of OSDs running, Ceph will not allow you to write data because there is no guarantee that Ceph can replicate your data.

If you receive an active+clean+inconsistent state, this may happen due to an error during scrubbing. Identify the inconsistent placement group(s) by executing the following:

cephuser@adm > ceph health detail
  HEALTH_ERR 1 pgs inconsistent; 2 scrub errors
  pg 0.6 is active+clean+inconsistent, acting [0,1,2]
  2 scrub errors

Or:

cephuser@adm > rados list-inconsistent-pg rbd
  ["0.6"]

There is only one consistent state, but in the worst case, there could be different inconsistencies in multiple perspectives found in more than one objects. If an object named foo in PG 0.6 is truncated, the output is:

cephuser@adm > rados list-inconsistent-obj 0.6 --format=json-pretty

  {
      "epoch": 14,
      "inconsistents": [
          {
              "object": {
                  "name": "foo",
                  "nspace": "",
                  "locator": "",
                  "snap": "head",
                  "version": 1
              },
              "errors": [
                  "data_digest_mismatch",
                  "size_mismatch"
              ],
              "union_shard_errors": [
                  "data_digest_mismatch_info",
                  "size_mismatch_info"
              ],
              "selected_object_info": "0:602f83fe:::foo:head(16'1 client.4110.0:1 dirty|data_digest|omap_digest s 968 uv 1 dd e978e67f od ffffffff alloc_hint [0 0 0])",
              "shards": [
                  {
                      "osd": 0,
                      "errors": [],
                      "size": 968,
                      "omap_digest": "0xffffffff",
                      "data_digest": "0xe978e67f"
                  },
                  {
                      "osd": 1,
                      "errors": [],
                      "size": 968,
                      "omap_digest": "0xffffffff",
                      "data_digest": "0xe978e67f"
                  },
                  {
                      "osd": 2,
                      "errors": [
                          "data_digest_mismatch_info",
                          "size_mismatch_info"
                      ],
                      "size": 0,
                      "omap_digest": "0xffffffff",
                      "data_digest": "0xffffffff"
                  }
              ]
          }
      ]
  }

In this case, we can learn from the output that the only inconsistent object is named foo, and it has inconsistencies. The inconsistencies fall into two categories:

Repair the inconsistent placement group by executing:

cephuser@adm > ceph pg repair placement-group-ID

This command overwrites the bad copies with the authoritative ones. In most cases, Ceph is able to choose authoritative copies from all available replicas using some predefined criteria but this does not always work. For example, the stored data digest could be missing, and the calculated digest will be ignored when choosing the authoritative copies. Use the above command with caution.

If read_error is listed in the errors attribute of a shard, the inconsistency is likely due to disk errors. You might want to check your disk used by that OSD.

If you receive active+clean+inconsistent states periodically due to clock skew, you may consider configuring your NTP daemons on your monitor hosts to act as peers.

When CRUSH fails to find enough OSDs to map to a PG, it will show as a 2147483647 which is ITEM_NONE or no OSD found. For instance:

[2,1,6,0,5,8,2147483647,7,4]

cephuser@adm > ceph osd erasure-code-profile set myprofile k=5 m=3
cephuser@adm > ceph osd pool create erasurepool erasure myprofile

cephuser@adm > ceph osd crush rule ls
  [
      "replicated_rule",
      "erasurepool"]
  $ ceph osd crush rule dump erasurepool
  { "rule_id": 1,
    "rule_name": "erasurepool",
    "ruleset": 1,
    "type": 3,
    "min_size": 3,
    "max_size": 20,
    "steps": [
          { "op": "take",
            "item": -1,
            "item_name": "default"},
          { "op": "chooseleaf_indep",
            "num": 0,
            "type": "host"},
          { "op": "emit"}]}

cephuser@adm > ceph osd erasure-code-profile set myprofile crush-failure-domain=osd
cephuser@adm > ceph osd pool create erasurepool erasure myprofile

cephuser@adm > ceph osd crush rule dump erasurepool
  { "rule_name": "erasurepool",
    "ruleset": 1,
    "type": 3,
    "min_size": 3,
    "max_size": 20,
    "steps": [
          { "op": "take",
            "item": -1,
            "item_name": "default"},
          { "op": "chooseleaf_indep",
            "num": 0,
            "type": "host"},
          { "op": "emit"}]}
  $ ceph osd getcrushmap > crush.map
  got crush map from osdmap epoch 13
  $ crushtool -i crush.map --test --show-bad-mappings \
     --rule 1 \
     --num-rep 9 \
     --min-x 1 --max-x $((1024 * 1024))
  bad mapping rule 8 x 43 num_rep 9 result [3,2,7,1,2147483647,8,5,6,0]
  bad mapping rule 8 x 79 num_rep 9 result [6,0,2,1,4,7,2147483647,5,8]
  bad mapping rule 8 x 173 num_rep 9 result [0,4,6,8,2,1,3,7,2147483647]

# crushtool --decompile crush.map > crush.txt

step set_choose_tries 100

  rule erasurepool {
          ruleset 1
          type erasure
          min_size 3
          max_size 20
          step set_chooseleaf_tries 5
          step set_choose_tries 100
          step take default
          step chooseleaf indep 0 type host
          step emit
  }

# crushtool --compile crush.txt -o better-crush.map

# crushtool -i better-crush.map --test --show-bad-mappings \
     --show-choose-tries \
     --rule 1 \
     --num-rep 9 \
     --min-x 1 --max-x $((1024 * 1024))
  ...
  11:        42
  12:        44
  13:        54
  14:        45
  15:        35
  16:        34
  17:        30
  18:        25
  19:        19
  20:        22
  21:        20
  22:        17
  23:        13
  24:        16
  25:        13
  26:        11
  27:        11
  28:        13
  29:        11
  30:        10
  31:         6
  32:         5
  33:        10
  34:         3
  35:         7
  36:         5
  37:         2
  38:         5
  39:         5
  40:         2
  41:         5
  42:         4
  43:         1
  44:         2
  45:         2
  46:         3
  47:         1
  48:         0
  ...
  102:         0
  103:         1
  104:         0
  ...

The admin socket allows you to interact with a given daemon directly using a Unix socket file. This file can be found in your monitor's run directory. By default, the admin socket will be kept in /var/run/ceph/ceph-mon.ID.asok but this can vary if you defined it otherwise. If you are unable to find it there, check your ceph.conf for an alternative path or run:

cephuser@adm > ceph-conf --name mon.ID --show-config-value admin_socket

Keep in mind that the admin socket is only available while the monitor is running. When the monitor is properly shutdown, the admin socket is removed. If however the monitor is not running and the admin socket still persists, it is likely that the monitor was improperly shutdown. Regardless, if the monitor is not running, you will not be able to use the admin socket, with ceph likely returning Error 111: Connection Refused. To accessing the admin socket run ceph tell on the daemon you are interested in. For example:

cephuser@adm > ceph tell mon.ID mon_status

This passes the command help to the running MON daemon ID via the admin socket, which is a file ending in .asok somewhere under /var/run/ceph. When you know the full path to the file, you can run the following:

cephuser@adm > ceph --admin-daemon PATH_TO_FILE COMMAND

Using help as the command to the ceph tool shows the supported commands available through the admin socket. Take a look at config get, config show, mon stat and quorum_status, as those can be enlightening when troubleshooting a monitor.

mon_status can be obtained via the admin socket. This command outputs a multitude of information about the monitor including the same output you would get with quorum_status. For example, the following example output of ceph tell mon.c mon_status:

  { "name": "c",
  "rank": 2,
  "state": "peon",
  "election_epoch": 38,
  "quorum": [
        1,
        2],
  "outside_quorum": [],
  "extra_probe_peers": [],
  "sync_provider": [],
  "monmap": { "epoch": 3,
      "fsid": "5c4e9d53-e2e1-478a-8061-f543f8be4cf8",
      "modified": "2013-10-30 04:12:01.945629",
      "created": "2013-10-29 14:14:41.914786",
      "mons": [
            { "rank": 0,
              "name": "a",
              "addr": "127.0.0.1:6789\/0"},
            { "rank": 1,
              "name": "b",
              "addr": "127.0.0.1:6790\/0"},
            { "rank": 2,
              "name": "c",
              "addr": "127.0.0.1:6795\/0"}]}}

This example shows that there are three montiors in the monmap (a, b and c), the quorum is formed by only two monitors, and c is in the quorum as a peon. This means that monitor a is out of quorum. This is because there are two monitors in this set: 1 and 2. These are not monitor names. These are monitor ranks, as established in the current monmap. It shows that the missing monitor is the one with a rank of 0, and according to the monmap that would be mon.a.

Ranks (re)calculated whenever you add or remove monitors and follow this rule: the greater the IP:PORT combination, the lower the rank is. In this case, considering that 127.0.0.1:6789 is lower than all the remaining IP:PORT combinations, mon.a has rank 0.

If the Ceph Monitors cannot form a quorum, cephadm will not be able to manage the cluster until the quorum is restored. In order to restore the Ceph Monitor quorum, remove unhealthy Ceph Monitors form the monmap by following these steps:

Note

You may wish to archive the removed monitors' data directories from /var/lib/ceph/mon in a safe location, or delete it if you are confident the remaining monitors are healthy and are sufficiently redundant.

cephadm requires a MGR daemon in order to manage the cluster. If the last MGR of a cluster was removed, follow these steps to deploy an example MGR daemon named mgr.hostname.smfvfd on a random host of your cluster manually:

Ceph is a distributed storage system that depends on networks to peer with OSDs, replicate objects, recover from faults and check heartbeats. Networking issues can cause OSD latency and flapping OSDs. See Section 4.8, “Flapping OSDs” for details.

Ensure that Ceph processes and Ceph-dependent processes are connected and listening.

 ss -a | grep ceph
 ss -l | grep ceph
 sudo ss -p | grep ceph

Check network statistics:

ss -s

NFS Ganesha has the following log levels: NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG, M_DBG, FULL_DEBUG and F_DBG.

Find the logs by running the following command:

cephuser@adm > cephadm logs --name nfs.SERVICE_ID.hostname

For example:

  INFO:cephadm:Inferring fsid e4d48df0-fc2a-11ea-8734-525400e03ff1 ​
  -- Logs begin at Mon 2020-09-21 18:29:25 CEST. -- ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] nfs_rpc_process_request :DISP :F_DBG :About to authenticate Prog=100003, vers=4, proc=1, xid=770234021, SVCXPRT=0x7> ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] nfs_rpc_process_request :DISP :F_DBG :Before SVCAUTH_CHECKSUM on SVCXPRT 0x7f081c0030c0 fd 53 ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :RW LOCK :F_DBG :Got read lock on 0x7f0872a64660 (&export_opt_lock) at /home/abuild/rpmbuild/BU> ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :EXPORT :M_DBG :EXPORT_DEFAULTS (options=03303002/00080000 , , , > ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :EXPORT :M_DBG :default options (options=03303002/ffffffff root_squash , ----, 34-, UDP, TCP,> ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :EXPORT :M_DBG :Final options (options=03303002/ffffffff root_squash , ----, 34-, UDP, TCP,> ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :RW LOCK :F_DBG :Unlocked 0x7f0872a64660 (&export_opt_lock) at /home/abuild/rpmbuild/BUILD/nfs-> ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] get_gsh_client :RW LOCK :F_DBG :Got read lock on 0x7f0872a5e5d0 (&client_by_ip.lock) at /home/abuild/rpmbuild/BUILD> ​
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] get_gsh_client :HT CACHE :DEBUG :client_mgr cache hit slot 859

  LOG {
      # The components block contains one or more logging components
      # and the setting to be used.
      components {
          # The ALL component is special. When set it defines the level
          # for all components and overrides any other setting in this block.
          ALL = FULL_DEBUG; # this will likely kill performance
      }
  } ​

cephuser@adm > rados --pool nfs-ganesha --namespace foo put debugconf-nfs.foo debugconf-nfs.foo​

%url "rados://nfs-ganesha/foo/export-1" ​
%url "rados://nfs-ganesha/foo/userconf-nfs-foo" ​
%url "rados://nfs-ganesha/foo/debugconf-nfs.foo"​

cephuser@adm > rados --pool nfs-ganesha --namespace foo put conf-nfs.foo conf-nfs.foo​

cephuser@adm > rados --pool nfs-ganesha --namespace foo notify conf-nfs.foo conf-nfs.foo ​
reply client.35746 cookie 94143722285376 : 0 bytes ​
reply client.35749 cookie 94618743368000 : 0 bytes​

To change the default port for NFS Ganesha you can use a user defined RADOS configuration object to add an NFS_CORE_PARAM block.

Set the NFS_Port configuration value to the port you want to change. This example uses port 12345.

  NFS_CORE_PARAM { ​
       Enable_NLM = false; ​
       Enable_RQUOTA = false; ​
       Protocols = 4; ​
       NFS_Port = 12345; ​
  }​

Put the file into a user defined RADOS configuration object. In this example, the object is named userconf-nfs.foo:

cephuser@adm > rados --pool nfs-ganesha --namespace foo put userconf-nfs.foo userconf-nfs.foo​

Add a reference to the parameter in your RADOS common configuration file:

%url "rados://nfs-ganesha/foo/export-1" ​
%url "rados://nfs-ganesha/foo/userconf-nfs.foo" ​

Note

This action is similar to how exports are included in the common configuration file.

Put the updated common configuration file into the RADOS common configuration object. In this example, the common configuration object is named conf-nfs.foo:

cephuser@adm > rados --pool nfs-ganesha --namespace foo put conf-nfs.foo conf-nfs.foo​

Once both configuration objects are placed into the RADOS recovery pool, restart the NFS Ganesha daemons using the Ceph orchestrator:

cephuser@adm > ceph orch restart nfs.foo ​
restart nfs.foo.node1 from host 'node1' ​
restart nfs.foo.node3 from host 'node3'

If you are unsure of the location of the Ceph Dashboard, run the following command:

cephuser@adm > ceph mgr services | grep dashboard
"dashboard": "https://host:port"

The command returns the URL where the Ceph Dashboard is located: (https://host:port/)

If you are unable to access the Ceph Dashboard, run through the following commands.

If you are unable to log into the Ceph Dashboard and you receive the following error, run through the procedural checks below:

Figure 10.1: Failed to execute login #

  

When an error occurs on the backend, you will usually receive an error notification on the frontend. Run through the following scenarios to debug.

The Web browser shows an error page when accessing any embedded Grafana pages using self-signed SSL certificates. Most browsers do not display the button to add an exception for those pages.

Follow the next steps to add the exception.

The most basic health check to test a running Object Gateway process is to simply point your browser or client at the Object Gateway endpoint. This should return an empty bucket results for the anonymous user:

root@master # curl -I

If the result returns a 405 status, this indicates MethodNotAllowed. The output will result in a NoSuchBucket XML file. If the result returns a 404 status, this means the Object Gateway DNS name is misconfigured or the requests were not made at an endpoint resolving to Object Gateway DNS name.

If the gateway is not running, usually restartinng the gateway (automatic under systemd) should restore the service. This can be achieved via:

cephuser@adm > ceph orch daemon restart rgw-daemon-name

For increasing the log level under the client.RGW_NAME section, the Object Gateway configurable can be increased from 1 (default) up to 20 (very verbose). For the messages sent to the cluster itself the messenger debug levels can be raised (off by default). This is controlled via debug ms. Usually a level of 1 is sufficient to gather enough information. For injecting arguments in a running process:

Note

This is a temporary change.

cephuser@adm > ceph-daemon CLIENT_RGW_NAME config set debug_rgw 20

For persistent changes, or a non-running process, it is best to set these lines in the ceph.conf file under the client.rgw-name section or, alternatively, using the Ceph CLI. In this case, the debug levels are extremely verbose, so it is best to do this only to capture error logs for a short time. To set the logs using the Ceph CLI, run:

   cephuser@adm > ceph config set CLIENT_RGW_NAME debug_rgw 20

If the Object Gateway process dies, you will normally see a connection refused at the client. In that situation, restarting Object Gateway will restore the service.

To diagnose the cause of the crash, check the log in /var/log/ceph or the core file (if one was generated).

If some (or all) Object Gateway requests appear to be blocked, you can get some insight into the internal state of the Object Gateway daemon via its admin socket. By default, there will be a socket configured to reside in /var/run/ceph, and the daemon can be queried with:

cephuser@adm > ceph daemon /var/run/ceph/client.rgw-name help
  help                list available commands
  objecter_requests   show in-progress osd requests
  perfcounters_dump   dump perfcounters value
  perfcounters_schema dump perfcounters schema
  version             get protocol version

Of particular interest:

cephuser@adm > ceph daemon /var/run/ceph/client.rgw objecter_requests
...

This will dump information about current in-progress requests with the RADOS cluster. This allows you to identify if any requests are blocked by a non-responsive OSD. For example, one might see:

{ "ops": [
      { "tid": 1858,
        "pg": "2.d2041a48",
        "osd": 1,
        "last_sent": "2012-03-08 14:56:37.949872",
        "attempts": 1,
        "object_id": "fatty_25647_object1857",
        "object_locator": "@2",
        "snapid": "head",
        "snap_context": "0=[]",
        "mtime": "2012-03-08 14:56:37.949813",
        "osd_ops": [
              "write 0~4096"]},
      { "tid": 1873,
        "pg": "2.695e9f8e",
        "osd": 1,
        "last_sent": "2012-03-08 14:56:37.970615",
        "attempts": 1,
        "object_id": "fatty_25647_object1872",
        "object_locator": "@2",
        "snapid": "head",
        "snap_context": "0=[]",
        "mtime": "2012-03-08 14:56:37.970555",
        "osd_ops": [
              "write 0~4096"]}],
"linger_ops": [],
"pool_ops": [],
"pool_stat_ops": [],
"statfs_ops": []}

In this dump, two requests are in progress. The last_sent field is the time the RADOS request was sent. If this is a while ago, it suggests that the OSD is not responding. For example, for request 1858, you could check the OSD status with:

cephuser@adm > ceph pg map 2.d2041a48
osdmap e9 pg 2.d2041a48 (2.0) -> up [1,0] acting [1,0]

This tells you to look at osd.1, the primary copy for this PG:

  ceph daemon osd.1 ops
  { "num_ops": 651,
   "ops": [
         { "description": "osd_op(client.4124.0:1858 fatty_25647_object1857 [write 0~4096] 2.d2041a48)",
           "received_at": "1331247573.344650",
           "age": "25.606449",
           "flag_point": "waiting for sub ops",
           "client_info": { "client": "client.4124",
               "tid": 1858}},
  ...

The flag_point field indicates that the OSD is currently waiting for replicas to respond, in this case osd.0.

If you receive a cluster warning for a large OMAP issue, it means that an object has exceeded one of the OSD deep scrub large OMAP object thresholds, usually set to 200,000 keys and 1 GiB of storage. The relevant pool or PG involved can be found by grepping for large OMAP object, as mentioned in the warning message. Depending on the pool and the object key involved it might indicate one of the following:

# radosgw-admin usage trim [--uid=user-id] --start-date=2019-01-01 --end-date=2019-03-03

If you are experiencing apparent hung operations, the first task is to identify where the problem is occurring: in the client, the MDS, or the network connecting them. Start by looking to see if either side has stuck operations and narrow it down from there.

If part of the CephFS metadata or data pools is unavailable and CephFS is not responding, it is probably because RADOS itself is unhealthy.

Check the cluster's status with the following command:

cephuser@adm > ceph status

Ceph will print the cluster status. Review Chapter 2, Troubleshooting logging and debugging, Chapter 6, Troubleshooting Ceph Monitors and Ceph Managers, Chapter 5, Troubleshooting placement groups (PGs), and Chapter 4, Troubleshooting OSDs for tips on what may be causing the issue.

If an operation is hung inside the MDS, it will eventually show up in ceph health, identifying “slow requests are blocked”. It may also identify clients as “failing to respond” or misbehaving in other ways. If the MDS identifies specific clients as misbehaving, we recommend investigating the root cause. Often it can be the result of the following:

cephuser@adm > ceph daemon mds.NAME dump_ops_in_flight

Because CephFS has a consistent cache, if your network connection is disrupted for a long enough time the client will be forcibly disconnected from the system. At this point, the kernel client is in a bind: it cannot safely write back dirty data, and many applications do not handle IO errors correctly on close(). At the moment, the kernel client will remount the FS, but outstanding filesystem IO may or may not be satisfied. In these cases, you may need to reboot your client system.

You can identify you are in this situation if dmesg/kern.log report something like:

  Jul 20 08:14:38 teuthology kernel: [3677601.123718] ceph: mds0 closed our session
  Jul 20 08:14:38 teuthology kernel: [3677601.128019] ceph: mds0 reconnect start
  Jul 20 08:14:39 teuthology kernel: [3677602.093378] ceph: mds0 reconnect denied
  Jul 20 08:14:39 teuthology kernel: [3677602.098525] ceph:  dropping dirty+flushing Fw state for ffff8802dc150518 1099935956631
  Jul 20 08:14:39 teuthology kernel: [3677602.107145] ceph:  dropping dirty+flushing Fw state for ffff8801008e8518 1099935946707
  Jul 20 08:14:39 teuthology kernel: [3677602.196747] libceph: mds0 172.21.5.114:6812 socket closed (con state OPEN)
  Jul 20 08:14:40 teuthology kernel: [3677603.126214] libceph: mds0 172.21.5.114:6812 connection reset
  Jul 20 08:14:40 teuthology kernel: [3677603.132176] libceph: reset on mds0

This is an area of ongoing work to improve the behavior. Kernels will soon be reliably issuing error codes to in-progress IO, although your application(s) may not deal with them well. In the longer-term, we hope to allow reconnect and reclaim of data in cases where it will not violate POSIX semantics.

The kernel since SUSE Linux Enterprise Server 15 SP2 includes a CephFS client that is able to take full advantage of all the features available on an SES7 cluster. All relevant features and bug fixes are backported to this operating system.

However, it may be necessary to access CephFS from other systems that may provide an older CephFS client, which may not support all the features required by an SUSE Enterprise Storage 7.1 cluster. When this happens, the kernel client will fail to mount the file system and will emit messages similar to the one shown below:

[ 4187.023633] libceph: mon0 192.168.122.150:6789 feature set mismatch, my 107b84a842aca < server's 40107b84a842aca, missing 400000000000000
[ 4187.023838] libceph: mon0 192.168.122.150:6789 missing required protocol features

The message above means that the MON identified 0x400000000000000 as the missing feature in the client (the value 0x107b84a842aca represents all the features supported by the client, while 0x40107b84a842aca represents the minimum set of features required by the cluster). From the following table, which shows the complete list of feature bits, we can see that the missing feature bit 58 (2^58 = 0x400000000000000) is CRUSH_TUNABLES5, NEW_OSDOPREPLY_ENCODING, or FS_FILE_LAYOUT_V2 (all these three features share the same feature bit).

A possible solution to allow an old kernel client to mount a recent CephFS is to modify the cluster CRUSH profile. CRUSH profiles define a set of CRUSH tunables that are named after the Ceph versions in which they were introduced. For example, the firefly tunables are first supported in the Firefly release (0.80), and older clients will not be able to access the cluster. Thus, to fix the problem shown above, the following command can be used:

cephuser@adm > ceph osd crush tunables hammer

This will adjust the CRUSH profile to the behaviour it had for the Hammer (0.94) release. Note however that this is not the optimal behaviour for the cluster. To change back to the optimal profile, run the following command:

cephuser@adm > ceph osd crush tunables optimal

The following table lists the available CRUSH profiles and which CRUSH tunables versions (the CRUSH_TUNABLE feature bits in the previous table) they correspond to. It also identifies the minimum kernel version required to use for each profile. Note however that Operating System vendors may choose to backport features to their kernels, so these kernel versions are valid for mainline kernels only. The kernel client included since SUSE Linux Enterprise Server 15 SP2, for example, includes backports of features and bug fixes relevant for usage in SUSE Enterprise Storage 7.1 clusters.

To identify possibly orphaned journal/WAL/DB volumes, follow these steps:

By default, Ceph performs light scrubbing daily (find more details in Book “Administration and Operations Guide”, Chapter 17 “Stored data management”, Section 17.6 “Scrubbing placement groups”) and deep scrubbing weekly. Light scrubbing checks object sizes and checksums to ensure that placement groups are storing the same object data. Deep scrubbing checks an object’s content with that of its replicas to ensure that the actual contents are the same. The price for checking data integrity is increased I/O load on the cluster during the scrubbing procedure.

The default settings allow Ceph OSDs to initiate scrubbing at inappropriate times, such as during periods of heavy loads. Customers may experience latency and poor performance when scrubbing operations conflict with their operations. Ceph provides several scrubbing settings that can limit scrubbing to periods with lower loads or during off-peak hours.

If the cluster experiences high loads during the day and low loads late at night, consider restricting scrubbing to night time hours, such as 11pm till 6am:

cephuser@adm > ceph config set osd osd_scrub_begin_hour 23
cephuser@adm > ceph config set osd osd_scrub_end_hour 6

If time restriction is not an effective method of determining a scrubbing schedule, consider using the osd_scrub_load_threshold option. The default value is 0.5, but it could be modified for low load conditions:

cephuser@adm > ceph config set osd osd_scrub_load_threshold 0.25

You may need to stop OSDs for maintenance periodically. If you do not want CRUSH to automatically rebalance the cluster in order to avoid huge data transfers, set the cluster to noout first:

root@minion > ceph osd set noout

When the cluster is set to noout, you can begin stopping the OSDs within the failure domain that requires maintenance work. To identify the unique FSID of the cluster, run ceph fsid. To identify the Object Gateway daemon name, run

cephuser@adm > ceph orch ps ---hostname HOSTNAME

# systemctl stop ceph-FSID@DAEMON_NAME

Find more information about operating Ceph services and identifying their names in Book “Administration and Operations Guide”, Chapter 14 “Operation of Ceph services”.

After you complete the maintenance, start OSDs again:

# systemctl start ceph-FSID@osd.SERVICE_ID.service

After OSD services are started, unset the cluster from noout:

cephuser@adm > ceph osd unset noout

When data is written to OSDs evenly, the cluster is considered balanced. Each OSD within a cluster is assigned its weight. The weight is a relative number and tells Ceph how much of the data should be written to the related OSD. The higher the weight, the more data will be written. If an OSD has zero weight, no data will be written to it. If the weight of an OSD is relatively high compared to other OSDs, a large portion of the data will be written there, which makes the cluster unbalanced.

Unbalanced clusters have poor performance, and in the case that an OSD with a high weight suddenly crashes, a lot of data needs to be moved to other OSDs, which slows down the cluster as well.

To avoid this, you should regularly check OSDs for the amount of data writing. If the amount is between 30% and 50% of the capacity of a group of OSDs specified by a given ruleset, you need to reweight the OSDs. Check for individual disks and find out which of them fill up faster than the others (or are generally slower), and lower their weight. The same is valid for OSDs where not enough data is written—you can increase their weight to have Ceph write more data to them. In the following example, you will find out the weight of an OSD with ID 13, and reweight it from 3 to 3.05:

cephuser@adm > ceph osd tree | grep osd.13
 13  hdd 3                osd.13  up  1.00000  1.00000

 cephuser@adm > ceph osd crush reweight osd.13 3.05
 reweighted item id 13 name 'osd.13' to 3.05 in crush map

cephuser@adm > ceph osd tree | grep osd.13
 13  hdd 3.05                osd.13  up  1.00000  1.00000

Tip: OSD reweight by utilization

The ceph osd reweight-by-utilization threshold command automates the process of reducing the weight of OSDs which are heavily overused. By default it will adjust the weights downward on OSDs which reached 120% of the average usage, but if you include threshold it will use that percentage instead.

For OSD daemons, the read/write operations are critical to keep the Ceph cluster balanced. They often need to have many files open for reading and writing at the same time. On the OS level, the maximum number of simultaneously open files is called 'maximum number of file descriptors'.

To prevent OSDs from running out of file descriptors, you can override the default and specify an appropriate value, for example:

cephuser@adm > ceph config set global max_open_files 131072

After you change max_open_files, you need to restart the OSD service on the relevant Ceph node.

We recommend protecting the network cluster communication with SUSE Firewall. You can edit its configuration by selecting YaST › Security and Users › Firewall › Allowed Services.

Following is a list of Ceph-related services and numbers of the ports that they normally use:

Both intermittent and complete network failures will impact a Ceph cluster. These two utilities can help in tracking down the cause and verifying expectations.

Tip: Sync Runners

Use the salt-run saltutil.sync_runners command if the Salt runner is reported as not available.

root@master # salt-run network.ping
Succeeded: 8 addresses from 7 minions average rtt 0.15 ms

root@master # salt-run network.ping6
Succeeded: 8 addresses from 7 minions average rtt 0.15 ms

root@master # salt-run network.jumbo_ping
Succeeded: 8 addresses from 7 minions average rtt 0.26 ms

root@master # salt-run network.iperf
Fastest 2 hosts:
    |_
      - 192.168.31.25
      - 11443 Mbits/sec
    |_
      - 172.16.31.25
      - 10363 Mbits/sec

Slowest 2 hosts:
    |_
      - 172.16.32.14
      - 10217 Mbits/sec
    |_
      - 192.168.121.164
      - 10113 Mbits/sec

root@master # salt-run network.iperf output=full
192.168.128.1:
    8644.0 Mbits/sec
192.168.128.2:
    10360.0 Mbits/sec
192.168.128.3:
    9336.0 Mbits/sec
192.168.128.4:
    9588.56 Mbits/sec
192.168.128.5:
    10187.0 Mbits/sec
192.168.128.6:
    10465.0 Mbits/sec

root@master # salt-run network.ping remove="192.168.121.0/24,192.168.1.0/24"
Succeeded: 20 addresses from 10 minions average rtt 14.16 ms

Ceph tracks which daemons manage which hardware storage devices (HDDs, SSDs), and collects health metrics about those devices in order to provide tools to predict and automatically respond to hardware failure.

You can blink the drive LEDs on hardware enclosures to make the replacement of failed disks easy and less error-prone. Use the following command:

cephuser@adm > ceph device light --enable=on --devid=string --light_type=ident --force

The DEVID parameter is the device identification. You can obtain it by running the following command:

cephuser@adm > ceph device ls

rados is a command line utility to manage RADOS object storage. For more information, see man 8 rados.

If you send a large object to a Ceph cluster with the rados utility, such as

cephuser@adm > rados -p mypool put myobject /file/to/send

it can fill up all the related OSD space and cause serious trouble to the cluster performance.

If you receive a Too Many PGs per OSD message after running ceph status, it means that the mon_pg_warn_max_per_osd value (300 by default) was exceeded. This value is compared to the number of PGs per OSD ratio. This means that the cluster setup is not optimal.

The number of PGs cannot be reduced after the pool is created. Pools that do not yet contain any data can safely be deleted and then re-created with a lower number of PGs. Where pools already contain data, the only solution is to add OSDs to the cluster so that the ratio of PGs per OSD becomes lower.

If you receive a stuck inactive status message after running ceph status, it means that Ceph does not know where to replicate the stored data to fulfill the replication rules. It can happen shortly after the initial Ceph setup and fix itself automatically. In other cases, this may require a manual interaction, such as bringing up a broken OSD, or adding a new OSD to the cluster. In very rare cases, reducing the replication level may help.

If the placement groups are stuck perpetually, you need to check the output of ceph osd tree. The output should look tree-structured, similar to the example in Section 4.6, “OSD is down”.

If the output of ceph osd tree is rather flat as in the following example

cephuser@adm > ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
-1         0.02939  root default
-3         0.00980      host doc-ses-node2
 0    hdd  0.00980          osd.0              up   1.00000  1.00000
-5         0.00980      host doc-ses-node3
 1    hdd  0.00980          osd.1              up   1.00000  1.00000
-7         0.00980      host doc-ses-node4
 2    hdd  0.00980          osd.2              up   1.00000  1.00000

You should check that the related CRUSH map has a tree structure. If it is also flat, or with no hosts as in the above example, it may mean that host name resolution is not working correctly across the cluster.

If the hierarchy is incorrect—for example the root contains hosts, but the OSDs are at the top level and are not themselves assigned to hosts—you will need to move the OSDs to the correct place in the hierarchy. This can be done using the ceph osd crush move and/or ceph osd crush set commands. For further details see Book “Administration and Operations Guide”, Chapter 17 “Stored data management”, Section 17.5 “CRUSH Map manipulation”.

As a general rule, time synchronization must be configured and running on all nodes. Once time synchronization is set up, the clocks should not get skewed. However, if a clock skew is to occur, this is likely due to the chronyd.service not running on one or more hosts.

Note

It is also possible that the battery on the motherboard has died, and the clock skew will be more pronounced. If this is the case, be aware that it will take quite some time for chronyd to re-synchronize the clocks.

If you receive a clock skew warning, confirm that the chronyd.service daemon is running on all cluster nodes. If not, restart the service and wait for chronyd to re-sync the clock.

Find more information on setting up time synchronization in https://documentation.suse.com/sles/15-SP3/html/SLES-all/cha-ntp.html#sec-ntp-yast.

There may be other reasons why cluster performance becomes weak, such as network problems. In such case, you may notice the cluster reaching quorum, OSD and monitor nodes going offline, data transfers taking a long time, or a lot of reconnect attempts.

To check whether cluster performance is degraded by network problems, inspect the Ceph log files under the /var/log/ceph directory.

To fix network issues on the cluster, focus on the following points:

Tip: Separate network

To ensure fast and safe network communication between cluster nodes, set up a separate network used exclusively by the cluster OSD and monitor nodes.

By default, the Salt Master saves every minion's result for every job in its job cache. The cache can then be used later to look up results from previous jobs. The cache directory defaults to /var/cache/salt/master/jobs/.

Each job return from every minion is saved in a single file. Over time this directory can grow very large, depending on the number of published jobs and the value of the keep_jobs option in the /etc/salt/master file. keep_jobs sets the number of hours (24 by default) to keep information about past minion jobs.

keep_jobs: 24

Important: Do not set keep_jobs: 0

Setting keep_jobs to '0' will cause the job cache cleaner to never run, possibly resulting in a full partition.

If you want to disable the job cache, set job_cache to 'False':

job_cache: False

Tip: Restoring partition full because of job cache

When the partition with job cache files gets full because of wrong keep_jobs setting, follow these steps to free disk space and improve the job cache settings:

1. Stop the Salt Master service:

   root@master # systemctl stop salt-master

2. Change the Salt Master configuration related to job cache by editing /etc/salt/master:

   job_cache: False
   keep_jobs: 1

3. Clear the Salt Master job cache:

   # rm -rfv /var/cache/salt/master/jobs/*

4. Start the Salt Master service:

   root@master # systemctl start salt-master

When your cluster is becoming 70% to 80% full, it is time to add more OSDs to it. When you increase the number of OSDs, you may consider increasing the number of placement groups as well.

Warning

Changing the number of PGs causes a lot of data transfer within the cluster.

To calculate the optimal value for your newly-resized cluster is a complex task.

A high number of PGs creates small chunks of data. This speeds up recovery after an OSD failure, but puts a lot of load on the monitor nodes as they are responsible for calculating the data location.

On the other hand, a low number of PGs takes more time and data transfer to recover from an OSD failure, but does not impose that much load on monitor nodes as they need to calculate locations for less (but larger) data chunks.

Find more information on the optimal number of PGs for your cluster in Book “Administration and Operations Guide”, Chapter 17 “Stored data management”, Section 17.4.2 “Determining the value of PG_NUM”.

Solid-state drives (SSD) are generally faster than hard disks. If you mix the two types of disks for the same write operation, the data writing to the SSD disk will be slowed down by the hard disk performance. Thus, you should never mix SSDs and hard disks for data writing following the same rule (see Book “Administration and Operations Guide”, Chapter 17 “Stored data management”, Section 17.3 “Rule sets” for more information on rules for storing data).

There are generally 2 cases where using SSD and hard disk on the same cluster makes sense:

Using SSDs for OSD journal(s) is better for performance as the journal is usually the bottleneck of hard disk-only OSDs. SSDs are often used to share journals of several OSDs.

Following is a list of potential disadvantages of using SSDs for OSD journal:

When a disk with a stored cluster data has a hardware problem and fails to operate, here is what happens:

Ceph can be configured to store journals or write ahead logs on devices separate from the OSDs. When a disk dedicated to a journal fails, the related OSD(s) fail as well (see Section 14.4, “What happens when a disk fails?”).

Warning: Hosting multiple journals on one disk

For performance boost, you can use a fast disk (such as SSD) to store journal partitions for several OSDs. We do not recommend to host journals for more than 4 OSDs on one disk, because in case of the journals' disk failure, you risk losing stored data for all the related OSDs' disks.