4 Troubleshooting OSDs #

If you have not changed the default path, you can find Ceph log files at /var/log/ceph:

cephuser@adm > ls /var/log/ceph

If you do not get enough detail, change your logging level. See Chapter 2, Troubleshooting logging and debugging for more information.

Use the admin socket tool to retrieve runtime information. For details, list the sockets for your Ceph processes:

cephuser@adm > ls /var/run/ceph

Execute the following:

cephuser@adm > ceph daemon DAEMON-NAME help

Alternatively, specify a SOCKET-FILE:

cephuser@adm > ceph daemon SOCKET-FILE help

The admin socket, among other things, allows you to:

Filesystem issues may arise. To display your file system’s free space, execute df.

cephuser@adm > df -h

Execute df --help for additional usage.

Use iostat to identify I/O-related issues.

cephuser@adm > iostat -x

To retrieve diagnostic messages, use dmesg with less, more, grep or tail. For example:

cephuser@adm > dmesg | grep scsi

If an OSD will not start when you start your cluster and you are unable to determine the reason, we recommend generating a supportconfig and open a support ticket. For more information, see How do I submit a support case?

When a ceph-osd process dies, the monitor learns about the failure from surviving ceph-osd daemons and reports it via the ceph health command:

cephuser@adm > ceph health
HEALTH_WARN 1/3 in osds are down

A warning displays whenever there are ceph-osd processes that are marked in and down. Identify which ceph-osds are down with:

cephuser@adm > ceph health detail
HEALTH_WARN 1/3 in osds are down
osd.0 is down since epoch 23, last address 192.168.106.220:6800/11080

If there is a disk failure or other fault preventing ceph-osd from functioning or restarting, an error message should be present in its log file. You can monitor the cephadm log in real time with the following command:

cephuser@adm > ceph -W cephadm

You can view the last few messages with:

cephuser@adm > ceph log last cephadm

If the daemon stopped because of a heartbeat failure, the underlying kernel file system may be unresponsive. Check dmesg output for disk or other kernel errors.

Ceph prevents writing to a full OSD so that you do not lose data. In an operational cluster, you should receive a warning when your cluster is getting near its full ratio. The mon osd full ratio defaults to 0.95, or 95% of capacity before it stops clients from writing data. The mon osd backfillfull ratio defaults to 0.90, or 90% of capacity when it blocks backfills from starting. The OSD nearfull ratio defaults to 0.85, or 85% of capacity when it generates a health warning. Change this using the following command:

cephuser@adm > ceph osd set-nearfull-ratio <float[0.0-1.0]>

Full cluster issues usually arise when testing how Ceph handles an OSD failure on a small cluster. When one node has a high percentage of the cluster’s data, the cluster can easily eclipse its nearfull and full ratio immediately. If you are testing how Ceph reacts to OSD failures on a small cluster, you should leave ample free disk space and consider temporarily lowering the OSD full ratio, OSD backfillfull ratio and OSD nearfull ratio using these commands:

cephuser@adm > ceph osd set-nearfull-ratio <float[0.0-1.0]>
cephuser@adm > ceph osd set-full-ratio <float[0.0-1.0]>
cephuser@adm > ceph osd set-backfillfull-ratio <float[0.0-1.0]>

Full ceph-osds will be reported by ceph health:

cephuser@adm > ceph health
  HEALTH_WARN 1 nearfull osd(s)

Or:

cephuser@adm > ceph health detail
  HEALTH_ERR 1 full osd(s); 1 backfillfull osd(s); 1 nearfull osd(s)
  osd.3 is full at 97%
  osd.4 is backfill full at 91%
  osd.2 is near full at 87%

We recommend adding new ceph-osds to deal with a full cluster, allowing the cluster to redistribute data to the newly available storage. If you cannot start an OSD because it is full, you may delete some data by deleting some placement group directories in the full OSD.

Important

If you choose to delete a placement group directory on a full OSD, do not delete the same placement group directory on another full OSD, or you may loose data. You must maintain at least one copy of your data on at least one OSD.

Check your disks for bad sectors and fragmentation. This can cause total throughput to drop substantially.

Monitors are generally light-weight processes but often perform fsync(). This can interfere with other workloads, particularly if monitors run on the same drive as the OSDs. Additionally, if you run monitors on the same host as the OSDs, you may incur performance issues related to:

In these cases, multiple OSDs running on the same host can drag each other down by doing lots of commits. That often leads to the bursty writes.

Spinning up co-resident processes such as a cloud-based solution, virtual machines and other applications that write data to Ceph while operating on the same hardware as OSDs can introduce significant OSD latency. We recommend optimizing a host for use with Ceph and using other hosts for other processes. The practice of separating Ceph operations from other applications may help improve performance and may streamline troubleshooting and maintenance.

If you turned logging levels up to track an issue and then forgot to turn logging levels back down, the OSD may be putting a lot of logs onto the disk. If you intend to keep logging levels high, you may consider mounting a drive to the default path for logging. For example, /var/log/ceph/$cluster-$name.log.

Depending upon your configuration, Ceph may reduce recovery rates to maintain performance or it may increase recovery rates to the point that recovery impacts OSD performance. Check to see if the OSD is recovering.

Check the kernel version you are running. Older kernels may not receive new backports that Ceph depends upon for better performance.

Try running one OSD per host to see if performance improves. Old kernels might not have a recent enough version of glibc to support syncfs(2).

Currently, we recommend deploying clusters with XFS.

We recommend 1.5 GB of RAM per TB of raw OSD capacity for each Object Storage Node. You may notice that during normal operations, the OSD only uses a fraction of that amount. Unused RAM makes it tempting to use the excess RAM for co-resident applications, VMs and so forth. However, when OSDs go into recovery mode, their memory utilization spikes. If there is no RAM available, the OSD performance will slow considerably.

If a ceph-osd daemon is slow to respond to a request, log messages will generate complaining about requests that are taking too long. The warning threshold defaults to 30 seconds, and is configurable via the osd op complaint time option. When this happens, the cluster log receives messages. Legacy versions of Ceph complain about old requests:

osd.0 192.168.106.220:6800/18813 312 : [WRN] old request \
  osd_op(client.5099.0:790 fatty_26485_object789 [write 0~4096] 2.5e54f643) \
  v4 received at 2012-03-06 15:42:56.054801 currently waiting for sub ops

Newer versions of Ceph complain about slow requests:

{date} {osd.num} [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.005692 secs
{date} {osd.num}  [WRN] slow request 30.005692 seconds old, received at \
{date-time}: osd_op(client.4240.0:8 benchmark_data_ceph-1_39426_object7 \
[write 0~4194304] 0.69848840) v4 currently waiting for subops from [610]

Possible causes include:

Possible solutions:

If you run ceph daemon osd.<id> dump_historic_ops or ceph daemon osd.<id> dump_ops_in_flight, you see a set of operations and a list of events each operation went through. These are briefly described below.

Events from the Messenger layer:

Events from the OSD as it prepares operations:

Events from the FileStore:

Events from the OSD after stuff has been given to local disk:

Many of these events are seemingly redundant, but cross important boundaries in the internal code (such as passing data across locks into new threads).