The following section details the statues that have been triggered and actions to take when the status is displayed.
MON_DOWN
One or more monitor daemons are down. The cluster requires a majority of the monitors in order to function. When one or more monitors are down, clients will initially have difficulty connecting to the cluster.
Restart the monitor daemon that is down as soon as possible to reduce the risk of a subsequent monitor failure.
MON_CLOCK_SKEW
The clocks on the hosts running the
ceph-mon
monitor daemons are not
well synchronized. This health alert is raised if the cluster detects a
clock skew greater than mon_clock_drift_allowed
. Resolve
this by synchronizing the clocks using either ntpd
or
chrony
. If it is impractical to keep the clocks closely
synchronized, the mon_clock_drift_allowed
threshold can
be increased, but this value must stay well below the
mon_lease
interval in order for monitor cluster to
function properly.
MON_MSGR2_NOT_ENABLED
The ms_bind_msgr2
option is enabled but one or more
monitors is not configured to bind to a v2 port in the cluster's monmap.
This means that features specific to the msgr2 protocol (for example,
encryption) are not available on some or all connections. In most cases
this can be corrected by issuing the following command:
cephuser@adm >
ceph mon enable-msgr2
This command changes any monitor configured for the old default port 6789 to continue to listen for v1 connections on 6789 and also listen for v2 connections on the new default 3300 port. If a monitor is configured to listen for v1 connections on a non-standard port (not 6789), then the monmap needs to be modified manually.
MGR_MODULE_DEPENDENCY
An enabled manager module is failing its dependency check. This health
check should come with a message from the module about the problem. For
example, a module might report that a required package is not installed.
In which case, the message will read: "Install the required package and
restart your manager daemons." This health check only applies to enabled
modules. If a module is not enabled, you can see whether it is reporting
dependency issues in the output of ceph module ls
.
MGR_MODULE_ERROR
A manager module has experienced an unexpected error. Typically, this
means an unhandled exception was raised from the module's
serve
function. The human-readable description of the
error may be obscurely worded if the exception did not provide a useful
description of itself. This health check may indicate a bug. Open a bug
report if you think you have encountered a bug. If you believe the error
is transient, you may restart your manager daemon(s), or use ceph
mgr fail
on the active daemon to prompt a failover to another
daemon.
OSD_DOWN
One or more OSDs are marked down. The
ceph-osd
daemon may have been
stopped, or peer OSDs may be unable to reach the OSD over the network.
Common causes include a stopped or crashed daemon, a down host, or a
network outage. Verify the host is healthy, the daemon is started, and
network is functioning. If the daemon has crashed, the daemon log file
(/var/log/ceph/ceph-osd.*
) may contain debugging
information.
OSD_CRUSH TYPE_DOWN
For example, OSD_HOST_DOWN
or
OSD_ROOT_DOWN
. All the OSDs within a particular CRUSH
subtree are marked down, for example all OSDs on a host.
OSD_ORPHAN
An OSD is referenced in the CRUSH Map hierarchy but does not exist. The OSD can be removed from the CRUSH hierarchy with:
cephuser@adm >
ceph osd crush rm osd.ID
OSD_OUT_OF_ORDER_FULL
The utilization thresholds for backfillfull
,
nearfull
, full
, and
failsafe_full
are not ascending. The thresholds can be
adjusted with:
cephuser@adm >
ceph osd set-backfillfull-ratio RATIOcephuser@adm >
ceph osd set-nearfull-ratio RATIOcephuser@adm >
ceph osd set-full-ratio RATIO
OSD_FULL
One or more OSDs have exceeded the full
threshold and
is preventing the cluster from servicing writes. Utilization by pool can
be checked with:
cephuser@adm >
ceph df
The currently defined full
ratio can be seen with:
cephuser@adm >
ceph osd dump | grep full_ratio
A short-term workaround to restore write availability is to raise the full threshold by a small amount:
cephuser@adm >
ceph osd set-full-ratio RATIO
New storage should be added to the cluster by deploying more OSDs or existing data should be deleted in order to free up space.
OSD_BACKFILLFULL
One or more OSDs have exceeded the backfillfull
threshold, preventing data from being allowed to rebalance to this device.
This is an early warning that rebalancing may not be able to complete and
that the cluster is approaching full
. Utilization by
pool can be checked with:
cephuser@adm >
ceph df
OSD_NEARFULL
One or more OSDs have exceeded the nearfull
threshold.
This is an early warning that the cluster is approaching
full
. Utilization by pool can be checked with:
cephuser@adm >
ceph df
OSDMAP_FLAGS
One or more cluster flags of interest has been set. These flags include:
The cluster is flagged as full
and cannot serve
writes
Paused reads or writes
OSDs are not allowed to start
OSD failure reports are being ignored and the monitors are not marking OSDs down
OSDs that were previously marked out are not being marked back in when they start
Down OSDs are not automatically marked out after the configured interval
Recovery or data rebalancing is suspended
Scrubbing is disabled
Cache tiering activity is suspended
With the exception of full
, these flags can be set or
cleared with:
cephuser@adm >
ceph osd set FLAGcephuser@adm >
ceph osd unset FLAG
OSD_FLAGS
One or more OSDs or CRUSH {nodes,device classes}
has a
flag of interest set. These flags include:
These OSDs are not allowed to start
Failure reports for these OSDs are ignored
If these OSDs were previously marked out automatically after a failure, they are not to be marked in when they start
If these OSDs are down they are not automatically marked out after the configured interval
These flags can be set and cleared in batch with:
cephuser@adm >
ceph osd set-group FLAG WHOcephuser@adm >
ceph osd unset-group FLAG WHO
For example:
cephuser@adm >
ceph osd set-group noup,noout osd.0 osd.1cephuser@adm >
ceph osd unset-group noup,noout osd.0 osd.1cephuser@adm >
ceph osd set-group noup,noout host-foocephuser@adm >
ceph osd unset-group noup,noout host-foocephuser@adm >
ceph osd set-group noup,noout class-hddcephuser@adm >
ceph osd unset-group noup,noout class-hdd
OLD_CRUSH_TUNABLES
The CRUSH Map is using old settings and should be updated. The oldest
tunables that can be used (for example, the oldest client version that can
connect to the cluster) without triggering this health warning are
determined by the mon_crush_min_required_version
config
option.
OLD_CRUSH_STRAW_CALC_VERSION
The CRUSH Map is using an older, sub-optimal method for calculating
intermediate weight values for straw buckets. The CRUSH Map requires an
update to use the newer method (straw_calc_version=1
).
CACHE_POOL_NO_HIT_SET
One or more cache pools are not configured with a hit set to track utilization. This prevents the tiering agent from identifying cold objects to flush and evict from the cache. Hit sets can be configured on the cache pool with the following:
cephuser@adm >
ceph osd pool set POOLNAME hit_set_type TYPEcephuser@adm >
ceph osd pool set POOLNAME hit_set_period PERIOD-IN-SECONDScephuser@adm >
ceph osd pool set POOLNAME hit_set_count NUMBER-OF-HITSETScephuser@adm >
ceph osd pool set POOLNAME hit_set_fpp TARGET-FALSE-POSITIVE-RATE
OSD_NO_SORTBITWISE
No SUSE Enterprise Storage 6 v12.y.z OSDs are running but the
sortbitwise
flag has not been set. Set the
sortbitwise
flag before v12.y.z or newer OSDs can start.
You can safely set the flag with:
cephuser@adm >
ceph osd set sortbitwise
POOL_FULL
One or more pools have reached the quota and are no longer allowing writes. Pool quotas and utilization can be seen with the following command:
cephuser@adm >
ceph df detail
You can either raise the pool quota with the following commands:
cephuser@adm >
ceph osd pool set-quota POOLNAME max_objects NUM-OBJECTScephuser@adm >
ceph osd pool set-quota POOLNAME max_bytes NUM-BYTES
Or, you can delete existing data to reduce utilization.
BLUEFS_SPILLOVER
One or more OSDs that use the BlueStore backend have been allocated db partitions (storage space for metadata, normally on a faster device) but that space has filled, such that metadata has overflowed onto the normal slow device. This is not necessarily an error condition or even unexpected, but if the administrator's expectation was that all metadata would fit on the faster device, it indicates that not enough space was provided. This warning can be disabled on all OSDs with the following command:
cephuser@adm >
ceph config set osd bluestore_warn_on_bluefs_spillover false
Alternatively, it can be disabled on a specific OSD with the following command:
cephuser@adm >
ceph config set osd.123 bluestore_warn_on_bluefs_spillover false
To provide more metadata space, the OSD in question can be destroyed and reprovisioned. This involves data migration and recovery. It is possible to expand the LVM logical volume backing the db storage. If the underlying LV has been expanded, the OSD daemon needs to be stopped and BlueFS informed of the device size change with the following command:
cephuser@adm >
ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-$ID
BLUEFS_AVAILABLE_SPACE
To check how much space is free for BlueFS, execute:
cephuser@adm >
ceph daemon osd.123 bluestore bluefs available
This provides output for up to 3 values; BDEV_DB free
,
BDEV_SLOW free
and
available_from_bluestore
. BDEV_DB
and
BDEV_SLOW
report the amount of space that has been
acquired by BlueFS and is considered free. Value
available_from_bluestore
denotes ability of BlueStore
to leave more space to BlueFS. It is normal that this value is different
from amount of BlueStore free space, as BlueFS allocation unit is
typically larger than BlueStore allocation unit. This means that only
part of BlueStore free space is acceptable for BlueFS.
BLUEFS_LOW_SPACE
If BlueFS is running low on available free space and there is little
available_from_bluestore
, consider reducing BlueFS'
allocation unit size. To simulate available space when the allocation unit
is different, execute:
cephuser@adm >
ceph daemon osd.123 bluestore bluefs available ALLOC-UNIT-SIZE
BLUESTORE_FRAGMENTATION
As BlueStore works, free space on underlying storage becomes fragmented. This is normal and unavoidable, but excessive fragmentation can cause slowdown. To inspect BlueStore fragmentation, execute:
cephuser@adm >
ceph daemon osd.123 bluestore allocator score block
Score is given in [0-1] range. [0.0 .. 0.4] tiny fragmentation [0.4 .. 0.7] small, acceptable fragmentation [0.7 .. 0.9] considerable, but safe fragmentation [0.9 .. 1.0] severe fragmentation, can impact BlueFS' ability to get space from BlueStore. If detailed report of free fragments is required, execute:
cephuser@adm >
ceph daemon osd.123 bluestore allocator dump block
If the OSD process does not perform fragmentation, inspect with
ceph-bluestore-tool
. Get the fragmentation score:
cephuser@adm >
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-score
Dump detailed free chunks:
cephuser@adm >
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-dump
BLUESTORE_LEGACY_STATFS
As of SUSE Enterprise Storage 6, BlueStore tracks its internal usage statistics on
a per-pool granular basis and one or more OSDs have BlueStore volumes
that were created prior to SUSE Enterprise Storage 7. If all OSDs are
older than SUSE Enterprise Storage 7, the per-pool metrics are not
available. However, if there is a mix of pre-SUSE Enterprise Storage 7
and post-SUSE Enterprise Storage 7 OSDs, the cluster usage statistics
reported by ceph df
will not be accurate. The old OSDs
can be updated to use the new usage tracking scheme by stopping each OSD,
running a repair operation, and the restarting it. For example, if
osd.123
requires an update, run the following command.
To identify the unique FSID of the cluster, run ceph
fsid
. To identify the Object Gateway daemon name, run ceph orch
ps ---hostname HOSTNAME
.
root #
systemctl stop ceph-FSID@osd.123cephuser@adm >
ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-123root #
systemctl start ceph-osd@123
This warning can be disabled with:
cephuser@adm >
ceph config set global bluestore_warn_on_legacy_statfs false
BLUESTORE_DISK_SIZE_MISMATCH
One or more OSDs using BlueStore has an internal inconsistency between the size of the physical device and the metadata tracking its size. This can lead to the OSD crashing in the future. The OSDs in question should be destroyed and re-deployed. To avoid putting any data at risk, re-deploy only one OSD at a time. For example, if OSD_ID has the error:
cephuser@adm >
ceph osd out osd.OSD_IDcephuser@adm >
while ! ceph osd safe-to-destroy osd.OSD_ID ; do sleep 1m ; donecephuser@adm >
ceph osd destroy osd.OSD_IDcephuser@adm >
cephadm ceph-volume lvm zap /path/to/devicecephuser@adm >
cephadm ceph-volume lvm create --osd-id OSD_ID--data /path/to/device
DEVICE_HEALTH
One or more devices are expected to fail. The warning threshold is
controlled by the mgr/devicehealth/warn_threshold
configuration option. This warning only applies to OSDs that are currently
marked in
. The expected response to this failure is to
mark the device out
. The data is then migrated off of the
device and the hardware is removed from the system. Marking out is
normally done automatically if
mgr/devicehealth/self_heal
is enabled based on the
mgr/devicehealth/mark_out_threshold
. Device health can
be checked with:
cephuser@adm >
ceph device info DEVICE-ID
Device life expectancy is set by a prediction model run by the Ceph Manager or by an external tool via the command:
cephuser@adm >
ceph device set-life-expectancy DEVICE-ID FROM TO
You can change the stored life expectancy manually, but that usually does not persist—the tool that originally set it reset and changing the stored value does not affect the actual health of the hardware device.
DEVICE_HEALTH_IN_USE
One or more devices are expected to fail and has been marked
out
of the cluster based on
mgr/devicehealth/mark_out_threshold
, but the devices
are still participating in one more PGs. This may be because it was only
recently marked as out
and the data is still migrating,
or because the data cannot be migrated off for some reason (for example,
the cluster is nearly full, or the CRUSH hierarchy is such that there is
not another suitable OSD to migrate the data to). This message can be
silenced by disabling the self heal behavior (setting
mgr/devicehealth/self_heal
to false), by adjusting the
mgr/devicehealth/mark_out_threshold
, or by addressing
what is preventing data from being migrated off of the ailing device.
DEVICE_HEALTH_TOOMANY
Too many devices are expected to fail and the
mgr/devicehealth/self_heal
behavior is enabled, such
that marking out
all of the ailing devices would exceed
the clusters mon_osd_min_in_ratio
ratio that prevents too
many OSDs from being automatically marked out
. This can
indicates that too many devices in the cluster are expected to fail and
action is required to add newer (healthier) devices before too many
devices fail and data is lost. The health message can also be silenced by
adjusting parameters like mon_osd_min_in_ratio
or
mgr/devicehealth/mark_out_threshold
, but be warned that
this increases the likelihood of unrecoverable data loss in the cluster.
PG_AVAILABILITY
Data availability is reduced and the cluster is unable to service potential read or write requests for some data in the cluster. Specifically, if one or more PGs are in a state that does not allow IO requests to be serviced. Problematic PG states include peering, stale, incomplete, and in-active (if those conditions do not clear quickly). Detailed information about which PGs are affected is available from:
cephuser@adm >
ceph health detail
In most cases the root cause is that one or more OSDs are currently down;
see the discussion for OSD_DOWN
above. The state of
specific problematic PGs can be queried with:
cephuser@adm >
ceph tell PG_ID query
PG_DEGRADED
Data redundancy is reduced for some data, meaning the cluster does not have the desired number of replicas for all data (for replicated pools) or erasure code fragments (for erasure coded pools). Specifically, if one or more PGs:
have a degraded or undersized flag set, meaning there are not enough instances of that placement group in the cluster;
have not had the clean flag set for some time.
PG_RECOVERY_FULL
Data redundancy can be reduced or at risk for some data due to a lack of
free space in the cluster. Specifically, one or more PGs have the
recovery_toofull
flag set, meaning that the cluster is
unable to migrate or recover data because one or more OSDs are above the
full threshold. See the discussion for OSD_FULL
above for
steps to resolve this condition.
PG_BACKFILL_FULL
Data redundancy can be reduced or at risk for some data due to a lack of
free space in the cluster. Specifically, one or more PGs have the
backfill_toofull
flag set, meaning that the cluster is
unable to migrate or recover data because one or more OSDs are above the
backfillfull threshold. See the discussion for
OSD_BACKFILLFULL
above for steps to resolve this
condition.
PG_DAMAGED
Data scrubbing has discovered some problems with data consistency in the
cluster. Specifically, one or more PGs have the inconsistent or
snaptrim_error
flag is set, indicating an earlier scrub
operation found a problem, or that the repair flag is set and a repair for
such an inconsistency is currently in progress.
OSD_SCRUB_ERRORS
Recent OSD scrubs have uncovered inconsistencies. This error is generally
paired with PG_DAMAGED
.
LARGE_OMAP_OBJECTS
One or more pools contain large omap objects as determined by
osd_deep_scrub_large_omap_object_key_threshold
(threshold
for number of keys to determine a large omap object) or
osd_deep_scrub_large_omap_object_value_sum_threshold
(the
threshold for summed size (bytes) of all key values to determine a large
omap object) or both. More information on the object name, key count, and
size in bytes can be found by searching the cluster log for ‘Large omap
object found'. Large omap objects can be caused by RGW bucket index
objects that do not have automatic resharding enabled. The thresholds can
be adjusted with:
cephuser@adm >
ceph config set osd osd_deep_scrub_large_omap_object_key_threshold KEYScephuser@adm >
ceph config set osd osd_deep_scrub_large_omap_object_value_sum_threshold BYTES
CACHE_POOL_NEAR_FULL
A cache tier pool is nearly full. Full is determined by the
target_max_bytes
and target_max_objects
properties on the cache pool. Once the pool reaches the target threshold,
write requests to the pool may block while data is flushed and evicted
from the cache, a state that normally leads to very high latencies and
poor performance. The cache pool target size can be adjusted with:
cephuser@adm >
ceph osd pool set CACHE-POOL-NAME target_max_bytes BYTEScephuser@adm >
ceph osd pool set CACHE-POOL-NAME target_max_objects OBJECTS
Normal cache flush and eviction activity can also be throttled due to reduced availability, performance of the base tier, or overall cluster load.
POOL_TOO_FEW_PGS
One or more pools should probably have more PGs, based on the amount of
data that is currently stored in the pool. This can lead to sub-optimal
distribution and balance of data across the OSDs in the cluster, and
similarly reduce overall performance. This warning is generated if the
pg_autoscale_mode
property on the pool is set to warn. To
disable the warning, you can disable auto-scaling of PGs for the pool
entirely with:
cephuser@adm >
ceph osd pool set POOL-NAME pg_autoscale_mode off
To allow the cluster to automatically adjust the number of PGs:
cephuser@adm >
ceph osd pool set POOL-NAME pg_autoscale_mode on
You can also manually set the number of PGs for the pool to the recommended amount with:
cephuser@adm >
ceph osd pool set POOL_NAME pg_num NEW_PG_NUM
TOO_MANY_PGS
The number of PGs in use in the cluster is above the configurable
threshold of mon_max_pg_per_osd
PGs per OSD. If this
threshold is exceeded, the cluster does not allow new pools to be created,
pool pg_num
to be increased, or pool replication to be
increased (any of which would lead to more PGs in the cluster). A large
number of PGs can lead to higher memory utilization for OSD daemons,
slower peering after cluster state changes (like OSD restarts, additions,
or removals), and higher load on the Ceph Manager and Ceph Monitor daemons. The simplest
way to mitigate the problem is to increase the number of OSDs in the
cluster by adding more hardware. The OSD count used for the purposes of
this health check is the number of in
OSDs, marking
out
OSDs in
(if there are any) can also
help:
cephuser@adm >
ceph osd in OSD_IDs
POOL_TOO_MANY_PGS
One or more pools require more PGs based on the amount of data that is
currently stored in the pool. This can lead to higher memory utilization
for OSD daemons, slower peering after cluster state changes (like OSD
restarts, additions, or removals), and higher load on the manager and
monitor daemons. This warning is generated if the
pg_autoscale_mode
property on the pool is set to warn. To
disable the warning, you can disable auto-scaling of PGs for the pool
entirely with:
cephuser@adm >
ceph osd pool set POOL_NAME pg_autoscale_mode off
To allow the cluster to automatically adjust the number of PGs:
cephuser@adm >
ceph osd pool set POOL_NAME pg_autoscale_mode on
You can also manually set the number of PGs for the pool to the recommended amount with:
cephuser@adm >
ceph osd pool set POOL_NAME pg_num NEW_PG_-NUM
POOL_TARGET_SIZE_RATIO_OVERCOMMITTED
One or more pools have a target_size_ratio
property set
to estimate the expected size of the pool as a fraction of total storage,
but the value(s) exceed the total available storage (either by themselves
or in combination with other pools' actual usage). This can indicate that
the target_size_ratio
value for the pool is too large,
and should be reduced or set to zero with:
cephuser@adm >
ceph osd pool set POOL-NAME target_size_ratio 0
POOL_TARGET_SIZE_BYTES_OVERCOMMITTED
One or more pools have a target_size_bytes
property set
to estimate the expected size of the pool, but the value(s) exceed the
total available storage (either by themselves or in combination with other
pools' actual usage). This indicates that the
target_size_bytes
value for the pool is too large and
should be reduced or set to zero with:
cephuser@adm >
ceph osd pool set POOL-NAME target_size_bytes 0
TOO_FEW_OSDS
The number of OSDs in the cluster is below the configurable threshold of
osd_pool_default_size
.
SMALLER_PGP_NUM
One or more pools have a pgp_num
value less than
pg_num
, indicating that the PG count was increased
without also increasing the placement behavior. To adjust the placement
group number, adjust pgp_num
and pg_num
.
Ensure that changing pgp_num
is performed first and does
not trigger the rebalance. To resolve, set pgp_num
to
match pg_num
and trigger the data migration with:
cephuser@adm >
ceph osd pool set POOL pgp_num PG_NUM_VALUE
MANY_OBJECTS_PER_PG
One or more pools has an average number of objects per PG that is
significantly higher than the overall cluster average. The specific
threshold is controlled by the
mon_pg_warn_max_object_skew
configuration value. This
indicates that the pool(s) containing most of the data in the cluster have
too few PGs, or that other pools that do not contain as much data have too
many PGs. The threshold can be raised to silence the health warning by
adjusting the mon_pg_warn_max_object_skew
configuration
option on the monitors.
POOL_APP_NOT_ENABLED
A pool exists that contains one or more objects but has not been tagged for use by a particular application. Resolve this warning by labeling the pool for use by an application. For example, if the pool is used by RBD:
cephuser@adm >
rbd pool init POOLNAME
If the pool is being used by a custom application FOO, you can also label via the low-level command:
cephuser@adm >
ceph osd pool application enable FOO
POOL_FULL
One or more pools has reached (or is very close to reaching) its quota.
The threshold to trigger this error condition is controlled by the
mon_pool_quota_crit_threshold
configuration option. Pool
quotas can be adjusted up or down (or removed) with:
cephuser@adm >
ceph osd pool set-quota POOL max_bytes BYTEScephuser@adm >
ceph osd pool set-quota POOL max_objects OBJECTS
Setting the quota value to 0 disables the quota.
POOL_NEAR_FULL
One or more pools are approaching its quota. The threshold to trigger this
warning condition is controlled by the
mon_pool_quota_warn_threshold
configuration option. Pool
quotas can be adjusted up or down (or removed) with:
cephuser@adm >
ceph osd pool set-quota POOL max_bytes BYTEScephuser@adm >
ceph osd pool set-quota POOL max_objects OBJECTS
OBJECT_MISPLACED
One or more objects in the cluster is not stored on the node the cluster would like it to be stored on. This is an indication that data migration due to some recent cluster change has not yet completed. Misplaced data is not a dangerous condition in and of itself. Data consistency is not at risk and old copies of objects are not removed until the desired number of new copies (in the desired locations) are present.
OBJECT_UNFOUND
One or more objects in the cluster cannot be found. Specifically, the OSDs know that a new or updated copy of an object should exist, but a copy of that version of the object has not been found on OSDs that are currently online. Read or write requests to unfound objects will block. Ideally, a down OSD can be brought back online that has the more recent copy of the unfound object. Candidate OSDs can be identified from the peering state for the PG(s) responsible for the unfound object:
cephuser@adm >
ceph tell PG_ID query
If the latest copy of the object is not available, the cluster can be told to roll back to a previous version of the object.
SLOW_OPS
One or more OSD requests is taking a long time to process. This can be an indication of extreme load, a slow storage device, or a software bug. The request queue on the OSD(s) in question can be queried with the following command, executed from the OSD host:
cephuser@adm >
ceph daemon osd.ID ops
A summary of the slowest recent requests can be seen with:
cephuser@adm >
ceph daemon osd.ID dump_historic_ops
The location of an OSD can be found with:
cephuser@adm >
ceph osd find osd.ID
PG_NOT_SCRUBBED
One or more PGs have not been scrubbed recently. PGs are normally scrubbed
every mon_scrub_interval
seconds and this warning
triggers when mon_warn_pg_not_deep_scrubbed_ratio
percentage of interval has elapsed without a scrub since it was due. PGs
do not scrub if they are not flagged as clean. This can happen if they are
misplaced or degraded (see PG_AVAILABILITY
and
PG_DEGRADED
above). You can manually initiate a scrub of
a clean PG with:
cephuser@adm >
ceph pg scrub PG_ID
PG_NOT_DEEP_SCRUBBED
One or more PGs have not been deep scrubbed recently. PGs are normally
scrubbed every osd_deep_scrub_interval
seconds and this
warning triggers when mon_warn_pg_not_deep_scrubbed_ratio
percentage of interval has elapsed without a scrub since it was due. PGs
do not (deep) scrub if they are not flagged as clean. This can happen if
they are misplaced or degraded (see PG_AVAILABILITY
and
PG_DEGRADED
above). You can manually initiate a scrub of
a clean PG with:
cephuser@adm >
ceph pg deep-scrub PG_ID
CEPHADM_PAUSED
cephadm background work has been paused with ceph orch
pause
. cephadm continues to perform passive monitoring
activities (for example, checking host and daemon status), but it will not
make any changes (for example, deploying or removing daemons).
Resume cephadm work with:
cephuser@adm >
ceph orch resume
CEPHADM_STRAY_HOST
One or more hosts have running Ceph daemons but are not registered as
hosts managed by cephadm. This means that those services cannot
currently be managed by cephadm. For example, restarted, upgraded,
included in ceph orch ps
.
You can manage the host(s) with:
cephuser@adm >
ceph orch host add hostname
You may need to configure SSH access to the remote host before this will work.
Alternatively, you can manually connect to the host and ensure that services on that host are removed or migrated to a host that is managed by cephadm.
You can also disable this warning entirely with:
cephuser@adm >
ceph config set mgr mgr/cephadm/warn_on_stray_hosts false
CEPHADM_STRAY_DAEMON
One or more Ceph daemons are running but not are not managed by
cephadm. This may be because they were deployed using a different tool,
or because they were started manually. Those services cannot currently be
managed by cephadm. For example, restarted, upgraded, or included in
ceph orch ps
..
If the daemon is a stateful one (MON or OSD), it should be adopted by
cephadm. For stateless daemons, it is usually easiest to provision a new
daemon with the ceph orch apply
command and then stop
the unmanaged daemon.
This warning can be disabled entirely with:
cephuser@adm >
ceph config set mgr mgr/cephadm/warn_on_stray_daemons false
CEPHADM_HOST_CHECK_FAILED
One or more hosts have failed the basic cephadm host check, which
verifies that the host is reachable and cephadm
can be
executed there, and that the host satisfies basic prerequisites, like a
working container runtime (podman or docker) and working time
synchronization. If this test fails, cephadm will not be able to manage
services on that host.
You can manually run this check with:
cephuser@adm >
ceph cephadm check-host hostname
You can remove a broken host from management with:
cephuser@adm >
ceph orch host rm hostname
You can disable this health warning with:
cephuser@adm >
ceph config set mgr mgr/cephadm/warn_on_failed_host_check false
RECENT_CRASH
One or more Ceph daemons have crashed recently, and the crash has not yet been archived or acknowledged by the administrator. This may indicate a software bug, a hardware problem (for example, a failing disk), or some other problem. New crashes can be listed with:
cephuser@adm >
ceph crash ls-new
Information about a specific crash can be examined with:
cephuser@adm >
ceph crash info CRASH-ID
This warning can be silenced by archiving the crash (perhaps after being examined by an administrator) so that it does not generate this warning:
cephuser@adm >
ceph crash archive CRASH-ID
Similarly, all new crashes can be archived with:
cephuser@adm >
ceph crash archive-all
Archived crashes are still visible via ceph crash ls
but not ceph crash ls-new
. The time period for what
recent means is controlled by the option
mgr/crash/warn_recent_interval
(default: two weeks).
These warnings can be disabled entirely with:
cephuser@adm >
ceph config set mgr/crash/warn_recent_interval 0
TELEMETRY_CHANGED
Telemetry has been enabled but the contents of the telemetry report have changed since that time, so telemetry reports are not sent. The Ceph developers periodically revise the telemetry feature to include new and useful information, or to remove information found to be useless or sensitive. If any new information is included in the report, Ceph requires the administrator to re-enable telemetry to ensure they have an opportunity to (re)review what information is shared. To review the contents of the telemetry report:
cephuser@adm >
ceph telemetry show
The telemetry report consists of several optional channels that are independently enabled or disabled. To re-enable telemetry (and make this warning go away):
cephuser@adm >
ceph telemetry on
To disable telemetry (and make this warning go away):
cephuser@adm >
ceph telemetry soff
groups: - name: cluster health rules: - alert: health error expr: ceph_health_status == 2 for: 5m labels: severity: critical type: ses_default annotations: description: Ceph in error for > 5m - alert: unhealthy expr: ceph_health_status != 0 for: 15m labels: severity: warning type: ses_default annotations: description: Ceph not healthy for > 5m - name: mon rules: - alert: low monitor quorum count expr: ceph_monitor_quorum_count < 3 labels: severity: critical type: ses_default annotations: description: Monitor count in quorum is low - name: osd rules: - alert: 10% OSDs down expr: sum(ceph_osd_down) / count(ceph_osd_in) >= 0.1 labels: severity: critical type: ses_default annotations: description: More then 10% of OSDS are down - alert: OSD down expr: sum(ceph_osd_down) > 1 for: 15m labels: severity: warning type: ses_default annotations: description: One or more OSDS down for more then 15 minutes - alert: OSDs near full expr: (ceph_osd_utilization unless on(osd) ceph_osd_down) > 80 labels: severity: critical type: ses_default annotations: description: OSD {{ $labels.osd }} is dangerously full, over 80% # alert on single OSDs flapping - alert: flap osd expr: rate(ceph_osd_up[5m])*60 > 1 labels: severity: warning type: ses_default annotations: description: > OSD {{ $label.osd }} was marked down at back up at least once a minute for 5 minutes. # alert on high deviation from average PG count - alert: high pg count deviation expr: abs(((ceph_osd_pgs > 0) - on (job) group_left avg(ceph_osd_pgs > 0) by (job)) / on (job) group_left avg(ceph_osd_pgs > 0) by (job)) > 0.35 for: 5m labels: severity: warning type: ses_default annotations: description: > OSD {{ $labels.osd }} deviates by more then 30% from average PG count # alert on high commit latency...but how high is too high - name: mds rules: # no mds metrics are exported yet - name: mgr rules: # no mgr metrics are exported yet - name: pgs rules: - alert: pgs inactive expr: ceph_total_pgs - ceph_active_pgs > 0 for: 5m labels: severity: critical type: ses_default annotations: description: One or more PGs are inactive for more then 5 minutes. - alert: pgs unclean expr: ceph_total_pgs - ceph_clean_pgs > 0 for: 15m labels: severity: warning type: ses_default annotations: description: One or more PGs are not clean for more then 15 minutes. - name: nodes rules: - alert: root volume full expr: node_filesystem_avail{mountpoint="/"} / node_filesystem_size{mountpoint="/"} < 0.1 labels: severity: critical type: ses_default annotations: description: Root volume (OSD and MON store) is dangerously full (< 10% free) # alert on nic packet errors and drops rates > 1 packet/s - alert: network packets dropped expr: irate(node_network_receive_drop{device!="lo"}[5m]) + irate(node_network_transmit_drop{device!="lo"}[5m]) > 1 labels: severity: warning type: ses_default annotations: description: > Node {{ $labels.instance }} experiences packet drop > 1 packet/s on interface {{ $lables.device }} - alert: network packet errors expr: irate(node_network_receive_errs{device!="lo"}[5m]) + irate(node_network_transmit_errs{device!="lo"}[5m]) > 1 labels: severity: warning type: ses_default annotations: description: > Node {{ $labels.instance }} experiences packet errors > 1 packet/s on interface {{ $lables.device }} # predict fs fillup times - alert: storage filling expr: ((node_filesystem_free - node_filesystem_size) / deriv(node_filesystem_free[2d]) <= 5) > 0 labels: severity: warning type: ses_default annotations: description: > Mountpoint {{ $lables.mountpoint }} will be full in less then 5 days assuming the average fillup rate of the past 48 hours. - name: pools rules: - alert: pool full expr: ceph_pool_used_bytes / ceph_pool_available_bytes > 0.9 labels: severity: critical type: ses_default annotations: description: Pool {{ $labels.pool }} at 90% capacity or over - alert: pool filling up expr: (-ceph_pool_used_bytes / deriv(ceph_pool_available_bytes[2d]) <= 5 ) > 0 labels: severity: warning type: ses_default annotations: description: > Pool {{ $labels.pool }} will be full in less then 5 days assuming the average fillup rate of the past 48 hours.