Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Enterprise Storage 6

18 Monitoring and Alerting Edit source

In SUSE Enterprise Storage 6, DeepSea no longer deploys a monitoring and alerting stack on the Salt master. Users have to define the Prometheus role for Prometheus and Alertmanager, and the Grafana role for Grafana. When multiple nodes are assigned with the Prometheus or Grafana role, a highly available setup is deployed.

  • Prometheus is the monitoring and alerting toolkit.

  • Alertmanager handles alerts sent by the Prometheus server.

  • Grafana is the visualization and alerting software.

  • The prometheus-node_exporter is the service running on all Salt minions.

The Prometheus configuration and scrape targets (exporting daemons) are setup automatically by DeepSea. DeepSea also deploys a list of default alerts, for example health error, 10% OSDs down, or pgs inactive.

18.1 Pillar Variables Edit source

The Salt pillar is a key-value store that provides information and configuration values to minions. It is available to all minions, each with differing content. The Salt pillar is pre-populated with default values and can be customized in two different ways:

  • /srv/pillar/ceph/stack/global.yml: to change pillar variables for all nodes.

  • /srv/pillar/ceph/stack/CLUSTER_NAME/minions/HOST: to change specific minion configurations.

The pillar variables below are available to all nodes by default:

  monitoring:
    alertmanager:
      config: salt://path/to/config
      additional_flags: ''
    grafana:
      ssl_cert: False # self-signed certs are created by default
      ssl_key: False # self-signed certs are created by default
    prometheus:
      # pass additional configration to prometheus
      additional_flags: ''
      alert_relabel_config: []
      rule_files: []
      # per exporter config variables
      scrape_interval:
        ceph: 10
        node_exporter: 10
        prometheus: 10
        grafana: 10
      relabel_config:
        alertmanager: []
        ceph: []
        node_exporter: []
        prometheus: []
        grafana: []
      metric_relabel_config:
        ceph: []
        node_exporter: []
        prometheus: []
        grafana: []
      target_partition:
        ceph: '1/1'
        node_exporter: '1/1'
        prometheus: '1/1'
        grafana: '1/1'

18.2 Grafana Edit source

18.2.1 Grafana SSL/TLS certificates Edit source

All traffic is encrypted through Grafana. You can either supply your own SSL certs or create self-signed one.

Grafana uses the following variables:

  • ssl_cert

  • ssl_key

The Ceph Dashboard embeds the Grafana dashboards via HTML iframe elements. If Grafana is configured without SSL/TLS support, or if SSL is using self-signed certificates, and if the SSL support in the dashboard has been enabled (which is the default configuration), then most browsers will block the embedding of insecure content into a secured web page. If you can not see the embedded Grafana dashboards in Ceph Dashboard, check your browser's documentation on how to unblock mixed content or how to accept self-signed certificates. Alternatively, consider enabling SSL/TLS support in Grafana, using a certificate that is issued by a certificate authority (CA) known to the browser.

For more information on supplying your own SSL certificates, see Section 13.1.3, “Certificates Signed by CA”. For generating a self-signed or trusted third-party certificate using OpenSSL, see Section 13.1.2, “Self-signed or Trusted Third-party Certificate with OpenSSL”. For creating your own CA-signed certificate, see Section 13.1.1, “Self-signed Certificates”. For creating your own custom CA signed certificate, see Section 13.1.4, “Certificates Signed with a Custom CA”.

18.2.2 Configuring Grafana frontend URL Edit source

The Ceph Dashboard backend requires the Grafana URL to be able to verify the existence of Grafana dashboards before the frontend even loads them. Due to the nature of how Grafana is implemented in Ceph Dashboard, this means that two working connections are required in order to be able to see Grafana graphs in Ceph Dashboard:

  • The backend (Ceph Manager module) needs to verify the existence of the requested graph. If this request succeeds, it lets the frontend know that it can safely access Grafana.

  • The frontend then requests the Grafana graphs directly from the user's browser using an iframe. The Grafana instance is accessed directly without any detour through Ceph Dashboard.

Now, it might be the case that your environment makes it difficult for the user's browser to directly access the URL configured in Ceph Dashboard. To solve this issue, a separate URL can be configured which will solely be used to tell the frontend (the user's browser) which URL it should use to access Grafana.

To change the URL that is returned to the frontend issue the following command:

cephadm@adm > ceph dashboard set-grafana-frontend-api-url GRAFANA-SERVER-URL

If no value is set for that option, it will simply fall back to the value of the GRAFANA_API_URL option, which is set automatically by DeepSea. If set, it will instruct the browser to use this URL to access Grafana.

18.3 Prometheus Edit source

The exporter based configuration that can be passed through the pillar. These groups map to exporters that provide data. The node exporter is present on all nodes, Ceph is exported by the Ceph Manager nodes, Prometheus and Grafana is exported by the respective Prometheus and Grafana nodes.

Prometheus uses the following variables:

  • scrape_interval: change the scrape interval, how often an exporter is to be scraped.

  • target_partition: partition scrape targets when multiple Prometheus instnaces are deployed and have some instances scrape only part of all exporter instances.

  • relabel_config: dynamically rewrites the label set of a target before it gets scraped. Multiple relabeling steps can be configured per scrape configuration.

  • metrics_relabel_config: applied to samples as the last step before ingestion.

18.3.1 Security Model Edit source

Prometheus' security model presumes that untrusted users have access to the Prometheus HTTP endpoint and logs. Untrusted users have access to all the (meta-)data Prometheus collects that is contained in the database, plus a variety of operational and debugging information.

However, Prometheus' HTTP API is limited to read-only operations. Configurations cannot be changed using the API, and secrets are not exposed. Moreover, Prometheus has some built-in measures to mitigate the impact of denial of service attacks.

18.4 Alertmanager Edit source

The Alertmanager handles alerts sent by the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver. It also takes care of silencing of alerts. Alertmanager is configured via the command line flags and a configuration file that defines inhibition rules, notification routing and notification receivers.

18.4.1 Configuration File Edit source

Alertmanager's configuration is different for each deployment. Therefore, DeepSea does not ship any related defaults. You need to provide your own alertmanager.yml configuration file. The alertmanager package by default installs a configuration file /etc/prometheus/alertmanager.yml which can serve as an example configuration. If you prefer to have your Alertmanager configuration managed by DeepSea, add the following key to your pillar, for example to the /srv/pillar/ceph/stack/ceph/minions/YOUR_SALT_MASTER_MINION_ID.sls file:

For a complete example of Alertmanager's configuration file, see Section 18.5, “Troubleshooting Alerts”.

monitoring:
  alertmanager:
    /path/to/your/alertmanager/config.yml

Alertmanager's configuration file is written in the YAML format. It follows the scheme described below. Parameters in brackets are optional. For non-list parameters the default value is used. The following generic placeholders are used in the scheme:

DURATION

A duration matching the regular expression [0-9]+(ms|[smhdwy])

LABELNAME

A string matching the regular expression [a-zA-Z_][a-zA-Z0-9_]*

LABELVALUE

A string of Unicode characters.

FILEPATH

A valid path in the current working directory.

BOOLEAN

A Boolean that can take the values 'true' or 'false'.

STRING

A regular string.

SECRET

A regular string that is a secret, for example a password.

TMPL_STRING

A string which is template-expanded before usage.

TMPL_SECRET

A secret string which is template-expanded before usage.

Example 18.1: Global Configuration

Parameters in the global: configuration are valid in all other configuration contexts. They also serve as defaults for other configuration sections.

global:
# the time after which an alert is declared resolved if it has not been updated
[ resolve_timeout: DURATION | default = 5m ]

# The default SMTP From header field.
[ smtp_from: TMPL_STRING ]
# The default SMTP smarthost used for sending emails, including port number.
# Port number usually is 25, or 587 for SMTP over TLS
# (sometimes referred to as STARTTLS).
# Example: smtp.example.org:587
[ smtp_smarthost: STRING ]
# The default host name to identify to the SMTP server.
[ smtp_hello: STRING | default = "localhost" ]
[ smtp_auth_username: STRING ]
# SMTP Auth using LOGIN and PLAIN.
[ smtp_auth_password: SECRET ]
# SMTP Auth using PLAIN.
[ smtp_auth_identity: STRING ]
# SMTP Auth using CRAM-MD5.
[ smtp_auth_secret: SECRET ]
# The default SMTP TLS requirement.
[ smtp_require_tls: BOOL | default = true ]

# The API URL to use for Slack notifications.
[ slack_api_url: STRING ]
[ victorops_api_key: STRING ]
[ victorops_api_url: STRING | default = "https://victorops.example.com/integrations/alert/" ]
[ pagerduty_url: STRING | default = "https://pagerduty.example.com/v2/enqueue" ]
[ opsgenie_api_key: STRING ]
[ opsgenie_api_url: STRING | default = "https://opsgenie.example.com/" ]
[ hipchat_api_url: STRING | default = "https://hipchat.example.com/" ]
[ hipchat_auth_token: SECRET ]
[ wechat_api_url: STRING | default = "https://wechat.example.com/cgi-bin/" ]
[ wechat_api_secret: SECRET ]
[ wechat_api_corp_id: STRING ]

# The default HTTP client configuration
[ http_config: HTTP_CONFIG ]

# Files from which custom notification template definitions are read.
# The last component may use a wildcard matcher, e.g. 'templates/*.tmpl'.
templates:
[ - FILEPATH ... ]

# The root node of the routing tree.
route: ROUTE

# A list of notification receivers.
receivers:
- RECEIVER ...

# A list of inhibition rules.
inhibit_rules:
[ - INHIBIT_RULE ... ]
Example 18.2: ROUTE

A ROUTE block defines a node in a routing tree. Unspecified parameters are inherited from its parent node. Every alert enters the routing tree at the configured top-level route, which needs to match all alerts. It then traverses the child nodes. If the continue option is set to 'false', the traversing stops after the first matched child. Setting the option to 'true' on a matched node, the alert will continue matching against subsequent siblings. If an alert does not match any children of a node, the alert is handled based on the configuration parameters of the current node.

[ receiver: STRING ]
[ group_by: '[' LABELNAME, ... ']' ]

# If an alert should continue matching subsequent sibling nodes.
[ continue: BOOLEAN | default = false ]

# A set of equality matchers an alert has to fulfill to match a node.
match:
 [ LABELNAME: LABELVALUE, ... ]

# A set of regex-matchers an alert has to fulfill to match a node.
match_re:
 [ LABELNAME: REGEX, ... ]

# Time to wait before sending a notification for a group of alerts.
[ group_wait: DURATION | default = 30s ]

# Time to wait before sending a notification about new alerts
# added to a group of alerts for which an initial notification has
# already been sent.
[ group_interval: DURATION | default = 5m ]

# Time to wait before re-sending a notification
[ repeat_interval: DURATION | default = 4h ]

# Possible child routes.
routes:
 [ - ROUTE ... ]
Example 18.3: INHIBIT_RULE

An inhibition rule mutes a target alert that matches a set of matchers when a source alert exists that matches another set of matchers. Both alerts need to share the same label values for the label names in the equal list.

Alerts can match and therefore inhibit themselves. Do not write inhibition rules where an alert matches both source and target.

# Matchers that need to be fulfilled for the alerts to be muted.
target_match:
 [ LABELNAME: LABELVALUE, ... ]
target_match_re:
 [ LABELNAME: REGEX, ... ]

# Matchers for which at least one alert needs to exist so that the
# inhibition occurs.
source_match:
 [ LABELNAME: LABELVALUE, ... ]
source_match_re:
 [ LABELNAME: REGEX, ... ]

# Labels with an equal value in the source and target
# alert for the inhibition to take effect.
[ equal: '[' LABELNAME, ... ']' ]
Example 18.4: HTTP_CONFIG

HTTP_CONFIG configures the HTTP client used by the receiver to communicate with API services.

Note that basic_auth, bearer_token and bearer_token_file options are mutually exclusive.

# Sets the 'Authorization' header with the user name and password.
basic_auth:
[ username: STRING ]
[ password: SECRET ]

# Sets the 'Authorization' header with the bearer token.
[ bearer_token: SECRET ]

# Sets the 'Authorization' header with the bearer token read from a file.
[ bearer_token_file: FILEPATH ]

# TLS settings.
tls_config:
# CA certificate to validate the server certificate with.
[ ca_file: FILEPATH ]
# Certificate and key files for client cert authentication to the server.
[ cert_file: FILEPATH ]
[ key_file: FILEPATH ]
# ServerName extension to indicate the name of the server.
# http://tools.ietf.org/html/rfc4366#section-3.1
[ server_name: STRING ]
# Disable validation of the server certificate.
[ insecure_skip_verify: BOOLEAN | default = false]

# Optional proxy URL.
[ proxy_url: STRING ]
Example 18.5: RECEIVER

Receiver is a named configuration for one or more notification integrations.

Instead of adding new receivers, we recommend implementing custom notification integrations using the webhook receiver (see Example 18.15, “WEBHOOK_CONFIG).

# The unique name of the receiver.
name: STRING

# Configurations for several notification integrations.
email_configs:
[ - EMAIL_CONFIG, ... ]
hipchat_configs:
[ - HIPCHAT_CONFIG, ... ]
pagerduty_configs:
[ - PAGERDUTY_CONFIG, ... ]
pushover_configs:
[ - PUSHOVER_CONFIG, ... ]
slack_configs:
[ - SLACK_CONFIG, ... ]
opsgenie_configs:
[ - OPSGENIE_CONFIG, ... ]
webhook_configs:
[ - WEBHOOK_CONFIG, ... ]
victorops_configs:
[ - VICTOROPS_CONFIG, ... ]
wechat_configs:
[ - WECHAT_CONFIG, ... ]
Example 18.6: EMAIL_CONFIG
# Whether to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = false ]

# The email address to send notifications to.
to: TMPL_STRING

# The sender address.
[ from: TMPL_STRING | default = global.smtp_from ]

# The SMTP host through which emails are sent.
[ smarthost: STRING | default = global.smtp_smarthost ]

# The host name to identify to the SMTP server.
[ hello: STRING | default = global.smtp_hello ]

# SMTP authentication details.
[ auth_username: STRING | default = global.smtp_auth_username ]
[ auth_password: SECRET | default = global.smtp_auth_password ]
[ auth_secret: SECRET | default = global.smtp_auth_secret ]
[ auth_identity: STRING | default = global.smtp_auth_identity ]

# The SMTP TLS requirement.
[ require_tls: BOOL | default = global.smtp_require_tls ]

# The HTML body of the email notification.
[ html: TMPL_STRING | default = '{{ template "email.default.html" . }}' ]
# The text body of the email notification.
[ text: TMPL_STRING ]

# Further headers email header key/value pairs. Overrides any headers
# previously set by the notification implementation.
[ headers: { STRING: TMPL_STRING, ... } ]
Example 18.7: HIPCHAT_CONFIG
# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = false ]

# The HipChat Room ID.
room_id: TMPL_STRING
# The authentication token.
[ auth_token: SECRET | default = global.hipchat_auth_token ]
# The URL to send API requests to.
[ api_url: STRING | default = global.hipchat_api_url ]

# A label to be shown in addition to the sender's name.
[ from:  TMPL_STRING | default = '{{ template "hipchat.default.from" . }}' ]
# The message body.
[ message:  TMPL_STRING | default = '{{ template "hipchat.default.message" . }}' ]
# Whether this message will trigger a user notification.
[ notify:  BOOLEAN | default = false ]
# Determines how the message is treated by the alertmanager and rendered inside HipChat. Valid values are 'text' and 'html'.
[ message_format:  STRING | default = 'text' ]
# Background color for message.
[ color:  TMPL_STRING | default = '{{ if eq .Status "firing" }}red{{ else }}green{{ end }}' ]

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]
Example 18.8: PAGERDUTY_CONFIG

The routing_key and service_key options are mutually exclusive.

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The PagerDuty integration key (when using 'Events API v2').
routing_key: TMPL_SECRET
# The PagerDuty integration key (when using 'Prometheus').
service_key: TMPL_SECRET

# The URL to send API requests to.
[ url: STRING | default = global.pagerduty_url ]

# The client identification of the Alertmanager.
[ client:  TMPL_STRING | default = '{{ template "pagerduty.default.client" . }}' ]
# A backlink to the notification sender.
[ client_url:  TMPL_STRING | default = '{{ template "pagerduty.default.clientURL" . }}' ]

# The incident description.
[ description: TMPL_STRING | default = '{{ template "pagerduty.default.description" .}}' ]

# Severity of the incident.
[ severity: TMPL_STRING | default = 'error' ]

# A set of arbitrary key/value pairs that provide further details.
[ details: { STRING: TMPL_STRING, ... } | default = {
 firing:       '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
 resolved:     '{{ template "pagerduty.default.instances" .Alerts.Resolved }}'
 num_firing:   '{{ .Alerts.Firing | len }}'
 num_resolved: '{{ .Alerts.Resolved | len }}'
} ]

# The HTTP client's configuration.
[ http_config: HTTP_CONFIG | default = global.http_config ]
Example 18.9: PUSHOVER_CONFIG
# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The recipient user key.
user_key: SECRET

# Registered application’s API token.
token: SECRET

# Notification title.
[ title: TMPL_STRING | default = '{{ template "pushover.default.title" . }}' ]

# Notification message.
[ message: TMPL_STRING | default = '{{ template "pushover.default.message" . }}' ]

# A supplementary URL displayed together with the message.
[ url: TMPL_STRING | default = '{{ template "pushover.default.url" . }}' ]

# Priority.
[ priority: TMPL_STRING | default = '{{ if eq .Status "firing" }}2{{ else }}0{{ end }}' ]

# How often the Pushover servers will send the same notification (at least 30 seconds).
[ retry: DURATION | default = 1m ]

# How long your notification will continue to be retried (unless the user
# acknowledges the notification).
[ expire: DURATION | default = 1h ]

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]
Example 18.10: SLACK_CONFIG
# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = false ]

# The Slack webhook URL.
[ api_url: SECRET | default = global.slack_api_url ]

# The channel or user to send notifications to.
channel: TMPL_STRING

# API request data as defined by the Slack webhook API.
[ icon_emoji: TMPL_STRING ]
[ icon_url: TMPL_STRING ]
[ link_names: BOOLEAN | default = false ]
[ username: TMPL_STRING | default = '{{ template "slack.default.username" . }}' ]
# The following parameters define the attachment.
actions:
[ ACTION_CONFIG ... ]
[ color: TMPL_STRING | default = '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}' ]
[ fallback: TMPL_STRING | default = '{{ template "slack.default.fallback" . }}' ]
fields:
[ FIELD_CONFIG ... ]
[ footer: TMPL_STRING | default = '{{ template "slack.default.footer" . }}' ]
[ pretext: TMPL_STRING | default = '{{ template "slack.default.pretext" . }}' ]
[ short_fields: BOOLEAN | default = false ]
[ text: TMPL_STRING | default = '{{ template "slack.default.text" . }}' ]
[ title: TMPL_STRING | default = '{{ template "slack.default.title" . }}' ]
[ title_link: TMPL_STRING | default = '{{ template "slack.default.titlelink" . }}' ]
[ image_url: TMPL_STRING ]
[ thumb_url: TMPL_STRING ]

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]
Example 18.11: ACTION_CONFIG for SLACK_CONFIG
# Provide a button to tell Slack you want to render a button.
type: TMPL_STRING
# Label for the button.
text: TMPL_STRING
# http or https URL to deliver users to. If you specify invalid URLs, the message will be posted with no button.
url: TMPL_STRING
#  If set to 'primary', the button will be green, indicating the best forward action to take
#  'danger' turns the button red, indicating a destructive action.
[ style: TMPL_STRING [ default = '' ]
Example 18.12: FIELD_CONFIG for SLACK_CONFIG
# A bold heading without markup above the value text.
title: TMPL_STRING
# The text of the field. It can span across several lines.
value: TMPL_STRING
# A flag indicating if value is short enough to be displayed together with other values.
[ short: BOOLEAN | default = slack_config.short_fields ]
Example 18.13: OPSGENIE_CONFIG
# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The API key to use with the OpsGenie API.
[ api_key: SECRET | default = global.opsgenie_api_key ]

# The host to send OpsGenie API requests to.
[ api_url: STRING | default = global.opsgenie_api_url ]

# Alert text (maximum is 130 characters).
[ message: TMPL_STRING ]

# A description of the incident.
[ description: TMPL_STRING | default = '{{ template "opsgenie.default.description" . }}' ]

# A backlink to the sender.
[ source: TMPL_STRING | default = '{{ template "opsgenie.default.source" . }}' ]

# A set of arbitrary key/value pairs that provide further detail.
[ details: { STRING: TMPL_STRING, ... } ]

# Comma separated list of team responsible for notifications.
[ teams: TMPL_STRING ]

# Comma separated list of tags attached to the notifications.
[ tags: TMPL_STRING ]

# Additional alert note.
[ note: TMPL_STRING ]

# Priority level of alert, one of P1, P2, P3, P4, and P5.
[ priority: TMPL_STRING ]

# Configuration of the HTTP.
[ http_config: HTTP_CONFIG | default = global.http_config ]
Example 18.14: VICTOROPS_CONFIG
# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The API key for talking to the VictorOps API.
[ api_key: SECRET | default = global.victorops_api_key ]

# The VictorOps API URL.
[ api_url: STRING | default = global.victorops_api_url ]

# A key used to map the alert to a team.
routing_key: TMPL_STRING

# Describes the behavior of the alert (one of 'CRITICAL', 'WARNING', 'INFO').
[ message_type: TMPL_STRING | default = 'CRITICAL' ]

# Summary of the alerted problem.
[ entity_display_name: TMPL_STRING | default = '{{ template "victorops.default.entity_display_name" . }}' ]

# Long explanation of the alerted problem.
[ state_message: TMPL_STRING | default = '{{ template "victorops.default.state_message" . }}' ]

# The monitoring tool the state message is from.
[ monitoring_tool: TMPL_STRING | default = '{{ template "victorops.default.monitoring_tool" . }}' ]

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]
Example 18.15: WEBHOOK_CONFIG

You can use the webhook receiver to configure a generic receiver.

# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = true ]

# The endpoint for sending HTTP POST requests.
url: STRING

# Configuration of the HTTP client.
[ http_config: HTTP_CONFIG | default = global.http_config ]

Alertmanager sends HTTP POST requests in the following JSON format:

{
 "version": "4",
 "groupKey": STRING, // identifycation of the group of alerts (to deduplicate)
 "status": "<resolved|firing>",
 "receiver": STRING,
 "groupLabels": OBJECT,
 "commonLabels": OBJECT,
 "commonAnnotations": OBJECT,
 "externalURL": STRING, // backlink to Alertmanager.
 "alerts": [
   {
     "status": "<resolved|firing>",
     "labels": OBJECT,
     "annotations": OBJECT,
     "startsAt": "<rfc3339>",
     "endsAt": "<rfc3339>",
     "generatorURL": STRING // identifies the entity that caused the alert
   },
   ...
 ]
}

The webhook receiver allows for integration with the following notification mechanisms:

  • DingTalk (https://github.com/timonwong/prometheus-webhook-dingtalk)

  • IRC Bot (https://github.com/multimfi/bot)

  • JIRAlert (https://github.com/free/jiralert)

  • Phabricator / Maniphest (https://github.com/knyar/phalerts)

  • prom2teams: forwards notifications to Microsoft Teams (https://github.com/idealista/prom2teams)

  • SMS: supports multiple providers (https://github.com/messagebird/sachet)

  • Telegram bot (https://github.com/inCaller/prometheus_bot)

  • SNMP trap (https://github.com/SUSE/prometheus-webhook-snmp)

Example 18.16: WECHAT_CONFIG
# Whether or not to notify about resolved alerts.
[ send_resolved: BOOLEAN | default = false ]

# The API key to use for the WeChat API.
[ api_secret: SECRET | default = global.wechat_api_secret ]

# The WeChat API URL.
[ api_url: STRING | default = global.wechat_api_url ]

# The corp id used to authenticate.
[ corp_id: STRING | default = global.wechat_api_corp_id ]

# API request data as defined by the WeChat API.
[ message: TMPL_STRING | default = '{{ template "wechat.default.message" . }}' ]
[ agent_id: STRING | default = '{{ template "wechat.default.agent_id" . }}' ]
[ to_user: STRING | default = '{{ template "wechat.default.to_user" . }}' ]
[ to_party: STRING | default = '{{ template "wechat.default.to_party" . }}' ]
[ to_tag: STRING | default = '{{ template "wechat.default.to_tag" . }}' ]

18.4.2 Custom Alerts Edit source

You can define your custom alert conditions to send notifications to an external service. Prometheus uses its own expression language for defining custom alerts. Following is an example of a rule with an alert:

groups:
- name: example
 rules:
  # alert on high deviation from average PG count
  - alert: high pg count deviation
   expr: abs(((ceph_osd_pgs > 0) - on (job) group_left avg(ceph_osd_pgs > 0) by (job)) / on (job) group_left avg(ceph_osd_pgs > 0) by (job)) > 0.35
   for: 5m
   labels:
    severity: warning
    type: ses_default
   annotations:
   description: >
    OSD {{ $labels.osd }} deviates by more then 30% from average PG count

The optional for clause specifies the time Prometheus will wait between first encountering a new expression output vector element and counting an alert as firing. In this case, Prometheus will check that the alert continues to be active for 5 minutes before firing the alert. Elements in a pending state are active, but not firing yet.

The labels clause specifies a set of additional labels attached to the alert. Conflicting labels will be overwritten. Labels can be templated (see Section 18.4.2.1, “Templates” for more details on templating).

The annotations clause specifies informational labels. You can use them to store additional information, for example alert descriptions or runbook links. Annotations can be templated (see Section 18.4.2.1, “Templates” for more details on templating).

To add your custom alerts to SUSE Enterprise Storage 6, either

  • place your YAML files with custom alerts in the /etc/prometheus/alerts directory

or

  • provide a list of paths to your custom alert files in the Pillar under the monitoring:custom_alerts key. DeepSea Stage 2 or the salt SALT_MASTER state.apply ceph.monitoring.prometheus command will add your alert files in the right place.

    Example 18.17: Adding Custom Alerts to SUSE Enterprise Storage

    A file with custom alerts is in /root/my_alerts/my_alerts.yml on the Salt master. If you add

    monitoring:
     custom_alerts:
       - /root/my_alerts/my_alerts.yml

    to the /srv/pillar/ceph/cluster/YOUR_SALT_MASTER_MINION_ID.sls file, DeepSea will create the /etc/prometheus/alerts/my_alerts.yml file and restart Prometheus.

18.4.2.1 Templates Edit source

You can use templates for label and annotation values. The $labels variable includes the label key/value pairs of an alert instance, while $value holds the evaluated value of an alert instance.

The following example inserts a firing element label and value:

{{ $labels.LABELNAME }}
{{ $value }}

18.4.2.2 Inspecting Alerts at Runtime Edit source

If you need to verify which alerts are active, you have several options:

  • Navigate to the Alerts tab of Prometheus. It will show you the exact label sets for which defined alerts are active. Prometheus also stores synthetic time series for pending and firing alerts. They have the following form:

    ALERTS{alertname="ALERT_NAME", alertstate="pending|firing", ADDITIONAL_ALERT_LABELS}

    The sample value is 1 if the alert is active (pending or firing). The series is marked 'stale' when the alert is inactive.

  • In the Prometheus Web interface at the URL address http://PROMETHEUS_HOST_IP:9090/alerts, inspect alerts and their state (INACTIVE, PENDING or FIRING).

  • In the Alertmanager Web interface at the URL address http://:PROMETHEUS_HOST_IP:9093/#/alerts, inspect alerts and silence them if desired.

18.4.3 SNMP Trap Receiver Edit source

If you want to get notified about Prometheus alerts via SNMP traps, then you can install the Prometheus Alertmanager SNMP trap receiver via DeepSea. To do so you need to enable it in the pillar under the monitoring:alertmanager_receiver_snmp:enabled key in your global.yml file. The configuration of the receiver must be set under the monitoring:alertmanager_receiver_snmp:config key.

Example 18.18: SNMP Trap Configuration
monitoring:
 alertmanager:
   receiver:
      snmp:
        enabled: True
        config:
          host: localhost
          port: 9099
          snmp_host: snmp.foo-bar.com
          snmp_community: private
          metrics: True

Refer to the receiver manual at https://github.com/SUSE/prometheus-webhook-snmp#global-configuration-file. for more details about the configuration options.

DeepSea Stage 2 or the salt SALT_MASTER state.apply ceph.monitoring.alertmanager command will install and configure the receiver in the appropriate location. Verify your settings with:

root@master # salt-call pillar.get 'monitoring:alertmanager_receiver_snmp:enabled'
root@master # salt-call pillar.get 'monitoring:alertmanager_receiver_snmp:config'

18.5 Troubleshooting Alerts Edit source

The following section details the alert that has been triggered and actions to take when the alert is displayed.

MONITOR
MON_DOWN

One or more monitor daemons are down. The cluster requires a majority of the monitors in order to function. When one or more monitors are down, clients will initially have difficulty connecting to the cluster.

Restart the monitor daemon that is down as soon as possible to reduce the risk of a subsequent monitor failure.

MON_CLOCK_SKEW

The clocks on the hosts running the ceph-mon monitor daemons are not well synchronized. This health alert is raised if the cluster detects a clock skew greater than mon_clock_drift_allowed. Resolve this by synchronizing the clocks using either ntpd or chrony. If it is impractical to keep the clocks closely synchronized, the mon_clock_drift_allowed threshold can be increased, but this value must stay well below the mon_lease interval in order for monitor cluster to function properly.

MON_MSGR2_NOT_ENABLED

The ms_bind_msgr2 option is enabled but one or more monitors is not configured to bind to a v2 port in the cluster’s monmap. This means that features specific to the msgr2 protocol (for example, encryption) are not available on some or all connections. In most cases this can be corrected by issuing the following command:

cephadm@adm > ceph mon enable-msgr2

This command changes any monitor configured for the old default port 6789 to continue to listen for v1 connections on 6789 and also listen for v2 connections on the new default 3300 port. If a monitor is configured to listen for v1 connections on a non-standard port (not 6789), then the monmap needs to be modified manually.

MANAGER
MGR_MODULE_DEPENDENCY

An enabled manager module is failing its dependency check. This health check should come with a message from the module about the problem. For example, a module might report that a required package is not installed. In which case, the message will read: "Install the required package and restart your manager daemons." This health check only applies to enabled modules. If a module is not enabled, you can see whether it is reporting dependency issues in the output of ceph module ls.

MGR_MODULE_ERROR

A manager module has experienced an unexpected error. Typically, this means an unhandled exception was raised from the module’s serve function. The human readable description of the error may be obscurely worded if the exception did not provide a useful description of itself. This health check may indicate a bug. Open a bug report if you think you have encountered a bug. If you believe the error is transient, you may restart your manager daemon(s), or use ceph mgr fail on the active daemon to prompt a failover to another daemon.

OSDS
OSD_DOWN

One or more OSDs are marked down. The ceph-osd daemon may have been stopped, or peer OSDs may be unable to reach the OSD over the network. Common causes include a stopped or crashed daemon, a down host, or a network outage. Verify the host is healthy, the daemon is started, and network is functioning. If the daemon has crashed, the daemon log file (/var/log/ceph/ceph-osd.*) may contain debugging information.

OSD_CRUSH TYPE_DOWN

For example, OSD_HOST_DOWN or OSD_ROOT_DOWN. All the OSDs within a particular CRUSH subtree are marked down, for example all OSDs on a host.

OSD_ORPHAN

An OSD is referenced in the CRUSH Map hierarchy but does not exist. The OSD can be removed from the CRUSH hierarchy with:

cephadm@adm > ceph osd crush rm osd.ID
OSD_OUT_OF_ORDER_FULL

The utilization thresholds for backfillfull, nearfull, full, and failsafe_full are not ascending. The thresholds can be adjusted with:

cephadm@adm > ceph osd set-backfillfull-ratio RATIO
cephadm@adm > ceph osd set-nearfull-ratio RATIO
cephadm@adm > ceph osd set-full-ratio RATIO
OSD_FULL

One or more OSDs have exceeded the full threshold and is preventing the cluster from servicing writes. Utilization by pool can be checked with:

cephadm@adm > ceph df

The currently defined full ratio can be seen with:

cephadm@adm > ceph osd dump | grep full_ratio

A short-term workaround to restore write availability is to raise the full threshold by a small amount:

cephadm@adm > ceph osd set-full-ratio RATIO

New storage should be added to the cluster by deploying more OSDs or existing data should be deleted in order to free up space.

OSD_BACKFILLFULL

One or more OSDs have exceeded the backfillfull threshold, preventing data from being allowed to rebalance to this device. This is an early warning that rebalancing may not be able to complete and that the cluster is approaching full. Utilization by pool can be checked with:

cephadm@adm > ceph df
OSD_NEARFULL

One or more OSDs have exceeded the nearfull threshold. This is an early warning that the cluster is approaching full. Utilization by pool can be checked with:

cephadm@adm > ceph df
OSDMAP_FLAGS

One or more cluster flags of interest has been set. These flags include:

full

The cluster is flagged as full and cannot serve writes

pauserd, pausewr

Paused reads or writes

noup

OSDs are not allowed to start

nodown

OSD failure reports are being ignored and the monitors are not marking OSDs down

noin

OSDs that were previously marked out are not being marked back in when they start

noout

Down OSDs are not automatically marked out after the configured interval

nobackfill, norecover, norebalance

Recovery or data rebalancing is suspended

noscrub, nodeep_scrub

Scrubbing is disabled

notieragent

Cache tiering activity is suspended

With the exception of full, these flags can be set or cleared with:

cephadm@adm > ceph osd set FLAG
cephadm@adm > ceph osd unset FLAG
OSD_FLAGS

One or more OSDs or CRUSH {nodes,device classes} has a flag of interest set. These flags include:

noup

These OSDs are not allowed to start

nodown

Failure reports for these OSDs are ignored

noin

If these OSDs were previously marked out automatically after a failure, they are not to be marked in when they start

noout

If these OSDs are down they are not automatically marked out after the configured interval

These flags can be set and cleared in batch with:

cephadm@adm > ceph osd set-group FLAG WHO
cephadm@adm > ceph osd unset-group FLAG WHO

For example:

cephadm@adm > ceph osd set-group noup,noout osd.0 osd.1
cephadm@adm > ceph osd unset-group noup,noout osd.0 osd.1
cephadm@adm > ceph osd set-group noup,noout host-foo
cephadm@adm > ceph osd unset-group noup,noout host-foo
cephadm@adm > ceph osd set-group noup,noout class-hdd
cephadm@adm > ceph osd unset-group noup,noout class-hdd
OLD_CRUSH_TUNABLES

The CRUSH Map is using old settings and should be updated. The oldest tunables that can be used (for example, the oldest client version that can connect to the cluster) without triggering this health warning are determined by the mon_crush_min_required_version config option.

OLD_CRUSH_STRAW_CALC_VERSION

The CRUSH Map is using an older, sub-optimal method for calculating intermediate weight values for straw buckets. The CRUSH Map requires an update to use the newer method (straw_calc_version=1).

CACHE_POOL_NO_HIT_SET

One or more cache pools are not configured with a hit set to track utilization. This prevents the tiering agent from identifying cold objects to flush and evict from the cache. Hit sets can be configured on the cache pool with the following:

cephadm@adm > ceph osd pool set POOLNAME hit_set_type TYPE
cephadm@adm > ceph osd pool set POOLNAME hit_set_period PERIOD-IN-SECONDS
cephadm@adm > ceph osd pool set POOLNAME hit_set_count NUMBER-OF-HITSETS
cephadm@adm > ceph osd pool set POOLNAME hit_set_fpp TARGET-FALSE-POSITIVE-RATE
OSD_NO_SORTBITWISE

No SUSE Enterprise Storage 5.5 v12.y.z OSDs are running but the sortbitwise flag has not been set. Set the sortbitwise flag before v12.y.z or newer OSDs can start. You can safely set the flag with:

cephadm@adm > ceph osd set sortbitwise
POOL_FULL

One or more pools have reached the quota and are no longer allowing writes. Pool quotas and utilization can be seen with the following command:

cephadm@adm > ceph df detail

You can either raise the pool quota with the following commands:

cephadm@adm > ceph osd pool set-quota POOLNAME max_objects NUM-OBJECTS
cephadm@adm > ceph osd pool set-quota POOLNAME max_bytes NUM-BYTES

Or, you can delete existing data to reduce utilization.

BLUEFS_SPILLOVER

One or more OSDs that use the BlueStore backend have been allocated db partitions (storage space for metadata, normally on a faster device) but that space has filled, such that metadata has overflowed onto the normal slow device. This is not necessarily an error condition or even unexpected, but if the administrator’s expectation was that all metadata would fit on the faster device, it indicates that not enough space was provided. This warning can be disabled on all OSDs with the following command:

cephadm@adm > ceph config set osd bluestore_warn_on_bluefs_spillover false

Alternatively, it can be disabled on a specific OSD with the following command:

cephadm@adm > ceph config set osd.123 bluestore_warn_on_bluefs_spillover false

To provide more metadata space, the OSD in question can be destroyed and reprovisioned. This involves data migration and recovery. It is possible to expand the LVM logical volume backing the db storage. If the underlying LV has been expanded, the OSD daemon needs to be stopped and BlueFS informed of the device size change with the following command:

cephadm@adm > ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-$ID
BLUEFS_AVAILABLE_SPACE

To check how much space is free for BlueFS, execute:

cephadm@adm > ceph daemon osd.123 bluestore bluefs available

This provides output for up to 3 values; BDEV_DB free, BDEV_SLOW free and available_from_bluestore. BDEV_DB and BDEV_SLOW report the amount of space that has been acquired by BlueFS and is considered free. Value available_from_bluestore denotes ability of BlueStore to leave more space to BlueFS. It is normal that this value is different from amount of BlueStore free space, as BlueFS allocation unit is typically larger than BlueStore allocation unit. This means that only part of BlueStore free space is acceptable for BlueFS.

BLUEFS_LOW_SPACE

If BlueFS is running low on available free space and there is little available_from_bluestore, consider reducing BlueFS' allocation unit size. To simulate available space when the allocation unit is different, execute:

cephadm@adm > ceph daemon osd.123 bluestore bluefs available ALLOC-UNIT-SIZE
BLUESTORE_FRAGMENTATION

As BlueStore works, free space on underlying storage becomes fragmented. This is normal and unavoidable, but excessive fragmentation can cause slowdown. To inspect BlueStore fragmentation, execute:

cephadm@adm > ceph daemon osd.123 bluestore allocator score block

Score is given in [0-1] range. [0.0 .. 0.4] tiny fragmentation [0.4 .. 0.7] small, acceptable fragmentation [0.7 .. 0.9] considerable, but safe fragmentation [0.9 .. 1.0] severe fragmentation, can impact BlueFS' ability to get space from BlueStore. If detailed report of free fragments is required, execute:

cephadm@adm > ceph daemon osd.123 bluestore allocator dump block

If the OSD process does not perform fragmentation, inspect with ceph-bluestore-tool. Get the fragmentation score:

cephadm@adm > ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-score

Dump detailed free chunks:

cephadm@adm > ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-dump
BLUESTORE_LEGACY_STATFS

As of SUSE Enterprise Storage 6, BlueStore tracks its internal usage statistics on a per-pool granular basis and one or more OSDs have BlueStore volumes that were created prior to SUSE Enterprise Storage 6. If all OSDs are older than SUSE Enterprise Storage 6, the per-pool metrics are not available. However, if there is a mix of pre-SUSE Enterprise Storage 6 and post-SUSE Enterprise Storage 6 OSDs, the cluster usage statistics reported by ceph df will not be accurate. The old OSDs can be updated to use the new usage tracking scheme by stopping each OSD, running a repair operation, and the restarting it. For example, if osd.123 requires an update:

root # systemctl stop ceph-osd@123
cephadm@adm > ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-123
root # systemctl start ceph-osd@123

This warning can be disabled with:

cephadm@adm > ceph config set global bluestore_warn_on_legacy_statfs false
BLUESTORE_DISK_SIZE_MISMATCH

One or more OSDs using BlueStore has an internal inconsistency between the size of the physical device and the metadata tracking its size. This can lead to the OSD crashing in the future. The OSDs in question should be destroyed and re-deployed. To avoid putting any data at risk, re-deploy only one OSD at a time. For example, if OSD_ID has the error:

cephadm@adm > ceph osd out osd.$N
while ! ceph osd safe-to-destroy osd.$N ; do sleep 1m ; done
ceph osd destroy osd.$N
ceph-volume lvm zap /path/to/device
ceph-volume lvm create --osd-id $N --data /path/to/device
DEVICE HEALTH
DEVICE_HEALTH

One or more devices are expected to fail. The warning threshold is controlled by the mgr/devicehealth/warn_threshold configuration option. This warning only applies to OSDs that are currently marked in. The expected response to this failure is to mark the device out. The data is then migrated off of the device and the hardware is removed from the system. Marking out is normally done automatically if mgr/devicehealth/self_heal is enabled based on the mgr/devicehealth/mark_out_threshold. Device health can be checked with:

cephadm@adm > ceph device info DEVICE-ID

Device life expectancy is set by a prediction model run by the Ceph Manager or by an external tool via the command:

cephadm@adm > ceph device set-life-expectancy DEVICE-ID FROM TO

You can change the stored life expectancy manually, but that usually does not persist—the tool that originally set it reset and changing the stored value does not affect the actual health of the hardware device.

DEVICE_HEALTH_IN_USE

One or more devices are expected to fail and has been marked out of the cluster based on mgr/devicehealth/mark_out_threshold, but the devices are still participating in one more PGs. This may be because it was only recently marked as out and the data is still migrating, or because the data cannot be migrated off for some reason (for example, the cluster is nearly full, or the CRUSH hierarchy is such that there is not another suitable OSD to migrate the data to). This message can be silenced by disabling the self heal behavior (setting mgr/devicehealth/self_heal to false), by adjusting the mgr/devicehealth/mark_out_threshold, or by addressing what is preventing data from being migrated off of the ailing device.

DEVICE_HEALTH_TOOMANY

Too many devices are expected to fail and the mgr/devicehealth/self_heal behavior is enabled, such that marking out all of the ailing devices would exceed the clusters mon_osd_min_in_ratio ratio that prevents too many OSDs from being automatically marked out. This can indicates that too many devices in the cluster are expected to fail and action is required to add newer (healthier) devices before too many devices fail and data is lost. The health message can also be silenced by adjusting parameters like mon_osd_min_in_ratio or mgr/devicehealth/mark_out_threshold, but be warned that this increases the likelihood of unrecoverable data loss in the cluster.

DATA HEALTH (POOLS AND PLACEMENT GROUPS)
PG_AVAILABILITY

Data availability is reduced and the cluster is unable to service potential read or write requests for some data in the cluster. Specifically, if one or more PGs are in a state that does not allow IO requests to be serviced. Problematic PG states include peering, stale, incomplete, and in-active (if those conditions do not clear quickly). Detailed information about which PGs are affected is available from:

cephadm@adm > ceph health detail

In most cases the root cause is that one or more OSDs are currently down; see the discussion for OSD_DOWN above. The state of specific problematic PGs can be queried with:

cephadm@adm > ceph tell PG_ID query
PG_DEGRADED

Data redundancy is reduced for some data, meaning the cluster does not have the desired number of replicas for all data (for replicated pools) or erasure code fragments (for erasure coded pools). Specifically, if one or more PGs:

  • have a degraded or undersized flag set, meaning there are not enough instances of that placement group in the cluster;

  • have not had the clean flag set for some time.

PG_RECOVERY_FULL

Data redundancy can be reduced or at risk for some data due to a lack of free space in the cluster. Specifically, one or more PGs have the recovery_toofull flag set, meaning that the cluster is unable to migrate or recover data because one or more OSDs are above the full threshold. See the discussion for OSD_FULL above for steps to resolve this condition.

PG_BACKFILL_FULL

Data redundancy can be reduced or at risk for some data due to a lack of free space in the cluster. Specifically, one or more PGs have the backfill_toofull flag set, meaning that the cluster is unable to migrate or recover data because one or more OSDs are above the backfillfull threshold. See the discussion for OSD_BACKFILLFULL above for steps to resolve this condition.

PG_DAMAGED

Data scrubbing has discovered some problems with data consistency in the cluster. Specifically, one or more PGs have the inconsistent or snaptrim_error flag is set, indicating an earlier scrub operation found a problem, or that the repair flag is set and a repair for such an inconsistency is currently in progress.

OSD_SCRUB_ERRORS

Recent OSD scrubs have uncovered inconsistencies. This error is generally paired with PG_DAMAGED.

LARGE_OMAP_OBJECTS

One or more pools contain large omap objects as determined by osd_deep_scrub_large_omap_object_key_threshold (threshold for number of keys to determine a large omap object) or osd_deep_scrub_large_omap_object_value_sum_threshold (the threshold for summed size (bytes) of all key values to determine a large omap object) or both. More information on the object name, key count, and size in bytes can be found by searching the cluster log for ‘Large omap object found’. Large omap objects can be caused by RGW bucket index objects that do not have automatic resharding enabled. The thresholds can be adjusted with:

cephadm@adm > ceph config set osd osd_deep_scrub_large_omap_object_key_threshold KEYS
cephadm@adm > ceph config set osd osd_deep_scrub_large_omap_object_value_sum_threshold BYTES
CACHE_POOL_NEAR_FULL

A cache tier pool is nearly full. Full is determined by the target_max_bytes and target_max_objects properties on the cache pool. Once the pool reaches the target threshold, write requests to the pool may block while data is flushed and evicted from the cache, a state that normally leads to very high latencies and poor performance. The cache pool target size can be adjusted with:

cephadm@adm > ceph osd pool set CACHE-POOL-NAME target_max_bytes BYTES
cephadm@adm > ceph osd pool set CACHE-POOL-NAME target_max_objects OBJECTS

Normal cache flush and eviction activity can also be throttled due to reduced availability, performance of the base tier, or overall cluster load.

POOL_TOO_FEW_PGS

One or more pools should probably have more PGs, based on the amount of data that is currently stored in the pool. This can lead to sub-optimal distribution and balance of data across the OSDs in the cluster, and similarly reduce overall performance. This warning is generated if the pg_autoscale_mode property on the pool is set to warn. To disable the warning, you can disable auto-scaling of PGs for the pool entirely with:

cephadm@adm > ceph osd pool set POOL-NAME pg_autoscale_mode off

To allow the cluster to automatically adjust the number of PGs:

cephadm@adm > ceph osd pool set POOL-NAME pg_autoscale_mode on

You can also manually set the number of PGs for the pool to the recommended amount with:

cephadm@adm > ceph osd pool set POOL-NAME pg_num NEW-PG-NUM
TOO_MANY_PGS

The number of PGs in use in the cluster is above the configurable threshold of mon_max_pg_per_osd PGs per OSD. If this threshold is exceeded, the cluster does not allow new pools to be created, pool pg_num to be increased, or pool replication to be increased (any of which would lead to more PGs in the cluster). A large number of PGs can lead to higher memory utilization for OSD daemons, slower peering after cluster state changes (like OSD restarts, additions, or removals), and higher load on the Ceph Manager and Ceph Monitor daemons. The simplest way to mitigate the problem is to increase the number of OSDs in the cluster by adding more hardware. The OSD count used for the purposes of this health check is the number of in OSDs, marking out OSDs in (if there are any) can also help:

cephadm@adm > ceph osd in OSD_IDs
POOL_TOO_MANY_PGS

One or more pools require more PGs based on the amount of data that is currently stored in the pool. This can lead to higher memory utilization for OSD daemons, slower peering after cluster state changes (like OSD restarts, additions, or removals), and higher load on the manager and monitor daemons. This warning is generated if the pg_autoscale_mode property on the pool is set to warn. To disable the warning, you can disable auto-scaling of PGs for the pool entirely with:

cephadm@adm > ceph osd pool set POOL-NAME pg_autoscale_mode off

To allow the cluster to automatically adjust the number of PGs:

cephadm@adm > ceph osd pool set POOL-NAME pg_autoscale_mode on

You can also manually set the number of PGs for the pool to the recommended amount with:

cephadm@adm > ceph osd pool set POOL-NAME pg_num NEW-PG-NUM
POOL_TARGET_SIZE_RATIO_OVERCOMMITTED

One or more pools have a target_size_ratio property set to estimate the expected size of the pool as a fraction of total storage, but the value(s) exceed the total available storage (either by themselves or in combination with other pools’ actual usage). This can indicate that the target_size_ratio value for the pool is too large and should be reduced or set to zero with:

cephadm@adm > ceph osd pool set POOL-NAME target_size_ratio 0
POOL_TARGET_SIZE_BYTES_OVERCOMMITTED

One or more pools have a target_size_bytes property set to estimate the expected size of the pool, but the value(s) exceed the total available storage (either by themselves or in combination with other pools’ actual usage). This indicates that the target_size_bytes value for the pool is too large and should be reduced or set to zero with:

cephadm@adm > ceph osd pool set POOL-NAME target_size_bytes 0
TOO_FEW_OSDS

The number of OSDs in the cluster is below the configurable threshold of osd_pool_default_size.

SMALLER_PGP_NUM

One or more pools have a pgp_num value less than pg_num, indicating that the PG count was increased without also increasing the placement behavior. To adjust the placement group number, adjust pgp_num and pg_num. Ensure that changing pgp_num is performed first and does not trigger the rebalance. To resolve, set pgp_num to match pg_num and trigger the data migration with:

cephadm@adm > ceph osd pool set POOL pgp_num PG-NUM-VALUE
MANY_OBJECTS_PER_PG

One or more pools has an average number of objects per PG that is significantly higher than the overall cluster average. The specific threshold is controlled by the mon_pg_warn_max_object_skew configuration value. This indicates that the pool(s) containing most of the data in the cluster have too few PGs, or that other pools that do not contain as much data have too many PGs. The threshold can be raised to silence the health warning by adjusting the mon_pg_warn_max_object_skew configuration option on the monitors.

POOL_APP_NOT_ENABLED

A pool exists that contains one or more objects but has not been tagged for use by a particular application. Resolve this warning by labeling the pool for use by an application. For example, if the pool is used by RBD:

cephadm@adm > rbd pool init POOLNAME

If the pool is being used by a custom application FOO, you can also label via the low-level command:

cephadm@adm > ceph osd pool application enable FOO
POOL_FULL

One or more pools has reached (or is very close to reaching) its quota. The threshold to trigger this error condition is controlled by the mon_pool_quota_crit_threshold configuration option. Pool quotas can be adjusted up or down (or removed) with:

cephadm@adm > ceph osd pool set-quota POOL max_bytes BYTES
cephadm@adm > ceph osd pool set-quota POOL max_objects OBJECTS

Setting the quota value to 0 disables the quota.

POOL_NEAR_FULL

One or more pools are approaching its quota. The threshold to trigger this warning condition is controlled by the mon_pool_quota_warn_threshold configuration option. Pool quotas can be adjusted up or down (or removed) with:

cephadm@adm > ceph osd pool set-quota POOL max_bytes BYTES
cephadm@adm > ceph osd pool set-quota POOL max_objects OBJECTS
OBJECT_MISPLACED

One or more objects in the cluster is not stored on the node the cluster would like it to be stored on. This is an indication that data migration due to some recent cluster change has not yet completed. Misplaced data is not a dangerous condition in and of itself. Data consistency is not at risk and old copies of objects are not removed until the desired number of new copies (in the desired locations) are present.

OBJECT_UNFOUND

One or more objects in the cluster cannot be found. Specifically, the OSDs know that a new or updated copy of an object should exist, but a copy of that version of the object has not been found on OSDs that are currently online. Read or write requests to unfound objects will block. Ideally, a down OSD can be brought back online that has the more recent copy of the unfound object. Candidate OSDs can be identified from the peering state for the PG(s) responsible for the unfound object:

cephadm@adm > ceph tell PG_ID query

If the latest copy of the object is not available, the cluster can be told to roll back to a previous version of the object.

SLOW_OPS

One or more OSD requests is taking a long time to process. This can be an indication of extreme load, a slow storage device, or a software bug. The request queue on the OSD(s) in question can be queried with the following command, executed from the OSD host:

cephadm@adm > ceph daemon osd.ID ops

A summary of the slowest recent requests can be seen with:

cephadm@adm > ceph daemon osd.ID dump_historic_ops

The location of an OSD can be found with:

cephadm@adm > ceph osd find osd.ID
PG_NOT_SCRUBBED

One or more PGs have not been scrubbed recently. PGs are normally scrubbed every mon_scrub_interval seconds and this warning triggers when mon_warn_pg_not_deep_scrubbed_ratio percentage of interval has elapsed without a scrub since it was due. PGs do not scrub if they are not flagged as clean. This can happen if they are misplaced or degraded (see PG_AVAILABILITY and PG_DEGRADED above). You can manually initiate a scrub of a clean PG with:

cephadm@adm > ceph pg scrub PG_ID
PG_NOT_DEEP_SCRUBBED

One or more PGs have not been deep scrubbed recently. PGs are normally scrubbed every osd_deep_scrub_interval seconds and this warning triggers when mon_warn_pg_not_deep_scrubbed_ratio percentage of interval has elapsed without a scrub since it was due. PGs do not (deep) scrub if they are not flagged as clean. This can happen if they are misplaced or degraded (see PG_AVAILABILITY and PG_DEGRADED above). You can manually initiate a scrub of a clean PG with:

cephadm@adm > ceph pg deep-scrub PG_ID
MISCELLANEOUS
RECENT_CRASH

One or more Ceph daemons have crashed recently, and the crash has not yet been archived or acknowledged by the administrator. This may indicate a software bug, a hardware problem (for example, a failing disk), or some other problem.

Note
Note

Encountering a crash is not normal, but can be observed on occasion. When a crash occurs, ceph crash will alert the administrator. If ceph crash is reporting an abnormal number of crashes, contact SUSE support for further assistance. supportconfig reports from the affected nodes will help SUSE address the issue. Also consider patching the cluster at a regular interval.

New crashes can be listed with:

cephadm@adm > ceph crash ls-new

Information about a specific crash can be examined with:

cephadm@adm > ceph crash info CRASH-ID

This warning can be silenced by archiving the crash (perhaps after being examined by an administrator) so that it does not generate this warning:

cephadm@adm > ceph crash archive CRASH-ID

Similarly, all new crashes can be archived with:

cephadm@adm > ceph crash archive-all

Archived crashes are still visible via ceph crash ls but not ceph crash ls-new. The time period for what recent means is controlled by the option mgr/crash/warn_recent_interval (default: two weeks). These warnings can be disabled entirely with:

cephadm@adm > ceph config set mgr/crash/warn_recent_interval 0
TELEMETRY_CHANGED

Telemetry has been enabled but the contents of the telemetry report have changed since that time, so telemetry reports are not sent. The Ceph developers periodically revise the telemetry feature to include new and useful information, or to remove information found to be useless or sensitive. If any new information is included in the report, Ceph requires the administrator to re-enable telemetry to ensure they have an opportunity to (re)review what information is shared. To review the contents of the telemetry report:

cephadm@adm > ceph telemetry show

The telemetry report consists of several optional channels that are independently enabled or disabled. To re-enable telemetry (and make this warning go away):

cephadm@adm > ceph telemetry on

To disable telemetry (and make this warning go away):

cephadm@adm > ceph telemetry off
 groups:
  - name: cluster health
   rules:
    - alert: health error
     expr: ceph_health_status == 2
     for: 5m
     labels:
      severity: critical
      type: ses_default
     annotations:
      description: Ceph in error for > 5m
    - alert: unhealthy
     expr: ceph_health_status != 0
     for: 15m
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: Ceph not healthy for > 5m
  - name: mon
   rules:
    - alert: low monitor quorum count
     expr: ceph_monitor_quorum_count < 3
     labels:
      severity: critical
      type: ses_default
     annotations:
      description: Monitor count in quorum is low
  - name: osd
   rules:
    - alert: 10% OSDs down
     expr: sum(ceph_osd_down) / count(ceph_osd_in) >= 0.1
     labels:
      severity: critical
      type: ses_default
     annotations:
      description: More then 10% of OSDS are down
    - alert: OSD down
     expr: sum(ceph_osd_down) > 1
     for: 15m
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: One or more OSDS down for more then 15 minutes
    - alert: OSDs near full
     expr: (ceph_osd_utilization unless on(osd) ceph_osd_down) > 80
     labels:
      severity: critical
      type: ses_default
     annotations:
      description: OSD {{ $labels.osd }} is dangerously full, over 80%
    # alert on single OSDs flapping
    - alert: flap osd
     expr: rate(ceph_osd_up[5m])*60 > 1
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: >
        OSD {{ $label.osd }} was marked down at back up at least once a
        minute for 5 minutes.
    # alert on high deviation from average PG count
    - alert: high pg count deviation
     expr: abs(((ceph_osd_pgs > 0) - on (job) group_left avg(ceph_osd_pgs > 0) by (job)) / on (job) group_left avg(ceph_osd_pgs > 0) by (job)) > 0.35
     for: 5m
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: >
        OSD {{ $labels.osd }} deviates by more then 30% from
        average PG count
    # alert on high commit latency...but how high is too high
  - name: mds
   rules:
   # no mds metrics are exported yet
  - name: mgr
   rules:
   # no mgr metrics are exported yet
  - name: pgs
   rules:
    - alert: pgs inactive
     expr: ceph_total_pgs - ceph_active_pgs > 0
     for: 5m
     labels:
      severity: critical
      type: ses_default
     annotations:
      description: One or more PGs are inactive for more then 5 minutes.
    - alert: pgs unclean
     expr: ceph_total_pgs - ceph_clean_pgs > 0
     for: 15m
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: One or more PGs are not clean for more then 15 minutes.
  - name: nodes
   rules:
    - alert: root volume full
     expr: node_filesystem_avail{mountpoint="/"} / node_filesystem_size{mountpoint="/"} < 0.1
     labels:
      severity: critical
      type: ses_default
     annotations:
      description: Root volume (OSD and MON store) is dangerously full (< 10% free)
    # alert on nic packet errors and drops rates > 1 packet/s
    - alert: network packets dropped
     expr: irate(node_network_receive_drop{device!="lo"}[5m]) + irate(node_network_transmit_drop{device!="lo"}[5m]) > 1
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: >
       Node {{ $labels.instance }} experiences packet drop > 1
       packet/s on interface {{ $lables.device }}
    - alert: network packet errors
     expr: irate(node_network_receive_errs{device!="lo"}[5m]) + irate(node_network_transmit_errs{device!="lo"}[5m]) > 1
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: >
       Node {{ $labels.instance }} experiences packet errors > 1
       packet/s on interface {{ $lables.device }}
    # predict fs fillup times
    - alert: storage filling
     expr: ((node_filesystem_free - node_filesystem_size) / deriv(node_filesystem_free[2d]) <= 5) > 0
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: >
       Mountpoint {{ $lables.mountpoint }} will be full in less then 5 days
       assuming the average fillup rate of the past 48 hours.
  - name: pools
   rules:
    - alert: pool full
     expr: ceph_pool_used_bytes / ceph_pool_available_bytes > 0.9
     labels:
      severity: critical
      type: ses_default
     annotations:
      description: Pool {{ $labels.pool }} at 90% capacity or over
    - alert: pool filling up
     expr: (-ceph_pool_used_bytes / deriv(ceph_pool_available_bytes[2d]) <= 5 ) > 0
     labels:
      severity: warning
      type: ses_default
     annotations:
      description: >
       Pool {{ $labels.pool }} will be full in less then 5 days
       assuming the average fillup rate of the past 48 hours.
Print this page