16 Monitoring and alerting #

Tip

This section describes how to change the configuration of container images which are used when services are deployed or updated. It does not include the commands necessary to deploy or re-deploy services.

The recommended method to deploy the monitoring stack is by applying its specification as described in Section 8.3.8, “Deploying the monitoring stack”.

To deploy custom or local container images, the images need to be set in cephadm. To do so, you will need to run the following command:

cephuser@adm > ceph config set mgr mgr/cephadm/OPTION_NAME VALUE

Where OPTION_NAME is any of the following names:

If no option is set or if the setting has been removed, the following images are used as VALUE:

For example:

cephuser@adm > ceph config set mgr mgr/cephadm/container_image_prometheus prom/prometheus:v1.4.1

Note

By setting a custom image, the default value will be overridden (but not overwritten). The default value changes when updates become available. By setting a custom image, you will not be able to update the component you have set the custom image for automatically. You will need to manually update the configuration (image name and tag) to be able to install updates.

If you choose to go with the recommendations instead, you can reset the custom image you have set before. After that, the default value will be used again. Use ceph config rm to reset the configuration option:

cephuser@adm > ceph config rm mgr mgr/cephadm/OPTION_NAME

For example:

cephuser@adm > ceph config rm mgr mgr/cephadm/container_image_prometheus

As mentioned in Section 16.1, “Configuring custom or local images”, cephadm is shipped with the URLs of the recommended and tested container images, and they are used by default.

By updating the Ceph packages, new versions of these URLs may be shipped. This just updates where the container images are pulled from but does not update any services.

After the URLs to the new container images have been updated, either manually as described in Section 16.1, “Configuring custom or local images”, or automatically through an update of the Ceph package, the monitoring services can be updated.

To do so, use ceph orch reconfig like so:

cephuser@adm > ceph orch reconfig node-exporter
cephuser@adm > ceph orch reconfig prometheus
cephuser@adm > ceph orch reconfig alertmanager
cephuser@adm > ceph orch reconfig grafana

Currently no single command to update all monitoring services exists. The order in which these services are updated is not important.

Note

If you use custom container images, the URLs specified for the monitoring services will not change automatically if the Ceph packages are updated. If you have specified custom container images, you will need to specify the URLs of the new container images manually. This may be the case if you use a local container registry.

You can find the URLs of the recommended container images to be used in the Section 16.1, “Configuring custom or local images” section.

To disable the monitoring stack, run the following commands:

cephuser@adm > ceph orch rm grafana
cephuser@adm > ceph orch rm prometheus --force   # this will delete metrics data collected so far
cephuser@adm > ceph orch rm node-exporter
cephuser@adm > ceph orch rm alertmanager
cephuser@adm > ceph mgr module disable prometheus

The Ceph Dashboard back-end requires the Grafana URL to be able to verify the existence of Grafana Dashboards before the front-end even loads them. Because of the nature of how Grafana is implemented in Ceph Dashboard, this means that two working connections are required in order to be able to see Grafana graphs in Ceph Dashboard:

Now, it might be the case that your environment makes it difficult for the user's browser to directly access the URL configured in Ceph Dashboard. To solve this issue, a separate URL can be configured which will solely be used to tell the front-end (the user's browser) which URL it should use to access Grafana.

To change the URL that is returned to the front-end issue the following command:

cephuser@adm > ceph dashboard set-grafana-frontend-api-url GRAFANA-SERVER-URL

If no value is set for that option, it will simply fall back to the value of the GRAFANA_API_URL option, which is set automatically and periodically updated by cephadm. If set, it will instruct the browser to use this URL to access Grafana.

The Prometheus Manager Module is a module inside Ceph that extends Ceph's functionality. The module reads (meta-)data from Ceph about its state and health, providing the (scraped) data in a consumable format to Prometheus.

Note

The Prometheus Manager Module needs to be restarted for the configuration changes to be applied.

cephuser@adm > ceph config set mgr mgr/prometheus/server_addr 0.0.0.0

cephuser@adm > ceph config set mgr mgr/prometheus/server_port 9283

16.5.2 Configuring scrape_interval #

  

By default, the Prometheus Manager Module is configured with a scrape interval of 15 seconds. We do not recommend using a scrape interval below 10 seconds. To set a different scrape interval in the Prometheus module, set scrape_interval to the desired value:

Important

To work properly and not cause any issues, the scrape_interval of this module should always be set to match the Prometheus scrape interval .

cephuser@adm > ceph config set mgr mgr/prometheus/scrape_interval 15

cephuser@adm > ceph config set mgr mgr/prometheus/stale_cache_strategy return

cephuser@adm > ceph config set mgr mgr/prometheus/stale_cache_strategy fail

cephuser@adm > ceph config set mgr mgr/prometheus/rbd_stats_pools "pool1,pool2,poolN"

cephuser@adm > ceph config set mgr mgr/prometheus/rbd_stats_pools_refresh_interval 600

Prometheus' security model presumes that untrusted users have access to the Prometheus HTTP endpoint and logs. Untrusted users have access to all the (meta-)data Prometheus collects that is contained in the database, plus a variety of operational and debugging information.

However, Prometheus' HTTP API is limited to read-only operations. Configurations cannot be changed using the API, and secrets are not exposed. Moreover, Prometheus has some built-in measures to mitigate the impact of denial-of-service attacks.

If you want to get notified about Prometheus alerts via SNMP traps, then you can install the Prometheus Alertmanager SNMP webhook via cephadm. To do so, you need to create a service and placement specification file with the following content:

Note

For more information on service and placement files, see Section 8.2, “Service and placement specification”.

service_type: container
service_id: prometheus-webhook-snmp
placement:
    ADD_PLACEMENT_HERE
image: registry.suse.com/ses/7/prometheus-webhook-snmp:latest
args:
    - "--publish 9099:9099"
envs:
    - ARGS="--debug --snmp-host=ADD_HOST_GATEWAY_HERE"
    - RUN_ARGS="--metrics"
EOF

Use this service specification to get the service running using its default settings.

You need to publish the port the Prometheus receiver is listening on by using the command line argument --publish HOST_PORT:CONTAINER_PORT when running the service, because the port is not exposed automatically by the container. This can be done by adding the following lines to the specification:

args:
    - "--publish 9099:9099"

Alternatively, connect the container to the host network by using the command line argument --network=host.

args:
    - "--network=host"

If the SNMP trap receiver is not installed on the same host as the container, then you must also specify the FQDN of the SNMP host. Use the container's network gateway to be able to receive SNMP traps outside the container/host:

envs:
    - ARGS="--debug --snmp-host=CONTAINER_GATEWAY"

envs:
    - ARGS="--debug --snmp-host=CONTAINER_GATEWAY"
    - RUN_ARGS="--metrics --port=9101"

files:
    etc/prometheus-webhook-snmp.conf:
        - "debug: True"
        - "snmp_host: ADD_HOST_GATEWAY_HERE"
        - "metrics: True"
volume_mounts:
    etc/prometheus-webhook-snmp.conf: /etc/prometheus-webhook-snmp.conf

cephuser@adm > ceph orch apply -i SERVICE_SPEC_FILE

16.7.2 Configuring the Prometheus Alertmanager for SNMP #

  

Finally, the Prometheus Alertmanager needs to be configured specifically for SNMP traps. If this service has not been deployed already, create a service specification file. You need to replace IP_OR_FQDN with the IP address or FQDN of the host where the Prometheus Alertmanager SNMP webhook has been installed. For example:

Note

If you have already deployed this service, then to ensure the Alertmanager is set up correctly for SNMP, re-deploy with the following settings.

  service_type: alertmanager
  placement:
    hosts:
    - HOSTNAME
  user_data:
    default_webhook_urls:
    - 'http://IP_OR_FQDN:9099/'

Apply the service specification with the following command:

cephuser@adm > ceph orch apply -i SERVICE_SPEC_FILE