Monitoring with Prometheus and Grafana

You can monitor your SUSE Manager environment using Prometheus and Grafana. SUSE Manager Server and Proxy are able to provide self-health metrics. You can also install and manage a number of Prometheus exporters on Salt clients.

Prometheus and Grafana packages are included in the SUSE Manager Client Tools for:

  • SUSE Linux Enterprise 12

  • SUSE Linux Enterprise 15

  • openSUSE Leap 15.x

You need to install Prometheus and Grafana on a machine separate from the SUSE Manager Server. We recommend to use a managed Salt SUSE client as your monitoring server. Other clients are not supported as a monitoring server.

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to monitored clients. Clients must have corresponding open ports and be reachable over the network. Alternatively, you can use reverse proxies to establish a connection.

You must have a monitoring add-on subscription for each client you want to monitor. Visit the SUSE Customer Center to manage your SUSE Manager subscriptions.

1. Prometheus and Grafana

1.1. Prometheus

Prometheus is an open-source monitoring tool that is used to record real-time metrics in a time-series database. Metrics are pulled via HTTP, enabling high performance and scalability.

Prometheus metrics are time series data, or timestamped values belonging to the same group or dimension. A metric is uniquely identified by its name and set of labels.

   metric name               labels                 timestamp     value
┌────────┴───────┐  ┌───────────┴───────────┐    ┌──────┴──────┐  ┌─┴─┐
http_requests_total{status="200", method="GET"}  @1557331801.111  42236

Each application or system being monitored must expose metrics in the format above, either through code instrumentation or Prometheus exporters.

1.2. Prometheus Exporters

Exporters are libraries that help with exporting metrics from third-party systems as Prometheus metrics. Exporters are useful whenever it is not feasible to instrument a given application or system with Prometheus metrics directly. Multiple exporters can run on a monitored host to export local metrics.

The Prometheus community provides a list of official exporters, and more can be found as community contributions. For more information and an extensive list of exporters, see https://prometheus.io/docs/instrumenting/exporters/.

1.3. Grafana

Grafana is a tool for data visualization, monitoring, and analysis. It is used to create dashboards with panels representing specific metrics over a set period of time. Grafana is commonly used together with Prometheus, but also supports other data sources such as ElasticSearch, MySQL, PostgreSQL, and Influx DB. For more information about Grafana, see https://grafana.com/docs/.

2. Set up the Monitoring Server

To set up your monitoring server, you need to install Prometheus and Grafana, and configure them.

2.1. Install Prometheus

If your monitoring server is a Salt client, you can install the Prometheus package using the SUSE Manager Web UI. Otherwise you can download and install the package on your monitoring server manually. The Prometheus software is also available for SUSE Manager Proxy and SUSE Manager for Retail Branch Server.

  • To access a shell inside the Server container run mgrctl term on the container host.

  • To copy files from inside the container use mgrctl cp.

Prometheus expects POSIX filesystem for storing data. Non-POSIX compliant filesystems are not supported. NFS filesystems are not supported.

Procedure: Installing Prometheus Using the Web UI
  1. In the SUSE Manager Web UI, open the details page of the system where Prometheus is to be installed, and navigate to the Formulas tab.

  2. Check the Prometheus checkbox to enable monitoring formulas, and click Save.

  3. Navigate to the Prometheus tab in the top menu.

  4. In the SUSE Manager Server section, enter valid SUSE Manager API credentials. Make sure that the credentials you have entered allow access to the set of systems you want to monitor.

  5. Customize any other configuration options according to your needs.

  6. Click Save Formula.

  7. Apply the highstate and confirm that it completes successfully.

  8. Check that the Prometheus interface loads correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

For more information about the monitoring formulas, see Monitoring Formula.

Procedure: Manually Installing and Configuring Prometheus
  1. On the monitoring server, install the golang-github-prometheus-prometheus package:

    zypper in golang-github-prometheus-prometheus
  2. Enable the Prometheus service:

    systemctl enable --now prometheus
  3. Check that the Prometheus interface loads correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

  4. Open the configuration file at /etc/prometheus/prometheus.yml and add this configuration information. Replace server.url with your SUSE Manager server URL and adjust username and password fields to match your SUSE Manager credentials.

    # {productname} self-health metrics
    scrape_configs:
    - job_name: 'mgr-server'
      static_configs:
        - targets:
          - 'server.url:9100'  # Node exporter
          - 'server.url:9187'  # PostgreSQL exporter
          - 'server.url:5556'  # JMX exporter (Tomcat)
          - 'server.url:5557'  # JMX exporter (Taskomatic)
          - 'server.url:9800'  # Taskomatic
        - targets:
          - 'server.url:80'    # Message queue
          labels:
            __metrics_path__: /rhn/metrics
    
    # Managed systems metrics:
    - job_name: 'mgr-clients'
      uyuni_sd_configs:
       - server: "http://server.url"
         username: "admin"
         password: "admin"
      relabel_configs:
       - source_labels: [__meta_uyuni_exporter]
         target_label: exporter
       - source_labels: [__address__]
         replacement: "No group"
         target_label: groups
       - source_labels: [__meta_uyuni_groups]
         regex: (.+)
         target_label: groups
       - source_labels: [__meta_uyuni_minion_hostname]
         target_label: hostname
       - source_labels: [__meta_uyuni_primary_fqdn]
         regex: (.+)
         target_label: hostname
       - source_labels: [hostname, __address__]
         regex: (.*);.*:(.*)
         replacement: ${1}:${2}
         target_label: __address__
       - source_labels: [__meta_uyuni_metrics_path]
         regex: (.+)
         target_label: __metrics_path__
       - source_labels: [__meta_uyuni_proxy_module]
         target_label: __param_module
       - source_labels: [__meta_uyuni_scheme]
         target_label: __scheme__
  5. Save the configuration file.

  6. Restart the Prometheus service:

    systemctl restart prometheus

For more information about the Prometheus configuration options, see the official Prometheus documentation at https://prometheus.io/docs/prometheus/latest/configuration/configuration/.

2.2. Install Grafana

If your monitoring server is a client managed by SUSE Manager, you can install the Grafana package using the SUSE Manager Web UI. Otherwise you can download and install the package on your monitoring server manually.

Grafana is not available on SUSE Manager Proxy.

Procedure: Installing Grafana Using the Web UI
  1. In the SUSE Manager Web UI, open the details page of the system where Grafana is to be installed, and navigate to the Formulas tab.

  2. Check the Grafana checkbox to enable monitoring formulas, and click Save.

  3. Navigate to the Grafana tab in the top menu.

  4. In the Enable and configure Grafana section, enter the admin credentials you want to use to log in Grafana.

  5. On the Datasources section, make sure that the Prometheus URL field points to the system where Prometheus is running.

  6. Customize any other configuration options according to your needs.

  7. Click Save Formula.

  8. Apply the highstate and confirm that it completes successfully.

  9. Check that the Grafana interface is loading correctly. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, http://example.com:3000).

SUSE Manager provides pre-built dashboards for server self-health, basic client monitoring, and more. You can choose which dashboards to provision in the formula configuration page.

Procedure: Manually Installing Grafana
  1. Install the grafana package:

    zypper in grafana
  2. Enable the Grafana service:

    systemctl enable --now grafana-server
  3. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, http://example.com:3000).

  4. On the login page, enter admin for username and password.

  5. Click Log in. If login is successful, then you will see a prompt to change the password.

  6. Click OK on the prompt, then change your password.

  7. Move your cursor to the cog icon on the side menu which will show the configuration options.

  8. Click Data sources.

  9. Click Add data source to see a list of all supported data sources.

  10. Choose the Prometheus data source.

  11. Make sure to specify the correct URL of the Prometheus server.

  12. Click Save & test.

  13. To import a dashboard click the + icon in the side menu, and then click Import.

  14. For SUSE Manager server overview load the dashboard ID: 17569.

  15. For SUSE Manager clients overview load the dashboard ID: 17570.

    monitoring grafana example

3. Configure SUSE Manager Monitoring

With SUSE Manager 4 and higher, you can enable the server to expose Prometheus self-health metrics, and also install and configure exporters on client systems.

3.1. Server Self Monitoring

The Server self-health metrics cover hardware, operating system and SUSE Manager internals. These metrics are made available by instrumentation of the Java application, combined with Prometheus exporters.

These exporters are shipped with SUSE Manager Server:

These exporter packages are shipped with SUSE Manager Proxy:

The exporters are pre-installed in SUSE Manager Server and Proxy, but their respective systemd daemons are disabled by default.

Procedure: Enabling Self Monitoring
  1. In the SUSE Manager Web UI, navigate to Admin  Manager Configuration  Monitoring.

  2. Click Enable services.

  3. Restart Tomcat and Taskomatic.

  4. Navigate to the URL of your Prometheus server, on port 9090 (for example, http://example.com:9090)

  5. In the Prometheus UI, navigate to Status  Targets and confirm that all the endpoints on the mgr-server group are up.

  6. If you have also installed Grafana with the Web UI, the server insights are visible on the SUSE Manager Server dashboard, in the Admin  Manager Configuration  Monitoring.

Only server self-health monitoring can be enabled using the Web UI. Metrics for a proxy are not automatically collected by Prometheus. To enable self-health monitoring on a proxy, you need to manually install exporters and enable them.

The following relevant metrics are collected on the SUSE Manager Server.

Table 1. Server statistics (port 80)
Metric Type Description

uyuni_all_systems

gauge

Number of all systems

uyuni_virtual_systems

gauge

Number of virtual systems

uyuni_inactive_systems

gauge

Number of inactive systems

uyuni_outdated_systems

gauge

Number of systems with outdated packages

Table 2. PostgreSQL exporter (port 9187)
Metric Type Description

pg_stat_database_tup_fetched

counter

Number of rows fetched by queries

pg_stat_database_tup_inserted

counter

Number of rows inserted by queries

pg_stat_database_tup_updated

counter

Number of rows updated by queries

pg_stat_database_tup_deleted

counter

Number of rows deleted by queries

mgr_serveractions_completed

gauge

Number of completed actions

mgr_serveractions_failed

gauge

Number of failed actions

mgr_serveractions_picked_up

gauge

Number of picked-up actions

mgr_serveractions_queued

gauge

Number of queued actions

Table 3. JMX exporter (Tomcat port 5556, Taskomatic port 5557)
Metric Type Description

java_lang_Threading_ThreadCount

gauge

Number of active threads

java_lang_Memory_HeapMemoryUsage_used

gauge

Current heap memory usage

Table 4. Server Message Queue (port 80)
Metric Type Description

message_queue_thread_pool_threads

counter

Number of message queue threads ever created

message_queue_thread_pool_threads_active

gauge

Number of currently active message queue threads

message_queue_thread_pool_task_count

counter

Number of tasks ever submitted

message_queue_thread_pool_completed_task_count

counter

Number of tasks ever completed

Table 5. Salt Queue (port 80)
Metric Type Description

salt_queue_thread_pool_size

gauge

Number of threads created per Salt queue

salt_queue_thread_pool_active_threads

gauge

Number of currently active Salt threads per queue

salt_queue_thread_pool_task_total

counter

Number of tasks ever submitted per queue

salt_queue_thread_pool_completed_task_total

counter

Number of tasks ever completed per queue

Every salt_queue value has a label named queue with the queue number as value.

Table 6. Taskomatic Scheduler (port 9800)
Metric Type Description

taskomatic_scheduler_threads

counter

Number of scheduler threads ever created

taskomatic_scheduler_threads_active

gauge

Number of currently active scheduler threads

taskomatic_scheduler_completed_task_count

counter

Number of tasks ever completed

3.2. Monitoring Managed Systems

Prometheus metrics exporters can be installed and configured on Salt clients using formulas. The packages are available from the SUSE Manager client tools channels, and can be enabled and configured directly in the SUSE Manager Web UI.

These exporters can be installed on managed systems:

On SLE Micro, only the Node exporter and the Blackbox exporter are available.

When you have the exporters installed and configured, you can start using Prometheus to collect metrics from the monitored systems. If you have configured your monitoring server with the Web UI, metrics collection happens automatically.

Procedure: Configuring Prometheus Exporters on a Client
  1. In the SUSE Manager Web UI, open the details page of the client to be monitored, and navigate to the Formulas tab.

  2. Check the Enabled checkbox on the Prometheus Exporters formula.

  3. Click Save.

  4. Navigate to the Formulas  Prometheus Exporters tab.

  5. Select the exporters you want to enable and customize arguments according to your needs. The Address field accepts either a port number preceded by a colon (:9100), or a fully resolvable address (example:9100).

  6. Click Save Formula.

  7. Apply the highstate.

Monitoring formulas can also be configured for System Groups, by applying the same configuration used for individual systems inside the corresponding group.

For more information about the monitoring formulas, see Monitoring Formula.

3.3. Change Grafana Password

To change the Grafana password follow the steps described in the Grafana documentation:

In case you have lost the Grafana administrator password you can reset it as root with the following command:

grafana-cli --configOverrides cfg:default.paths.data=/var/lib/grafana --homepath /usr/share/grafana admin reset-admin-password <new_password>

4. Network Boundaries

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to monitored clients. By default, Prometheus uses these ports:

  • Node exporter: 9100

  • PostgreSQL exporter: 9187

  • Apache exporter: 9117

Additionally, if you are running the alert manager on a different host than where you run Prometheus, you also need to open port 9093. The alert manager is part of Prometheus solution. It handles alerts sent by client applications such as the Prometheus server instance. For more information about the alert manager, see https://prometheus.io/docs/alerting/latest/alertmanager/.

For clients installed on cloud instances, you can add the required ports to a security group that has access to the monitoring server.

Alternatively, you can deploy a Prometheus instance in the exporters' local network, and configure federation. This allows the main monitoring server to scrape the time series from the local Prometheus instance. If you use this method, you only need to open the Prometheus API port, which is 9090.

For more information on Prometheus federation, see https://prometheus.io/docs/prometheus/latest/federation/.

You can also proxy requests through the network boundary. Tools like PushProx deploy a proxy and a client on both sides of the network barrier and allow Prometheus to work across network topologies such as NAT.

For more information on PushProx, see https://github.com/RobustPerception/PushProx.

4.1. Reverse Proxy Setup

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to each exporter on the monitored clients. To simplify your firewall configuration, you can use reverse proxy for your exporters to expose all metrics on a single port.

Procedure: Installing Prometheus Exporters with Reverse Proxy
  1. In the SUSE Manager Web UI, open the details page of the system to be monitored, and navigate to the Formulas tab.

  2. Check the Prometheus Exporters checkbox to enable the exporters formula, and click Save.

  3. Navigate to the Prometheus Exporters tab in the top menu.

  4. Check the Enable reverse proxy option, and enter a valid reverse proxy port number. For example, 9999.

  5. Customize the other exporters according to your needs.

  6. Click Save Formula.

  7. Apply the highstate and confirm that it completes successfully.

For more information about the monitoring formulas, see Monitoring Formula.

5. Security

Prometheus server and Prometheus node exporter offer a built-in mechanism to secure their endpoints with TLS encryption and authentication. SUSE Manager Web UI simplifies the configuration of all involved components. The TLS certificates have to be provided and deployed by the user. SUSE Manager offers enabling the following security model:

  • Node exporter: TLS encryption and client certificate based authentication

  • Prometheus: TLS encryption and basic authentication

For more information about configuring all available options, see Monitoring Formula.

5.1. Generating TLS certificates

By default, SUSE Manager does not provide any certificates for securing monitoring configuration. For providing security, you can generate or import custom certificates, self-signed or signed by a third party certificate authority (CA).

This section demonstrates how to generate client/server certificates for Prometheus and Node exporter minions self-signed with SUSE Manager CA.

Procedure: Creating server/client TLS certificate
  1. On the SUSE Manager Server, at the command prompt, run following command:

    rhn-ssl-tool --gen-server --dir="/root/ssl-build" --set-country="COUNTRY" \
    --set-state="STATE" --set-city="CITY" --set-org="ORGANIZATION" \
    --set-org-unit="ORGANIZATION UNIT" --set-email="name@example.com" \
    --set-hostname="minion.example.com" --set-cname="minion.example.com" --no-rpm

    Ensure that the set-cname parameter is the fully qualified domain name (FQDN) of your Salt client. You can use the the set-cname parameter multiple times if you require multiple aliases.

  2. Copy server.crt and server.key files to the Salt minion and provide read access for prometheus user.