Monitoring with Prometheus and Grafana

You can monitor your SUSE Manager environment using Prometheus and Grafana. SUSE Manager Server and Proxy are able to provide self-health metrics. You can also install and manage a number of Prometheus exporters on Salt clients.

Prometheus and Grafana packages are included in the SUSE Manager Client Tools for:

  • SUSE Linux Enterprise 12

  • SUSE Linux Enterprise 15

  • openSUSE Leap 15.x

You need to install Prometheus and Grafana on a machine separate from the SUSE Manager Server. We recommend to use a managed Salt SUSE client as your monitoring server. Other clients are not supported as a monitoring server.

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to monitored clients. Clients must have corresponding open ports and be reachable over the network. Alternatively, you can use reverse proxies to establish a connection.

You must have a monitoring add-on subscription for each client you want to monitor. Visit the SUSE Customer Center to manage your SUSE Manager subscriptions.

1. Prometheus and Grafana

1.1. 普罗米修斯

Prometheus is an open-source monitoring tool that is used to record real-time metrics in a time-series database. Metrics are pulled via HTTP, enabling high performance and scalability.

Prometheus metrics are time series data, or timestamped values belonging to the same group or dimension. A metric is uniquely identified by its name and set of labels.

   metric name               labels                 timestamp     value
┌────────┴───────┐  ┌───────────┴───────────┐    ┌──────┴──────┐  ┌─┴─┐
http_requests_total{status="200", method="GET"}  @1557331801.111  42236

Each application or system being monitored must expose metrics in the format above, either through code instrumentation or Prometheus exporters.

1.2. Prometheus Exporters

Exporters are libraries that help with exporting metrics from third-party systems as Prometheus metrics. Exporters are useful whenever it is not feasible to instrument a given application or system with Prometheus metrics directly. Multiple exporters can run on a monitored host to export local metrics.

The Prometheus community provides a list of official exporters, and more can be found as community contributions. For more information and an extensive list of exporters, see https://prometheus.io/docs/instrumenting/exporters/.

1.3. Grafana

Grafana is a tool for data visualization, monitoring, and analysis. It is used to create dashboards with panels representing specific metrics over a set period of time. Grafana is commonly used together with Prometheus, but also supports other data sources such as ElasticSearch, MySQL, PostgreSQL, and Influx DB. For more information about Grafana, see https://grafana.com/docs/.

2. Set up the Monitoring Server

To set up your monitoring server, you need to install Prometheus and Grafana, and configure them.

2.1. Install Prometheus

If your monitoring server is a Salt client, you can install the Prometheus package using the SUSE Manager Web UI. Otherwise you can download and install the package on your monitoring server manually. The Prometheus software is also available for SUSE Manager Proxy and SUSE Manager for Retail Branch Server.

Prometheus expects POSIX filesystem for storing data. Non-POSIX compliant filesystems are not supported. NFS filesystems are explicitly not supported.
Procedure: Installing Prometheus Using the Web UI

  1. In the SUSE Manager Web UI, open the details page of the system where Prometheus is to be installed, and navigate to the Formulas tab.

  2. Check the Prometheus checkbox to enable monitoring formulas, and click Save.

  3. Navigate to the Prometheus tab in the top menu.

  4. In the SUSE Manager Server section, enter valid SUSE Manager API credentials. Make sure that the credentials you have entered allow access to the set of systems you want to monitor.

  5. Customize any other configuration options according to your needs.

  6. Click Save Formula.

  7. Apply the highstate and confirm that it completes successfully.

  8. Check that the Prometheus interface loads correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

For more information about the monitoring formulas, see Monitoring Formula.

Procedure: Manually Installing and Configuring Prometheus

  1. On the monitoring server, install the golang-github-prometheus-prometheus package:

    zypper in golang-github-prometheus-prometheus

  2. Enable the Prometheus service:

    systemctl enable --now prometheus

  3. Check that the Prometheus interface loads correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

  4. Open the configuration file at /etc/prometheus/prometheus.yml and add this configuration information. Replace server.url with your SUSE Manager server URL and adjust username and password fields to match your SUSE Manager credentials.

    # {productname} self-health metrics
scrape_configs:
- job_name: 'mgr-server'
  static_configs:
    - targets:
      - 'server.url:9100'  # Node exporter
      - 'server.url:9187'  # PostgreSQL exporter
      - 'server.url:5556'  # JMX exporter (Tomcat)
      - 'server.url:5557'  # JMX exporter (Taskomatic)
      - 'server.url:9800'  # Taskomatic
    - targets:
      - 'server.url:80'    # Message queue
      labels:
        __metrics_path__: /rhn/metrics

# Managed systems metrics:
- job_name: 'mgr-clients'
  uyuni_sd_configs:
   - host: "http://server.url"
     username: "admin"
     password: "admin"

  5. Save the configuration file.

  6. Restart the Prometheus service:

    systemctl restart prometheus

For more information about the Prometheus configuration options, see the official Prometheus documentation at https://prometheus.io/docs/prometheus/latest/configuration/configuration/

2.2. Install Grafana

If your monitoring server is a Salt client, you can install the Grafana package using the SUSE Manager Web UI. Otherwise you can download and install the package on your monitoring server manually.

Grafana is not available on SUSE Manager Proxy.
Procedure: Installing Grafana Using the Web UI

  1. In the SUSE Manager Web UI, open the details page of the system where Grafana is to be installed, and navigate to the Formulas tab.

  2. Check the Grafana checkbox to enable monitoring formulas, and click Save.

  3. Navigate to the Grafana tab in the top menu.

  4. In the Enable and configure Grafana section, enter the admin credentials you want to use to log in Grafana.

  5. On the Datasources section, make sure that the Prometheus URL field points to the system where Prometheus is running.

  6. Customize any other configuration options according to your needs.

  7. Click Save Formula.

  8. Apply the highstate and confirm that it completes successfully.

  9. Check that the Grafana interface is loading correctly. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, http://example.com:3000).

SUSE Manager provides pre-built dashboards for server self-health, basic client monitoring, and more. You can choose which dashboards to provision in the formula configuration page.

For more information about the monitoring formulas, see Monitoring Formula.

Procedure: Manually Installing Grafana

  1. Install the grafana package:

    zypper in grafana

  2. Enable the Grafana service:

    systemctl enable --now grafana-server

  3. Check that the Grafana interface is loading correctly. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, http://example.com:3000).

monitoring grafana example

For more information on how to manually install and configure Grafana, see https://grafana.com/docs.

For more information about the monitoring formulas with forms, see Monitoring Formula.

3. Configure SUSE Manager Monitoring

With SUSE Manager 4 and higher, you can enable the server to expose Prometheus self-health metrics, and also install and configure exporters on client systems.

3.1. Server Self Monitoring

The Server self-health metrics cover hardware, operating system and SUSE Manager internals. These metrics are made available by instrumentation of the Java application, combined with Prometheus exporters.

These exporter packages are shipped with SUSE Manager Server:

These exporter packages are shipped with SUSE Manager Proxy:

The exporter packages are pre-installed in SUSE Manager Server and Proxy, but their respective systemd daemons are disabled by default.

Procedure: Enabling Self Monitoring

  1. In the SUSE Manager Web UI, navigate to Admin  Manager Configuration  Monitoring.

  2. Click Enable services.

  3. Restart Tomcat and Taskomatic.

  4. Navigate to the URL of your Prometheus server, on port 9090 (for example, http://example.com:9090)

  5. In the Prometheus UI, navigate to menu:[Status > Targets] and confirm that all the endpoints on the mgr-server group are up.

  6. If you have also installed Grafana with the Web UI, the server insights are visible on the SUSE Manager Server dashboard.

monitoring enable services

Only server self-health monitoring can be enabled using the Web UI. Metrics for a proxy are not automatically collected by Prometheus. To enable self-health monitoring on a proxy, you need to manually install exporters and enable them.

3.2. Monitoring Managed Systems

Prometheus metrics exporters can be installed and configured on Salt clients using formulas. The packages are available from the SUSE Manager client tools channels, and can be enabled and configured directly in the SUSE Manager Web UI.

These exporters can be installed on managed systems:

When you have the exporters installed and configured, you can start using Prometheus to collect metrics from monitored systems. If you have configured your monitoring server with the Web UI, metrics collection happens automatically.

Procedure: Configuring Prometheus Exporters on a Client

  1. In the SUSE Manager Web UI, open the details page of the client to be monitored, and navigate to the menu:Formulas tab.

  2. Check the Enabled checkbox on the Prometheus Exporters formula.

  3. Click Save.

  4. Navigate to the Formulas  Prometheus Exporters tab.

  5. Select the exporters you want to enable and customize arguments according to your needs. The Address field accepts either a port number preceded by a colon (:9100), or a fully resolvable address (example:9100).

  6. Click Save Formula.

  7. Apply the highstate.

monitoring configure formula

Monitoring formulas can also be configured for System Groups, by applying the same configuration used for individual systems inside the corresponding group.

For more information about the monitoring formulas, see Monitoring Formula.

4. Network Boundaries

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to monitored clients. By default, Prometheus uses these ports:

  • Node exporter: 9100

  • PostgreSQL exporter: 9187

  • Apache exporter: 9117

Additionally, if you are running the alert manager on a different host than where you run Prometheus, you also need to open port 9093.

For clients installed on cloud instances, you can add the required ports to a security group that has access to the monitoring server.

Alternatively, you can deploy a Prometheus instance in the exporters' local network, and configure federation. This allows the main monitoring server to scrape the time series from the local Prometheus instance. If you use this method, you only need to open the Prometheus API port, which is 9090.

For more information on Prometheus federation, see https://prometheus.io/docs/prometheus/latest/federation/.

You can also proxy requests through the network boundary. Tools like PushProx deploy a proxy and a client on both sides of the network barrier and allow Prometheus to work across network topologies such as NAT.

For more information on PushProx, see https://github.com/RobustPerception/PushProx.

4.1. Reverse Proxy Setup

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to each exporter on the monitored clients. To simplify your firewall configuration, you can use reverse proxy for your exporters to expose all metrics on a single port.

Procedure: Installing Prometheus Exporters with Reverse Proxy

  1. In the SUSE Manager Web UI, open the details page of the system to be monitored, and navigate to the Formulas tab.

  2. Check the Prometheus Exporters checkbox to enable the exporters formula, and click Save.

  3. Navigate to the Prometheus Exporters tab in the top menu.

  4. Check the Enable reverse proxy option, and enter a valid reverse proxy port number. For example, 9999.

  5. Customize the other exporters according to your needs.

  6. Click Save Formula.

  7. Apply the highstate and confirm that it completes successfully.

For more information about the monitoring formulas, see Monitoring Formula.

5. Security

Prometheus server and Prometheus node exporter offer a built-in mechanism to secure their endpoints with TLS encryption and authentication. SUSE Manager Web UI simplifies the configuration of all involved components. The TLS certificates have to be provided and deployed by the user. SUSE Manager offers enabling the following security model:

  • Node exporter: TLS encryption and client certificate based authentication

  • Prometheus: TLS encryption and basic authentication

For more information about configuring all available options, see Monitoring Formula.

5.1. Generating TLS certificates

By default, SUSE Manager does not provide any certificates for securing monitoring configuration. For providing security, you can generate or import custom certificates, self-signed or signed by a third party certificate authority (CA).

This section demonstrates how to generate client/server certificates for Prometheus and Node exporter minions self-signed with SUSE Manager CA.

Procedure: Creating server/client TLS certificate

  1. On the SUSE Manager Server, at the command prompt, run following command:

    rhn-ssl-tool --gen-server --dir="/root/ssl-build" --set-country="COUNTRY" \
--set-state="STATE" --set-city="CITY" --set-org="ORGANIZATION" \
--set-org-unit="ORGANIZATION UNIT" --set-email="name@example.com" \
--set-hostname="minion.example.com" --set-cname="minion.example.com" --no-rpm

    Ensure that the set-cname parameter is the fully qualified domain name (FQDN) of your Salt client. You can use the the set-cname parameter multiple times if you require multiple aliases.

  2. Copy server.crt and server.key files to the Salt minion and provide read access for prometheus user.