6 Monitoring and logging #

In this procedure, replace MNTRNODE with the host name or IP address of the server where Prometheus and Grafana are installed.

1. Install the Prometheus and Grafana packages:

   monitor # zypper in golang-github-prometheus-prometheus grafana

2. Enable and start Prometheus:

   monitor # systemctl enable --now prometheus

3. Verify that Prometheus works:

   • In a browser, navigate to MNTRNODE:9090/config, or:

   • In a terminal, run the following command:

     > wget MNTRNODE:9090/config --output-document=-

   Either of these methods should show the default contents of the /etc/prometheus/prometheus.yml file.

4. Enable and start Grafana:

   monitor # systemctl enable --now grafana-server

5. Log in to the Grafana web server at MNTRNODE:3000.

   Use admin for both the user name and password, then change the password when prompted.

6. On the left panel, select the gear icon () and click Data Sources.

7. Click Add data source.

8. Find Prometheus and click Select.

9. In the URL field, enter http://localhost:9090. The default settings for the other fields can remain unchanged.

   

   If Prometheus and Grafana are installed on different servers, replace localhost with the host name or IP address of the server where Prometheus is installed.

10. Click Save & Test.

In this procedure, replace MGMTSERVER with the host name or IP address of the server where the Slurm exporter is installed, and replace MNTRNODE with the host name or IP address of the server where Grafana is installed.

1. Install the Slurm exporter:

   management # zypper in golang-github-vpenso-prometheus_slurm_exporter

2. Enable and start the Slurm exporter:

   management # systemctl enable --now prometheus-slurm_exporter

   

   Important: Slurm exporter fails when GPU monitoring is enabled

   In Slurm 20.11, the Slurm exporter fails when GPU monitoring is enabled.

   This feature is disabled by default. Do not enable it for this version of Slurm.

3. Verify that the Slurm exporter works:

   • In a browser, navigate to MNGMTSERVER:8080/metrics, or:

   • In a terminal, run the following command:

     > wget MGMTSERVER:8080/metrics --output-document=-

   Either of these methods should show output similar to the following:

   # HELP go_gc_duration_seconds A summary of the GC invocation durations.
   # TYPE go_gc_duration_seconds summary
   go_gc_duration_seconds{quantile="0"} 1.9521e-05
   go_gc_duration_seconds{quantile="0.25"} 4.5717e-05
   go_gc_duration_seconds{quantile="0.5"} 7.8573e-05
   ...

4. On the server where Prometheus is installed, edit the scrape_configs section of the /etc/prometheus/prometheus.yml file to add a job for the Slurm exporter:

     - job_name: slurm-exporter
        scrape_interval: 30s
        scrape_timeout: 30s
        static_configs:
          - targets: ['MGMTSERVER:8080']

   Set the scrape_interval and scrape_timeout to 30s to avoid overloading the server.

5. Restart the Prometheus service:

   monitor # systemctl restart prometheus

6. Log in to the Grafana web server at MNTRNODE:3000.

7. On the left panel, select the plus icon () and click Import.

8. In the Import via grafana.com field, enter the dashboard ID 4323, then click Load.

9. From the Select a Prometheus data source drop-down box, select the Prometheus data source added in Procedure 6.1, “Installing Prometheus and Grafana”, then click Import.

10. Review the Slurm dashboard. The data might take some time to appear.

11. If you made any changes, click Save dashboard when prompted, optionally describe your changes, then click Save.

In this procedure, replace the example node names with the host names or IP addresses of the nodes, and replace MNTRNODE with the host name or IP address of the server where Grafana is installed.

1. Install the node exporter on each compute node. You can do this on multiple nodes at once by running the following command:

   management # pdsh -R ssh -u root -w "NODE1,NODE2" \
   "zypper in -y golang-github-prometheus-node_exporter"

2. Enable and start the node exporter. You can do this on multiple nodes at once by running the following command:

   management # pdsh -R ssh -u root -w "NODE1,NODE2" \
   "systemctl enable --now prometheus-node_exporter"

3. Verify that the node exporter works:

   • In a browser, navigate to NODE1:9100/metrics, or:

   • In a terminal, run the following command:

     > wget NODE1:9100/metrics --output-document=-

   Either of these methods should show output similar to the following:

   # HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
   # TYPE go_gc_duration_seconds summary
   go_gc_duration_seconds{quantile="0"} 2.3937e-05
   go_gc_duration_seconds{quantile="0.25"} 3.5456e-05
   go_gc_duration_seconds{quantile="0.5"} 8.1436e-05
   ...

4. On the server where Prometheus is installed, edit the scrape_configs section of the /etc/prometheus/prometheus.yml file to add a job for the node exporter:

     - job_name: node-exporter
       static_configs:
         - targets: ['NODE1:9100']
         - targets: ['NODE2:9100']

   Add a target for every node that has the node exporter installed.

5. Restart the Prometheus service:

   monitor # systemctl restart prometheus

6. Log in to the Grafana web server at MNTRNODE:3000.

7. On the left panel, select the plus icon () and click Import.

8. In the Import via grafana.com field, enter the dashboard ID 405, then click Load.

9. From the Select a Prometheus data source drop-down box, select the Prometheus data source added in Procedure 6.1, “Installing Prometheus and Grafana”, then click Import.

10. Review the node dashboard. Click the node drop-down box to select the nodes you want to view. The data might take some time to appear.

11. If you made any changes, click Save dashboard when prompted. To keep the currently selected nodes next time you access the dashboard, activate Save current variable values as dashboard default. Optionally describe your changes, then click Save.