4 SUSE OpenStack Cloud Monitoring #

For accessing SUSE OpenStack Cloud Crowbar Monitoring and performing monitoring tasks, you must have access to the OpenStack platform as a user with the monasca-user or monasca-read-only-user role in the monasca tenant.

Log in to OpenStack horizon with your user name and password. The functions you can use in OpenStack horizon depend on your access permissions. To access logs and metrics, switch to the monasca tenant in horizon. This allows you to access all monitoring data for SUSE OpenStack Cloud Crowbar Monitoring.

Figure 4.1: SUSE OpenStack Cloud horizon Dashboard—Monitoring #

 

SUSE OpenStack Cloud Crowbar Monitoring provides one convenient access point to your monitoring data. Use Monitoring > Overview to keep track of your services and servers and quickly check their status. The overview also indicates any irregularities in the log data of the system components you are monitoring.

On the Overview page, you can:

A Metrics Agent can be installed and configured on each physical and virtual server where cloud resources are to be monitored. The agent is responsible for querying metrics and sending the data to the Monitoring Service for further processing.

Metrics are self-describing data structures that are uniquely identified by a name and a set of dimensions. Each dimension consists of a key/value pair that allows for a flexible and concise description of the data to be monitored, for example, region, availability zone, service tier, or resource ID.

The Metrics Agent supports various types of metrics including the following:

All SUSE OpenStack Cloud Crowbar Monitoring user groups work with a graphical user interface that is seamlessly integrated into their cloud infrastructure. Based on OpenStack horizon, the user interface enables access to all monitoring functionality and the resulting large-scale monitoring data.

A comfortable dashboard visualizes the health and status of the cloud resources. It allows SUSE OpenStack Cloud Crowbar Monitoring users to experiment with many ways of analyzing the performance of their cloud resources in real-time. They cannot only view but also share and explore visualizations of their monitoring data.

SUSE OpenStack Cloud Crowbar Monitoring supports GUI-based alarm and notification management. Template-based alarm definitions allow for monitoring a dynamically changing set of resources without the need for reconfiguration. While the number of underlying virtual machines is changing, for example, this ensures the efficient monitoring of scalable cloud services. Alarm definitions allow you to specify expressions that are evaluated based on the metrics data that is received. Alarm definitions can be combined to form compound alarms. Compound alarms allow you to track and process even more complex events. Notifications can be configured in order to inform SUSE OpenStack Cloud Crowbar Monitoring users when an alarm is triggered.

SUSE OpenStack Cloud Crowbar Monitoring is a highly scalable and fault tolerant monitoring solution for OpenStack-based cloud infrastructures.

The system operator of the cloud infrastructure and the service providers do not have to care for system monitoring software any longer. They use SUSE OpenStack Cloud Crowbar Monitoring to check whether their services and servers are working appropriately.

SUSE OpenStack Cloud Crowbar Monitoring provides comprehensive and configurable metrics with reasonable defaults for monitoring the status, capacity, throughput, and latency of cloud systems. SUSE OpenStack Cloud Crowbar Monitoring users can set their own warnings and critical thresholds and can combine multiple warnings and thresholds to support the processing of complex events. Combined with a notification system, these alerting features enable them to quickly analyze and resolve problems in the cloud infrastructure.

The Metrics agent is responsible for querying metrics and sending them to the Monitoring Service for further processing.

Metrics are self-describing data structures that are uniquely identified by a name and a set of dimensions. Each dimension consists of a key/value pair that allows for a flexible and concise description of the data to be monitored, for example, region, availability zone, service tier, or resource ID.

The Metrics Agent supports various types of metrics including the following:

Your individual agent configuration determines which metrics are available for monitoring your services and servers. For details on installing and configuring a Metrics Agent, see Deployment Guide using Crowbar.

As soon as an agent is available, you have access to the SUSE OpenStack Cloud Crowbar Monitoring monitoring features. You work with a graphical user interface that is seamlessly integrated into your cloud infrastructure. Based on OpenStack horizon, the user interface enables access to all monitoring functionality and the resulting large-scale monitoring data. A comfortable dashboard visualizes the health and status of your cloud resources.

SUSE OpenStack Cloud Crowbar Monitoring provides functions for alarm and notification management. Template-based alarm definitions allow for monitoring a dynamically changing set of resources without the need for reconfiguration. While the number of underlying virtual machines is changing, for example, this ensures the efficient monitoring of scalable cloud services. Alarm definitions allow you to specify expressions that are evaluated based on the metrics data that is received. Alarm definitions can be combined to form compound alarms. Compound alarms allow you to track and process even more complex events. Notifications can be configured in order to inform SUSE OpenStack Cloud Crowbar Monitoring users when an alarm is triggered.

With the increasing complexity of cloud infrastructures, it is becoming more and more difficult and time-consuming for the system operator to gather, store, and query the large amounts of log data manually. To cope with these problems, SUSE OpenStack Cloud Crowbar Monitoring provides centralized log management features.

SUSE OpenStack Cloud Crowbar Monitoring stores the log data in a central database. This forms the basis for visualizing the log data for the SUSE OpenStack Cloud Crowbar Monitoring users. Advanced data analysis and visualization of the log data is supported in a variety of charts, tables, and maps. Visualizations can easily be combined in dynamic dashboards that display changes to search queries in real time.

The log data from a large number of sources can be accessed from a single dashboard. Integrated search, filter, and graphics options enable system operators to isolate problems and narrow down potential root causes. SUSE OpenStack Cloud Crowbar Monitoring thus provides valuable insights into the log data, even with large amounts of data resulting from highly complex environments.

Based on OpenStack horizon, the customizable dashboards are seamlessly integrated into your cloud infrastructure. They enable user access to all log management functionality.

GUI-based alarm and notification management is also supported for log data. Based on a template mechanism, you can configure alarms and notifications to monitor the number of critical log events over time. Compound alarms can be created to analyze more complex log events. This automation of log handling guarantees that you can identify problems in your their infrastructure early and find the root cause quickly.

SUSE OpenStack Cloud Crowbar Monitoring is integrated with OpenStack core services. These include:

The Monitoring Service is the central SUSE OpenStack Cloud Crowbar Monitoring component. It is responsible for receiving, persisting, and processing metrics and log data, as well as providing the data to the users.

The Monitoring Service relies on monasca. It uses monasca for high-speed metrics querying and integrates the streaming alarm engine and the notification engine of monasca. For details, refer to the monasca Wiki.

SUSE OpenStack Cloud Crowbar Monitoring comes with a plugin for the OpenStack horizon dashboard. The horizon plugin extends the main dashboard in OpenStack with a view for monitoring. This enables SUSE OpenStack Cloud Crowbar Monitoring users to access the monitoring and log management functions from a central Web-based graphical user interface. Metrics and log data are visualized on a comfortable and easy-to-use dashboard.

For details, refer to the OpenStack horizon documentation.

Based on OpenStack horizon, the monitoring data is visualized on a comfortable and easy-to-use dashboard which fully integrates with the following applications:

A Metrics Agent is required for retrieving metrics data from the host on which it runs and sending the metrics data to the Monitoring Service. The agent supports metrics from a variety of sources as well as a number of built-in system and service checks.

A Metrics Agent can be installed on each virtual or physical server to be monitored.

The agent functionality is fully integrated into the source code base of the monasca project. For details, refer to the monasca Wiki.

A Log Agent is needed for collecting log data from the host on which it runs and forwarding the log data to the Monitoring Service for further processing. It can be installed on each virtual or physical server from which log data is to be retrieved.

The agent functionality is fully integrated into the source code base of the monasca project. For details, refer to the monasca Wiki.

CMM is fully integrated with keystone, the identity service which serves as the common authentication and authorization system in OpenStack.

The CMM integration with keystone requires any CMM user to be registered as an OpenStack user. All authentication and authorization in CMM is done through keystone. If a user requests monitoring data, for example, CMM verifies that the user is a valid user in OpenStack and allowed to access the requested metrics.

CMM users are created and administrated in OpenStack:

For details on user management, refer to the OpenStack documentation.

Metrics data is stored in the Metrics and Alarms InfluxDB Database. InfluxDB features an SQL-like query language for querying data and performing aggregations on that data.

The Metrics Agent configuration defines the metrics and types of measurement for which data is stored. For each measurement, a so-called series is written to the InfluxDB database. A series consists of a timestamp, the metrics, and the value measured.

Every series can be assigned key tags. In the case of SUSE OpenStack Cloud Crowbar Monitoring, this is the _tenant_id tag. This tag identifies the OpenStack project for which the metrics data has been collected.

From time to time, you may want to delete outdated or unnecessary metrics data from the Metrics and Alarms Database, for example, to save space or remove data for metrics you are no longer interested in. To delete data, you use the InfluxDB command line interface, the interactive shell that is provided for the InfluxDB database.

Proceed as follows to delete metrics data from the database:

Log data is stored in the Elasticsearch database. Elasticsearch stores the data in indices. One index per day is created for every OpenStack project.

By default, the indices are stored in the following directory on the host where the Monitoring Service is installed:

/var/data/elasticsearch/<cluster-name>/nodes/<node-name>

Example:

/var/data/elasticsearch/elasticsearch/nodes/0

Note

If your system uses a different directory, look up the path.data parameter in the Elasticsearch configuration file, /etc/elasticsearch/elasticsearch.yml.

If you want to delete outdated or unnecessary log data from the Elasticsearch database, proceed as follows:

Note

Be aware that the -XDELETE command immediately deletes the index file!

Both, for -XHEAD and -XDELETE, you can use wildcards for processing several indices. For example, you can delete all indices of a specific project for the whole month of July, 2015:

curl -XDELETE -i 'http://localhost:9200/abc123-2015-07-*'

Note

Take extreme care when using wildcards for the deletion of indices. You could delete all existing indices with one single command!

In case of trouble with the SUSE OpenStack Cloud Crowbar Monitoring services, you can study their log files to find the reason. The log files are also useful if you need to contact your support organization. For storing the log files, the default installation uses the /var/log directory on the hosts where the agents or services are installed.

You can use systemd, a system and session manager for LINUX, and journald, a LINUX logging interface, for addressing dispersed log files.

The SUSE OpenStack Cloud Crowbar Monitoring installer automatically puts all SUSE OpenStack Cloud Crowbar Monitoring services under the control of systemd. journald provides a centralized management solution for the logging of all processes that are controlled by systemd. The logs are collected and managed in a so-called journal controlled by the journald daemon.

For details on the systemd and journald utilities, refer to the https://documentation.suse.com/sles/15-SP1/single-html/SLES-admin/#part-system.

Typical tasks of the Monitoring Service operator are to make regular backups, particularly of the data created during operation.

At regular intervals, you should make a backup of all:

SUSE OpenStack Cloud Crowbar Monitoring does not offer integrated backup and recovery mechanisms. Instead, use the mechanisms and procedures of the individual components.

systemctl stop apache2

 kafka-consumer-groups.sh --zookeeper 192.168.56.81:2181 --list
 kafka-consumer-groups.sh --zookeeper 192.168.56.81:2181 --describe \
  --group 1_metrics | column -t -s ','
 kafka-consumer-groups.sh --zookeeper 192.168.56.81:2181 --describe \
  --group transformer-logstash-consumer | column -t -s ','
 kafka-consumer-groups.sh --zookeeper 192.168.56.81:2181 --describe \
  --group thresh-metric | column -t -s ','

systemctl start apache2

mysqldump -u root -p mon > dumpfile.sql

/etc/influxdb/influxdb.conf
 /etc/kafka/server.properties
 /etc/my.cnf
 /etc/my.cnf.d/client.cnf
 /etc/my.cnf.d/mysql-clients.cnf
 /etc/my.cnf.d/server.cnf
 /etc/monasca/agent/agent.yaml
 /etc/monasca/agent/conf.d/*
 /etc/monasca/agent/supervisor.conf
 /etc/monasca/api-config.conf
 /etc/monasca/log-api-config.conf
 /etc/monasca/log-api-config.ini
 /etc/monasca-log-persister/monasca-log-persister.conf
 /etc/monasca-log-transformer/monasca-log-transformer.conf
 /etc/monasca-log-agent/agent.conf
 /etc/monasca-notification/monasca-notification.yaml
 /etc/monasca-persister/monasca-persister.yaml
 /etc/monasca-thresh/thresh.yaml
 /etc/elasticsearch/elasticsearch.yml
 /etc/elasticsearch/logging.yml
 /etc/kibana/kibana.yml

As an OpenStack operator, you use the Dashboard option on the Overview page to view the metrics data on the OpenStack services. The Monitoring Service operator uses the Dashboard option to view the metrics data on the Monitoring Service.

To monitor OpenStack, the preconfigured dashboard shows the following:

Each metrics dashboard is composed of one or more panels that are arranged in one or more rows. A row serves as a logical divider within a dashboard. It organizes your panels in groups. The panel is the basic building block for visualizing your metrics data.

For building dashboards, you have two options:

The following sections provide introductory information on dashboards, rows, and panels, and make you familiar with the first steps involved in building a dashboard. For additional information, you can also refer to the Grafana documentation.

To create a new dashboard, you use Open Dashboard in the top right corner of your dashboard window. The option provides access to various features for administrating dashboards. Use New to create an empty dashboard that serves as a starting point for adding rows and panels.

On the left side of an empty dashboard, there is a green rectangle displayed. Hover over this rectangle to access a Row menu. To insert your first panel, you can use the options in the Add Panel submenu. See below for details on the available panel types.

As soon as you have inserted an empty panel, you can add additional rows. For this purpose, use the Add Row option on the right side of the dashboard.

Features for editing rows can be accessed via the green rectangle that is displayed to the left of each row.

In addition to adding panels to a row, you can collapse or remove a row, move the position of the row within your dashboard, or set the row height. Row settings allows you, for example, to insert a row title or to hide the Row menu so that the row can no longer be edited.

Grafana distinguishes between three panel types:

Panels of type Graph are used to visualize metrics data. A query editor is provided to define the data to be visualized. The editor allows you to combine multiple queries. This means that any number of metrics and data series can be visualized in one panel.

Panels of type Singlestat are also used to visualize metrics data, yet they reduce a single query to a single number. The single number can be, for example, the minimum, maximum, average, or sum of values of the data series. The single number can be translated into a text value, if required.

Panels of type Text are used to insert static text. The text may, for example, provide information for the dashboard users. Text panels are not connected to any metrics data.

As soon as you have added a panel to your dashboard, you can access the options for editing the panel content. For this purpose, click the panel title and use Edit:

A large number of display and formatting features are provided to customize how the content is presented in a panel. Click the panel title to access the corresponding options. The menu that is displayed also allows you to duplicate or remove a panel. To change the size of a panel, click the + and - icons.

You can move panels on your dashboard by simply dragging and dropping them within and between rows.

By default, the time range for panels is controlled by dashboard settings. Use the time picker in the top right corner of your dashboard window to define relative or absolute time ranges. You can also set an auto-refresh interval, or manually refresh the data that is displayed.

CMM allows you to save your metrics dashboards locally. Saving a dashboard means exporting it to a JSON file. The JSON file can be edited, it can be shared with other users, and it can be imported to CMM again.

To save a dashboard, use Save in the top right corner of your dashboard window. The option allows you to directly view the JSON syntax and export the dashboard to a JSON file. The JSON file can be forwarded to other users, if required. To import a JSON file, use Open dashboard in the top left corner of the dashboard window.

For an alarm definition, you specify the following details:

Figure 4.2: Creating an Alarm Definition #

 

The expression defines how to evaluate a metrics. The expression syntax is based on a simple expressive grammar. For details, refer to the monasca API documentation.

To define an alarm expression, proceed as follows:

Note

You can also edit the expression syntax directly. For this purpose, save your alarm definition and update it using the Edit Alarm Definition option.

By default, an alarm definition is evaluated every minute. When updating the alarm definition, you can change this interval. For syntax details, refer to the monasca API documentation on Alarm Definition Expressions.

You can enable notifications for an alarm definition. As soon as an alarm is triggered, the enabled notifications will be sent.

The Notifications tab allows you to select the notifications from the ones that are predefined in your environment. For a selected notification, you specify whether you want to send it for a status transition to Alarm, OK, and/or Undetermined.

SUSE OpenStack Cloud Crowbar Monitoring supports the following standard metrics for monitoring servers and networks. These metrics usually do not require specific settings. The metrics are grouped by metrics types. Each metrics type references a set of related metrics.

Metrics on CPU usage, e.g. the percentage of time the CPU is idle when no I/O requests are in progress, or the percentage of time the CPU is used at system level or user level.

Metrics on disk space, e.g. the percentage of disk space that is used on a device, or the total amount of disk space aggregated across all the disks on a particular node.

Metrics on the average system load over different periods (e.g. 1 minute, 5 minutes, or 15 minutes).

Metrics on memory usage, e.g. the number of megabytes of total memory or free memory, or the percentage of free swap memory.

Metrics on the network, e.g. the number of network bytes received or sent per second, or the number of network errors on incoming or outgoing network traffic per second.

These metrics are configured automatically on all machines and nodes that have the monasca-agent role assigned. This applies not only to network.yaml but also to all metrics covered in this chapter.

In addition to the standard metrics, SUSE OpenStack Cloud Crowbar Monitoring automatically adds the following additional metrics to the monasca agent configuration on the OpenStack Controller.

HTTP endpoint checks perform up/down checks on HTTP endpoints. Based on a list of URLs, the agent sends an HTTP request and reports success or failure to the Monitoring Service.

The following barclamps will automatically create an HTTP check for the API services they deploy if the monasca barclamp is active:

By default, the monitoring dashboard is configured to display the service status for the following services:

The status visualization for additional services can be added manually.

Postgres checks gather various CRUD and system statistics for a database hosted by a PostgreSQL DBMS.

The following barclamps will automatically create Postgres checks for their service database if the monasca barclamp is active: