Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE OpenStack Cloud Monitoring

1 Introduction to SUSE OpenStack Cloud Monitoring

As more and more applications are deployed on cloud systems and cloud systems are growing in complexity, managing the cloud infrastructure is becoming increasingly difficult. SUSE OpenStack Cloud Monitoring helps mastering this challenge by providing a sophisticated Monitoring as a Service solution that is operated on top of OpenStack-based cloud computing platforms.

The component architecture of OpenStack provides for high flexibility, yet it increases the burden of system operation because multiple services must be handled. SUSE OpenStack Cloud Monitoring offers an integrated view of all services and assembles and presents related metrics and log data in one convenient access point. While being flexible and scalable to instantly reflect changes in the OpenStack platform, SUSE OpenStack Cloud Monitoring provides the ways and means required to ensure multi-tenancy, high availability, and data security.

SUSE OpenStack Cloud Monitoring covers all aspects of a Monitoring as a Service solution:

  • Central management of monitoring and log data from medium and large-size OpenStack deployments.

  • Storage of metrics and log data in a resilient way.

  • Multi-tenancy architecture to ensure the secure isolation of metrics and log data.

  • Horizontal and vertical scalability to support constantly evolving cloud infrastructures. When physical and virtual servers are scaled up or down to varying loads, the monitoring and log management solution can be adapted accordingly.

1.1 Basic Usage Scenario

The basic usage scenario of setting up and using the monitoring features of SUSE OpenStack Cloud Monitoring looks as follows:

metricslogs.png

An application operator acts as a service provider in the OpenStack environment. He books virtual machines to provide services to end users or to host services that he needs for his own development activities. SUSE OpenStack Cloud Monitoring helps application operators ensure that their services and the servers on which they are provided are configured and working as required.

The OpenStack operator is a special application operator who is responsible for administrating and maintaining the underlying OpenStack platform. The monitoring and log management services of SUSE OpenStack Cloud Monitoring enable him to ensure the availability and quality of the platform. He uses SUSE OpenStack Cloud Monitoring for:

  • Monitoring physical and virtual servers, hypervisors, and OpenStack services.

  • Monitoring middleware components, for example, database services.

  • Retrieving and analyzing the log data of the OpenStack services and servers, the middleware components, and the operating system.

As the Monitoring Service operator, you are responsible for providing the monitoring and log management features to the application operators and the OpenStack operator. This enables the application operators and the OpenStack operator to focus on operation and the quality of their services and servers without having to carry out the tedious tasks implied by setting up and administrating their own monitoring software. You use the monitoring features yourself for ensuring the quality of SUSE OpenStack Cloud Monitoring.

The Monitoring Service Operator's Tasks

As the Monitoring Service operator, you have the following responsibilities:

  • Deploying the Monitoring Service, thus providing the monitoring features to the application operators, and the monitoring and log management features to the OpenStack operator.

  • Regular maintenance of the components and services SUSE OpenStack Cloud Monitoring consists of.

  • Backup of the SUSE OpenStack Cloud Monitoring databases, configuration files, and customized dashboards.

  • Monitoring of SUSE OpenStack Cloud Monitoring to ensure the SUSE OpenStack Cloud Monitoring quality. For monitoring, you use a graphical user interface that is seamlessly integrated into the cloud infrastructure. Based on OpenStack Horizon, the user interface visualizes the health and status of your cloud resources and enables user access to all monitoring and log management functionality.

The tasks of the Monitoring Service operator and the OpenStack Operator can jointly be performed by one system operator. In this case, also refer to the OpenStack Operator's Guide.

1.2 SUSE OpenStack Cloud Monitoring Architecture and Components

The following illustration provides an overview of the main components of SUSE OpenStack Cloud Monitoring and their interaction:

cmm-architecture.png
OpenStack

SUSE OpenStack Cloud Monitoring relies on OpenStack as technology for building cloud computing platforms for public and private clouds. OpenStack consists of a series of interrelated projects delivering various components for a cloud infrastructure solution and allowing for the deployment and management of Infrastructure as a Service (IaaS) platforms.

Monitoring Service

The Monitoring Service is the central SUSE OpenStack Cloud Monitoring component. It is responsible for receiving, persisting, and processing monitoring and log data, as well as providing the data to the users.

The Monitoring Service relies on Monasca, an open source Monitoring as a Service solution. It uses Monasca for high-speed metrics querying and integrates the Threshold Engine (streaming alarm engine) and the Notification Engine of Monasca.

The Monitoring Service consists of the following components:

Monitoring API

A RESTful API for monitoring. It is primarily focused on the following areas:

  • Metrics: Store and query massive amounts of metrics in real time.

  • Statistics: Provide statistics for metrics.

  • Alarm Definitions: Create, update, query, and delete alarm definitions.

  • Alarms: Query and delete the alarm history.

  • Notification Methods: Create and delete notification methods and associate them with alarms. Users can be notified directly when alarms are triggered, for example, via email.

Message Queue

A component that primarily receives published metrics from the Monitoring API, alarm state transition messages from the Threshold Engine, and log data from the Log API. The data is consumed by other components, such as the Persister, the Notification Engine, and the Log Persister. The Message Queue is also used to publish and consume other events in the system. It is based on Kafka, a high-performance, distributed, fault-tolerant, and scalable message queue with durability built-in. For administrating the Message Queue, SUSE OpenStack Cloud Monitoring uses Zookeeper, a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Persister

A Monasca component that consumes metrics and alarm state transitions from the Message Queue and stores them in the Metrics and Alarms Database (InfluxDB).

Notification Engine

A Monasca component that consumes alarm state transition messages from the Message Queue and sends notifications for alarms, such as emails.

Threshold Engine

A Monasca component that computes thresholds on metrics and publishes alarms to the Message Queue when they are triggered. The Threshold Engine is based on Apache Storm, a free and open distributed real-time computation system.

Metrics and Alarms Database

An InfluxDB database used for storing metrics and the alarm history.

Config Database

A MariaDB database used for storing configuration information, alarm definitions, and notification methods.

Log API

A RESTful API for log management. It gathers log data from the Log Agents and forwards it to the Message Queue.

The SUSE OpenStack Cloud Monitoring log management is based on Logstash, a tool for receiving, processing, and publishing all kinds of logs. It provides a powerful pipeline for querying and analyzing logs. Elasticsearch is used as the back-end datastore, and Kibana as the front-end tool for retrieving and visualizing the log data.

Log Transformer

A Logstash component that consumes the log data from the Message Queue, performs transformation and aggregation operations on the data, and publishes the data that it creates back to the Message Queue.

Log Metrics

A Monasca component that consumes log data from the Message Queue, filters the data according to severity, and generates metrics for specific severities, for example, for errors or warnings. The generated metrics are published to the Message Queue and can be further processed by the Threshold Engine like any other metrics.

Log Persister

A Logstash component that consumes the transformed and aggregated log data from the Message Queue and stores it in the Log Database.

Kibana Server

A Web browser-based analytics and search interface to the Log Database.

Log Database

An Elasticsearch database for storing the log data.

Horizon Plugin

SUSE OpenStack Cloud Monitoring comes with a plugin for the OpenStack Horizon dashboard. The plugin extends the main dashboard in OpenStack with a view for monitoring. This enables SUSE OpenStack Cloud Monitoring users to access the monitoring functions from a central Web-based graphical user interface. For details, refer to the OpenStack Horizon documentation (http://docs.openstack.org/developer/horizon/).

Based on OpenStack Horizon, the monitoring data is visualized on a comfortable and easy-to-use dashboard which fully integrates with the following applications:

Grafana (for metrics data)

An open source application for visualizing large-scale measurement data.

Kibana (for log data)

An open source analytics and visualization platform designed to work with Elasticsearch.

Metrics Agent

A Metrics Agent is required for retrieving metrics data from the host on which it runs and sending the metrics data to the Monitoring Service. The agent supports metrics from a variety of sources as well as a number of built-in system and service checks.

A Metrics Agent can be installed on each virtual or physical server to be monitored.

The agent functionality is fully integrated into the source code base of the Monasca project. For details, refer to the Monasca Wiki (https://wiki.openstack.org/wiki/Monasca).

Log Agent

A Log Agent is needed for collecting log data from the host on which it runs and forwarding the log data to the Monitoring Service for further processing. It can be installed on each virtual or physical server from which log data is to be retrieved.

The agent functionality is fully integrated into the source code base of the Monasca project. For details, refer to the Monasca Wiki (https://wiki.openstack.org/wiki/Monasca).

1.2.1 Agents and Services

Service NameDescription
zookeeper Centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
storm-nimbus Storm is a distributed real-time computation system for processing large volumes of high-velocity data. The Storm Nimbus daemon is responsible for distributing code around a cluster, assigning tasks to machines, and monitoring for failures.
storm-supervisor The Storm supervisor listens for work assigned to its machine and starts and stops worker processes as necessary based on what Nimbus has assigned to it.
mariadb MariaDB database service. SUSE OpenStack Cloud Monitoring stores configuration information in this database.
kafka Message queue service.
influxdb InfluxDB database service. SUSE OpenStack Cloud Monitoring stores metrics and alarms in this database.
elasticsearch Elasticsearch database service. SUSE OpenStack Cloud Monitoring stores the log data in this database.
memcached Memcached service. SUSE OpenStack Cloud Monitoring uses it for caching authentication and authorization information required for the communication between the Log API and OpenStack Keystone.
openstack-monasca-notification Notification Engine.
openstack-monasca-thresh Threshold Engine.
openstack-monasca-log-transformer Log Transformer.
apache2 Log and monitoring API.
openstack-monasca-persister Persister.
openstack-monasca-agent Metrics Agent.
kibana Kibana server.
openstack-monasca-log-persister Log Persister.
openstack-monasca-log-metrics Log Metrics.
openstack-monasca-log-agent Log Agent.

1.3 User Management

SUSE OpenStack Cloud Monitoring is fully integrated with Keystone, the identity service which serves as the common authentication and authorization system in OpenStack.

The SUSE OpenStack Cloud Monitoring integration with Keystone requires any SUSE OpenStack Cloud Monitoring user, including the Monitoring Service operator, to be registered as an OpenStack user. All authentication and authorization in SUSE OpenStack Cloud Monitoring is done through Keystone. If a user requests monitoring data, for example, SUSE OpenStack Cloud Monitoring verifies that the user is a valid user in OpenStack and allowed to access the requested metrics.

SUSE OpenStack Cloud Monitoring users are created and administrated in OpenStack:

  • Each user assumes a role in OpenStack to perform a specific set of operations. The OpenStack role specifies a set of rights and privileges.

  • Each user is assigned to at least one project in OpenStack. A project is an organizational unit that defines a set of resources which can be accessed by the assigned users.

    Application operators in SUSE OpenStack Cloud Monitoring can monitor the set of resources that is defined for the projects to which they are assigned.

For details on user management, refer to the OpenStack documentation (http://docs.openstack.org/newton/).

1.4 Distribution Media

SUSE OpenStack Cloud Monitoring is shipped as part of SUSE OpenStack Cloud with a maintenance update. Note that you need to buy a separate subscription for SUSE OpenStack Cloud Monitoring to be entitled to get support for SUSE OpenStack Cloud Monitoring.

Print this page