13 Managing Monitoring, Logging, and Usage Reporting #

The current percentage of CPU used by each worker and in total by all workers combined

service=block-storage, hostname, cluster, cloud_name, control_plane, component, backends

service=block-storage, hostname, cluster, cloud_name, control_plane, component, backendname

service=block-storage, hostname, cluster, cloud_name, control_plane, component, backendname

service=block-storage, hostname, cluster, cloud_name, control_plane, component

service=block-storage, hostname, cluster, cloud_name, control_plane, component, sub_component, logical_drive, controller_slot, array

The HPE Smart Storage Administrator (HPE SSA) CLI component will have to be installed for SSACLI status to be reported. To download and install the SSACLI utility to enable management of disk controllers, please refer to: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3d16386b418a443388c18da82f

service=block-storage, hostname, cluster, cloud_name, control_plane, component, box, bay, controller_slot

service=block-storage, hostname, cluster, cloud_name, control_plane, component, sub_component, model

service=block-storage, hostname, cluster, cloud_name, control_plane, component, model

Checks that all services are running heartbeats (uses nova user and to list services then sets up checks for each. For example, nova-scheduler, nova-conductor, nova-compute)

Indicates the number of primary shards in your cluster. This is an aggregate total across all indices.

Aggregate total of all shards across all indices, which includes replica shards.

Cluster health status.

The count of shards that are being freshly created.

Number of data nodes.

Number of nodes.

Shows the number of shards that are currently moving from one node to another node.

The number of unassigned shards from the master node.

Utilized physical host cpu core capacity for one or all hosts by time interval (defaults to a hour).

Available as total or per host

Total physical host cpu core capacity for one or all hosts by time interval (defaults to a hour)

Available as total or per host

Total physical host memory capacity by time interval (defaults to a hour)

Utilized physical host disk capacity by time interval (defaults to a hour)

CPUs allocated across all virtual machines by time interval (defaults to a hour)

Virtual CPUs allocated capacity for virtual machines of one or all projects by time interval (defaults to a hour)

Available as total or per host

Memory allocated to all virtual machines by time interval (defaults to a hour)

Memory utilized by virtual machines of one or all projects by time interval (defaults to an hour)

Available as total or per host

Memory allocated to virtual machines of one or all projects by time interval (defaults to an hour)

Available as total or per host

CPU utilized by all virtual machines by project by time interval (defaults to an hour)

Disk space allocated to all virtual machines by time interval (defaults to an hour)

Disk allocation for virtual machines of one or all projects by time interval (defaults to a hour)

Available as total or per host

Total available object storage capacity by time interval (defaults to a hour)

Available as total or per host

Remaining object storage capacity by time interval (defaults to a hour)

Available as total or per host

Rate of change of object storage usage by time interval (defaults to a hour)

Used object storage capacity by time interval (defaults to a hour)

The number of free pages, in bytes. This value is calculated by multiplying Innodb_buffer_pool_pages_free and Innodb_page_size of the server status variable.

The total size of buffer pool, in bytes. This value is calculated by multiplying Innodb_buffer_pool_pages_total and Innodb_page_size of the server status variable.

The number of used pages, in bytes. This value is calculated by subtracting Innodb_buffer_pool_pages_total away from Innodb_buffer_pool_pages_free of the server status variable.

Corresponding to current row locks of the server status variable.

Corresponding to Innodb_data_reads of the server status variable.

Corresponding to Innodb_data_writes of the server status variable.

Corresponding to the OS waits of the server status variable.

Corresponding to spinlock rounds of the server status variable.

Corresponding to the spin waits of the server status variable.

Corresponding to Innodb_os_log_fsyncs of the server status variable.

Corresponding to Innodb_row_lock_time of the server status variable.

Corresponding to Innodb_row_lock_waits of the server status variable.

Corresponding to Connections of the server status variable.

Corresponding to Max_used_connections of the server status variable.

Corresponding to Com_delete of the server status variable.

Corresponding to Com_delete_multi of the server status variable.

Corresponding to Com_insert of the server status variable.

Corresponding to Com_insert_select of the server status variable.

Corresponding to Com_replace_select of the server status variable.

Corresponding to Com_select of the server status variable.

Corresponding to Com_update of the server status variable.

Corresponding to Com_update_multi of the server status variable.

Corresponding to Created_tmp_disk_tables of the server status variable.

Corresponding to Created_tmp_files of the server status variable.

Corresponding to Created_tmp_tables of the server status variable.

The kernel time for the databases performance, in seconds.

Corresponding to Open_files of the server status variable.

Corresponding to Qcache_hits of the server status variable.

Corresponding to Queries of the server status variable.

Corresponding to Question of the server status variable.

Corresponding to Slow_queries of the server status variable.

Corresponding to Table_locks_waited of the server status variable.

Corresponding to Threads_connected of the server status variable.

The CPU user time for the databases performance, in seconds.

Value of the "publish_out" field of "message_stats" object

Value of the "rate" field of "message_stats/publish_out_details" object

Value of the "publish_in" field of "message_stats" object

Value of the "rate" field of "message_stats/publish_in_details" object

Value of the "fd_used" field in the response of /api/nodes

Value of the "mem_used" field in the response of /api/nodes

Value of the "run_queue" field in the response of /api/nodes

Value of the "sockets_used" field in the response of /api/nodes

Sum of ready and unacknowledged messages (queue depth)

Value of the "rate" field of "message_stats/deliver_details" object

Value of the "rate" field of "message_stats/publish_details" object

Value of the "rate" field of "message_stats/redeliver_details" object

This metric is the number of bytes read from objects in GET requests processed by this host during the last minute. Only successful GET requests to objects are counted. GET requests to the account or container is not included.

This metric is a count of the all the API requests made to swift that were processed by this host during the last minute.

This metric is the number of bytes written to objects in PUT or POST requests processed by this host during the last minute. Only successful requests to objects are counted. Requests to the account or container is not included.

This metric reports whether the swiftlm-access-log-tailer program is running normally.

This metric is a count of the all the API requests made to swift that were processed by this host during the last minute to a given project id.

This metric is the number of bytes read from objects in GET requests processed by this host for a given project during the last minute. Only successful GET requests to objects are counted. GET requests to the account or container is not included.

This metric is the number of bytes written to objects in PUT or POST requests processed by this host for a given project during the last minute. Only successful requests to objects are counted. Requests to the account or container is not included.

This metric reports the total length of all async pending queues in the system.

When a container update fails, the update is placed on the async pending queue. An update may fail becuase the container server is too busy or because the server is down or failed. Later the system will “replay” updates from the queue – so eventually, the container listings will show all objects known to the system.

If you know that container servers are down, it is normal to see the value of async pending increase. Once the server is restored, the value should return to zero.

A non-zero value may also indicate that containers are too large. Look for “lock timeout” messages in /var/log/swift/swift.log. If you find such messages consider reducing the container size or enable rate limiting.

The total exception string is truncated if longer than 1919 characters and an ellipsis is prepended in the first three characters of the message. If there is more than one error reported, the list of errors is paired to the last reported error and the operator is expected to resolve failures until no more are reported. Where there are no further reported errors, the Value Class is emitted as ‘Ok’.

Is the average utilization of all drives in the system. The value is a percentage (example: 30.0 means 30% of the total space is used).

Is the highest utilization of all drives in the system. The value is a percentage (example: 80.0 means at least one drive is 80% utilized). The value is just as important as swiftlm.diskusage.usage.avg. For example, if swiftlm.diskusage.usage.avg is 70% you might think that there is plenty of space available. However, if swiftlm.diskusage.usage.max is 100%, this means that some objects cannot be stored on that drive. swift will store replicas on other drives. However, this will create extra overhead.

Is the lowest utilization of all drives in the system. The value is a percentage (example: 10.0 means at least one drive is 10% utilized)

Is the size in bytes of available (unused) space of all drives in the system. Only drives used by swift are included.

Is the size in bytes of raw size of all drives in the system.

Is the size in bytes of used space of all drives in the system. Only drives used by swift are included.

This metric reports the average percent usage of all swift filesystems on a host.

This metric reports the percent usage of a swift filesystem that is most used (full) on a host. The value is the max of the percentage used of all swift filesystems.

This metric reports the percent usage of a swift filesystem that is least used (has free space) on a host. The value is the min of the percentage used of all swift filesystems.

This metric reports the number of bytes available (free) in a swift filesystem. The value is an integer (units: Bytes)

This metric reports the size in bytes of a swift filesystem. The value is an integer (units: Bytes)

This metric reports the percent usage of a swift filesystem. The value is a floating point number in range 0.0 to 100.0

This metric reports the number of used bytes in a swift filesystem. The value is an integer (units: Bytes)

This is the averaged value of the five minutes system load average of all nodes in the swift system.

This is the five minute load average of the busiest host in the swift system.

This is the five minute load average of the least loaded host in the swift system.

This metric reports the 5 minute load average of a host. The value is derived from /proc/loadavg.

If you are in the middle of deploying new rings, it is normal for this to be in the failed state.

However, if you are not in the middle of a deployment, you need to investigate the cause. Use “swift-recon –md5 -v” to identify the problem hosts.

This is the average across all servers for the account replicator to complete a cycle. As the system becomes busy, the time to complete a cycle increases. The value is in seconds.

This is the average across all servers for the container replicator to complete a cycle. As the system becomes busy, the time to complete a cycle increases. The value is in seconds.

This is the average across all servers for the object replicator to complete a cycle. As the system becomes busy, the time to complete a cycle increases. The value is in seconds.

This is the number of seconds since the account replicator last completed a scan on the host that has the oldest completion time. Normally the replicators runs periodically and hence this value will decrease whenever a replicator completes. However, if a replicator is not completing a cycle, this value increases (by one second for each second that the replicator is not completing). If the value remains high and increasing for a long period of time, it indicates that one of the hosts is not completing the replication cycle.

This is the number of seconds since the container replicator last completed a scan on the host that has the oldest completion time. Normally the replicators runs periodically and hence this value will decrease whenever a replicator completes. However, if a replicator is not completing a cycle, this value increases (by one second for each second that the replicator is not completing). If the value remains high and increasing for a long period of time, it indicates that one of the hosts is not completing the replication cycle.

This is the number of seconds since the object replicator last completed a scan on the host that has the oldest completion time. Normally the replicators runs periodically and hence this value will decrease whenever a replicator completes. However, if a replicator is not completing a cycle, this value increases (by one second for each second that the replicator is not completing). If the value remains high and increasing for a long period of time, it indicates that one of the hosts is not completing the replication cycle.

If an unrecoverable read error (URE) occurs on a filesystem, the error is logged in the kernel log. The swift-drive-audit program scans the kernel log looking for patterns indicating possible UREs.

To get more information, log onto the node in question and run:

sudoswift-drive-audit/etc/swift/drive-audit.conf

UREs are common on large disk drives. They do not necessarily indicate that the drive is failed. You can use the xfs_repair command to attempt to repair the filesystem. Failing this, you may need to wipe the filesystem.

If UREs occur very often on a specific drive, this may indicate that the drive is about to fail and should be replaced.

This metric reports if a directory or file has the appropriate owner. The check looks at swift configuration directories and files. It also looks at the top-level directories of mounted file systems (for example, /srv/node/disk0 and /srv/node/disk0/objects).

This metric reports if a directory or file has the appropriate owner. The check looks at swift configuration directories and files. It also looks at the top-level directories of mounted file systems (for example, /srv/node/disk0 and /srv/node/disk0/objects).

This indicates of the swiftlm monasca-agent Plug-in is running normally. If the status is failed, it probable that some or all metrics are no longer being reported.

This reports how long (in seconds) since the replicator process last finished a replication run. If the replicator is stuck, the time will keep increasing forever. The time a replicator normally takes depends on disk sizes and how much data needs to be replicated. However, a value over 24 hours is generally bad.

This reports how long (in seconds) since the replicator process last finished a replication run. If the replicator is stuck, the time will keep increasing forever. The time a replicator normally takes depends on disk sizes and how much data needs to be replicated. However, a value over 24 hours is generally bad.

This reports how long (in seconds) since the replicator process last finished a replication run. If the replicator is stuck, the time will keep increasing forever. The time a replicator normally takes depends on disk sizes and how much data needs to be replicated. However, a value over 24 hours is generally bad.

This metric reports of the process as named in the component dimension and the msg value_meta is running or not.

Use the swift-start.yml playbook to attempt to restart the stopped process (it will start any process that has stopped – you do not need to specifically name the process).

This metric reports the mount state of each drive that should be mounted on this node.

This metric reports if a server can connect to a VIPs. Currently the following VIPs are checked:

This metric reports if memcached on the host as specified by the hostname dimension is accepting connections from the host running the check. The following value_meta.msg are used:

We successfully connected to <hostname> on port <target_port>

{
  "dimensions": {
    "hostname": "ardana-ccp-c1-m1-mgmt",
    "observer_host": "ardana-ccp-c1-m1-mgmt",
    "service": "object-storage",
    "target_port": "11211"
  },
  "metric": "swiftlm.systems.connectivity.memcache_check",
  "timestamp": 1449084058,
  "value": 0,
  "value_meta": {
    "msg": "ardana-ccp-c1-m1-mgmt:11211 ok"
  }
}

We failed to connect to <hostname> on port <target_port>

{
  "dimensions": {
    "fail_message": "[Errno 111] Connection refused",
    "hostname": "ardana-ccp-c1-m1-mgmt",
    "observer_host": "ardana-ccp-c1-m1-mgmt",
    "service": "object-storage",
    "target_port": "11211"
  },
  "metric": "swiftlm.systems.connectivity.memcache_check",
  "timestamp": 1449084150,
  "value": 2,
  "value_meta": {
    "msg": "ardana-ccp-c1-m1-mgmt:11211 [Errno 111] Connection refused"
  }
}

This metric reports if rsyncd on the host as specified by the hostname dimension is accepting connections from the host running the check. The following value_meta.msg are used:

We successfully connected to <hostname> on port <target_port>:

{
  "dimensions": {
    "hostname": "ardana-ccp-c1-m1-mgmt",
    "observer_host": "ardana-ccp-c1-m1-mgmt",
    "service": "object-storage",
    "target_port": "873"
  },
  "metric": "swiftlm.systems.connectivity.rsync_check",
  "timestamp": 1449082663,
  "value": 0,
  "value_meta": {
    "msg": "ardana-ccp-c1-m1-mgmt:873 ok"
  }
}

We failed to connect to <hostname> on port <target_port>:

{
  "dimensions": {
    "fail_message": "[Errno 111] Connection refused",
    "hostname": "ardana-ccp-c1-m1-mgmt",
    "observer_host": "ardana-ccp-c1-m1-mgmt",
    "service": "object-storage",
    "target_port": "873"
  },
  "metric": "swiftlm.systems.connectivity.rsync_check",
  "timestamp": 1449082860,
  "value": 2,
  "value_meta": {
    "msg": "ardana-ccp-c1-m1-mgmt:873 [Errno 111] Connection refused"
  }
}

Reports the average value of N-iterations of the latency values recorded for a component.

This metric reports the state of each component after N-iterations of checks. If the initial check succeeds, the checks move onto the next component until all components are queried, then the checks sleep for ‘main_loop_interval’ seconds. If a check fails, it is retried every second for ‘retries’ number of times per component. If the check fails ‘retries’ times, it is reported as a fail instance.

A successful state will be reported in JSON:

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.check.state",
    "timestamp": 1453111805,
    "value": 0
},

A failed state will report a “fail” value and the value_meta will provide the http response error.

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.check.state",
    "timestamp": 1453112841,
    "value": 2,
    "value_meta": {
        "msg": "HTTPConnectionPool(host='192.168.245.9', port=8080): Max retries exceeded with url: /v1/AUTH_76538ce683654a35983b62e333001b47 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fd857d7f550>: Failed to establish a new connection: [Errno 110] Connection timed out',))"
    }
}

This metric reports the maximum response time in seconds of a REST call from the observer to the component REST API listening on the reported host

A response time query will be reported in JSON:

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.max.latency_sec",
    "timestamp": 1453111805,
    "value": 0.2772650718688965
}

A failed query will have a much longer time value:

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.max.latency_sec",
    "timestamp": 1453112841,
    "value": 127.288015127182
}

This metric reports the minimum response time in seconds of a REST call from the observer to the component REST API listening on the reported host

A response time query will be reported in JSON:

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.min.latency_sec",
    "timestamp": 1453111805,
    "value": 0.10025882720947266
}

A failed query will have a much longer time value:

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.min.latency_sec",
    "timestamp": 1453112841,
    "value": 127.25378203392029
}

This metric reports the average of all the collected records in the swiftlm.umon.target.val.avail_minute metric data. This is a walking average data set of these approximately per-minute states of the swift Object Store. The most basic case is a whole day of successful per-minute records, which will average to 100% availability. If there is any downtime throughout the day resulting in gaps of data which are two minutes or longer, the per-minute availability data will be “back filled” with an assumption of a down state for all the per-minute records which did not exist during the non-reported time. Because this is a walking average of approximately 24 hours worth of data, any outtage will take 24 hours to be purged from the dataset.

A 24-hour average availability report:

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.val.avail_day",
    "timestamp": 1453645405,
    "value": 7.894736842105263
}

A value of 100 indicates that swift-uptime-monitor was able to get a token from keystone and was able to perform operations against the swift API during the reported minute. A value of zero indicates that either keystone or swift failed to respond successfully. A metric is produced every minute that swift-uptime-monitor is running.

An “up” minute report value will report 100 [percent]:

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.val.avail_minute",
    "timestamp": 1453645405,
    "value": 100.0
}

A “down” minute report value will report 0 [percent]:

{
    "dimensions": {
        "component": "rest-api",
        "hostname": "ardana-ccp-vip-admin-SWF-PRX-mgmt",
        "observer_host": "ardana-ccp-c1-m1-mgmt",
        "service": "object-storage",
        "url": "http://ardana-ccp-vip-admin-SWF-PRX-mgmt:8080"
    },
    "metric": "swiftlm.umon.target.val.avail_minute",
    "timestamp": 1453649139,
    "value": 0.0
}

This metric reports the firmware version of a component of a Smart Array controller.

This reports the status of various sub-components of a Smart Array Controller.

A failure is considered to have occured if:

This reports the status of a disk drive attached to a Smart Array controller.

This reports the status of a LUN presented by a Smart Array controller.

A LUN is considered failed if the LUN has failed or if the LUN cache is not enabled and working.

Username that will be required for logging into the Kibana UI.

Password generated during installation that is used to login to the Kibana UI.

Add a shared disk partition to each of the Elasticsearch controller nodes.

The default partition name used for backup is

/var/lib/esbackup

You can change this by:

Ensure the shared disk has enough storage to retain backups for the desired retention period.

barbican-api

barbican-worker

INFO (default)

DEBUG

cinder-api

cinder-scheduler

cinder-backup

cinder-volume

INFO (default)

DEBUG

ceilometer-collector

ceilometer-agent-notification

ceilometer-polling

ceilometer-expirer

INFO (default)

DEBUG