This is unreleased documentation for SUSE® Storage 1.13 (Dev).

SUSE® Storage Metrics for Monitoring

Volume

Name Description Example

longhorn_volume_actual_size_bytes

Actual space used by each replica of the volume on the corresponding node

longhorn_volume_actual_size_bytes{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 1.1917312e+08

longhorn_volume_capacity_bytes

Configured size in bytes for this volume

longhorn_volume_capacity_bytes{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 6.442450944e+09

longhorn_volume_encrypted

Encrypted volume

longhorn_volume_encrypted{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 1

longhorn_volume_state

Volume state. This metric uses the state label to indicate the current volume state. The value is 1 for the current state and 0 for others. States: creating, attached, detached, attaching, detaching, deleting

longhorn_volume_state{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol",state="attached"} 1

longhorn_volume_robustness

Volume robustness. This metric uses the state label to indicate the current robustness. The value is 1 for the current state and 0 for others. States: unknown, healthy, degraded, faulted

longhorn_volume_robustness{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol",state="healthy"} 1

longhorn_volume_read_throughput

Read throughput of this volume (Bytes/s)

longhorn_volume_read_throughput{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 5120000

longhorn_volume_write_throughput

Write throughput of this volume (Bytes/s)

longhorn_volume_write_throughput{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 512000

longhorn_volume_read_iops

Read IOPS of this volume

longhorn_volume_read_iops{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 100

longhorn_volume_write_iops

Write IOPS of this volume

longhorn_volume_write_iops{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 100

longhorn_volume_read_latency

Read latency of this volume (ns)

longhorn_volume_read_latency{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 100000

longhorn_volume_write_latency

Write latency of this volume (ns)

longhorn_volume_write_latency{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 100000

longhorn_volume_file_system_read_only

This metric indicates that the volume is now in read-only mode. The metric is either 1 or no record for each volume

longhorn_volume_file_system_read_only{node="worker-2",pvc="testvol",pvc_namespace="default",volume="testvol"} 1

longhorn_volume_last_backup_at

Unix timestamp of the last successful backup of this volume, or 0 if no such backup exists

longhorn_volume_last_backup_at{pvc_namespace="default",node="worker-2",pvc="testvol",volume="testvol"} 1.766365578e+09

Node

Name Description Example

longhorn_node_status

Status of this node: 1=true, 0=false

longhorn_node_status{condition="ready",condition_reason="",node="worker-2"} 1

longhorn_node_count_total

Total number of nodes in the Longhorn system

longhorn_node_count_total 4

longhorn_node_cpu_capacity_millicpu

The maximum allocatable CPU on this node

longhorn_node_cpu_capacity_millicpu{node="worker-2"} 2000

longhorn_node_cpu_usage_millicpu

The CPU usage on this node

longhorn_node_cpu_usage_millicpu{node="pworker-2"} 186

longhorn_node_memory_capacity_bytes

The maximum allocatable memory on this node

longhorn_node_memory_capacity_bytes{node="worker-2"} 4.031229952e+09

longhorn_node_memory_usage_bytes

The memory usage on this node

longhorn_node_memory_usage_bytes{node="worker-2"} 1.833582592e+09

longhorn_node_storage_capacity_bytes

The storage capacity of this node

longhorn_node_storage_capacity_bytes{node="worker-3"} 8.3987283968e+10

longhorn_node_storage_usage_bytes

The used storage of this node

longhorn_node_storage_usage_bytes{node="worker-3"} 9.060941824e+09

longhorn_node_storage_reservation_bytes

The reserved storage for other applications and system on this node

longhorn_node_storage_reservation_bytes{node="worker-3"} 2.519618519e+10

Replica

Name

Description

Example

longhorn_replica_info

Static metadata for each Replica CR

longhorn_replica_info{replica="testvol-r-abc", volume="testvol", node="node-1", disk_path="/dev/xda", data_engine="v2"} 1

longhorn_replica_state

Current runtime state of the replica: running, stopped, error, starting, stopping, unknown

longhorn_replica_state{replica="testvol-r-abc", volume="testvol", node="node-1", state="running"} 1

Engine

Name Description Example

longhorn_engine_info

Static metadata for each Engine CR

longhorn_engine_info{engine="testvol-e-0", volume="testvol", node="node-1", data_engine="v2", frontend="blockdev", image="longhorn-instance-manager:latest"} 1

longhorn_engine_state

Runtime state of an engine: running, stopped, error, starting, stopping, unknown

longhorn_engine_state{engine="testvol-e-0", volume="testvol", node="node-1", state="running"} 1

longhorn_engine_replica_mode

The mode reported for each replica by the engine: RW, WO, ERR

longhorn_engine_replica_mode{volume="testvol", engine="testvol-e-0", replica="testvol-r-abc", mode="RW"} 1

longhorn_engine_rebuild_progress

Engine rebuild progress, ranging from 0 to 100 percent. This metric is visible only when a replica is rebuilding.

longhorn_engine_rebuild_progress{pvc_namespace="default",pvc="testvol",engine="testvol-e-0",rebuild_src="10.42.1.215:20036",rebuild_dst="10.42.0.131:20922"} 42

Disk

Name Description Example

longhorn_disk_capacity_bytes

The storage capacity of this disk

longhorn_disk_capacity_bytes{disk="default-disk-8b28ee3134628183",node="worker-3"} 8.3987283968e+10

longhorn_disk_usage_bytes

The used storage of this disk

longhorn_disk_usage_bytes{disk="default-disk-8b28ee3134628183",node="worker-3"} 9.060941824e+09

longhorn_disk_reservation_bytes

The reserved storage for other applications and system on this disk

longhorn_disk_reservation_bytes{disk="default-disk-8b28ee3134628183",node="worker-3"} 2.519618519e+10

longhorn_disk_status

The status of this disk

longhorn_disk_status{condition="ready",condition_reason="",disk="default-disk-ca0300000000",node="worker-3"}

longhorn_disk_read_throughput

Read throughput of this disk (Bytes/s)

longhorn_disk_read_throughput{disk="default-disk-8b28ee3134628183",node="worker-3",disk_path="/dev/sda"} 10485760

longhorn_disk_write_throughput

Write throughput of this disk (Bytes/s)

longhorn_disk_write_throughput{disk="default-disk-8b28ee3134628183",node="worker-3",disk_path="/dev/sda"} 2097152

longhorn_disk_read_iops

Read IOPS of this disk

longhorn_disk_read_iops{disk="default-disk-8b28ee3134628183",node="worker-3",disk_path="/dev/sda"} 200

longhorn_disk_write_iops

Write IOPS of this disk

longhorn_disk_write_iops{disk="default-disk-8b28ee3134628183",node="worker-3",disk_path="/dev/sda"} 150

longhorn_disk_read_latency

Read latency of this disk (nanoseconds)

longhorn_disk_read_latency{disk="default-disk-8b28ee3134628183",node="worker-3",disk_path="/dev/sda"} 85000

longhorn_disk_write_latency

Write latency of this disk (nanoseconds)

longhorn_disk_write_latency{disk="default-disk-8b28ee3134628183",node="worker-3",disk_path="/dev/sda"} 95000

longhorn_disk_health

Disk health status (1 = healthy, 0 = unhealthy). See Disk Health Monitoring for details

longhorn_disk_health{disk="default-disk-8b28ee3134628183",node="worker-3"} 1

longhorn_disk_health_attribute_raw

Raw SMART health attribute value for the disk. Available only when SMART data is supported. See Disk Health Monitoring for details

longhorn_disk_health_attribute_raw{attribute="Temperature Celsius",attribute_id="",disk="default-disk-db0dcb3d43bdcd05",node="longhorn-worker"} 37

Instance Manager

Name Description Example

longhorn_instance_manager_cpu_usage_millicpu

The cpu usage of this longhorn instance manager

longhorn_instance_manager_cpu_usage_millicpu{instance_manager="instance-manager-e-2189ed13",instance_manager_type="engine",node="worker-2"} 80

longhorn_instance_manager_cpu_requests_millicpu

Requested CPU resources in kubernetes of this Longhorn instance manager

longhorn_instance_manager_cpu_requests_millicpu{instance_manager="instance-manager-e-2189ed13",instance_manager_type="engine",node="worker-2"} 250

longhorn_instance_manager_memory_usage_bytes

The memory usage of this longhorn instance manager

longhorn_instance_manager_memory_usage_bytes{instance_manager="instance-manager-e-2189ed13",instance_manager_type="engine",node="worker-2"} 2.4072192e+07

longhorn_instance_manager_memory_requests_bytes

Requested memory in Kubernetes of this longhorn instance manager

longhorn_instance_manager_memory_requests_bytes{instance_manager="instance-manager-e-2189ed13",instance_manager_type="engine",node="worker-2"} 0

longhorn_instance_manager_proxy_grpc_connection

The number of proxy gRPC connection of this longhorn instance manager

longhorn_instance_manager_proxy_grpc_connection{instance_manager="instance-manager-e-814dfd05", instance_manager_type="engine", node="worker-2"} 0

Manager

Name Description Example

longhorn_manager_cpu_usage_millicpu

The CPU usage of this Longhorn Manager

longhorn_manager_cpu_usage_millicpu{manager="longhorn-manager-5rx2n",node="worker-2"} 27

longhorn_manager_memory_usage_bytes

The memory usage of this Longhorn Manager

longhorn_manager_memory_usage_bytes{manager="longhorn-manager-5rx2n",node="worker-2"} 2.6144768e+07

Workqueue

Longhorn Manager uses workqueues to process updates of the watched Longhorn and Kubernetes resources. Each Longhorn Manager exposes the metrics of its workqueues, which are important indicators of how well Longhorn Manager is performing. The name label is the name of the workqueue, which matches the corresponding controller (for example, longhorn-volume, longhorn-engine, and longhorn-replica).

Name Description Example

longhorn_workqueue_depth

Current depth of the workqueue

longhorn_workqueue_depth{name="longhorn-volume"} 0

longhorn_workqueue_adds_total

Total number of adds handled by the workqueue

longhorn_workqueue_adds_total{name="longhorn-volume"} 1024

longhorn_workqueue_queue_duration_seconds

Histogram of how long in seconds an item stays in the workqueue before being requested

longhorn_workqueue_queue_duration_seconds_bucket{name="longhorn-volume",le="0.001"} 1000

longhorn_workqueue_work_duration_seconds

Histogram of how long in seconds processing an item from the workqueue takes

longhorn_workqueue_work_duration_seconds_bucket{name="longhorn-volume",le="0.1"} 1010

longhorn_workqueue_unfinished_work_seconds

How many seconds of work has been done that is in progress and has not been observed by work_duration. Large values indicate stuck threads. You can deduce the number of stuck threads by observing the rate at which this value increases

longhorn_workqueue_unfinished_work_seconds{name="longhorn-volume"} 0

longhorn_workqueue_longest_running_processor_seconds

How many seconds the longest running processor for the workqueue has been running

longhorn_workqueue_longest_running_processor_seconds{name="longhorn-volume"} 0

longhorn_workqueue_retries_total

Total number of retries handled by the workqueue

longhorn_workqueue_retries_total{name="longhorn-volume"} 12

Kubernetes Client

Each Longhorn Manager exposes the metrics of the requests sent by its Kubernetes client (client-go) to the Kubernetes API server. These metrics help you understand the latency, rate limiting, and results of the requests issued by Longhorn Manager.

Name Description Example

longhorn_rest_client_request_latency_seconds

Histogram of request latency in seconds, broken down by verb and URL

longhorn_rest_client_request_latency_seconds_bucket{url="https://10.43.0.1:443/apis/longhorn.io/v1beta2/namespaces/%7Bnamespace%7D/volumes/%7Bname%7D",verb="PUT",le="0.016"} 250

longhorn_rest_client_rate_limiter_latency_seconds

Histogram of client-side rate limiter latency in seconds, broken down by verb and URL

longhorn_rest_client_rate_limiter_latency_seconds_bucket{url="https://10.43.0.1:443/apis/longhorn.io/v1beta2/namespaces/%7Bnamespace%7D/volumes/%7Bname%7D",verb="PUT",le="0.001"} 250

longhorn_rest_client_requests_total

Number of HTTP requests, partitioned by status code, method, and host

longhorn_rest_client_requests_total{code="200",host="10.43.0.1:443",method="GET"} 12345

Histogram metrics are exposed as _bucket, _sum, and _count series. For example, you can calculate the 99th percentile of the time spent processing items for each workqueue by using the following PromQL query:

histogram_quantile(0.99, sum by (name, le) (rate(longhorn_workqueue_work_duration_seconds_bucket[5m])))

Backup

Name Description Example

longhorn_backup_actual_size_bytes

Actual size of this backup

longhorn_backup_actual_size_bytes{backup="backup-4ab66eca0d60473e",volume="testvol", recurring_job="backup"} 6.291456e+07

longhorn_backup_state

State of this backup: 0=New, 1=Pending, 2=InProgress, 3=Completed, 4=Error, 5=Unknown

longhorn_backup_state{backup="backup-4ab66eca0d60473e",volume="testvol", recurring_job=""} 3

longhorn_backup_uploaded_data_size_bytes

Uploaded data size of this backup

longhorn_backup_uploaded_data_size_bytes{backup="backup-4ab66eca0d60473e",volume="testvol",recurring_job=""} 2.012112434e+09

Backup Target

Name Description Example

longhorn_backup_target_backup_volume_count

Number of backup volumes on this backup target

longhorn_backup_target_backup_volume_count{backup_target="default"} 1

Backup Volume

Name Description Example

longhorn_backup_volume_backups_count

Number of backups belonging to this backup volume

longhorn_backup_volume_backups_count{backup_volume="testvol"} 1

Snapshot

Name Description Example

longhorn_snapshot_actual_size_bytes

Actual size of this snapshot

longhorn_snapshot_actual_size_bytes{snapshot="f4468111-2efa-45f5-aef6-63109e30d92c",user_created="false",volume="testvol"} 1.048576e+07

BackingImage

Name Description Example

longhorn_backing_image_actual_size_bytes

Actual size of this backing image

longhorn_backing_image_actual_size_bytes{backing_image="parrot",disk="ca203ce8-2cad-4cd1-92a7-542851f50518",node="kworker1"} 3.3554432e+07

longhorn_backing_image_state

State of this backing image: 0=Pending, 1=Starting, 2=InProgress, 3=ReadyForTransfer, 4=Ready, 5=Failed, 6=FailedAndCleanUp, 7=Unknown

longhorn_backing_image_state{backing_image="parrot",disk="ca203ce8-2cad-4cd1-92a7-542851f50518",node="kworker1"} 4

BackupBackingImage

Name Description Example

longhorn_backup_backing_image_actual_size_bytes

Actual size of this backup backing image

longhorn_backup_backing_image_actual_size_bytes{backup_backing_image="parrot"} 3.3554432e+07

longhorn_backup_backing_image_state

State of this backup backing image: 0=New, 1=Pending, 2=InProgress, 3=Completed, 4=Error, 5=Unknown

longhorn_backup_backing_image_state{backup_backing_image="parrot"} 3

CSI

The CSI sidecar component has built-in metrics for users to get insights into CSI operations. The CSI operations metrics cover total count, error count, and call latency. Longhorn enables the metrics by adding the flag --http-endpoint for each CSI sidecar component. You can use Prometheus’s PodMonitor to collect these metrics.

Name Port

longhorn-csi-attacher

8000

longhorn-csi-provisioner

8000

longhorn-csi-resizer

8000

longhorn-csi-snapshotter

8000

The metrics provided by the CSI sidecar component are provided in a histogram format. For example, you can obtain metrics observing the time it takes to create a Longhorn Volume for the PVC.

csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="0.1"} 0
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="0.25"} 0
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="0.5"} 0
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="1"} 0
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="2.5"} 3
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="5"} 3
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="10"} 3
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="15"} 9
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="25"} 9
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="50"} 9
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="120"} 9
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="300"} 9
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="600"} 9
csi_sidecar_operations_seconds_bucket{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume",le="+Inf"} 9
csi_sidecar_operations_seconds_sum{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume"} 66.816478825
csi_sidecar_operations_seconds_count{driver_name="driver.longhorn.io",grpc_status_code="OK",method_name="/csi.v1.Controller/ControllerPublishVolume"} 9