Run SUSE Observability on Longhorn

This page describes the Longhorn decisions that are specific to SUSE Observability. Use it together with the Longhorn documentation for installation, capacity planning, monitoring, backup targets, and version-specific procedures.

Before installation

Use SSD-backed disks that meet the SUSE Observability system requirements. Provide at least three eligible Longhorn storage nodes and spread them across the failure domains that the installation must tolerate. Configure Longhorn replica node and zone anti-affinity so that this separation is enforced, rather than only preferred. Keep enough free capacity to rebuild all replicas from one failed or drained node.

Do not use Longhorn replicas as a replacement for backups. Configure SUSE Observability backups in external object storage and test the restore procedure.

StorageClass strategy

Use two Longhorn replicas by default for durable SUSE Observability volumes in both Non-HA and HA installations. This follows the Longhorn I/O performance best practices, which recommend two replicas for data availability with lower disk-space usage and less impact on system performance, especially for data-intensive applications. After one replica-host failure, a two-replica volume has only one healthy copy until Longhorn completes the rebuild; do not continue maintenance while any volume is degraded.

An HA installation can reduce replication traffic, and improve write performance, by using one Longhorn replica for selected services that already maintain independent application-level replicas. This optimization makes the loss of one storage node a service-specific recovery operation: the application must rebuild the lost member after its PVC is replaced. Use it only when that procedure has been tested.

Keep the HA profile’s default hard pod anti-affinity enabled when using one-replica Longhorn volumes. It spreads pods of each stateful service across different nodes. Pod anti-affinity and Longhorn replica anti-affinity protect different layers and don’t replace each other. Best-effort data locality does not guarantee that each volume is on its pod’s node: verify that different application members' one-replica volumes also occupy independent storage nodes. Ensure that the cluster has enough eligible nodes and spare capacity to satisfy the scheduling rules during maintenance; otherwise, replacement pods remain pending. See Configure Kubernetes affinities.
Strategy Example StorageClass Longhorn replicas Data locality Use

Default durable

longhorn-durable-2

2

Disabled

Use for all durable Non-HA and HA data unless applying the optional HA overrides below. Keep single-replica ClickHouse, HDFS NameNodes, VictoriaMetrics, Workload Observer, AI Assistant SQLite data, anomaly-detection manager data, settings, backups, StackPacks, and other state without a tested peer-recovery procedure on this class.

Application-replicated HA

longhorn-ha-database

1

Best effort

HA installations only: Elasticsearch members, HDFS DataNodes, Kafka brokers, ZooKeeper members, and replicated ClickHouse with verified application replication and volume placement. ClickHouse has additional prerequisites below. Losing the volume loses that member and requires application-level recovery.

Transient

longhorn-transient

1

Best effort

Temporary buffers and working data, such as vmagent and temporary component PVCs. Data in these volumes can be lost during a node failure.

The mixed-class examples below require SUSE Observability 2.11.1 or later. That release includes the Elasticsearch StorageClass override and Kafka and ZooKeeper per-component precedence over global.storageClass. On earlier releases, use the global durable class and do not apply the mixed-class HA examples below.

The following example creates the durable, HA database, and transient classes. Adapt the names and other Longhorn parameters to your environment.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-durable-2
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "2"
  dataLocality: disabled
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-ha-database
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "1"
  dataLocality: best-effort
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-transient
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  numberOfReplicas: "1"
  dataLocality: best-effort

Set the two-replica durable class as the default for the chart:

global:
  storageClass: longhorn-durable-2

Changing the chart’s StorageClass value doesn’t migrate existing PVCs.

For the optional HA optimization, after meeting the release and recovery requirements above, add these overrides:

elasticsearch:
  volumeClaimTemplate:
    storageClassName: longhorn-ha-database

hbase:
  hdfs:
    datanode:
      persistence:
        storageClass: longhorn-ha-database

kafka:
  persistence:
    storageClass: longhorn-ha-database

zookeeper:
  persistence:
    storageClass: longhorn-ha-database

stackstate:
  components:
    checks:
      tmpToPVC:
        storageClass: longhorn-transient
    healthSync:
      tmpToPVC:
        storageClass: longhorn-transient
    state:
      tmpToPVC:
        storageClass: longhorn-transient
    sync:
      tmpToPVC:
        storageClass: longhorn-transient
    vmagent:
      persistence:
        storageClass: longhorn-transient

Replicated ClickHouse

The 4000-ha sizing profile runs ClickHouse with multiple replicas. For this profile, use the HA database class when each shard has at least two healthy replicas of its application data:

clickhouse:
  persistence:
    storageClass: longhorn-ha-database

Keep ClickHouse on longhorn-durable-2 when it has only one replica per shard, including the default 150-ha, 250-ha, and 500-ha profiles.

Optional component transaction-log PVCs

Component transaction-log PVCs store debugging data. If you enable this feature, use longhorn-transient:

stackstate:
  features:
    storeTransactionLogsToPVC:
      enabled: true
      storageClass: longhorn-transient

These PVCs do not inherit global.storageClass; the setting above applies only to component transaction logs. Most other component PVCs, including AI Assistant SQLite storage, inherit global.storageClass. See Storage classes and customization for the complete set of storage settings. After installation, verify that every PVC uses the intended class and that every Longhorn volume is healthy.

Understand the failure modes

Failure Likely effect Protection

One node or disk holding a replica fails

A two-replica volume becomes degraded but remains available, with no remaining storage redundancy until it rebuilds. A one-replica volume becomes unavailable and its data is lost if the disk cannot be recovered.

Spread replicas across independent nodes and failure domains. Restore the replica count before starting maintenance elsewhere.

Failures exceed the available replica count

Volumes become unavailable and may require restore. SUSE Observability components that depend on them become unavailable.

Maintain failure-domain separation and keep tested backups outside the cluster.

A one-replica application member is lost

The remaining application replicas may continue serving, but capacity or quorum is reduced. The lost PVC and member must be replaced and rebuilt.

Use this layout only with the HA profile, monitor application health, and test the component recovery procedure.

A disk fills or there is no destination for rebuilding

Longhorn cannot restore the requested replica count. A second failure can then cause unavailability or data loss.

Reserve rebuild capacity, alert on disk pressure and degraded volumes, and stop node maintenance until all volumes are healthy.

A PVC is deleted

With the Delete reclaim policy, the Longhorn volume can also be deleted.

Restrict destructive access and rely on external backups for recovery.

The cluster or an entire failure domain is lost

Local replicas and snapshots can become unreachable together.

Place replicas according to the required failure boundary and store SUSE Observability backups outside that boundary.

Maintain or rotate nodes

This guidance applies regardless of Kubernetes distribution or cluster management platform.

HA maintenance limitation

The current database liveness and readiness probes do not verify that application-level replication has recovered after a node disruption. Pod or node Ready status, a successful drain, and satisfied PodDisruptionBudgets are not sufficient checks. Healthy Longhorn volumes do not establish that application-level replication is complete.

Operate on one node hosting stateful SUSE Observability workloads or Longhorn replicas at a time. Before each node, confirm that:

  • SUSE Observability is healthy.

  • Application replication checks pass for HDFS, Elasticsearch, Kafka, ClickHouse, and ZooKeeper. Use Check replication status to check the deployed databases.

  • All Longhorn volumes are healthy and have their expected replica count.

  • Other nodes have enough storage and compute capacity for the workloads and replica rebuilds.

  • A recent external backup is available.

Automated Kubernetes upgrades, node rotations, and node restarts must perform these checks before disrupting the next node. Integrate sts-backup replication check --wait into the workflow as described below; SUSE Observability does not add this step automatically. Stop further node disruptions if a check fails or times out. If the workflow cannot perform these checks, pause it between nodes and verify recovery before continuing; a fixed delay is not a substitute.

Check replication status

The replication command runs read-only checks and does not require enabled backups, backup credentials, or changes to GitOps reconciliation. Replace observability with the namespace containing your SUSE Observability installation. No Helm release name is required; the checker expects one installation per namespace.

Starting with SUSE Observability 2.11.2, HA sizing profiles enable a separate replication-checker Deployment by default. It uses no persistent storage and runs checks only when invoked. Non-HA profiles and installations without a sizing profile leave it disabled. If you retain older generated sizing files, regenerate them or set stackstate.components.replicationChecker.enabled: true explicitly. Use it through kubectl without installing sts-backup locally:

kubectl -n observability get deployments -l app.kubernetes.io/component=replication-checker
kubectl -n observability exec deployment/<replication-checker-deployment> -c replication-checker -- \
  sts-backup replication check --namespace observability

Replace <replication-checker-deployment> with the Deployment name from the first command. The checker has a dedicated service account with namespace-scoped permissions to list Pods and StatefulSets and create pods/exec requests. Your Kubernetes identity needs access to get the Deployment, find its Pods, and exec into the checker container. Although the checker runs read-only queries, its service account’s pods/exec permission permits arbitrary commands in namespace Pods. Set stackstate.components.replicationChecker.enabled: false to remove the Deployment and its dedicated RBAC resources. An explicit true or false overrides the built-in profile default.

Alternatively, install sts-backup v0.10.0 or later; see Download the backup CLI. The local CLI uses the current kubeconfig context; add --kubeconfig <path> to select another configuration. Its Kubernetes identity needs permission to list Pods and StatefulSets and create pods/exec requests in the installation namespace.

sts-backup replication check --namespace observability

The report includes each database’s status and diagnostic messages:

Status Meaning

healthy

The applicable replication checks passed.

degraded

Replication or workload availability has not recovered. Inspect the reported component before continuing maintenance.

unknown

The checker could not verify the state, for example because of permissions, a timeout, or an unsupported configuration. Resolve the reported cause and rerun.

not_applicable

The deployed configuration does not require that replication check, for example a single-replica database or HBase mono. Only Kubernetes availability is checked; this does not establish data redundancy.

Wait for recovery during maintenance

Run the wait command before the first node disruption and after each node returns or replacement capacity is ready to schedule workloads. For an in-place restart, uncordon the returned node first. Keep the configured database replica counts unchanged and allow only one maintenance workflow to operate on the installation at a time.

The following shell step waits for recovery and stops the workflow on failure:

if ! kubectl -n observability exec deployment/<replication-checker-deployment> -c replication-checker -- \
  sts-backup replication check \
  --namespace observability \
  --wait --stable-for 30s --timeout 15m \
  --output json > replication.json; then
  printf '%s\n' 'Replication could not be verified. Stop node maintenance and inspect replication.json.' >&2
  exit 1
fi

For a locally installed CLI, run sts-backup replication check with the same arguments instead of the kubectl exec command. The Pod must be available; if it is rescheduling or an exec session is interrupted, wait for a replacement Pod and rerun the complete check successfully before continuing maintenance. Progress goes to stderr; replication.json contains the final report. Exit code 0 means every selected component is healthy or not_applicable; a failed check, timeout, or cancellation returns nonzero. Configure the maintenance system to honor that exit code and run its remaining checks before starting the next node.

With applicable replication checks, --wait requires successful observations spanning the --stable-for period. An unhealthy observation or a relevant database topology, readiness, or restart change resets that period. After stability, the checker performs a final HDFS file/block audit when applicable and rechecks health before returning success. If all components are not_applicable, it returns after the first successful availability check. Choose --timeout for your recovery window, including the final HDFS audit and subsequent checks.

If a database is intentionally disabled, use --components to select the databases you actually run. Omitted components are not checked; do not exclude a failing database to continue maintenance. The v0.10.0 checker supports the chart’s standard database layouts and containers. Custom images or container names, overridden database labels, external databases, alternative NameNode topologies, erasure-coded HDFS, Kafka SASL/TLS, Elasticsearch HTTPS, and ZooKeeper TLS-only or nonstandard quorum layouts are unsupported. See the replication checker documentation for the full requirements and limitations. If a check returns unknown, resolve the cause before continuing automated maintenance.

A successful report describes observed replication; it does not reserve a maintenance window or certify that a particular node can be removed. Continue checking Longhorn volume health and placement, spare capacity, application health, and backups separately. The checker does not verify HBase region recovery, durability of the latest active WAL writes, VictoriaMetrics redundancy, or quorum survival after removing the next node.

Configure Longhorn node draining

The official Longhorn node maintenance and drain policy guide is the source of truth for maintenance procedures and drain policy behavior. The following commands are examples for planning a procedure; verify them against the documentation for your installed Longhorn version and test the complete procedure before using it in production.

Longhorn drain policies act when a node is cordoned, even if it is not subsequently drained. Change the policy before cordoning the first node and restore the previous value after maintenance. Record the original value in your maintenance log and keep the same shell open; capture it only once per maintenance window.

Short restart or in-place upgrade

For a short restart, operating-system update, or in-place Kubernetes upgrade, Block For Eviction If Contains Last Replica automatically relocates the last healthy replica of a volume. This allows one-replica volumes to drain while avoiding the evacuation of every replica on the node.

For example:

LONGHORN_DRAIN_POLICY="$(kubectl -n longhorn-system get settings.longhorn.io node-drain-policy -o jsonpath='{.value}')"
: "${LONGHORN_DRAIN_POLICY:?Could not read the original Longhorn drain policy. Stop maintenance.}"
printf 'Original Longhorn drain policy: %s\n' "$LONGHORN_DRAIN_POLICY"

kubectl -n longhorn-system patch settings.longhorn.io node-drain-policy --type=merge -p '{"value":"block-for-eviction-if-contains-last-replica"}'

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --timeout=0

Then:

  1. If the drain is blocked, inspect Longhorn events and the replicas on the node. Do not force the drain past Longhorn’s data-protection checks.

  2. Complete the maintenance and return the node.

  3. Uncordon the node so workloads can be scheduled, then run the replication wait command and verify Longhorn has restored the expected replica count before proceeding to the next node.

allow-if-replica-is-stopped avoids rebuilding a one-replica volume when the same node and disk are expected to return, but the volume is lost if they do not return. Use that policy only when this risk is explicitly accepted.

Permanent node removal or replacement

  1. Add replacement capacity before removing capacity.

  2. Temporarily set the Longhorn node drain policy to Block For Eviction.

  3. Cordon and drain the old node. Longhorn automatically requests eviction and blocks the drain until all replicas have moved.

  4. Confirm that no volumes remain attached, then remove the node from Longhorn and Kubernetes.

  5. Ensure replacement capacity can schedule workloads, run the replication wait command, and verify Longhorn has restored the expected replica count before removing another node.

For example:

LONGHORN_DRAIN_POLICY="$(kubectl -n longhorn-system get settings.longhorn.io node-drain-policy -o jsonpath='{.value}')"
: "${LONGHORN_DRAIN_POLICY:?Could not read the original Longhorn drain policy. Stop maintenance.}"
printf 'Original Longhorn drain policy: %s\n' "$LONGHORN_DRAIN_POLICY"

kubectl -n longhorn-system patch settings.longhorn.io node-drain-policy --type=merge -p '{"value":"block-for-eviction"}'

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --timeout=0

Follow the Longhorn replica eviction procedure. Eviction rebuilds data over the storage network and can take a long time; schedule it as a storage-intensive maintenance operation.

After the maintenance window, restore the previous policy. If you changed shells, first set LONGHORN_DRAIN_POLICY to the original value recorded in your maintenance log; reading the current setting would capture the temporary policy instead. If the original value is unavailable, confirm the intended policy with your cluster administrator before restoring it:

: "${LONGHORN_DRAIN_POLICY:?Set LONGHORN_DRAIN_POLICY to the original value recorded before maintenance.}"
kubectl -n longhorn-system patch settings.longhorn.io node-drain-policy --type=merge -p '{"value":"'"${LONGHORN_DRAIN_POLICY}"'"}'

Do not use Always Allow in production.

Operational checks

Monitor degraded or faulted Longhorn volumes, unschedulable replicas, disk pressure, node readiness, and replica rebuild failures. Treat a degraded volume as reduced fault tolerance, not as a normal steady state.

Before and after maintenance, list the relevant resources:

kubectl -n longhorn-system get nodes.longhorn.io
kubectl -n longhorn-system get volumes.longhorn.io
kubectl -n <suse-observability-namespace> get pvc

Run the replication wait command before proceeding to the next node, and confirm Longhorn has restored the expected replica count and the remaining maintenance checks pass. Pod readiness alone does not establish that the databases have finished rebuilding their data.