Documentation survey

Disaster Recovery Volumes

Ensuring data resilience is important when working with containerized applications. A Longhorn Disaster Recovery (DR) volume is a special type of volume designed to maintain a standby copy of data in a secondary Kubernetes cluster. It is created from backups of a primary volume and kept in sync to enable rapid recovery if the primary cluster becomes unavailable.

The DR volume stores a geographically separated replica of the data. The backup frequency determines how current the DR volume is and, consequently, the potential amount of data loss in the event of a site failure.

How it works

The functionality of DR volumes relies on asynchronous replication through a shared backup store.

Shared backup target

Your primary and secondary Kubernetes clusters must be configured to use the exact same external backup target (for example, an S3-compatible object store or an NFS share).

Incremental backup and restore

A DR volume is created from an existing backup. It continuously polls the backup target for newer backups from the source volume and restores them incrementally. The Last Backup field in the UI shows the most recent backup that has been restored.

To keep the DR volume updated, configure recurring jobs on the source volume to perform regular incremental backups. These recurring backups provide the DR volume with new backups to restore, helping ensure minimal data loss in the event of a disaster.

Standby state

The DR volume remains in a passive standby state. It is not mounted or accessible by any workloads, which prevents data inconsistencies. The UI indicates the status of the volume with an icon:

  • Gray Icon: The volume is busy restoring data and cannot be activated.

  • Blue Icon: The volume is fully synchronized and ready for activation.

Activation

In a disaster, you manually activate the DR volume. This process converts it into a standard, writable Longhorn volume that you can attach to your applications in the recovery cluster.

Creating a DR Volume

You can create a DR volume using either the SUSE Storage UI or kubectl.

Set up two Kubernetes clusters, referred to as Cluster A and Cluster B. Install SUSE Storage on both clusters, and configure the same backup target on both. For assistance with setting the backup target, refer to configure backup target page.

Using SUSE Storage UI

  1. In your primary cluster, ensure the source volume has at least one backup.

  2. In the SUSE Storage UI of your secondary (recovery) cluster, navigate to the Backup page.

  3. Select the desired backup from the list and choose Create Disaster Recovery Volume. We recommend using the same name as the original volume.

  4. SUSE Storage will create the volume, which will appear on the Volume page with a Standby status.

Using kubectl command

  1. Get the Backup URL: First, copy the full URL of the source backup from the Backup page in the SUSE Storage UI. The format of this URL depends on your configured backup target (for example, S3 or NFS).

  2. Create a YAML Manifest: Create a file (for example, dr-volume.yaml) with the following content. Replace the placeholder URL and adjust the name, size, accessMode, etc., to match your source volume. In this file, the standby: true field defines the volume as a DR standby volume.

apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: example-dr-volume
  namespace: longhorn-system
spec:
  size: "2147483648"
  accessMode: rwo
  numberOfReplicas: 3
  fromBackup: "nfs://longhorn-nfs-server.example.com:/opt/backupstore?backup=backup-b69a1249e97f4a27&volume=pvc-33509786-92d7-427c-9b5a-b6d61d56b063"
  # This flag is essential to create a standby volume
  Standby: true

  1. Apply the Manifest: Apply the manifest to your secondary cluster to create the volume.

Activating a DR Volume

When a failover is necessary, activate the DR volume to make it writable.

SUSE Storage supports activation under the following conditions:

When the Allow Volume Creation with Degraded Availability setting is disabled, attempting to activate a degraded DR volume causes the volume to become stuck in the Attached state.

After enabling the setting, the DR volume will activate and convert into a normal volume, remaining in the Detached state.

Using SUSE Storage UI

  1. Go to the Volume page in the SUSE Storage UI of your secondary cluster.

  2. Select the DR volume you want to activate.

  3. Click the Activate Disaster Recovery Volume button in the Operation dropdown menu.

  4. The volume will transition to the Detached state, and you can attach it with your workloads.

Using kubectl command

  1. Run the following command to activate the DR volume and update the frontend:

kubectl patch volume example-dr-volume1 -n longhorn-system --type='json' -p='[
  {"op": "replace", "path": "/spec/Standby", "value": false},
  {"op": "replace", "path": "/spec/frontend", "value": "blockdev"}
]'

  1. The volume will transition to the Detached state, and you can attach it with your workloads.

Limitations

Because the primary purpose of a DR volume is to restore data from backups, the following actions are not supported until the volume is activated:

  • Creating, deleting, or reverting snapshots

  • Creating backups

  • Creating persistent volumes (PVs)

  • Creating persistent volume claims (PVCs)