This is unreleased documentation for SUSE® Storage 1.12 (Dev).

Multiple Disks

SUSE Storage supports using more than one disk on the nodes to store the volume data.

Disk Types

SUSE Storage supports two types of disks:

  • Filesystem-type disk (diskType: filesystem): It is used by the V1 Data Engine. The disk must be formatted with an extent-based file system (for example, Ext4 or xfs) and mounted to a directory on the host.

  • Block-type disk (diskType: block): It is used by the V2 Data Engine. The disk is used as a raw block device without a file system.

Each disk type is tied to its corresponding data engine:

  • V1 volumes are scheduled only to file system-type disks.

  • V2 volumes are scheduled only to block-type disks.

Default Disk Behavior

When a node is first added to SUSE Storage, the system creates a default file system-type disk at /var/lib/longhorn. This default disk is used for V1 volumes unless you disable scheduling for it or add other file system-type disks.

Add a Disk

Add a File System-Type Disk

File system-type disks are used for V1 volumes. Before adding one to SUSE Storage, you must prepare and mount it on the host.

  1. Choose a Disk: Select the physical or virtual disk for SUSE Storage storage and format it with an extent-based file system (for example, Ext4 or XFS).

  2. Mount the Disk: Mount the disk to a directory on the host, such as /mnt/example-disk. Ensure the directory is accessible and correctly configured.

After the disk is mounted, you can add it to SUSE Storage by using either the UI or the kubectl command-line tool.

Using the SUSE Storage UI

  1. Go to the Nodes tab, select a node, and choose Edit Disks from the dropdown menu.

  2. Add the disk’s mount path to the disk list.

Using kubectl

  1. Run kubectl edit node.longhorn.io <node-name> to modify the Longhorn node resource.

  2. Add the disk path to spec.disks. For example:

    ...
spec:
  ...
  disks:
    ...
    example-disk:
      allowScheduling: true
      diskDriver: ""
      diskType: filesystem
      evictionRequested: false
      path: /mnt/example-disk
      storageReserved: 0
      tags: []
...

  3. Save and exit the editor.

Once a disk is added:

  • SUSE Storage automatically detects the disk’s storage details, such as maximum and available capacity.

  • If the disk is suitable for storing volume data, SUSE Storage begins scheduling volumes to it.

  1. You cannot add a disk path that another SUSE Storage disk already uses.

  2. SUSE Storage uses the file system ID to detect duplicate mounts. Therefore, you cannot add a disk with the same file system ID as another disk on the same node.

    For more information, see issue #2477.

Use an Alternative Path for a File System-Type Disk on the Node

If you prefer to use a different path for a file system-type disk (rather than the original mount point), you can use mount --bind to create an alternative path. Do not use symbolic link (lh -s), as these are not properly resolved inside Longhorn pods.

Make sure the alternative path is remounted after a node reboot, for example, by adding it to /etc/fstab.

Add a Block-Type Disk

Block-type disks are used for V2 volumes. The V2 Data Engine requires raw block devices for direct access.

SUSE Storage supports several ways to manage block-type disks on a node:

  • NVMe disks: It is managed with the nvme driver.

  • VirtIO disks: It is managed with the virtio-blk driver.

  • Other block devices: It is managed with the aio driver.

Prerequisites

  • The V2 Data Engine must be enabled. See V2 Data Engine Quick Start.

  • SUSE Storage prevents adding block disks that contain an existing file system or partition table. Clean the disk first:

    wipefs -a /path/to/block/device

Steps

You can add a block-type disk using the SUSE Storage UI or kubectl.

  • Using the SUSE Storage UI:

    1. Go to the Nodes tab, select a node, and choose Edit Disks from the dropdown menu.

    2. Add the block device path and set the disk type to Block.

  • Using kubectl Command:

    1. Run kubectl edit node.longhorn.io <node-name> to edit the Longhorn node resource.

    2. Add the block disk to spec.disks. For example:

      ...
spec:
  ...
  disks:
    ...
    block-disk:
      allowScheduling: true
      diskDriver: auto
      diskType: block
      evictionRequested: false
      path: /dev/sdb
      storageReserved: 0
      tags: []
...

    3. Save and exit the editor.

Disk Driver

The diskDriver field controls how SUSE Storage accesses the block device. Setting it to auto allows SUSE Storage to detect the appropriate driver automatically:

  • NVMe disks: It uses the nvme driver. The path can be a BDF (Bus Device Function) notation (for example, 0000:05:00.0).

  • VirtIO disks: It uses the virtio-blk driver. The path can also be a BDF notation.

  • Other disks: It falls back to the aio (Linux AIO) driver. The path should be a standard device path (for example, /dev/sdb).

After the disk is added, check node.status.diskStatus to confirm that SUSE Storage detected the disk without errors and assigned the expected diskDriver.

When a disk is managed by the nvme or virtio-blk driver, it is no longer exposed through the original Linux kernel device node such as /dev/nvmeXnY or /dev/vdX.
Using AIO Disks

When a block device is neither NVMe nor VirtIO, or when you want to force Linux AIO, configure it with diskDriver: aio and use a standard device path.

aio-disk:
  allowScheduling: true
  diskDriver: aio
  diskType: block
  evictionRequested: false
  path: /dev/sdb
  storageReserved: 0
  tags: []

  1. Block-type disks are exclusively used by the V2 Data Engine and cannot be used for V1 volumes.

  2. The disk must be clean and must not contain an existing file system or partition table before it is added. Use wipefs -a /path/to/block/device to clean it.

  3. For disk-specific configuration examples, see Using NVMe Disks, Using VirtIO Disks, and Using AIO Disks.
Using NVMe Disks

To let SUSE Storage use the nvme driver, add the disk by its BDF instead of the Linux device path.

  1. Identify the available disks on the node. For example:

    ls -al /sys/block/

    Example output:

    lrwxrwxrwx  1 root root 0 Jul 30 12:20 loop0 -> ../devices/virtual/block/loop0
lrwxrwxrwx  1 root root 0 Jul 30 12:20 nvme0n1 -> ../devices/pci0000:00/0000:00:01.2/0000:02:00.0/nvme/nvme0/nvme0n1
lrwxrwxrwx  1 root root 0 Jul 30 12:20 nvme1n1 -> ../devices/pci0000:00/0000:00:01.2/0000:05:00.0/nvme/nvme1/nvme1n1

  2. Find the BDF of the target NVMe disk. In the example above, the BDF of /dev/nvme1n1 is 0000:05:00.0.

  3. Add the disk to spec.disks. For example:

    nvme-disk:
  allowScheduling: true
  diskType: block
  diskDriver: auto
  evictionRequested: false
  path: 0000:05:00.0
  storageReserved: 0
  tags: []

If you instead add the same disk using /dev/nvme1n1, SUSE Storage treats it as an aio disk instead of an nvme disk.

Using VirtIO Disks

VirtIO disks follow the same pattern. To let SUSE Storage use the virtio-blk driver, add the disk by its BDF.

  1. Identify the available disks on the node. For example:

    ls -al /sys/block/

    Example output:

    lrwxrwxrwx  1 root root 0 Feb 22 14:04 vda -> ../devices/pci0000:00/0000:00:02.3/0000:04:00.0/virtio2/block/vda
lrwxrwxrwx  1 root root 0 Feb 22 14:24 vdb -> ../devices/pci0000:00/0000:00:02.6/0000:07:00.0/virtio5/block/vdb

  2. Find the BDF of the target VirtIO disk. In the example above, the BDF of /dev/vdb is 0000:07:00.0.

  3. Add the disk to spec.disks. For example:

    virtio-disk:
  allowScheduling: true
  diskType: block
  diskDriver: auto
  evictionRequested: false
  path: 0000:07:00.0
  storageReserved: 0
  tags: []

If you instead add the same disk using /dev/vdb, SUSE Storage treats it as an aio disk instead of a virtio-blk disk.

Root Disk Reservation

You can use the Space Reserved field in the UI or spec.disks.<disk-name>.storageReserved to reserve a portion of any disk’s space (in bytes) for other purposes. This reserved space will not be used by SUSE Storage for volume data.

To maintain node stability when compute resources (for example, memory or disk) are under pressure, the kubelet requires some space to remain free. If these critical resources are exhausted, it can lead to node instability.

For the default file system-type disk at /var/lib/longhorn, SUSE Storage reserves 30% of the root disk space by default to help prevent issues such as DiskPressure conditions from the kubelet, particularly after scheduling multiple volumes. This behavior is controlled by the storage-reserved-percentage-for-default-disk setting.

This automatic reservation applies only to the default file system-type disk created for V1 volumes. Block-type disks for V2 volumes must be configured explicitly, including any storageReserved value that you want to apply.

Remove a Disk

Nodes and disks can be excluded from future scheduling. Note that any storage already scheduled on a node would not be automatically released when scheduling is disabled for that node.

To remove a disk:

  1. Disable scheduling for the disk.

  2. Ensure that no replicas or backing images remain on the disk, including any in an error state. For instructions on how to evict replicas from disabled disks, see Select Disks or Nodes for Eviction.

Once the disk is empty and scheduling is disabled, you can safely remove it from the node configuration.

Disk-Scheduling Configuration

There are two global settings that affect disk scheduling.

These settings apply to both file system-type and block-type disks. However, the V1 Data Engine uses sparse files for thin provisioning, so over-provisioning behavior is commonly discussed in the context of file system-type disks.

  • StorageOverProvisioningPercentage defines the maximum total storage that can be scheduled on a disk, relative to its usable capacity. The formula is:

ScheduledStorage / (MaximumStorage - ReservedStorage)

By default, this setting is 100%.

For example, on a 200 GiB disk with 50 GiB reserved, SUSE Storage considers 150 GiB of usable space. With the default setting, SUSE Storage can schedule up to 150 GiB of volume data.

Since workloads typically do not consume the entire allocated volume size, increasing this setting can help optimize disk utilization. For V1 volumes, higher values are commonly used because the V1 Data Engine uses sparse files for thin provisioning.

  • StorageMinimalAvailablePercentage specifies the minimum percentage of free space that must remain on a disk to schedule new replicas. The formula is:

AvailableStorage / MaximumStorage

By default, this setting is 25%.

For example, for a 200 GiB disk with 50 GiB reserved, SUSE Storage stops scheduling new replicas if available space falls below 37.5 GiB (25% of 150 GiB). A new volume also will not be scheduled if its size would push available space below that limit.

This setting helps prevent disks from becoming too full, which could lead to scheduling failures or volume operation issues.

SUSE Storage currently cannot fully enforce the StorageMinimalAvailablePercentage limit in all scenarios because:

  1. Longhorn volumes might use more space than their requested size, especially when snapshots are taken.

  2. SUSE Storage allows over-provisioning by default.