This is unreleased documentation for SUSE® Storage 1.12 (Dev).

Requirements

Installation Methods

SUSE Storage can be installed on a Kubernetes cluster in several ways:

For air gapped environments, refer to Air Gap Installation.

For customizing SUSE Storage’s default settings, refer to Customizing Default Settings.

For deploying SUSE Storage on specific nodes and rejecting general workloads for those nodes, refer to Taints and Tolerations.

Installation Requirements

Unless otherwise noted, the requirements in this section apply to both the V1 and V2 Data Engines. Engine-specific requirements are listed in V1 Data Engine Requirements and V2 Data Engine Requirements.

Each node in the Kubernetes cluster where SUSE Storage is installed must fulfill the following requirements:

  • A container runtime compatible with Kubernetes (Docker v1.13+, containerd v1.3.7+, etc.)

  • Kubernetes v1.25 or later.

  • RWX support requires that each node has a NFSv4 client installed.

  • bash, curl, findmnt, grep, awk, blkid, lsblk must be installed.

  • Mount propagation must be enabled.

The SUSE Storage workloads must be able to run as root for SUSE Storage to be deployed and operated properly.

The Longhorn Command Line Tool can be used to check the Longhorn environment for potential issues.

For the minimum recommended hardware, refer to the best practices guide.

Platform Specific Configuration

You must perform additional setups before using SUSE Storage with certain operating systems and distributions.

  • Google Kubernetes Engine (GKE). See GKE.

  • K3s clusters. See K3s.

Kubernetes Version

Use the following command to check your Kubernetes server version:

kubectl version

Result:

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.10", GitCommit:"b8609d4dd75c5d6fba4a5eaa63a5507cb39a6e99", GitTreeState:"clean", BuildDate:"2023-10-18T11:44:31Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.10+k3s2", GitCommit:"cb5cb5557f34e240e38c68a8c4ca2506c68b1d86", GitTreeState:"clean", BuildDate:"2023-11-08T03:21:46Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}

The Server Version should be greater than or equal to v1.25.

Pod Security Policy

SUSE Storage is shipped with a default Pod Security Policy that gives SUSE Storage the necessary privileges to be able to run properly.

No special configuration is needed for SUSE Storage to work properly on clusters with Pod Security Policy enabled.

Mount Propagation

If your Kubernetes cluster was provisioned by Rancher v2.0.7+ or later, the MountPropagation feature is enabled by default.

If MountPropagation is disabled, Base Image feature will be disabled.

Root and Privileged Permission

SUSE Storage components require root access with privileged permissions to achieve volume operations and management, because SUSE Storage relies on system resources on the host across different namespaces. For example, SUSE Storage uses nsenter to understand block devices' usage or encrypt or decrypt volumes on the host.

Below are the directories SUSE Storage components requiring access with root and privileged permissions :

  • Longhorn Manager

    • /boot: Get information about required modules from /boot/config-$(uname -r) on the host.

    • /dev: Block devices created by Longhorn are under the /dev path.

    • /proc: Find the recognized host process like container runtime, then use nsenter to access the mounts on the host to understand disks usage.

    • /var/lib/longhorn: The default path for storing volume data on a host.

  • Longhorn Engine Image

    • /var/lib/longhorn/engine-binaries: The default path for storing the Longhorn engine binaries.

  • Longhorn Instance Manager

    • /: Access any data path on this node and access Longhorn engine binaries.

    • /dev: Block devices created by Longhorn are under the /dev path.

    • /proc: Find the recognized host process like container runtime, then use nsenter to manage iSCSI targets and initiators, also some file system

  • Longhorn Share Manager

    • /dev: Block devices created by Longhorn are under the /dev path.

    • /lib/modules: Kernel modules required by cryptsetup for volume encryption.

    • /proc: Find the recognized host process like container runtime, then use nsenter for volume encryption.

    • /sys: Support volume encryption by cryptsetup.

  • Longhorn CSI Plug-in

    • /: For host checks via the NFS customer mounter (deprecated). Note: this is going be removed in the future release.

    • /dev: Block devices created by Longhorn are under the /dev path.

    • /lib/modules: Kernel modules required by Longhorn CSI plug-in.

    • /sys: Support volume encryption by cryptsetup.

    • /var/lib/kubelet/plugins/kubernetes.io/csi: The path where the Longhorn CSI plug-in creates the staging path (via NodeStageVolume) of a block device. The staging path is bind-mounted to the target path /var/lib/kubelet/pods (via NodePublishVolume) for support single volume could be mounted to multiple Pods.

    • /var/lib/kubelet/plugins_registry: The path where the node-driver-registrar registers the CSI plug-in with kubelet.

    • /var/lib/kubelet/plugins/driver.longhorn.io: The path where the socket for the communication between kubelet Longhorn CSI driver.

    • /var/lib/kubelet/pods: The path where the Longhorn CSI driver mounts volume from the target path (via NodePublishVolume).

  • Longhorn CSI Attacher/Provisioner/Resizer/Snapshotter

    • /var/lib/kubelet/plugins/driver.longhorn.io: The path where the socket for the communication between kubelet Longhorn CSI driver.

  • Longhorn Backing Image Manager

    • /var/lib/longhorn: The default path for storing data on the host.

  • Longhorn Backing Image Data Source

    • /var/lib/longhorn: The default path for storing data on the host.

  • Longhorn System Restore Rollout

    • /var/lib/longhorn/engine-binaries: The default path for storing the Longhorn engine binaries.

In rare cases, it may be required to modify the installed SELinux policy to get SUSE Storage working. If you are running an up-to-date version of a Fedora downstream distribution (for example, Fedora, RHEL, Rocky, CentOS, etc.) and plan to leave SELinux enabled, see Troubleshooting: Volume Attachment Fails Due to SELinux Denials for details.

Install NFSv4 client

In Longhorn system, backup feature requires NFSv4, v4.1 or v4.2, and ReadWriteMany (RWX) volume feature requires NFSv4.1. Before installing NFSv4 client userspace daemon and utilities, make sure the client kernel support is enabled on each Longhorn node.

  • Check if NFSv4 support is enabled in the kernel:

    cat /boot/config-`uname -r`| grep CONFIG_NFS_V4

  • Check if NFSv4.1 support is enabled in the kernel:

    cat /boot/config-`uname -r`| grep CONFIG_NFS_V4_1

  • Check if NFSv4.2 support is enabled in the kernel:

    cat /boot/config-`uname -r`| grep CONFIG_NFS_V4_2

The command used to install a NFSv4 client differs depending on the Linux distribution.

You can also use the Longhorn Command Line Tool to install nfs-client automatically.

These steps only verify that the kernel supports NFSv4, v4.1, or v4.2. To verify the NFS version in use, run mount | grep nfs or nfsstat -m to confirm the mounted version. Using the correct NFS version is required for backup and RWX volume features in SUSE Storage.

Install Cryptsetup and LUKS

Cryptsetup is an open-source utility used to conveniently set up dm-crypt based device-mapper targets and Longhorn uses Linux Unified Key Setup (LUKS2) format that is the standard for Linux disk encryption to support volume encryption.

The command used to install the cryptsetup tool differs depending on the Linux distribution.

  • For Debian and Ubuntu, use this command:

    apt-get install cryptsetup

  • For RHEL, CentOS, Rocky Linux and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use this command:

    yum install cryptsetup

  • For SUSE and OpenSUSE, use this command:

    zypper install cryptsetup

Install Device Mapper Userspace Tool

The device mapper is a framework provided by the Linux kernel for mapping physical block devices onto higher-level virtual block devices. It forms the foundation of the dm-crypt disk encryption and provides the linear dm device on the top of V2 volume. The device mapper is typically included by default in many Linux distributions. Some lightweight or highly customized distributions or a minimal installation of a distribution might exclude it to save space or reduce complexity

The command used to install the device mapper differs depending on the Linux distribution.

  • For Debian and Ubuntu, use this command:

    apt-get install dmsetup

  • For RHEL, CentOS, Rocky Linux and EKS with EKS Kubernetes Worker AMI with AmazonLinux2 image, use this command:

    yum install device-mapper

  • For SUSE/OpenSUSE, use this command:

    zypper install device-mapper

V1 Data Engine Requirements

This is the default installation path for SUSE Storage. If you have completed the Installation Requirements and installed SUSE Storage using the Installation Methods, SUSE Storage runs with the V1 Data Engine by default.

In this default mode:

  • Volumes use the V1 Data Engine.

  • Volume data is stored on file system-type disks.

  • SUSE Storage uses the default file system-type disk at /var/lib/longhorn unless you configure additional file system-type disks.

No additional data-engine-specific steps are required after installing SUSE Storage.

Installing open-iscsi

The command used to install open-iscsi differs depending on the Linux distribution.

For GKE, we recommend using Ubuntu as the guest OS image since it contains open-iscsi already.

You may need to edit the cluster security group to allow SSH access.

  • SUSE and openSUSE:

    zypper install open-iscsi
systemctl enable iscsid
systemctl start iscsid

  • Debian and Ubuntu:

    apt-get install open-iscsi

  • RHEL, CentOS, and EKS (EKS Kubernetes Worker AMI with AmazonLinux2 image):

    yum --setopt=tsflags=noscripts install iscsi-initiator-utils
echo "InitiatorName=$(/sbin/iscsi-iname)" > /etc/iscsi/initiatorname.iscsi
systemctl enable iscsid
systemctl start iscsid

Please ensure that the iscsi_tcp module has been loaded before the iscsid service starts. Generally, it should be automatically loaded along with the package installation.

modprobe iscsi_tcp

On SUSE and openSUSE, the iscsi_tcp module is included only in the kernel-default package. If the kernel-default-base package is installed on your system, you must replace it with kernel-default.

We also provide an iscsi installer to make it easier for users to install open-iscsi automatically. You can use the Longhorn CLI to install the prerequisites.

V2 Data Engine Requirements

This section is for clusters that uses the V2 Data Engine. Complete the shared Installation Requirementsfirst, then prepare each V2 node and enable the V2 Data Engine.

SUSE Storage’s V2 Data Engine utilizes the Storage Performance Development Kit (SPDK) to deliver enhanced performance with lower I/O latency and higher IOPS and throughput.

Before you enable the V2 Data Engine, ensure that each Longhorn node that is going to host V2 volumes meets the following requirements:

  • AMD64 or ARM64 CPU

    • AMD64 CPUs require SSE4.2 instruction support.

  • Linux kernel 6.7 or later for NVMe/TCP support and better stability.

  • Required kernel modules:

    • vfio_pci

    • uio_pci_generic

    • nvme-tcp

  • Huge page support:

    • 2 GiB of 2 MiB-sized pages on each Longhorn node

  • Raw block disks for V2 volumes

    • Local NVMe disks are strongly recommended for best performance.

When the V2 Data Engine is enabled, each V2 instance-manager pod typically consumes one dedicated CPU core because the spdk_tgt process uses intensive polling.

After confirming these prerequisites, configure the V2 environment on each node and then enable the V2 Data Engine in SUSE Storage.

IOMMU Group Isolation Requirement

For the V2 Data Engine (SPDK) to claim a disk, the NVMe device must be isolatable. Because SPDK uses vfio-pci, the following hardware constraints apply:

  • VFIO must claim the entire IOMMU group.

  • If a group contains multiple devices, all devices in that group must be bound to VFIO.

  • The Linux kernel does not allow binding a PCIe bridge or switch port to vfio-pci.

If your hardware topology places an NVMe device in the same IOMMU group as its parent PCIe bridge, SPDK cannot initialize the device. In such cases, the disk must be used in AIO mode instead of the SPDK NVMe path.

Load Kernel Modules

For Debian and Ubuntu, install Linux kernel extra modules before loading the required kernel modules:

apt install -y linux-modules-extra-`uname -r`

To configure the necessary kernel modules and huge pages for SPDK, you can use the Longhorn CLI.

Or, load the required modules manually on each Longhorn node:

modprobe vfio_pci
modprobe uio_pci_generic
modprobe nvme-tcp

To avoid reloading these modules after every reboot, configure your operating system to load them automatically during boot.

Enable HugePages

SPDK uses huge pages for performance and memory efficiency. You must configure 2 MiB-sized huge pages on each Longhorn node. Specifically, 1024 pages must be available on each node, which is equivalent to 2 GiB in total.

To allocate huge pages temporarily:

echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
Configure Huge Pages Persistently

Huge page allocations made through /sys/kernel/mm/hugepages/…​ are not persistent across reboots. To make the allocation persistent, configure the kernel boot parameters.

  1. Update /etc/default/grub and append the required huge page parameters:

    GRUB_CMDLINE_LINUX="hugepagesz=2M hugepages=1024"

  2. Apply the GRUB configuration:

    • BIOS systems:

      sudo update-grub

    • RHEL or SUSE with GRUB2:

      sudo grub2-mkconfig -o /boot/grub2/grub.cfg

    • UEFI systems:

      sudo grub2-mkconfig -o /boot/efi/EFI/<distro>/grub.cfg

  3. Reboot the node:

    sudo reboot

  4. Verify the huge pages:

    grep Huge /proc/meminfo

    Expected output:

    HugePages_Total:    1024
Hugepagesize:       2048 kB

  5. Verify the Kubernetes node resources:

    kubectl describe node <node-name>

    Expected in Capacity and Allocatable:

    hugepages-2Mi: 2Gi

Restart kubelet

Restart kubelet on each node so Kubernetes can detect and use the configured huge pages.

Enable V2 Data Engine

After SUSE Storage is installed, enable the V2 Data Engine by changing the v2-data-engine setting to true.

You can do this in the SUSE Storage UI under Settings > V2 Data Engine.

After the setting is enabled, the instance-manager pods are automatically restarted.

When the V2 Data Engine is enabled, each instance-manager pod for the V2 Data Engine typically consumes one dedicated CPU core because the spdk_tgt process uses intensive polling.

Add block-type Disks in Longhorn Nodes

Unlike filesystem-type disks that are designed for legacy volumes, volumes using the V2 Data Engine are persistent on block-type disks. Therefore, nodes that host V2 volumes must provide block-type disks. For more information about block-type disks, see Block-Type Disks.

Longhorn Command Line Tool

You can use longhornctl to check and install prerequisites for either the default V1 installation path or the V2 Data Engine installation path.

Download longhornctl

Download the longhornctl binary for your platform:

The longhornctl tool is a CLI for Longhorn operations. For more information, see Command Line Tool (longhornctl).

# For AMD64 platform
curl -sSfL -o longhornctl https://github.com/longhorn/cli/releases/download/v{{< current-version >}}/longhornctl-linux-amd64
# For ARM platform
curl -sSfL -o longhornctl https://github.com/longhorn/cli/releases/download/v{{< current-version >}}/longhornctl-linux-arm64

chmod +x longhornctl

Check Prerequisites

Use the base preflight check for the default V1 installation path. If you plan to use the V2 Data Engine, run the same check with --enable-spdk to validate the additional V2 requirements.

For V1 Data Engine:

./longhornctl check preflight

Example of result:

./longhornctl check preflight
INFO[2024-01-01T00:00:01Z] Initializing preflight checker
INFO[2024-01-01T00:00:01Z] Cleaning up preflight checker
INFO[2024-01-01T00:00:01Z] Running preflight checker
INFO[2024-01-01T00:00:02Z] Retrieved preflight checker result:
worker1:
  info:
  - Service iscsid is running
  - NFS4 is supported
  - Package nfs-common is installed
  - Package open-iscsi is installed
  warn:
  - multipathd.service is running. Please refer to https://longhorn.io/kb/troubleshooting-volume-with-multipath/ for more information.
worker2:
  info:
  - Service iscsid is running
  - NFS4 is supported
  - Package nfs-common is not installed
  - Package open-iscsi is installed

For V2 Data Engine:

./longhornctl check preflight --enable-spdk

This command validates, among other things:

  • CPU instruction requirements.

  • HugePages availability.

  • SPDK-related kernel modules.

  • Base SUSE Storage dependencies such as open-iscsi and NFS support.

Install Prerequisites

Use the base install command for the default V1 installation path. If you plan to use the V2 Data Engine, run the install command with --enable-spdk to install the additional V2 prerequisites.

For V1 Data Engine:

longhornctl --kubeconfig ~/.kube/config --image longhornio/longhorn-cli:v{{< current-version >}} install preflight

Example of result:

INFO[2025-03-11T08:17:57+08:00] Initializing preflight installer
INFO[2025-03-11T08:17:57+08:00] Cleaning up preflight installer
INFO[2025-03-11T08:17:57+08:00] Running preflight installer
INFO[2025-03-11T08:17:57+08:00] Installing dependencies with package manager
INFO[2025-03-11T08:18:28+08:00] Installed dependencies with package manager
INFO[2025-03-11T08:18:28+08:00] Cleaning up preflight installer
INFO[2025-03-11T08:18:28+08:00] Completed preflight installer. Use 'longhornctl check preflight' to check the result.

Some immutable Linux distributions, such as SUSE Linux Enterprise Micro (SLE Micro), require you to reboot worker nodes after running the install sub-command. After the reboot, you must run the install sub-command again to complete the operation.

The documentation of the Linux distribution you are using should outline such requirements. For example, the SLE Micro documentation explains how all changes made by the transactional-update command become active only after the node is rebooted.

For V2 Data Engine:

longhornctl --kubeconfig ~/.kube/config --image longhornio/longhorn-cli:v{{< current-version >}} install preflight --enable-spdk

After installation, run the check command again to verify that all V2-related prerequisites are correctly configured.