Settings

Define the policy to use when a node with the last healthy replica of a volume is drained. Available options:

Each option has benefits and drawbacks. See Node Drain Policy Recommendations for help deciding which is most appropriate in your environment.

Longhorn will automatically detach volumes that are manually attached to the nodes which are cordoned. This prevent the draining process stuck by the PDB of instance-manager which still has running engine on the node.

Longhorn will generate system snapshot during replica rebuild, and if a user doesn’t setup a recurring snapshot schedule, all the system generated snapshots would be left in the replica, and user has to delete them manually, this setting allow Longhorn to automatically cleanup system generated snapshot before and after replica rebuild.

If enabled, when running a recurring backup job, Longhorn takes a new snapshot before creating the backup. Longhorn retains only the snapshot used by the last backup job even if the value of the retain parameter is not 1.

If disabled, this setting ensures that the retained snapshots directly correspond to the backups on the remote backup target.

If enabled, Longhorn will automatically delete the workload pod that is managed by a controller (e.g. deployment, statefulset, daemonset, etc…​) when Longhorn volume is detached unexpectedly (e.g. during Kubernetes upgrade, Docker reboot, or network disconnect). By deleting the pod, its controller restarts the pod and Kubernetes handles volume reattachment and remount.

If disabled, Longhorn will not delete the workload pod that is managed by a controller. You will have to manually restart the pod to reattach and remount the volume.

If enabled, volumes will be automatically salvaged when all the replicas become faulty e.g. due to network disconnection. Longhorn will try to figure out which replica(s) are usable, then use them for the volume.

This setting controls how Longhorn automatically upgrades volumes' engines to the new default engine image after upgrading Longhorn manager. The value of this setting specifies the maximum number of engines per node that are allowed to upgrade to the default engine image at the same time. If the value is 0, Longhorn will not automatically upgrade volumes' engines to default version.

This setting controls how many volumes on a node can restore the backup concurrently.

Longhorn blocks the backup restore once the restoring volume count exceeds the limit.

Set the value to 0 to disable backup restore.

If no other disks exist, create the default disk automatically, only on nodes with the Kubernetes label node.longhorn.io/create-default-disk=true .

If disabled, the default disk will be created on all new nodes when the node is detected for the first time.

This option is useful if you want to scale the cluster but don’t want to use storage on the new nodes, or if you want to customize disks for Longhorn nodes.

The current customer resource’s API version, e.g. longhorn.io/v1beta2. Set by manager automatically.

We say a Longhorn volume has data locality if there is a local replica of the volume on the same node as the pod which is using the volume. This setting specifies the default data locality when a volume is created from the Longhorn UI. For Kubernetes configuration, update the dataLocality in the StorageClass

The available modes are:

Default path to use for storing data on a host.

Can be used with Create Default Disk on Labeled Nodes option, to make Longhorn only use the nodes with specific storage mounted at, for example, /opt/longhorn when scaling the cluster.

The default engine image used by the manager. Can be changed on the manager starting command line only.

Every Longhorn release will ship with a new Longhorn engine image. If the current Longhorn volumes are not using the default engine, a green arrow will show up, indicate this volume needs to be upgraded to use the default engine.

The storageClassName is for persistent volumes (PVs) and persistent volume claims (PVCs) when creating PV/PVC for an existing Longhorn volume. Notice that it’s unnecessary for users to create the related StorageClass object in Kubernetes since the StorageClass would only be used as matching labels for PVC bounding purpose. By default 'longhorn-static'.

The default number of replicas when creating the volume from Longhorn UI. For Kubernetes, update the numberOfReplicas in the StorageClass

The recommended way of choosing the default replica count is: if you have three or more nodes for storage, use 3; otherwise use 2. Using a single replica on a single node cluster is also OK, but the high availability functionality wouldn’t be available. You can still take snapshots/backups of the volume.

This flag protects Longhorn from unexpected uninstallation which leads to data loss. Set this flag to true to allow Longhorn uninstallation. If this flag is false, the Longhorn uninstallation job will fail.

Allows engine controller and engine replica to disable revision counter file update for every data write. This improves the data path performance. See Revision Counter for details.

Upgrade Checker will check for a new Longhorn version periodically. When there is a new version available, it will notify the user in the Longhorn UI.

The latest version of Longhorn available. Automatically updated by the Upgrade Checker.

Enabling this setting will allow Longhorn to provide valuable usage metrics to https://metrics.longhorn.io/.

This information will help us gain insights how Longhorn is being used, which will ultimately contribute to future improvements.

Node Information collected from all cluster nodes includes:

Cluster Information collected from one of the cluster nodes includes:

Defines the Longhorn action when a Volume is stuck with a StatefulSet/Deployment Pod on a node that is down.

The Kubernetes Secret name.

When there is at least one failed replica volume in a degraded volume, this interval in seconds determines how long Longhorn will wait at most in order to reuse the existing data of the failed replicas rather than directly creating a new replica for this volume.

This setting defines the Image Pull Policy of Longhorn system managed pods, e.g. instance manager, engine image, CSI driver, etc.

Notice that the new Image Pull Policy will only apply after the system managed pods restart.

This setting definition is exactly the same as that of in Kubernetes. Here are the available options:

This interval in minutes determines how long Longhorn will wait before cleaning up the backing image file when there is no replica in the disk using it.

The interval in seconds determines how long Longhorn will wait before re-downloading the backing image file when all disk files of this backing image become failed or unknown.

The default minimum number of backing image copies Longhorn maintains.

The value in seconds specifies the timeout of the engine to the replica(s), and the value should be between 8 to 30 seconds.

Longhorn uses the support bundle manager image to generate the support bundles.

There will be a default image given during installation and upgrade. You can also change it in the settings.

An example of the support bundle manager image:

This setting specifies how many failed support bundles can exist in the cluster.

The retained failed support bundle is for analysis purposes and needs to clean up manually.

Longhorn blocks support bundle creation when reaching the upper bound of the limitation. You can set this value to 0 to have Longhorn automatically purge all failed support bundles.

Number of minutes Longhorn allows for collection of node information and node logs for the support bundle.

If the collection process is not completed within the allotted time, Longhorn continues generating the support bundle without the uncollected node data.

The setting enables fast replica rebuilding feature. It relies on the checksums of snapshot disk files, so setting the snapshot-data-integrity to enable or fast-check is a prerequisite.

The value in seconds specifies the timeout of the HTTP client to the replica’s file sync server used for replica rebuilding, volume cloning, snapshot cloning, etc.

Number of seconds that Longhorn allows for the completion of replica rebuilding and snapshot cloning operations.

Setting that allows you to enable the V1 Data Engine.

Enable improved ReadWriteMany volume HA by shortening the time it takes to recover from a node failure.

Setting that allows you to enable the V2 Data Engine, which is based on the Storage Performance Development Kit (SPDK). The V2 Data Engine is a preview feature and should not be used in production environments. For more information, see V2 Data Engine (Preview Feature).

Maximum huge page size (in MiB) for the V2 Data Engine.

Number of millicpus on each node to be reserved for each instance manager pod when the V2 Data Engine is enabled. By default, the Storage Performance Development Kit (SPDK) target daemon within each instance manager pod uses 1 CPU core. Configuring a minimum CPU usage value is essential for maintaining engine and replica stability, especially during periods of high node workload.

This setting allows users to enable or disable snapshot hashing and data integrity checking. Available options are:

Hashing snapshot disk files impacts the performance of the system. The immediate snapshot hashing and checking can be disabled to minimize the impact after creating a snapshot.

Unix-cron string format. The setting specifies when Longhorn checks the data integrity of snapshot disk files.

Maximum snapshot count for a volume. The value should be between 2 to 250.

This setting only applies to volumes with the Kubernetes volume mode Filesystem. When enabled, Longhorn freezes the volume’s filesystem immediately before creating a user-initiated snapshot. When disabled or when the Kubernetes volume mode is Block, Longhorn instead attempts a system sync before creating a user-initiated snapshot.

Snapshots created when this setting is enabled are more likely to be consistent because the filesystem is in a consistent state at the moment of creation. However, under very heavy I/O, freezing the filesystem may take a significant amount of time and may cause workload activity to pause.

When this setting is disabled, all data is flushed to disk just before the snapshot is created, but Longhorn cannot completely block write attempts during the brief interval between the system sync and snapshot creation. I/O is not paused during the system sync, so workloads likely do not notice that a snapshot is being created.

The default option for this setting is false because kernels with version v5.17 or earlier may not respond correctly when a volume crashes while a freeze is ongoing. This is not likely to happen but if it does, an affected kernel will not allow you to unmount the filesystem or stop processes using the filesystem without rebooting the node. Only enable this setting if you plan to use kernels with version 5.17 or later, and ext4 or XFS filesystems.

You can override this setting (using the field freezeFilesystemForSnapshot) for specific volumes through the Longhorn UI, a StorageClass, or direct changes to an existing volume. freezeFilesystemForSnapshot accepts the following values:

This setting allows Longhorn to automatically delete the orphan resource and its orphaned data like volume replica.

If this setting is enabled, Longhorn automatically attaches the volume and takes snapshot/backup when it is the time to do recurring snapshot/backup.

Endpoint used to access a backupstore. Longhorn supports AWS S3, Azure, GCP, CIFS and NFS. See Setting a Backup Target for details.

The Kubernetes secret associated with the backup target. See Setting a Backup Target for details.

The interval in seconds to poll the backup store for updating volumes' Last Backup field. Set to 0 to disable the polling. See Setting up Disaster Recovery Volumes for details.

For more information on how the backupstore poll interval affects the recovery time objective and recovery point objective, refer to the concepts section.

The interval in minutes to keep the backup resource that was failed. Set to 0 to disable the auto-deletion.

Failed backups will be checked and cleaned up during backupstore polling which is controlled by Backupstore Poll Interval setting. Hence this value determines the minimal wait interval of the cleanup. And the actual cleanup interval is multiple of Backupstore Poll Interval. Disabling Backupstore Poll Interval also means to disable failed backup auto-deletion.

This setting specifies how many failed backup or snapshot job histories should be retained.

History will not be retained if the value is 0.

This setting specifies how many successful backup or snapshot job histories should be retained.

History will not be retained if the value is 0.

This setting allows restoring the recurring jobs of a backup volume from the backup target during a volume restoration if they do not exist on the cluster. This is also a volume-specific setting with the below options. Users can customize it for each volume to override the global setting.

This setting allows users to specify backup compression method.

This setting controls how many worker threads per backup concurrently.

This setting controls how many worker threads per restore concurrently.

This setting allows user to create and attach a volume that doesn’t have all the replicas scheduled at the time of creation.

When this setting is checked, the Longhorn Manager will not schedule replicas on Kubernetes cordoned nodes.

When this setting is un-checked, the Longhorn Manager will schedule replicas on Kubernetes cordoned nodes.

When this setting is checked, the Longhorn Manager will allow scheduling on nodes with existing healthy replicas of the same volume.

When this setting is un-checked, Longhorn Manager will forbid scheduling on nodes with existing healthy replicas of the same volume.

When this setting is checked, the Longhorn Manager will allow scheduling new replicas of a volume to the nodes in the same zone as existing healthy replicas.

When this setting is un-checked, Longhorn Manager will forbid scheduling new replicas of a volume to the nodes in the same zone as existing healthy replicas.

When this setting is checked, the Longhorn Manager will allow scheduling new replicas of a volume to the same disks as existing healthy replicas.

When this setting is un-checked, Longhorn Manager will forbid scheduling new replicas of a volume to the same disks as existing healthy replicas.

Enable this setting automatically rebalances replicas when discovered an available node.

The available global options are:

Longhorn also supports customizing for individual volume. The setting can be specified in UI or with Kubernetes manifest volume.spec.replicaAutoBalance, this overrules the global setting. The available volume spec options are:

Percentage of currently used storage that triggers automatic replica rebalancing.

When the threshold is reached, Longhorn automatically rebuilds replicas that are under disk pressure on another disk within the same node.

To disable this setting, set the value to 0.

This setting takes effect only when the following conditions are met:

This setting is not affected by Replica Node Level Soft Anti_Affinity, which can prevent Longhorn from rebuilding a replica on the same node. Regardless of that setting’s value, this setting still allows Longhorn to attempt replica rebuilding on a different disk on the same node for migration purposes.

With the default setting of 25, the Longhorn Manager will allow scheduling new replicas only after the amount of disk space has been subtracted from the available disk space (Storage Available) and the available disk space is still over 25% of actual disk capacity (Storage Maximum). Otherwise the disk becomes unschedulable until more space is freed up.

See Multiple Disks Support for details.

The over-provisioning percentage defines the amount of storage that can be allocated relative to the hard drive’s capacity.

By increase this setting, the Longhorn Manager will allow scheduling new replicas only after the amount of disk space has been added to the used disk space (storage scheduled), and the used disk space (Storage Maximum - Storage Reserved) is not over the over-provisioning percentage of the actual usable disk capacity.

It’s worth noting that a volume replica may require more storage space than the volume’s actual size, as the snapshots also require storage. You can regain space by deleting unnecessary snapshots.

The reserved percentage specifies the percentage of disk space that will not be allocated to the default disk on each new Longhorn node.

This setting only affects the default disk of a new adding node or nodes when installing Longhorn.

This setting allows replica of the volume without node selector to be scheduled on node with tags.

This setting allows replica of the volume without disk selector to be scheduled on disk with tags.

This setting controls how many replicas on a node can be rebuilt simultaneously.

Typically, Longhorn can block the replica starting once the current rebuilding count on a node exceeds the limit. But when the value is 0, it means disabling the replica rebuilding.

This setting controls how many backing image copies on a node can be replenished simultaneously.

Typically, Longhorn can block the backing image copy starting once the current replenishing count on a node exceeds the limit. But when the value is 0, it means disabling the backing image replenish.

If you want to dedicate nodes to just store Longhorn replicas and reject other general workloads, you can set tolerations for all Longhorn components and add taints to the nodes dedicated for storage.

Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.) This setting only sets taint tolerations for system managed components. Depending on how you deployed Longhorn, you need to set taint tolerations for user deployed components in Helm chart or deployment YAML file.

To apply the modified toleration setting immediately, ensure that all Longhorn volumes are detached. When volumes are in use, Longhorn components are not restarted, and you need to reconfigure the settings after detaching the remaining volumes; otherwise, you can wait for the setting change to be reconciled in an hour. We recommend setting tolerations during Longhorn deployment because the Longhorn system cannot be operated during the update.

Multiple tolerations can be set here, and these tolerations are separated by semicolon. For example:

By default, Longhorn workloads run with the same priority as other pods in the cluster, meaning in cases of node pressure, such as a node running out of memory, Longhorn workloads will be at the same priority as other Pods for eviction.

The Priority Class setting will specify a Priority Class for the Longhorn workloads to run as. This can be used to set the priority for Longhorn workloads higher so that they will not be the first to be evicted when a node is under pressure.

Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.).

Note that this setting only sets Priority Class for system managed components. Depending on how you deployed Longhorn, you need to set Priority Class for user deployed components in Helm chart or deployment YAML file.

See Priority Class for details.

If you want to restrict Longhorn components to only run on a particular set of nodes, you can set node selector for all Longhorn components.

Longhorn system contains user deployed components (e.g, Longhorn manager, Longhorn driver, Longhorn UI) and system managed components (e.g, instance manager, engine image, CSI driver, etc.) You need to set node selector for both of them. This setting only sets node selector for system managed components. Follow the instruction at Node Selector to change node selector.

Setting the Kubernetes Cluster Autoscaler Enabled to true allows Longhorn to unblock the Kubernetes Cluster Autoscaler scaling.

See Kubernetes Cluster Autoscaler Support for details.

The storage network uses Multus NetworkAttachmentDefinition to segregate the in-cluster data traffic from the default Kubernetes cluster network.

By default, the this setting applies only to RWO (Read-Write-Once) volumes. For RWX (Read-Write-Many) volumes, see Storage Network for RWX Volume Enabled setting.

See Storage Network for details.

This setting allows Longhorn to use the storage network for RWX volumes.

For more information, see Storage Network.

This setting allows Longhorn filesystem trim feature to automatically mark the latest snapshot and its ancestors as removed and stops at the snapshot containing multiple children.

Since Longhorn filesystem trim feature can be applied to the volume head and the followed continuous removed or system snapshots only.

Notice that trying to trim a removed files from a valid snapshot will do nothing but the filesystem will discard this kind of in-memory trimmable file info. Later on if you mark the snapshot as removed and want to retry the trim, you may need to unmount and remount the filesystem so that the filesystem can recollect the trimmable file info.

See Trim Filesystem for details.

Percentage of the total allocatable CPU resources on each node to be reserved for each instance manager pod when the V1 Data Engine is enabled. For example, Longhorn reserves 10% of the total allocatable CPU resources if you specify a value of 10. This setting is essential for maintaining engine and replica stability, especially during periods of high node workload.

In order to prevent an unexpected volume instance (engine/replica) crash as well as guarantee a relatively acceptable I/O performance, you can use the following formula to calculate a value for this setting:

The result of above calculation doesn’t mean that’s the maximum CPU resources the Longhorn workloads require. To fully exploit the Longhorn volume I/O performance, you can allocate/guarantee more CPU resources via this setting.

If it’s hard to estimate the usage now, you can leave it with the default value, which is 12%. Then you can tune it when there is no running workload using Longhorn volumes.

When set to true, temporarily prevent all attempts to purge volume snapshots.

Longhorn typically purges snapshots during replica rebuilding and user-initiated snapshot deletion. While purging, Longhorn coalesces unnecessary snapshots into their newer counterparts, freeing space consumed by historical data.

Allowing snapshot purging during normal operations is ideal, but this process temporarily consumes additional disk space. If insufficient disk space prevents the process from continuing, consider temporarily disabling purging while data is moved to other disks.

When set to true, the snapshot used by the backup will be automatically cleaned up when the backup is deleted.