17 Managing multipath I/O for devices #

The traditional, generic implementation of multipathing under Linux uses the device mapper framework. For most device types like SCSI devices, device mapper multipathing is the only available implementation. Device mapper multipath is highly configurable and flexible.

The Linux NVM Express (NVMe) kernel subsystem implements multipathing natively in the kernel. This implementation creates less computational overhead for NVMe devices, which are typically fast devices with very low latencies. Native NVMe multipathing requires no user space component. For details, refer to Section 16.2.4, “Multipathing”.

This chapter documents device mapper multipath and its user-space component, multipath-tools. multipath-tools also has limited support for native NVMe multipathing (see Section 17.13, “Miscellaneous options”).

Device mapper multipath is a generic technology. Multipath device detection requires only that the low-level (for example, SCSI) devices are detected by the kernel, and that device properties reliably identify multiple low-level devices as being different “paths” to the same volume rather than actually different devices.

The multipath-tools package detects storage arrays by their vendor and product names. It provides built-in configuration defaults for a large variety of storage products. Consult the hardware documentation of your storage array: some vendors provide specific recommendations for Linux multipathing configuration.

If you need to apply changes to the built-in configuration for your storage array, read Section 17.8, “Multipath configuration”.

Important: Disclaimer about built-in hardware properties

multipath-tools has built-in presets for many storage arrays. The existence of such presets for a given storage product does not imply that the vendor of the storage product has tested the product with dm-multipath, nor that the vendor endorses or supports the use of dm-multipath with the product. Always consult the original vendor documentation for support-related questions.

Some storage arrays require special commands for failover from one path to the other, or non-standard error-handling methods. These special commands and methods are implemented by hardware handlers in the Linux kernel. Modern SCSI storage arrays support the “Asymmetric Logical Unit Access” (ALUA) hardware handler defined in the SCSI standard. Besides ALUA, the SLE kernel contains hardware handlers for Netapp E-Series (RDAC), the Dell/EMC CLARiiON CX family of arrays, and legacy arrays from HP.

Since Linux kernel 4.4, the Linux kernel has automatically detected hardware handlers for most arrays, including all arrays supporting ALUA. The only requirement is that the device handler modules are loaded at the time the respective devices are probed. The multipath-tools package ensures this by installing appropriate configuration files. Once a device handler is attached to a given device, it cannot be changed anymore.

We distinguish installation types by the way the root device is handled. Section 17.4, “Installing SUSE Linux Enterprise Server on multipath systems” describes how the different setups are created during and after installation.

Use third-party SAN array management tools or the user interface of your storage array to create logical devices and assign them to hosts. Make sure to configure the host credentials correctly on both sides.

You can add or remove volumes to a running host, but detecting the changes may require rescanning SCSI targets and reconfiguring multipathing on the host. See Section 17.14.6, “Scanning for new devices without rebooting”.

Note: Storage processors

On some disk arrays, the storage array manages the traffic through storage processors. One processor is active and the other one is passive until there is a failure. If you are connected to the passive storage processor, you might not see the expected LUNs, or you might see the LUNs but encounter I/O errors when you try to access them.

If a disk array has more than one storage processor, ensure that the SAN switch has a connection to the active storage processor that owns the LUNs you want to access.

Multipathing is set up on top of basic storage devices such as SCSI disks. In a multi-layered storage stack, multipathing is always the bottom layer. Other layers such as software RAID, Logical Volume Management, block device encryption, etc. are layered on top of it. Therefore, for each device that has multiple I/O paths and that you plan to use in a software RAID, you must configure the device for multipathing before you attempt to create the software RAID device.

High-availability solutions for clustering storage resources run on top of the multipathing service on each node. Make sure that the configuration settings in the /etc/multipath.conf file on each node are consistent across the cluster.

Make sure that multipath devices have the same name across all devices. Refer to Section 17.12, “Multipath device names and WWIDs” for details.

The Distributed Replicated Block Device (DRBD) high-availability solution for mirroring devices across a LAN runs on top of multipathing. For each device that has multiple I/O paths and that you plan to use in a DRDB solution, you must configure the device for multipathing before you configure DRBD.

Special care must be taken when using multipathing together with clustering software that relies on shared storage for fencing, such as pacemaker with sbd. See Section 17.9.2, “Queuing policy on clustered servers” for details.

You may want to perform installation on a local disk, without configuring the fabric and the storage first, with the intention to add multipath SAN devices to the system later. In this case, the installation will proceed like on a non-multipath system. After installation, multipath-tools will be installed, but the systemd service multipathd.service will be disabled. The system will be configured as described in Multipath disabled in the initramfs in Section 17.3.2.2, “Root file system on a local disk”. Before adding SAN hardware, you will need to enable and start multipathd.service. We recommend creating a blacklist entry in the /etc/multipath.conf for the root device (see Section 17.11.1, “The blacklist section in multipath.conf”).

If multipath devices are connected to the system at installation time, YaST will detect them and display a pop-up window asking you whether multipath should be enabled before entering the partitioning stage.

If you select “No” at this prompt (not recommended), the installation will proceed as in Section 17.4.1, “Installing without connected multipath devices”. In the partitioning stage, do not use/edit devices that will later be part of a multipath map.

If you select “Yes” at the multipath prompt, multipathd will run during the installation. No device will be added to the blacklist section of /etc/multipath.conf, thus all SCSI and DASD devices, including local disks, will appear as multipath devices in the partitioning dialogs. After installation, all SCSI and DASD devices will be multipath devices, as described in Section 17.3.2.1, “Root file system on multipath (SAN-boot)”.

The Device Mapper Multipath (DM-MP) module dm-multipath.ko provides the generic multipathing capability for Linux. DM-MPIO is the preferred solution for multipathing on SUSE Linux Enterprise Server for SCSI and DASD devices, and can be used for NVMe devices as well.

Note: Using DM-MP for NVMe devices

To disable native NVMe multipathing and use device mapper multipath instead (not recommended), boot with the kernel parameter nvme-core.multipath=0.

The Device Mapper Multipath module handles the following tasks:

The following tasks are handled by the user-space components in the multipath-tools package, not by the Device Mapper Multipath module:

For details about the components from the multipath-tools package, refer to Section 17.6.2, “The multipathd daemon”.

Note: Failures that multipath prevents

DM-MPIO protects against failures in the paths to the device, and not failures in the device itself, such as media errors. The latter kind of errors must be prevented by other means, such as replication.

multipathd is the most important part of a modern Linux device mapper multipath setup. It is normally started through the systemd service multipathd.service (see Section 17.7.1, “Enabling, starting, and stopping multipath services”).

multipathd serves the following tasks (some of them depend on the configuration):

multipathd also serves as a command-line client to process interactive commands by sending them to the running daemon. The general syntax to send commands to the daemon is as follows:

tux > sudo multipathd COMMAND

or

tux > sudo multipathd -k'COMMAND'

There is also an interactive mode that allows sending multiple subsequent commands:

tux > sudo multipathd -k

Note: How multipath and multipathd work together

Many multipathd commands have multipath equivalents. For example, multipathd show topology does the same thing as multipath -ll. The notable difference is that the multipathd command inquires the internal state of the running multipathd daemon, whereas multipath obtains information directly from the kernel and I/O operations.

If the multipath daemon is running, we recommend making modifications to the system by using the multipathd commands. Otherwise, the daemon may notice configuration changes and react to them. In some situations, the daemon might even try to undo the applied changes. multipath automatically delegates certain possibly dangerous commands, like destroying and flushing maps, to multipathd if a running daemon is detected.

The list below describes frequently used multipathd commands:

Additional commands are available to modify path states, enable or disable queuing, and more. See multipathd(8) for details.

Even though multipath setup is mostly automatic and handled by multipathd, multipath is still useful for some administration tasks. Several examples of the command usage follow:

The option -v controls the verbosity of the output. The provided value overrides the verbosity option in /etc/multipath.conf. See Section 17.13, “Miscellaneous options”.

The mpathpersist utility is used to manage SCSI persistent reservations on Device Mapper Multipath devices. Persistent reservations serve to restrict access to SCSI Logical Units to certain SCSI initiators. In multipath configurations, it is important to use the same reservation keys for all I_T nexuses (paths) for a given volume; otherwise, creating a reservation on one path device would cause I/O errors on other paths.

Use this utility with the reservation_key attribute in the /etc/multipath.conf file to set persistent reservations for SCSI devices. If (and only if) this option is set, the multipathd daemon checks persistent reservations for newly discovered paths or reinstated paths.

You can add the attribute to the defaults section or the multipaths section of multipath.conf. For example:

multipaths {
    multipath {
        wwid             3600140508dbcf02acb448188d73ec97d
        alias            yellow
        reservation_key  0x123abc
    }
}

After setting the reservation_key parameter for all mpath devices applicable for persistent management, reload the configuration using multipathd reconfigure.

Note: Using “reservation_key file”

If the special value reservation_key file is used in the defaults section of multipath.conf, reservation keys can be managed dynamically in the file /etc/multipath/prkeys using mpathpersist.

This is the recommended way to handle persistent reservations with multipath maps.

Use the command mpathpersist to query and set persistent reservations for multipath maps consisting of SCSI devices. Refer to the manual page mpathpersist(8) for details. The command-line options are the same as those of the sg_persist from the sg3_utils package. The sg_persist(8) manual page explains the semantics of the options in detail.

In the following examples, DEVICE denotes a device mapper multipath device like /dev/mapper/mpatha. The commands below are listed with long options for better readability. All options have single-letter replacements, like in mpathpersist -oGS 123abc DEVICE.

To enable multipath services to start at boot time, run the following command:

tux > sudo systemctl enable multipathd

To manually start the service in the running system, enter:

tux > sudo systemctl start multipathd

To restart the service, enter:

tux > sudo systemctl restart multipathd

In most situations, restarting the service is not necessary. To simply have multipathd reload its configuration, run:

tux > sudo systemctl reload multipathd

To check the status of the service, enter:

tux > sudo systemctl status multipathd

To stop the multipath services in the current session, run:

tux > sudo systemctl stop multipathd multipathd.socket

Stopping the service does not remove existing multipath maps. To remove unused maps, run:

tux > sudo multipath -F

Warning: Keep multipathd.service enabled

We strongly recommend keeping multipathd.service always enabled and running on systems with multipath hardware. The service does support systemd's socket activation mechanism, but it is discouraged to rely on that. Multipath maps will not be set up during boot if the service is disabled.

Note: Disabling multipath

If you need to disable multipath despite the warning above, for example, because a third-party multipathing software is going to be deployed, proceed as follows. Be sure that the system uses no hard-coded references to multipath devices (see Section 17.15.2, “Understanding device referencing issues”).

To disable multipathing just for a single system boot, use the kernel parameter multipath=off. This affects both the booted system and the initramfs, which does not need to be rebuilt in this case.

To disable multipathd services permanently, so that they will not be started on future system boots, run the following commands:

tux > sudo systemctl disable multipathd multipathd.socket
tux > sudo dracut --force --omit multipath

(Whenever you disable or enable the multipath services, rebuild the initramfs. See Section 17.7.4, “Keeping the initramfs synchronized”.)

If you want to make sure multipath devices do not get set up, even when running multipath manually, add the following lines at the end of /etc/multipath.conf before rebuilding the initramfs:

blacklist {
    wwid .*
}

Before configuring multipath I/O for your SAN devices, prepare the SAN devices, as necessary, by doing the following:

If multipath devices are detected and multipathd.service is enabled, multipath maps should be created automatically. If this does not happen, Section 17.15.3, “Troubleshooting steps in emergency mode” lists some shell commands that can be used to examine the situation. When the LUNs are not seen by the HBA driver, check the zoning setup in the SAN. In particular, check whether LUN masking is active and whether the LUNs are correctly assigned to the server.

If the HBA driver can see LUNs, but no corresponding block devices are created, additional kernel parameters may be needed. See TID 3955167: Troubleshooting SCSI (LUN) Scanning Issues in the SUSE Knowledge base at https://www.suse.com/support/kb/doc.php?id=3955167.

Multipath maps can have partitions like their path devices. Partition table scanning and device node creation for partitions is done in user space by the kpartx tool. kpartx is automatically invoked by udev rules; there is usually no need to run it manually. See Section 17.12.4, “Referring to multipath maps” for ways to refer to multipath partitions.

Note: Disabling invocation of kpartx

The skip_kpartx option in /etc/multipath.conf can be used to disable invocation of kpartx on selected multipath maps. This may be useful on virtualization hosts, for example.

Partition tables and partitions on multipath devices can be manipulated as usual, using YaST or tools like fdisk or parted. Changes applied to the partition table will be noted by the system when the partitioning tool exits. If this does not work (usually because a device is busy), try multipathd reconfigure, or reboot the system.

Important

Make sure that the initial RAM file system (initramfs) and the booted system behave consistently regarding the use of multipathing for all block devices. Rebuild the initramfs after applying multipath configuration changes.

If multipathing is enabled in the system, it also needs to be enabled in the initramfs and vice versa. The only exception to this rule is the option Multipath disabled in the initramfs in Section 17.3.2.2, “Root file system on a local disk”.

The multipath configuration must be synchronized between the booted system and the initramfs. Therefore, if you change any of the files: /etc/multipath.conf, /etc/multipath/wwids, /etc/multipath/bindings, or other configuration files, or udev rules related to device identification, rebuild initramfs using the command:

tux > sudo dracut -f

If the initramfs and the system are not synchronized, the system will not boot properly, and the start-up procedure may result in an emergency shell. See Section 17.15, “Troubleshooting MPIO” for instructions on how to avoid or repair such a scenario.

tux > sudo dracut --force --add multipath

tux > sudo dracut --force --omit multipath

/dev/mapper/3600a098000aad73f00000a3f5a275dc8-part1

tux > sudo dracut --force --omit multipath --persistent-policy=by-uuid

It is recommended that you create a minimal /etc/multipath.conf that just contains those settings you want to change. In many cases, you do not need to create /etc/multipath.conf at all.

If you prefer working with a configuration template that contains all possible configuration directives, run:

multipath -T >/etc/multipath.conf

See also Section 17.14.1, “Best practices for configuration”.

The /etc/multipath.conf file uses a hierarchy of sections, subsections, and option/value pairs.

The following example illustrates the syntax:

section {
    subsection {
        option1 value
        option2      "complex value!"
        option3    "value with ""quoted"" word"
    } ! subsection end
} # section end

The /etc/multipath.conf file is organized into the following sections. Some options can occur in more than one section. See multipath.conf(5) for details.

To apply the configuration changes, run:

tux > sudo multipathd reconfigure

Do not forget to synchronize with the configuration in the initramfs. See Section 17.7.4, “Keeping the initramfs synchronized”.

Warning: Do not apply settings using multipath

Do not apply new settings with the multipath command while multipathd is running. This may result in an inconsistent and possibly broken setup.

Note: Verifying a modified setup

It is possible to test modified settings first before they are applied, by running:

tux > sudo multipath -d -v2

This command shows new maps to be created with the proposed topology, but not whether maps will be removed/flushed. To obtain more information, run with increased verbosity:

tux > sudo multipath -d -v3 2>&1 | less

When you configure multipath I/O for a stand-alone server, a no_path_retry setting with value queue protects the server operating system from receiving I/O errors as long as possible. It queues messages until a multipath failover occurs. If “infinite” queuing is not desired (see above), select a numeric value that is deemed high enough for the storage paths to recover under ordinary circumstances (see above).

When you configure multipath I/O for a node in a high-availability cluster, you want multipath to report the I/O failure to trigger the resource failover instead of waiting for a multipath failover to be resolved. In cluster environments, you must modify the no_path_retry setting so that the cluster node receives an I/O error in relation to the cluster verification process (recommended to be 50% of the heartbeat tolerance) if the connection is lost to the storage system. In addition, you want the multipath failback to be set to manual or followover to avoid a ping-pong of resources because of path failures.

The /etc/multipath.conf file can contain a blacklist section that lists all devices that should be ignored by multipathd and multipath. The following example illustrates possible ways of excluding devices:

blacklist {
    wwid 3600605b009e7ed501f0e45370aaeb77f 1
    device {  2
        vendor ATA
        product .*
    }
    protocol scsi:sas 3
    property SCSI_IDENT_LUN_T10 4
    devnode "!^dasd[a-z]*" 5
}

tux > sudo multipathd show paths format "%d %P"

By default, multipath-tools ignores all devices except SCSI, DASD, or NVMe. Technically, the built-in devnode exclude list is this negated regular expression:

    devnode !^(sd[a-z]|dasd[a-z]|nvme[0-9])

Sometimes it is desired to configure only very specific devices for multipathing. In this case, devices are excluded by default, and exceptions are defined for devices that should be part of a multipath map. The blacklist_exceptions section exists for this purpose. It is typically used like in the following example, which excludes everything except storage with product string “NETAPP”:

blacklist {
     wwid .*
}
blacklist_exceptions {
     device {
         vendor ^NETAPP$
         product .*
     }
}

The blacklist_exceptions section supports all methods described for the blacklist section above.

The property directive in blacklist_exceptions is mandatory because every device must have at least one of the “allowed” udev properties to be considered a path device for multipath (the value of the property does not matter). The built-in default for property is

    property (SCSI_IDENT_|ID_WWN)

Only devices that have at least one udev property matching this regular expression will be included.

Besides the blacklist options, there are several other settings in /etc/multipath.conf that affect which devices are considered multipath path devices.

It is crucial for multipath operation to reliably detect devices that represent paths to the same storage volume. multipath-tools uses the device's World Wide Identification (WWID) for this purpose (sometimes also referred to as Universally Unique ID (UUID) or Unique ID (UID — do not confuse with “User ID”). The WWID of a map device is always the same as the WWID of its path devices.

By default, WWIDs of path devices are inferred from udev properties of the devices, which are determined in udev rules, either by reading device attributes from the sysfs file system or by using specific I/O commands. To see the udev properties of a device, run:

tux > udevadm info /dev/sdx

The udev properties used by multipath-tools to derive WWIDs are:

Warning: Avoid changing WWIDs

It is impossible to change the WWID of a multipath map which is in use. If the WWID of mapped path devices changes because of a configuration change, the map needs to be destroyed, and a new map needs to be set up with the new WWID. This cannot be done while the old map is in use. In extreme cases, data corruption may result from WWID changes. It must therefore be strictly avoided to apply configuration changes that would cause map WWIDs to change.

It is allowed to enable the uid_attrs option in /etc/multipath.conf, see Section 17.13, “Miscellaneous options”.

Arbitrary map names can be set in the multipaths section of /etc/multipath.conf as follows:

multipaths {
    multipath {
        wwid 3600a098000aad1e3000064e45f2c2355
        alias postgres
    }
}

Aliases are expressive, but they need to be assigned to each map individually, which may be cumbersome on large systems.

multipath-tools also supports autogenerated aliases, so-called “user-friendly names”. The naming scheme of the aliases follows the pattern: mpathINDEX, where INDEX is a lower case letter (starting with a). So the first autogenerated alias is mpatha, the next one is mpathb, mpathc to mpathz. After that follows mpathaa, mpathab, and so on.

Map names are only useful if they are persistent. multipath-tools keeps track of the assigned names in the file /etc/multipath/bindings (the “bindings file”). When a new map is created, the WWID is first looked up in this file. If it is not found, the lowest available user-friendly name is assigned to it.

Explicit aliases as discussed in Section 17.12.2, “Setting aliases for multipath maps” take precedence over user-friendly names.

The following options in /etc/multipath.conf affect user-friendly names:

Warning: Map names in high-availability clusters

For cluster operations, device names must be identical across all nodes in the cluster. The multipath-tools configuration must be kept synchronized between nodes. If user_friendly_names is used, multipathd may modify the /etc/multipath/bindings file at runtime. Such modifications must be replicated dynamically to all nodes. The same applies to /etc/multipath/wwids (see Section 17.11.3, “Other options affecting device selection”).

Note: Changing map names at runtime

It is possible to change map names at runtime. Use any of the methods described in this section and run multipathd reconfigure, and the map names will change without disrupting the system operation.

Technically, multipath maps are Device Mapper devices, which have generic names of the form /dev/dm-n with an integer number n. These names are not persistent. They should never be used to refer to the multipath maps. udev creates various symbolic links to these devices, which are more suitable as persistent references. These links differ with respect to their invariance against certain configuration changes. The following typical example shows various symlinks all pointing to the same device.

/dev/disk/by-id/dm-name-mpathb1 -> ../../dm-1
/dev/disk/by-id/dm-uuid-mpath-3600a098000aad73f00000a3f5a275dc82 -> ../../dm-1
/dev/disk/by-id/scsi-3600a098000aad73f00000a3f5a275dc83 -> ../../dm-1
/dev/disk/by-id/wwn-0x600a098000aad73f00000a3f5a275dc84 -> ../../dm-1
/dev/mapper/mpathb5 -> ../dm-1

filter = [ "a|/dev/disk/by-id/dm-uuid-mpath-.*|", "r|.*|" ]

For partitions on multipath maps created by the kpartx tool, there are similar symbolic links, derived from the parent device name or WWID and the partition number:

/dev/disk/by-id/dm-name-mpatha-part2 -> ../../dm-5
/dev/disk/by-id/dm-uuid-part2-mpath-3600a098000aad1e300000b4b5a275d45 -> ../../dm-5
/dev/disk/by-id/scsi-3600a098000aad1e300000b4b5a275d45-part2 -> ../../dm-5
/dev/disk/by-id/wwn-0x600a098000aad1e300000b4b5a275d45-part2 -> ../../dm-5
/dev/disk/by-partuuid/1c2f70e0-fb91-49f5-8260-38eacaf7992b -> ../../dm-5
/dev/disk/by-uuid/f67c49e9-3cf2-4bb7-8991-63568cb840a4 -> ../../dm-5
/dev/mapper/mpatha-part2 -> ../dm-5

Note that partitions often have by-uuid links, too, referring not to the device itself but to the file system it contains. These links are often preferable. They are invariant even if the file system is copied to a different device or partition.

Warning: Map names in the initramfs

When dracut builds an initramfs, it creates hard-coded references to devices in the initramfs, using /dev/mapper/$MAP_NAME references by default. These hard-coded references will not be found during boot if the map names used in the initramfs do not match the names used during building the initramfs, causing boot failure. Normally this will not happen, because dracut will add all multipath configuration files to the initramfs. But problems can occur if the initramfs is built from a different environment, for example, in the rescue system or during an offline update. To prevent this boot failure, change dracut's persistent_policy setting, as explained in Section 17.7.4.2, “Persistent device names in the initramfs”.

Unstable conditions in the fabric can cause path devices to behave erratically. They exhibit frequent I/O errors, recover, and fail again. Such path devices are also denoted “marginal” or “shaky” paths. This section summarizes the options that multipath-tools provides to deal with this problem.

Note: multipathd's marginal path checking algorithm

If a path device exhibits a second failure (good → bad transition) before marginal_path_double_failed_time elapses after the first failure, multipathd starts monitoring the path at a rate of 10 requests per second, for a monitoring period of marginal_path_err_sample_time. If the error rate during the monitoring period exceeds marginal_path_err_rate_threshold, the path is classified as marginal. After marginal_path_err_recheck_gap_time, the path transitions to normal state again.

This algorithm is used if all four numeric marginal_path_ parameters are set to a positive value, and marginal_pathgroups is not set to fpin.

A simpler algorithm using the parameters san_path_err_threshold, san_path_err_forget_rate, and san_path_err_recovery time is also available. See the “Shaky paths detection” section in multipath.conf(5).

The large number of configuration directives is daunting at first. Usually, you can get good results with an empty configuration, unless you are in a clustering environment.

Here are some general recommendations for stand-alone servers. They are not mandatory. See the documentation of the respective parameters in the previous sections for background information.

defaults {
    deferred_remove      yes
    find_multipaths      yes
    enable_foreign       nvme
    
}
devices {
    # A catch-all device entry.
    device {
        vendor                .*
        product               .*
        dev_loss_tmo          infinity
        no_path_retry         60            # 5 minutes
        path_grouping_policy  group_by_prio
        path_selector         "historical-service-time 0"
        reservation_key       file          # if using SCSI persistent reservations
    }
    # Follow up with specific device entries below, they will take precedence.
}

After modifying the /etc/multipath.conf file, apply your settings as described in Section 17.8.4, “Applying multipath.conf modifications”.

For a quick overview of the multipath subsystem, use multipath -ll or multipathd show topology. The output of these commands has the same format. The former command reads the kernel state, while the latter prints the status of the multipath daemon. Normally both states are equal. Here is an example of the output:

tux > sudo multipathd show topology
mpatha1 (3600a098000aad1e300000b4b5a275d452) dm-03 NETAPP,INF-01-004
size=64G features='3 queue_if_no_path pg_init_retries 50'5 hwhandler='1 alua'6 wp=rw7
|-+- 8policy='historical-service-time 2'9 prio=5010 status=active11
| |-12 3:0:0:113 sdb 8:1614 active15 ready16 running17
| `- 4:0:0:1 sdf 8:80  active ready running
`-+- policy='historical-service-time 2' prio=10 status=enabled
  `- 4:0:1:1 sdj 8:144 active ready running

The multipath path device states are:

LVM2 has built-in support for detecting multipath devices. It is activated by default in /etc/lvm/lvm.conf:

    multipath_component_detection=1

This works reliably only if LVM2 is also configured to obtain information about device properties from udev:

    external_device_info_source="udev"

It is also possible (although normally not necessary) to create a filter expression for LVM2 to ignore all devices except multipath devices. See Section 17.12.4, “Referring to multipath maps”.

If all paths fail concurrently and I/O is queued, applications may stall for a long time. To resolve this, you can use the following procedure:

MD RAID arrays on top of multipathing are set up automatically by the system's udev rules. No special configuration in /etc/mdadm.conf is necessary.

If your system has already been configured for multipathing and you need to add storage to the SAN, you can use the rescan-scsi-bus.sh script to scan for the new devices. The general syntax for the command follows:

tux > sudo rescan-scsi-bus.sh [-a] [-r] --hosts=2-3,5

Where the options have the following meaning:

Run rescan-scsi-bus.sh --help for help on additional options.

If multipathd is running and new SAN devices are discovered, they should be automatically set up as multipath maps according to the configuration described in Section 17.11, “Selecting devices for multipathing”.

Warning: Dell/EMC PowerPath environments

In EMC PowerPath environments, do not use the rescan-scsi-bus.sh utility provided with the operating system or the HBA vendor scripts for scanning the SCSI buses. To avoid potential file system corruption, EMC requires that you follow the procedure provided in the vendor documentation for EMC PowerPath for Linux.

A block device can only either be part of a multipath map or be used directly (mounted as file system, used as swap, LVM physical volume, or otherwise). If a device is already mounted, an attempt by multipathd to make it part of a multipath map will fail with a “Device or resource busy” error. Vice-versa, the same error results if systemd attempts to mount a device which has already been made part of a multipath map.

Storage device activation during boot is handled by a complex interaction between systemd, udev, multipathd and some other tools. udev rules play a central role. They set device properties that indicate to other subsystems how a device should be used. The multipath-related udev rules set the following properties for devices that are selected for multipathing:

SYSTEMD_READY=0
DM_MULTIPATH_DEVICE_PATH=1

Partition devices inherit these properties from their parents.

If these properties are not set correctly, if some tool does not respect them, or if they get set too late, a race condition between multipathd and some other subsystem may result. Only one of the contenders can win the race; the other one will see a “Device or resource busy” error.

One problem in this context is that the tools of the LVM2 suite do not evaluate udev properties by default. They rely on their own logic for determining whether a device is a multipath component, which sometimes does not match the logic of the rest of the system. A workaround for this is described in Section 17.14.3, “Using LVM2 on multipath devices”.

Note: Example of boot deadlock

Consider a system with multipathing where the root device is not multipathed, and no devices are excluded from multipath (see Multipath disabled in the initramfs in Section 17.3.2.2, “Root file system on a local disk”). The root file system is mounted in the initramfs. systemd switches to the root file system and multipathd starts up. Because the device is already mounted, multipathd fails to set up the multipath map for it. Because the root device is not configured in blacklist, it is considered a multipath device, and SYSTEMD_READY=0 is set for it.

Later in the boot process, the system attempts to mount additional file systems like /var and /home. Usually, these file systems will be on the same device as the root file system, by default as BTRFS subvolumes of the root file system itself. But systemd cannot mount them because of SYSTEMD_READY=0. We are in a deadlock: The dm-multipath device cannot be created, and the underlying device is blocked for systemd. The additional file systems cannot be mounted, resulting in boot failure.

A solution to this problem already exists. multipathd detects this situation and releases the device to systemd which can then proceed mounting the file system. However, it is important to understand the general problem, which can still occur in more subtle ways.

An example of a device referencing issue has been given in Section 17.7.4.2, “Persistent device names in the initramfs”. Typically, there are multiple symbolic links pointing to a device node (see Section 17.12.4, “Referring to multipath maps”). But these links do not always exist; udev creates them according to the current udev rules. For example, if multipathing is off, symbolic links under /dev/mapper/ for multipath devices will be missing. Thus, any reference to a /dev/mapper/ device will fail.

Such references can appear in various places, notably in /etc/fstab and /etc/crypttab, in the initramfs, or even on the kernel command line.

The safest way to circumvent this problem is to avoid using the kind of device references that are not persistent between boots or depend on system configuration. We generally recommend referring to file systems (and similar entities like swap space) by properties of the file system itself (like UUID or label) rather than the containing device. If such references are not available and device references are required, for example, in /etc/crypttab, the options should be evaluated carefully. For example, in Section 17.12.4, “Referring to multipath maps”, the best option might be the /dev/disk/by-id/wwn- link because it would also work with multipath=off.

As there are many error situations that differ in subtle ways, it is impossible to provide a step-by-step recovery guide. But with the background knowledge from the previous subsections, you should be able to figure out the problem if a system runs into emergency mode because of multipathing issues. Before you begin debugging, make sure you have checked the following questions:

If you are unsure with respect to the last question, here is a sample dracut emergency prompt as it would be printed before switching root:

Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.

You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

Give root password for maintenance
(or press Control-D to continue):

The mention of rdsosreport.txt is a clear indication that the system is still running from the initramfs. If you are still uncertain, log in and check for the existence of the file /etc/initrd-release. This file exists only in an initramfs environment.

If emergency mode is entered after switching root, the emergency prompt looks similar, but rdsosreport.txt is not mentioned:

Timed out waiting for device dev-disk-by\x2duuid-c4a...cfef77d.device.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Postfix Mail Transport Agent.
Welcome to emergency shell
Give root password for maintenance
(or press Control-D to continue):

For more information about troubleshooting multipath I/O issues on SUSE Linux Enterprise Server, see the following Technical Information Documents (TIDs) in the SUSE Knowledgebase: