Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / SUSE Linux Enterprise High Availability Documentation / Administration Guide / Storage and data replication / Cluster multi-device (Cluster MD)
Applies to SUSE Linux Enterprise High Availability 15 SP3

24 Cluster multi-device (Cluster MD)

The cluster multi-device (Cluster MD) is a software based RAID storage solution for a cluster. Currently, Cluster MD provides the redundancy of RAID1 mirroring to the cluster. With SUSE Linux Enterprise High Availability 15 SP3, RAID10 is included as a technology preview. If you want to try RAID10, replace mirror with 10 in the related mdadm command. This chapter shows you how to create and use Cluster MD.

24.1 Conceptual overview

The Cluster MD provides support for use of RAID1 across a cluster environment. The disks or devices used by Cluster MD are accessed by each node. If one device of the Cluster MD fails, it can be replaced at runtime by another device and it is re-synced to provide the same amount of redundancy. The Cluster MD requires Corosync and Distributed Lock Manager (DLM) for co-ordination and messaging.

A Cluster MD device is not automatically started on boot like the rest of the regular MD devices. A clustered device needs to be started using resource agents to ensure the DLM resource has been started.

24.2 Creating a clustered MD RAID device

  • A running cluster with pacemaker.

  • A resource agent for DLM (see Section 19.2, “Configuring DLM cluster resources”).

  • At least two shared disk devices. You can use an additional device as a spare which will fail over automatically in case of device failure.

  • An installed package cluster-md-kmp-default.

Warning: Always use persistent device names

Always use cluster-wide persistent device names, such as /dev/disk/by-id/DEVICE_ID. Unstable device names like /dev/sdX or /dev/dm-X might become mismatched on different nodes, causing major problems across the cluster.

  1. Make sure the DLM resource is up and running on every node of the cluster and check the resource status with the command:

    # crm_resource -r dlm -W
  2. Create the Cluster MD device:

    • If you do not have an existing normal RAID device, create the Cluster MD device on the node running the DLM resource with the following command:

      # mdadm --create /dev/md0 --bitmap=clustered \
      --metadata=1.2 --raid-devices=2 --level=mirror \
      /dev/disk/by-id/DEVICE_ID1 /dev/disk/by-id/DEVICE_ID2

      As Cluster MD only works with version 1.2 of the metadata, it is recommended to specify the version using the --metadata option. For other useful options, refer to the man page of mdadm. Monitor the progress of the re-sync in /proc/mdstat.

    • If you already have an existing normal RAID, first clear the existing bitmap and then create the clustered bitmap:

      # mdadm --grow /dev/mdX --bitmap=none
      # mdadm --grow /dev/mdX --bitmap=clustered
    • Optionally, to create a Cluster MD device with a spare device for automatic failover, run the following command on one cluster node:

      # mdadm --create /dev/md0 --bitmap=clustered --raid-devices=2 \
      --level=mirror --spare-devices=1 --metadata=1.2 \
      /dev/disk/by-id/DEVICE_ID1 /dev/disk/by-id/DEVICE_ID2 /dev/disk/by-id/DEVICE_ID3
  3. Get the UUID and the related md path:

    # mdadm --detail --scan

    The UUID must match the UUID stored in the superblock. For details on the UUID, refer to the mdadm.conf man page.

  4. Open /etc/mdadm.conf and add the md device name and the devices associated with it. Use the UUID from the previous step:

    DEVICE /dev/disk/by-id/DEVICE_ID1 /dev/disk/by-id/DEVICE_ID2
    ARRAY /dev/md0 UUID=1d70f103:49740ef1:af2afce5:fcf6a489
  5. Open Csync2's configuration file /etc/csync2/csync2.cfg and add /etc/mdadm.conf:

    group ha_group
       # ... list of files pruned ...
       include /etc/mdadm.conf

24.3 Configuring a resource agent

Configure a CRM resource as follows:

  1. Create a Raid1 primitive:

    crm(live)configure# primitive raider Raid1 \
      params raidconf="/etc/mdadm.conf" raiddev=/dev/md0 \
      force_clones=true \
      op monitor timeout=20s interval=10 \
      op start timeout=20s interval=0 \
      op stop timeout=20s interval=0
  2. Add the raider resource to the base group for storage that you have created for DLM:

    crm(live)configure# modgroup g-storage add raider

    The add sub-command appends the new group member by default.

    If not already done, clone the g-storage group so that it runs on all nodes:

    crm(live)configure# clone cl-storage g-storage \
        meta interleave=true target-role=Started
  3. Review your changes with show.

  4. If everything seems correct, submit your changes with commit.

24.4 Adding a device

To add a device to an existing, active Cluster MD device, first ensure that the device is visible on each node with the command cat /proc/mdstat. If the device is not visible, the command will fail.

Use the following command on one cluster node:

# mdadm --manage /dev/md0 --add /dev/disk/by-id/DEVICE_ID

The behavior of the new device added depends on the state of the Cluster MD device:

  • If only one of the mirrored devices is active, the new device becomes the second device of the mirrored devices and a recovery is initiated.

  • If both devices of the Cluster MD device are active, the new added device becomes a spare device.

24.5 Re-adding a temporarily failed device

Quite often the failures are transient and limited to a single node. If any of the nodes encounters a failure during an I/O operation, the device will be marked as failed for the entire cluster.

This could happen, for example, because of a cable failure on one of the nodes. After correcting the problem, you can re-add the device. Only the outdated parts will be synchronized as opposed to synchronizing the entire device by adding a new one.

To re-add the device, run the following command on one cluster node:

# mdadm --manage /dev/md0 --re-add /dev/disk/by-id/DEVICE_ID

24.6 Removing a device

Before removing a device at runtime for replacement, do the following:

  1. Make sure the device is failed by introspecting /proc/mdstat. Look for an (F) before the device.

  2. Run the following command on one cluster node to make a device fail:

    # mdadm --manage /dev/md0 --fail /dev/disk/by-id/DEVICE_ID
  3. Remove the failed device using the command on one cluster node:

    # mdadm --manage /dev/md0 --remove /dev/disk/by-id/DEVICE_ID

24.7 Assembling Cluster MD as normal RAID at the disaster recovery site

In the event of disaster recovery, you might face the situation that you do not have a Pacemaker cluster stack in the infrastructure on the disaster recovery site, but applications still need to access the data on the existing Cluster MD disks, or from the backups.

You can convert a Cluster MD RAID to a normal RAID by using the --assemble operation with the -U no-bitmap option to change the metadata of the RAID disks accordingly.

Find an example below of how to assemble all arrays on the data recovery site:

while read i; do
   NAME=`echo $i | sed 's/.*name=//'|awk '{print $1}'|sed 's/.*://'`
   UUID=`echo $i | sed 's/.*UUID=//'|awk '{print $1}'`
   mdadm -AR "/dev/md/$NAME" -u $UUID -U no-bitmap
   echo "NAME =" $NAME ", UUID =" $UUID ", assembled."
done < <(mdadm -Es)