23 Cluster Logical Volume Manager (Cluster LVM) #

Start a shell and log in as root.

Check the current configuration of the cluster resources:

# crm configure show

If you have already configured a DLM resource (and a corresponding base group and base clone), continue with Procedure 23.2, “Creating an lvmlockd Resource”.

Otherwise, configure a DLM resource and a corresponding base group and base clone as described in Procedure 19.1, “Configuring a Base Group for DLM”.

Start a shell and log in as root.

Run the following command to see the usage of this resource:

# crm configure ra info lvmlockd

Configure a lvmlockd resource as follows:

# crm configure primitive lvmlockd lvmlockd \
  op start timeout="90" \
  op stop timeout="100" \
  op monitor interval="30" timeout="90"

To ensure the lvmlockd resource is started on every node, add the primitive resource to the base group for storage you have created in Procedure 23.1, “Creating a DLM Resource”:

# crm configure modgroup g-storage add lvmlockd

Review your changes:

# crm configure show

Check if the resources are running well:

# crm status full

Start a shell and log in as root.

Assuming you already have two shared disks, create a shared VG with them:

# vgcreate --shared vg1 /dev/disk/by-id/DEVICE_ID1 /dev/disk/by-id/DEVICE_ID2

Create an LV and do not activate it initially:

# lvcreate -an -L10G -n lv1 vg1

Start a shell and log in as root.

Run the following command to see the usage of this resource:

# crm configure ra info LVM-activate

This resource manages the activation of a VG. In a shared VG, LV activation has two different modes: exclusive and shared mode. The exclusive mode is the default and should be used normally, when a local file system like ext4 uses the LV. The shared mode should only be used for cluster file systems like OCFS2.

Configure a resource to manage the activation of your VG. Choose one of the following options according to your scenario:

Make sure the VG can only be activated on nodes where the DLM and lvmlockd resources are already running:

Check if the resources are running well:

# crm status full

Run YaST and click Network Services › iSCSI LIO Target to start the iSCSI Server module.

If you want to start the iSCSI target whenever your computer is booted, choose When Booting, otherwise choose Manually.

If you have a firewall running, enable Open Port in Firewall.

Switch to the Global tab. If you need authentication, enable incoming or outgoing authentication or both. In this example, we select No Authentication.

Add a new iSCSI target:

1. Switch to the Targets tab.

2. Click Add.

3. Enter a target name. The name needs to be formatted like this:

   iqn.DATE.DOMAIN

   For more information about the format, refer to Section 3.2.6.3.1. Type "iqn." (iSCSI Qualified Name) at http://www.ietf.org/rfc/rfc3720.txt.

4. If you want a more descriptive name, you can change it as long as your identifier is unique for your different targets.

5. Click Add.

6. Enter the device name in Path and use a Scsiid.

7. Click Next twice.

Confirm the warning box with Yes.

Open the configuration file /etc/iscsi/iscsid.conf and change the parameter node.startup to automatic.

Run YaST and click Network Services › iSCSI Initiator.

If you want to start the iSCSI initiator whenever your computer is booted, choose When Booting, otherwise set Manually.

Change to the Discovery tab and click the Discovery button.

Add the IP address and the port of your iSCSI target (see Procedure 23.5, “Configuring iSCSI Targets (SAN)”). Normally, you can leave the port as it is and use the default value.

If you use authentication, insert the incoming and outgoing user name and password, otherwise activate No Authentication.

Select Next. The found connections are displayed in the list.

Proceed with Finish.

Open a shell, log in as root.

Test if the iSCSI initiator has been started successfully:

# iscsiadm -m discovery -t st -p 192.168.3.100
192.168.3.100:3260,1 iqn.2010-03.de.jupiter:san1

Establish a session:

# iscsiadm -m node -l -p 192.168.3.100 -T iqn.2010-03.de.jupiter:san1
Logging in to [iface: default, target: iqn.2010-03.de.jupiter:san1, portal: 192.168.3.100,3260]
Login to [iface: default, target: iqn.2010-03.de.jupiter:san1, portal: 192.168.3.100,3260]: successful

See the device names with lsscsi:

...
[4:0:0:2]    disk    IET      ...     0     /dev/sdd
[5:0:0:1]    disk    IET      ...     0     /dev/sde

Look for entries with IET in their third column. In this case, the devices are /dev/sdd and /dev/sde.

Open a root shell on one of the nodes you have run the iSCSI initiator from Procedure 23.6, “Configuring iSCSI Initiators”.

Create the shared volume group on disks /dev/sdd and /dev/sde, using their stable device names (for example, in /dev/disk/by-id/):

# vgcreate --shared testvg /dev/disk/by-id/DEVICE_ID /dev/disk/by-id/DEVICE_ID

Create logical volumes as needed:

# lvcreate --name lv1 --size 500M testvg

Check the volume group with vgdisplay:

  --- Volume group ---
      VG Name               testvg
      System ID
      Format                lvm2
      Metadata Areas        2
      Metadata Sequence No  1
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                0
      Open LV               0
      Max PV                0
      Cur PV                2
      Act PV                2
      VG Size               1016,00 MB
      PE Size               4,00 MB
      Total PE              254
      Alloc PE / Size       0 / 0
      Free  PE / Size       254 / 1016,00 MB
      VG UUID               UCyWw8-2jqV-enuT-KH4d-NXQI-JhH3-J24anD

Check the shared state of the volume group with the command vgs:

# vgs
  VG       #PV #LV #SN Attr   VSize     VFree
  vgshared   1   1   0 wz--ns 1016.00m  1016.00m

The Attr column shows the volume attributes. In this example, the volume group is writable (w), resizeable (z), the allocation policy is normal (n), and it is a shared resource (s). See the man page of vgs for details.

Create a primary/primary DRBD resource:

1. First, set up a DRBD device as primary/secondary as described in Procedure 22.2, “Manually Configuring DRBD”. Make sure the disk state is up-to-date on both nodes. Check this with drbdadm status.

2. Add the following options to your configuration file (usually something like /etc/drbd.d/r0.res):

   resource r0 {
     net {
        allow-two-primaries;
     }
     ...
   }

3. Copy the changed configuration file to the other node, for example:

   # scp /etc/drbd.d/r0.res venus:/etc/drbd.d/

4. Run the following commands on both nodes:

   # drbdadm disconnect r0
   # drbdadm connect r0
   # drbdadm primary r0

5. Check the status of your nodes:

   # drbdadm status r0

Include the lvmlockd resource as a clone in the pacemaker configuration, and make it depend on the DLM clone resource. See Procedure 23.1, “Creating a DLM Resource” for detailed instructions. Before proceeding, confirm that these resources have started successfully on your cluster. Use crm status or the Web interface to check the running services.

Prepare the physical volume for LVM with the command pvcreate. For example, on the device /dev/drbd_r0 the command would look like this:

# pvcreate /dev/drbd_r0

Create a shared volume group:

# vgcreate --shared testvg /dev/drbd_r0

Create logical volumes as needed. You probably want to change the size of the logical volume. For example, create a 4 GB logical volume with the following command:

# lvcreate --name lv1 -L 4G testvg

The logical volumes within the VG are now available as file system mounts for raw usage. Ensure that services using them have proper dependencies to collocate them with and order them after the VG has been activated.

On node alice, execute the following steps:

1. Convert the mirror logical volume test-lv to a linear logical volume:

   # lvconvert -m0 cluster-vg2/test-lv /dev/vdc

2. Remove the physical volume /dev/vdc from the volume group cluster-vg2:

   # vgreduce cluster-vg2 /dev/vdc

3. Remove this physical volume from LVM:

   # pvremove /dev/vdc

   When you run lsblk now, you get:

   NAME                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   vda                     253:0    0   40G  0 disk
   ├─vda1                  253:1    0    4G  0 part [SWAP]
   └─vda2                  253:2    0   36G  0 part /
   vdb                     253:16   0   20G  0 disk
   └─cluster--vg2-test--lv 254:5    0   12G  0 lvm
   vdc                     253:32   0   20G  0 disk

4. Create a cluster MD device /dev/md0 with the disk /dev/vdc:

   # mdadm --create /dev/md0 --bitmap=clustered \
   --metadata=1.2 --raid-devices=1 --force --level=mirror \
   /dev/vdc --data-offset=128

   For details on why to use the data-offset option, see Important: Avoiding Migration Failures.

On node bob, assemble this MD device:

# mdadm --assemble md0 /dev/vdc

If your cluster consists of more than two nodes, execute this step on all remaining nodes in your cluster.

Back on node alice:

1. Initialize the MD device /dev/md0 as physical volume for use with LVM:

   # pvcreate /dev/md0

2. Add the MD device /dev/md0 to the volume group cluster-vg2:

   # vgextend cluster-vg2 /dev/md0

3. Move the data from the disk /dev/vdb to the /dev/md0 device:

   # pvmove /dev/vdb /dev/md0

4. Remove the physical volume /dev/vdb from the volume group cluster-vg2:

   # vgreduce cluster-vg2 /dev/vdb

5. Remove the label from the device so that LVM no longer recognizes it as physical volume:

   # pvremove /dev/vdb

6. Add /dev/vdb to the MD device /dev/md0:

   # mdadm --grow /dev/md0 --raid-devices=2 --add /dev/vdb