Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / SUSE Linux Enterprise High Availability Extension – Dokumentation / Administration Guide / Storage and data replication / Disaster recovery with ReaR (Relax-and-Recover)
Applies to SUSE Linux Enterprise High Availability 15 SP3

26 Disaster recovery with ReaR (Relax-and-Recover)

 

Relax-and-Recover (“ReaR”) is a disaster recovery framework for use by system administrators. It is a collection of Bash scripts that need to be adjusted to the specific production environment that is to be protected in case of disaster.

No disaster recovery solution will work out-of-the-box. Therefore it is essential to take preparations before any disaster happens.

26.1 Conceptual overview

 

The following sections describe the general disaster recovery concept and the basic steps you need to execute for successful recovery with ReaR. They also provide some guidance on ReaR requirements, some limitations to be aware of, and scenarios and backup tools.

26.1.1 Creating a disaster recovery plan

 

Before the worst scenario happens, take action: analyze your IT infrastructure for any substantial risks, evaluate your budget, and create a disaster recovery plan. If you do not already have a disaster recovery plan at hand, find some information on each step below:

  • Risk analysis.  Conduct a solid risk analysis of your infrastructure. List all the possible threats and evaluate how serious they are. Determine how likely these threats are and prioritize them. It is recommended to use a simple categorization: probability and impact.

  • Budget planning.  The outcome of the analysis is an overview, which risks can be tolerated and which are critical for your business. Ask yourself how you can minimize risks and how much will it cost. Depending on how big your company is, spend two to fifteen percent of the overall IT budget on disaster recovery.

  • Disaster recovery plan development.  Make checklists, test procedures, establish and assign priorities, and inventory your IT infrastructure. Define how to deal with a problem when some services in your infrastructure fail.

  • Test.  After defining an elaborate plan, test it. Test it at least once a year. Use the same testing hardware as your main IT infrastructure.

26.1.2 What does disaster recovery mean?

 

If a system in a production environment has been destroyed (for whatever reasons—be it broken hardware, a misconfiguration or software problems), you need to re-create the system. The recreation can be done either on the same hardware or on compatible replacement hardware. Re-creating a system means more than restoring files from a backup. It also includes preparing the system's storage (with regard to partitioning, file systems, and mount points), and reinstalling the boot loader.

26.1.3 How does disaster recovery with ReaR work?

 

While the system is up and running, create a backup of the files and create a recovery system on a recovery medium. The recovery system contains a recovery installer.

In case the system has been destroyed, replace broken hardware (if needed), boot the recovery system from the recovery medium and launch the recovery installer. The recovery installer re-creates the system: First, it prepares the storage (partitioning, file systems, mount points), then it restores the files from the backup. Finally, it reinstalls the boot loader.

26.1.4 ReaR requirements

 

To use ReaR you need at least two identical systems: the machine that runs your production environment and an identical test machine. “Identical” in this context means that you can, for example, replace a network card with another one using the same Kernel driver.

Warning
Warning: Identical drivers required

If a hardware component does not use the same driver as the one in your production environment, it is not considered identical by ReaR.

26.1.5 ReaR version updates

 

To see which versions of ReaR are available with SUSE Linux Enterprise High Availability 15 SP3, run the following command: 

# zypper search --type package --verbose rear
Note
Note: Find important information in changelogs

Any information about bug fixes, incompatibilities, and other issues can be found in the changelogs of the packages. It is recommended to review also later package versions of ReaR in case you need to re-validate your disaster recovery procedure.

Be aware of the following issues with ReaR:

  • To allow disaster recovery on UEFI systems, you need at least ReaR version 1.18.a and the package ebiso. Only this version supports the new helper tool /usr/bin/ebiso. This helper tool is used to create a UEFI-bootable ReaR system ISO image.

  • If you have a tested and fully functional disaster recovery procedure with one ReaR version, do not update ReaR. Keep the ReaR package and do not change your disaster recovery method!

  • Version updates for ReaR are provided as separate packages that intentionally conflict with each other to prevent your installed version getting accidentally replaced with another version.

In the following cases you need to completely re-validate your existing disaster recovery procedure:

  • For each ReaR version update.

  • When you update ReaR manually.

  • For each software that is used by ReaR.

  • If you update low-level system components such as parted, btrfs and similar.

26.1.6 Limitations with Btrfs

 

The following limitations apply if you use Btrfs.

Your system includes subvolumes, but no snapshot subvolumes

At least ReaR version 1.17.2.a is required. This version supports re-creating “normal” Btrfs subvolume structure (no snapshot subvolumes).

Your system includes snapshot subvolumes
Warning
Warning

Btrfs snapshot subvolumes cannot be backed up and restored as usual with file-based backup software.

While recent snapshot subvolumes on Btrfs file systems need almost no disk space (because of Btrfs's copy-on-write functionality), those files would be backed up as complete files when using file-based backup software. They would end up twice in the backup with their original file size. Therefore, it is impossible to restore the snapshots as they have been before on the original system.

Your SLE system needs matching ReaR configuration

For example, the setup in SLE12 GA, SLE12 SP1, and SLE12 SP2 have several different incompatible Btrfs default structures. As such, it is crucial to use a matching ReaR configuration file. See the example files /usr/share/rear/conf/examples/SLE12*-btrfs-example.conf.

26.1.7 Scenarios and backup tools

 

ReaR can create a disaster recovery system (including a system-specific recovery installer) that can be booted from a local medium (like a hard disk, a flash disk, a DVD/CD-R) or via PXE. The backup data can be stored on a network file system, for example NFS, as described in Example 26.1.

ReaR does not replace a file backup, but complements it. By default, ReaR supports the generic tar command, and several third-party backup tools (such as Tivoli Storage Manager, QNetix Galaxy, Symantec NetBackup, EMC NetWorker, or HP DataProtector). Refer to Example 26.3 for an example configuration of using ReaR with EMC NetWorker as backup tool.

26.1.8 Basic steps

 

For a successful recovery with ReaR in case of disaster, you need to execute the following basic steps:

Setting up ReaR and your backup solution

This includes tasks like editing the ReaR configuration file, adjusting the Bash scripts, and configuring the backup solution that you want to use.

Creating the recovery installation system

While the system to be protected is up and running, create a file backup and generate a recovery system that contains a system-specific ReaR recovery installer.

For basic backups with tar, use the rear mkbackup command to create both the backup and the recovery system.

For third-party backup tools, use the rear mkrescue command to create the recovery system, and use the third-party backup tool to create the backup.

Testing the recovery process

Whenever you have created a disaster recovery medium with ReaR, test the disaster recovery process thoroughly. It is essential to use a test machine that has identical hardware like the one that is part of your production environment. For details, refer to Section 26.1.4, “ReaR requirements”.

Recovering from disaster

After a disaster has occurred, replace any broken hardware (if necessary). Then boot the ReaR recovery system and start the recovery installer with the rear recover command.

26.2 Setting up ReaR and your backup solution

 

To install the latest version of ReaR, run the following command: 

# zypper install rear

If you need to install an earlier version of ReaR, you can specify the package version. For example: 

# zypper install rear23a

To configure ReaR, add your settings to the ReaR configuration file /etc/rear/local.conf and, if needed, edit the Bash scripts that are part of the ReaR framework. In particular, you need to define the following tasks that ReaR should do:

  • How to back up files and how to create and store the disaster recovery system.  This needs to be configured in /etc/rear/local.conf.

  • What to re-create (partitioning, file systems, mount points, etc.).  This can be defined in /etc/rear/local.conf. To re-create non-standard systems, you may need to enhance the Bash scripts.

  • How the recovery process works.  To change how ReaR generates the recovery installer, or to adapt what the ReaR recovery installer does, you need to edit the Bash scripts.

All ReaR configuration variables and their default values are explained in /usr/share/rear/conf/default.conf.

Below are some examples of useful configuration options. You can also find example files for different scenarios in the /usr/share/rear/conf/examples/ subdirectory. Use these to help you create your /etc/rear/local.conf file.

Example 26.1: Using an NFS server to store the file backup
 

  1. Set up an NFS server with YaST as described in the Administration Guide for SUSE Linux Enterprise Server 15 SP3.

  2. Define the configuration for your NFS server in the /etc/exports file. Make sure the directory on the NFS server has the right mount options. For example, you might need no_root_squash as the rear mkbackup command runs as root. For more information, see man exports.

  3. Adjust the various BACKUP parameters in the configuration file /etc/rear/local.conf to make ReaR store the file backup on the respective NFS server. Find examples in your installed system under /usr/share/rear/conf/examples/SLE*-example.conf.

Example 26.2: Backing up Btrfs subvolumes with tar
 

Btrfs is set up with subvolumes, but tar creates backups with the --one-file-system option, which excludes subvolumes. Therefore, the mount points for the subvolumes must be explicitly included in the ReaR configuration.

Add this configuration snippet to /etc/rear/local.conf and adjust it according to your setup. For more information and additional options, see the example files at /usr/share/rear/conf/examples/SLE*-btrfs-example.conf. 

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://host.example.com/path/to/rear/backup
BACKUP_PROG_INCLUDE=( /root /boot/grub2/i386-pc /tmp /opt /var /boot/grub2/x86_64-efi /srv /usr/local )
Tip
Tip: Mount points for BACKUP_PROG_INCLUDE

You can find the mount points for the subvolumes on your specific system by running the following command: 

# findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash'
Example 26.3: Using third-party backup tools like EMC NetWorker
 

Using third-party backup tools instead of tar requires appropriate settings in the ReaR configuration file.

The following is an example configuration for EMC NetWorker. Add this configuration snippet to /etc/rear/local.conf and adjust it according to your setup: 

BACKUP=NSR
OUTPUT=ISO
BACKUP_URL=nfs://host.example.com/path/to/rear/backup
OUTPUT_URL=nfs://host.example.com/path/to/rear/backup
NSRSERVER=backupserver.example.com
RETENTION_TIME="Month"

For more information about supported third-party backup tools, see man rear, section BACKUP SOFTWARE INTEGRATION.

Example 26.4: Backing up multipath devices
 

By default, ReaR ignores file systems on multipath devices because it assumes they are on remote storage, not part of the local system. You can make ReaR include these file systems by adding the following line to /etc/rear/local.conf: 

AUTOEXCLUDE_MULTIPATH=n
Example 26.5: Booting your system with UEFI
 

If your system boots with a UEFI boot loader, additional configuration is required:

  1. Install the package ebiso: 

    # zypper install ebiso

  2. Add the following line to /etc/rear/local.conf: 

    ISO_MKISOFS_BIN="/usr/bin/ebiso"

  3. If your system boots with UEFI Secure Boot, you must also add the following line: 

    SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/sles/shim.efi"

For more information about ReaR configuration variables for UEFI, see the /usr/share/rear/conf/default.conf file.

26.3 Creating the recovery installation system

 

After you have configured ReaR as described in Section 26.2, create the recovery installation system (including the ReaR recovery installer) plus the file backup.

Example 26.6: Creating a recovery system with a basic tar backup
 

Run the rear mkbackup command: 

# rear -d -D mkbackup

This command performs the following steps:

  • Analyzing the target system and gathering information, in particular about the disk layout (partitioning, file systems, mount points) and about the boot loader.

  • Creating a bootable recovery system with the information gathered in the first step. The resulting ReaR recovery installer is specific to the system that you want to protect from disaster. It can only be used to re-create this specific system.

  • Calling the internal backup tool to back up system files and user files.

Example 26.7: Creating a recovery system with a third-party backup
 

  1. Run the rear mkrescue command: 

    # rear -d -D mkrescue

    This command analyzes the target and creates a recovery system, but does not create a backup of the files.

  2. Create a file backup using your third-party backup tool.

26.4 Testing the recovery process

 

After having created the recovery system, test the recovery process on a test machine with identical hardware. See also Section 26.1.4, “ReaR requirements”. Make sure the test machine is correctly set up and can serve as a replacement for your main machine.

Warning
Warning: Extensive testing on identical hardware

Thorough testing of the disaster recovery process on machines is required. Test the recovery procedure on a regular basis to ensure everything works as expected.

Procedure 26.1: Performing a disaster recovery on a test machine
 

  1. Create a recovery medium by burning the recovery system that you have created in Section 26.3 to a DVD or CD. Alternatively, you can use a network boot via PXE.

  2. Boot the test machine from the recovery medium.

  3. From the menu, select Recover.

  4. Log in as root (no password needed).

  5. Enter the following command to start the recovery installer: 

    rear -d -D recover

    For details about the steps that ReaR takes during the process, see Recovery process.

  6. After the recovery process has finished, check whether the system has been successfully re-created and can serve as a replacement for your original system in the production environment.

26.5 Recovering from disaster

 

In case a disaster has occurred, replace any broken hardware if necessary. Then proceed as described in Procedure 26.1, using either the repaired machine (or a tested, identical machine that can serve as a replacement for your original system).

The rear recover command will execute the following steps:

Recovery process
 

  1. Restoring the disk layout (partitions, file systems, and mount points).

  2. Restoring the system and user files from the backup.

  3. Restoring the boot loader.

26.6 For more information