Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Linux Enterprise High Availability Extension 11 SP4

19 Disaster Recovery with Rear

Relax-and-Recover (formerly ReaR, in this chapter abbreviated as Rear) is a disaster recovery framework for use by system administrators. It is a collection of Bash scripts that need to be adjusted to the specific production environment that is to be protected in case of disaster.

No disaster recovery solution will just work out-of-the-box. Therefore it is essential to take preparations before any disaster happens.

19.1 Conceptual Overview

The following sections describe the general disaster recovery concept and the basic steps you need to execute for successful recovery with Rear. They also provide some guidance on Rear requirements, some limitations to be aware of, and scenarios and backup tools.

19.1.1 Creating a Disaster Recovery Plan

Before the worst scenario happens, take action: analyze your IT infrastructure for any substantial risks, evaluate your budget, and create a disaster recovery plan. If you do not already have a disaster recovery plan at hand, find some information on each step below:

  • Risk Analysis.  Conduct a solid risk analysis of your infrastructure. List all the possible threats and evaluate how serious they are. Determine how likely these threats are and prioritize them. It is recommended to use a simple categorization: probability and impact.

  • Budget Planning.  The outcome of the analysis is an overview, which risks can be tolerated and which are critical for your business. Ask yourself how you can minimize risks and how much will it cost. Depending on how big your company is, spend two to fifteen percent of the overall IT budget on disaster recovery.

  • Disaster Recovery Plan Development.  Make checklists, test procedures, establish and assign priorities, and inventory your IT infrastructure. Define how to deal with a problem when some services in your infrastructure fail.

  • Test.  After defining an elaborate plan, test it. Test it at least once a year. Use the same testing hardware as your main IT infrastructure.

19.1.2 What Does Disaster Recovery Mean?

If a system in a production environment has been destroyed (for whatever reasons—be it broken hardware, a misconfiguration or software problems), you need to recreate the system. The recreation can be done either on the same hardware or on compatible replacement hardware. Recreating a system means more than just restoring files from a backup. It also includes preparing the system's storage (with regards to partitioning, file systems, and mount points), and reinstalling the boot loader.

19.1.3 How Does Disaster Recovery With Rear Work?

While the system is up and running, create a backup of the files and create a recovery system on a recovery medium. The recovery system contains a recovery installer.

In case the system has been destroyed, replace broken hardware (if needed), boot the recovery system from the recovery medium and launch the recovery installer. The recovery installer recreates the system: First, it prepares the storage (partitioning, file systems, mount points), then it restores the files from the backup. Finally, it reinstalls the boot loader.

19.1.4 Rear Requirements

To use Rear you need at least two identical systems: the machine that runs your production environment and an identical test machine. Identical in this context means that you can, for example, replace a network card with another one using the same Kernel driver.

Warning
Warning: Identical Drivers Required

If a hardware component does not use the same driver like the one in your production environment, it is not considered identical by Rear.

19.1.5 Rear Versions

SUSE Linux Enterprise High Availability Extension 11 SP4 ships two Rear versions in parallel:

  • Rear version 1.10.0 (included in RPM package: rear). For the documentation of this Rear version, see the High Availability Guide for SUSE Linux Enterprise High Availability Extension 11 SP3. It is available from http://documentation.suse.com/.

  • Rear version 1.16 (included in RPM package rear116). This version is documented in the current chapter.

Important
Important

If you have a tested and fully functional disaster recovery procedure with Rear version 1.10.0 there is no need to upgrade. In that case, keep the Rear package and do not change your disaster recovery method!

However, if Rear version 1.10.0 does not fulfill your particular needs (see also Section 19.1.6), you can manually upgrade to Rear version 1.16. The packages rear and rear116 intentionally conflict with each other to prevent that your installed version gets accidentally replaced with another version.

For each Rear version upgrade you must carefully re-validate whether your particular disaster recovery procedure still works.

19.1.6 Limitations with Btrfs

The following limitations apply if you use Btrfs.

Your System Includes Subvolumes, but No Snapshots Subvolumes

Rear version 1.16 is required. This version supports recreating normal Btrfs subvolume structure (no snapshot subvolumes).

Your System Includes Snapshot Subvolumes
Warning
Warning

Btrfs snapshot subvolumes cannot be backed up and restored as usual with file-based backup software.

While recent snapshot subvolumes on Btrfs file systems need almost no disk space (because of Btrfs' copy-on-write functionality), those files would be backed up as complete files when using file-based backup software. They would end up twice in the backup with their original file size. Therefore, it is impossible to restore the snapshots as they have been before on the original system.

19.1.7 Scenarios and Backup Tools

Rear can create a disaster recovery system (including a system-specific recovery installer) that can be booted from a local medium (like a hard disk, a flash disk, a DVD/CD-R) or via PXE. The backup data can be stored on a network file system, for example NFS, as described in Example 19.1.

Rear does not replace a file backup, but complements it. By default, Rear supports the generic tar command, and several third-party backup tools (such as Tivoli Storage Manager, QNetix Galaxy, Symantec NetBackup, EMC NetWorker, or HP DataProtector). Refer to Example 19.2 for an example configuration of using Rear with EMC NetWorker as backup tool.

19.1.8 Basic Steps

For a successful recovery with Rear in case of disaster, you need to execute the following basic steps:

Setting Up Rear and Your Backup Solution

This includes tasks like editing the Rear configuration file, adjusting the Bash scripts, and configuring the backup solution that you want to use.

Creating the Recovery Installation System

While the system to be protected is up and running use the rear mkbackup command to create a file backup and to generate a recovery system that contains a system-specific Rear recovery installer.

Testing the Recovery Process

Whenever you have created a disaster recovery medium with Rear, test the disaster recovery process thoroughly. It is essential to use a test machine that has identical hardware like the one that is part of your production environment. For details, refer to Section 19.1.4, “Rear Requirements”.

Recovering from Disaster

After a disaster has occurred, replace any broken hardware (if necessary). Then boot the Rear recovery system and start the recovery installer with the rear recover command.

19.2 Setting Up Rear and Your Backup Solution

To set up Rear, you need to edit at least the Rear configuration file /etc/rear/local.conf and, if needed, the Bash scripts that are part of the Rear framework.

In particular, you need to define the following tasks that Rear should do:

  • How to back up files and how to create and store the disaster recovery system.  This needs to be configured in /etc/rear/local.conf.

  • What to recreate exactly (partitioning, file systems, mount points etc.).  This can be defined in /etc/rear/local.conf (for example, what to exclude). To recreate non-standard systems, you may have to enhance the Bash scripts.

  • How the recovery process works.  To change how Rear generates the recovery installer, or to adapt what the Rear recovery installer does, you have to edit the Bash scripts.

To configure Rear, add your options to the /etc/rear/local.conf configuration file. (The former configuration file /etc/rear/sites.conf has been removed from the package. However, if you have such a file from your last setup, Rear will still use it).

A default Rear configuration for reference is available at /usr/share/rear/conf/default.conf. Other example configurations (*example.conf) are available in the same directory. Find more information in the Rear man page.

After you have changed the Rear configuration file, run the following command and check its output:

rear dump
Example 19.1: Using an NFS Server to Store the File Backup

Rear can be used in different scenarios. The following example uses an NFS server as storage for the file backup.

  1. Set up an NFS server with YaST as described in the SUSE Linux Enterprise Server 11 SP4 Administration Guide, chapter Sharing File Systems with NFS. It is available from http://documentation.suse.com/.

  2. Define the configuration for your NFS server in the /etc/exports file. Make sure the directory on the NFS server (where you want the backup data to be available), has the right mount options. For example:

    /srv/nfs *([...],rw,no_root_squash,[...])

    Replace /srv/nfs with the path to your backup data on the NFS server and adjust the mount options. You will probably need no_root_squash to access the backup data as the rear mkbackup command runs as root.

  3. Adjust the various BACKUP parameters in the configuration file /etc/rear/local.conf to make Rear store the file backup on the respective NFS server. Find an example in your installed system under /usr/share/rear/conf/SLE11-ext3-example.conf.

Example 19.2: Using Third-Party Backup Tools Like EMC NetWorker

Using third-party backup tools instead of tar requires appropriate settings in the Rear configuration file.

The following is an example configuration for EMC NetWorker. Add this configuration snippet to /etc/rear/local.conf and adjust it according to your setup:

BACKUP=NSR
    OUTPUT=ISO
    BACKUP_URL=nfs://host.example.com/path/to/rear/backup
    OUTPUT_URL=nfs://host.example.com/path/to/rear/backup
    NSRSERVER=backupserver.example.com
    RETENTION_TIME="Month"

19.3 Creating the Recovery Installation System

After you have configured Rear as described in Section 19.2, create the recovery installation system (including the Rear recovery installer) plus the file backup with the following command:

rear -d  -D mkbackup

It executes the following steps:

  1. Analyzing the target system and gathering information, in particular about the disk layout (partitioning, file systems, mount points) and about the boot loader.

  2. Creating a bootable recovery system with the information gathered in the first step. The resulting Rear recovery installer is specific to the system that you want to protect from disaster. It can only be used to recreate this specific system.

  3. Calling the configured backup tool to back up system and user files.

19.4 Testing the Recovery Process

After having created the recovery system, test the recovery process on a test machine with identical hardware. See also Section 19.1.4, “Rear Requirements”. Make sure the test machine is correctly set up and can serve as a replacement for your main machine.

Warning
Warning: Extensive Testing on Identical Hardware

Thorough testing of the disaster recovery process on machines is required. Test the recovery procedure on a regular basis to ensure everything works as expected.

Procedure 19.1: Performing a Disaster Recovery on a Test Machine
  1. Create a recovery medium by burning the recovery system that you have created in Section 19.3 to a DVD or CD . Alternatively, you can use a network boot via PXE.

  2. Boot the test machine from the recovery medium.

  3. From the menu, select Recover.

  4. Log in as root (no password needed).

  5. Enter the following command to start the recovery installer:

    rear -d -D recover

    For details about the steps that Rear takes during the process, see Recovery Process.

  6. After the recovery process has finished, check if the system has been successfully recreated and can serve as a replacement for your original system in the production environment.

19.5 Recovering from Disaster

In case a disaster has occurred, exchange any broken hardware if necessary. Then proceed as described in Procedure 19.1, using either the repaired machine (or a tested, identical machine that can serve as a replacement for your original system).

The rear recover command will execute the following steps:

Recovery Process
  1. Restoring the disk layout (partitions, file systems, and mount points).

  2. Restoring the system and user files from the backup.

  3. Restoring the boot loader.

19.6 For More Information

Print this page