Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com › Documentación de SUSE Linux Enterprise High Availability Extension › Administration Guide › Storage and Data Replication › Disaster Recovery with Relax-and-Recover (Rear)
ContentsContents
Administration Guide
  1. About This Guide
  2. I Installation, Setup and Upgrade
    1. 1 Product Overview
    2. 2 System Requirements and Recommendations
    3. 3 Installing the High Availability Extension
    4. 4 Using the YaST Cluster Module
    5. 5 Upgrading Your Cluster and Updating Software Packages
  3. II Configuration and Administration
    1. 6 Configuration and Administration Basics
    2. 7 Configuring and Managing Cluster Resources with Hawk2
    3. 8 Configuring and Managing Cluster Resources (Command Line)
    4. 9 Adding or Modifying Resource Agents
    5. 10 Fencing and STONITH
    6. 11 Storage Protection and SBD
    7. 12 Access Control Lists
    8. 13 Network Device Bonding
    9. 14 Load Balancing
    10. 15 Geo Clusters (Multi-Site Clusters)
    11. 16 Executing Maintenance Tasks
  4. III Storage and Data Replication
    1. 17 Distributed Lock Manager (DLM)
    2. 18 OCFS2
    3. 19 GFS2
    4. 20 DRBD
    5. 21 Cluster Logical Volume Manager (cLVM)
    6. 22 Cluster Multi-device (Cluster MD)
    7. 23 Samba Clustering
    8. 24 Disaster Recovery with Relax-and-Recover (Rear)
  5. IV Appendix
    1. A Troubleshooting
    2. B Naming Conventions
    3. C Cluster Management Tools (Command Line)
    4. D Running Cluster Reports Without root Access
  6. Glossary
  7. E GNU Licenses
Navigation←→
documentation.suse.com › Documentación de SUSE Linux Enterprise High Availability Extension › Administration Guide › Storage and Data Replication › Disaster Recovery with Relax-and-Recover (Rear)
Top
←→
Applies to SUSE Linux Enterprise High Availability Extension 12 SP5

24 Disaster Recovery with Relax-and-Recover (Rear)

Abstract

Relax-and-Recover (“ReaR”, in this chapter abbreviated as Rear) is a disaster recovery framework for use by system administrators. It is a collection of Bash scripts that need to be adjusted to the specific production environment that is to be protected in case of disaster.

No disaster recovery solution will work out-of-the-box. Therefore it is essential to take preparations before any disaster happens.

24.1 Conceptual Overview
24.2 Setting Up Rear and Your Backup Solution
24.3 Creating the Recovery Installation System
24.4 Testing the Recovery Process
24.5 Recovering from Disaster
24.6 For More Information

24.1 Conceptual Overview

The following sections describe the general disaster recovery concept and the basic steps you need to execute for successful recovery with Rear. They also provide some guidance on Rear requirements, some limitations to be aware of, and scenarios and backup tools.

Note
Note: Understanding Rear

Understanding Rear's complex functionality is essential for making the tool work as intended. Therefore, read this chapter carefully and familiarize yourself with Rear before a disaster strikes. You should also be aware of Rear's known limitations and test your system in advance.

24.1.1 Creating a Disaster Recovery Plan

Before the worst scenario happens, take action: analyze your IT infrastructure for any substantial risks, evaluate your budget, and create a disaster recovery plan. If you do not already have a disaster recovery plan at hand, find some information on each step below:

  • Risk Analysis.  Conduct a solid risk analysis of your infrastructure. List all the possible threats and evaluate how serious they are. Determine how likely these threats are and prioritize them. It is recommended to use a simple categorization: probability and impact.

  • Budget Planning.  The outcome of the analysis is an overview, which risks can be tolerated and which are critical for your business. Ask yourself how you can minimize risks and how much will it cost. Depending on how big your company is, spend two to fifteen percent of the overall IT budget on disaster recovery.

  • Disaster Recovery Plan Development.  Make checklists, test procedures, establish and assign priorities, and inventory your IT infrastructure. Define how to deal with a problem when some services in your infrastructure fail.

  • Test.  After defining an elaborate plan, test it. Test it at least once a year. Use the same testing hardware as your main IT infrastructure.

24.1.2 What Does Disaster Recovery Mean?

If a system in a production environment has been destroyed (for whatever reasons—be it broken hardware, a misconfiguration or software problems), you need to re-create the system. The recreation can be done either on the same hardware or on compatible replacement hardware. Re-creating a system means more than restoring files from a backup. It also includes preparing the system's storage (with regard to partitioning, file systems, and mount points), and reinstalling the boot loader.

24.1.3 How Does Disaster Recovery With Rear Work?

While the system is up and running, create a backup of the files and create a recovery system on a recovery medium. The recovery system contains a recovery installer.

In case the system has been destroyed, replace broken hardware (if needed), boot the recovery system from the recovery medium and launch the recovery installer. The recovery installer re-creates the system: First, it prepares the storage (partitioning, file systems, mount points), then it restores the files from the backup. Finally, it reinstalls the boot loader.

24.1.4 Rear Requirements

To use Rear you need at least two identical systems: the machine that runs your production environment and an identical test machine. “Identical” in this context means that you can, for example, replace a network card with another one using the same Kernel driver.

Warning
Warning: Identical Drivers Required

If a hardware component does not use the same driver as the one in your production environment, it is not considered identical by Rear.

24.1.5 Rear Version Updates

To be compatible with older service packs, SUSE Linux Enterprise High Availability Extension 12 SP5 ships with different Rear versions: 1.16 (included in RPM package rear116), 1.17.2.a (rear1172a), 1.18.a (rear118a), and 2.4 (rear23a). The latest version contains some later enhancements from the upstream GitHub project.

Note
Note: Find Important Information in Changelogs

Any information about bugfixes, incompatibilities, and other issues can be found in the changelogs of the packages. It is recommended to review also later package versions of Rear in case you need to re-validate your disaster recovery procedure.

Be aware of the following issues with Rear:

  • To allow disaster recover on UEFI systems, you need version 1.18.a and the package ebiso. Only this version supports the new helper tool /usr/bin/ebiso. This helper tool is used to create a UEFI-bootable Rear system ISO image.

  • If you have a tested and fully functional disaster recovery procedure with one Rear version, do not update Rear. Keep the Rear package and do not change your disaster recovery method!

  • Version updates for Rear are provided as separate packages that intentionally conflict with each other to prevent your installed version getting accidentally replaced with another version.

In the following cases you need to completely re-validate your existing disaster recovery procedure:

  • For each Rear version update.

  • When you update Rear manually.

  • For each software that is used by Rear.

  • If you update low-level system components such as parted, btrfs and similar.

24.1.6 Limitations with Btrfs

The following limitations apply if you use Btrfs.

Your System Includes Subvolumes, but No Snapshots Subvolumes

At least Rear version 1.17.2.a is required. This version supports re-creating “normal” Btrfs subvolume structure (no snapshot subvolumes).

Your System Includes Snapshot Subvolumes
Warning
Warning

Btrfs snapshot subvolumes cannot be backed up and restored as usual with file-based backup software.

While recent snapshot subvolumes on Btrfs file systems need almost no disk space (because of Btrfs's copy-on-write functionality), those files would be backed up as complete files when using file-based backup software. They would end up twice in the backup with their original file size. Therefore, it is impossible to restore the snapshots as they have been before on the original system.

Your SLE12 System Needs Matching Rear Configuration

The setup in SLE12 GA, SLE12 SP1, and SLE12 SP2 have several incompatible Btrfs default structures. As such, it is crucial to use a matching Rear configuration file. See the example files /usr/share/rear/conf/examples/SLE12*-btrfs-example.conf.

24.1.7 Scenarios and Backup Tools

Rear can create a disaster recovery system (including a system-specific recovery installer) that can be booted from a local medium (like a hard disk, a flash disk, a DVD/CD-R) or via PXE. The backup data can be stored on a network file system, for example NFS, as described in Example 24.1.

Rear does not replace a file backup, but complements it. By default, Rear supports the generic tar command, and several third-party backup tools (such as Tivoli Storage Manager, QNetix Galaxy, Symantec NetBackup, EMC NetWorker, or HP DataProtector). Refer to Example 24.2 for an example configuration of using Rear with EMC NetWorker as backup tool.

24.1.8 Basic Steps

For a successful recovery with Rear in case of disaster, you need to execute the following basic steps:

Setting Up Rear and Your Backup Solution

This includes tasks like editing the Rear configuration file, adjusting the Bash scripts, and configuring the backup solution that you want to use.

Creating the Recovery Installation System

While the system to be protected is up and running use the rear mkbackup command to create a file backup and to generate a recovery system that contains a system-specific Rear recovery installer.

Testing the Recovery Process

Whenever you have created a disaster recovery medium with Rear, test the disaster recovery process thoroughly. It is essential to use a test machine that has identical hardware like the one that is part of your production environment. For details, refer to Section 24.1.4, “Rear Requirements”.

Recovering from Disaster

After a disaster has occurred, replace any broken hardware (if necessary). Then boot the Rear recovery system and start the recovery installer with the rear recover command.

24.2 Setting Up Rear and Your Backup Solution

To set up Rear, you need to edit at least the Rear configuration file /etc/rear/local.conf and, if needed, the Bash scripts that are part of the Rear framework.

In particular, you need to define the following tasks that Rear should do:

  • When your system is booted with UEFI.  If your system boots with a UEFI boot loader, install the package ebiso and add the following line into /etc/rear/local.conf: 

    ISO_MKISOFS_BIN=/usr/bin/ebiso

  • How to back up files and how to create and store the disaster recovery system.  This needs to be configured in /etc/rear/local.conf.

  • What to re-create exactly (partitioning, file systems, mount points, etc.).  This can be defined in /etc/rear/local.conf (for example, what to exclude). To re-create non-standard systems, you may need to enhance the Bash scripts.

  • How the recovery process works.  To change how Rear generates the recovery installer, or to adapt what the Rear recovery installer does, you need to edit the Bash scripts.

To configure Rear, add your options to the /etc/rear/local.conf configuration file. (The former configuration file /etc/rear/sites.conf has been removed from the package. However, if you have such a file from your last setup, Rear will still use it.)

All Rear configuration variables and their default values are set in /usr/share/rear/conf/default.conf. Some example files (*example.conf) for user configurations (for example, what is set in /etc/rear/local.conf) are available in the examples subdirectory. Find more information in the Rear man page.

You should start with a matching example configuration file as template and adapt it as needed to create your particular configuration file. Copy various options from several example configuration files and paste them into your specific /etc/rear/local.conf file that matches your particular system. Do not use original example configuration files, because they provide an overview of variables that can be used for specific setups.

After you have changed the Rear configuration file, run the following command and check its output: 

rear dump
Example 24.1: Using an NFS Server to Store the File Backup

Rear can be used in different scenarios. The following example uses an NFS server as storage for the file backup.

  1. Set up an NFS server with YaST as described in the SUSE Linux Enterprise Server 12 SP5 Administration Guide: https://documentation.suse.com/sles-12/html/SLES-all/cha-nfs.html.

  2. Define the configuration for your NFS server in the /etc/exports file. Make sure the directory on the NFS server (where you want the backup data to be available), has the right mount options. For example: 

    /srv/nfs *([...],rw,no_root_squash,[...])

    Replace /srv/nfs with the path to your backup data on the NFS server and adjust the mount options. You will probably need no_root_squash to access the backup data as the rear mkbackup command runs as root.

  3. Adjust the various BACKUP parameters in the configuration file /etc/rear/local.conf to make Rear store the file backup on the respective NFS server. Find examples in your installed system under /usr/share/rear/conf/examples/SLE12-*-example.conf.

Example 24.2: Using Third-Party Backup Tools Like EMC NetWorker

Using third-party backup tools instead of tar requires appropriate settings in the Rear configuration file.

The following is an example configuration for EMC NetWorker. Add this configuration snippet to /etc/rear/local.conf and adjust it according to your setup: 

BACKUP=NSR
    OUTPUT=ISO
    BACKUP_URL=nfs://host.example.com/path/to/rear/backup
    OUTPUT_URL=nfs://host.example.com/path/to/rear/backup
    NSRSERVER=backupserver.example.com
    RETENTION_TIME="Month"

24.3 Creating the Recovery Installation System

After you have configured Rear as described in Section 24.2, create the recovery installation system (including the Rear recovery installer) plus the file backup with the following command: 

rear -d  -D mkbackup

It executes the following steps:

  1. Analyzing the target system and gathering information, in particular about the disk layout (partitioning, file systems, mount points) and about the boot loader.

  2. Creating a bootable recovery system with the information gathered in the first step. The resulting Rear recovery installer is specific to the system that you want to protect from disaster. It can only be used to re-create this specific system.

  3. Calling the configured backup tool to back up system and user files.

24.4 Testing the Recovery Process

After having created the recovery system, test the recovery process on a test machine with identical hardware. See also Section 24.1.4, “Rear Requirements”. Make sure the test machine is correctly set up and can serve as a replacement for your main machine.

Warning
Warning: Extensive Testing on Identical Hardware

Thorough testing of the disaster recovery process on machines is required. Test the recovery procedure on a regular basis to ensure everything works as expected.

Procedure 24.1: Performing a Disaster Recovery on a Test Machine

  1. Create a recovery medium by burning the recovery system that you have created in Section 24.3 to a DVD or CD. Alternatively, you can use a network boot via PXE.

  2. Boot the test machine from the recovery medium.

  3. From the menu, select Recover.

  4. Log in as root (no password needed).

  5. Enter the following command to start the recovery installer: 

    rear -d -D recover

    For details about the steps that Rear takes during the process, see Recovery Process.

  6. After the recovery process has finished, check whether the system has been successfully re-created and can serve as a replacement for your original system in the production environment.

24.5 Recovering from Disaster

In case a disaster has occurred, replace any broken hardware if necessary. Then proceed as described in Procedure 24.1, using either the repaired machine (or a tested, identical machine that can serve as a replacement for your original system).

The rear recover command will execute the following steps:

Recovery Process

  1. Restoring the disk layout (partitions, file systems, and mount points).

  2. Restoring the system and user files from the backup.

  3. Restoring the boot loader.

24.6 For More Information

→Part IV Appendix←Chapter 23 Samba Clustering
Print this page

© 2024 SUSE