Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE OpenStack Cloud 8

14 Backup and Restore Edit source

Information about how to back up and restore your cloud.

Freezer is a Backup and Restore as a Service platform that helps you automate the backup and restore process for your data. This backup and restore component (Freezer) executes backups and restores as jobs, and executes these jobs independently and/or as managed sessions (multiple jobs in multiple machines sharing a state).

There are a number of things you must do before installing your cloud so that you achieve the backup plan you need.

First, you should consider Section 14.4, “Enabling Default Backups of the Control Plane to an SSH Target” in case you lose cloud servers that the Freezer backup and restore service uses by default.

Second, you can prevent the Freezer backup and restore service from being installed completely, or designate which services should be backed up by default. Section 14.11, “Disabling Backup/Restore before Deployment”.

SUSE OpenStack Cloud 8 supports backup and restore of control plane services. It comes with playbooks and procedures to recover the control plane from various disaster scenarios.

The following features are supported:

  • Backup of your file system using a point-in-time snapshot.

  • Strong encryption: AES-256-CFB.

  • Backup of your MySQL database with LVM snapshot.

  • Restoring your data from a specific date automatically to your file system.

  • Low storage consumption: the backups are uploaded as a stream.

  • Flexible backup policy (both incremental and differential).

  • Data archived in GNU Tar format for file-based incremental.

  • Multiple compression algorithm support (zlib, bzip2, xz).

  • Removal old backup automatically according the provided parameters.

  • Multiple storage media support (Swift, local file system, SSH).

  • Management of multiple jobs (multiple backups on the same node).

  • Synchronization of backup and restore on multiple nodes.

  • Execution of scripts/commands before or after a job execution.

14.1 Architecture Edit source

The backup and restore service/Freezer uses GNU Tar under the hood to execute incremental backup and restore. When a key is provided, it uses Open SSL to encrypt data (AES-256-CFB).

The architecture consists of the following components:

ComponentDescription
Freezer Scheduler

A client-side component running on the node from where the data backup is executed. It consists of a daemon that retrieves the data from the freezer API and executes jobs (that is, backups, restore, admin actions, info actions, and pre- and/or post- job scripts) by running the Freezer Agent. The metrics and exit codes returned by the Freezer Agent are captured and sent to the Freezer API.

The scheduler manages the execution and synchronization of multiple jobs executed on a single node or multiple nodes. The status of the execution of all the nodes is saved through the API.

The Freezer scheduler takes care of uploading jobs to the API by reading job files on the file system. It also has its own configuration file where job sessions or other settings such as the Freezer API polling interval can be configured.

Freezer Agent

Multiprocessing Python software that runs on the client side where the data backup is executed. It can be executed as a standalone or by the Freezer Scheduler. The freezer-agent provides a flexible way to execute backup, restore, and perform other actions on a running system.

To provide flexibility in terms of data integrity, speed, performance, resource usage, and so on, the Freezer Agent offers a wide range of options to execute optimized backup according the available resources, such as:

  • Segments size (the amount of memory used)

  • Queues size (optimize backups where I/O, bandwidth, memory, or CPU is a constraint)

  • I/O affinity and process priority (can be used with real time I/O and maximum user-level process priority)

  • Bandwidth limitation

  • Client-side Encryption (AES-256-CFB)

  • Compression (multiple algorithms supported as zlib, bzip2, xz/lzma)

  • Parallel upload to pluggable storage media (that is, upload backup to Swift and to a remote node by SSH, or upload to two or more independent Swift instances with different credentials, and so on)

  • Execute file-based incremental (such as tar), block-based incremental (such as rsync algorithm), and differential-based backup and restore

  • Multi platform: you can run it on SUSE Linux, Windows, *BSD, and OSX

  • Automatic removal of old backups

Freezer API Stores and provides metadata to the Freezer Scheduler. Also stores session information for multi node backup synchronization. Workload data is not stored in the API .
DB ElasticsearchAPI uses the backend to store and retrieve metrics metadata sessions information job status, and so on.

14.2 Architecture of the Backup/Restore Service Edit source

ComponentDescriptionRuns on
APIAPI service to add / fetch Freezer jobsController nodes with Elasticsearch
SchedulerDaemon that stores and retrieves backup/restore jobs and executes themNodes needing backup/restore (controllers, Cloud Lifecycle Manager)
AgentThe agent that backs up and restores to and from targets. Invoked from scheduler or manually.Nodes needing backup/restore (controllers, Cloud Lifecycle Manager)

14.3 Default Automatic Backup Jobs Edit source

By default, the following are automatically backed up. You do not have to do anything for these backup jobs to run. However if you want to back up to somewhere outside the cluster, you do need to Section 14.4, “Enabling Default Backups of the Control Plane to an SSH Target”.

  • Cloud Lifecycle Manager Data.  All important information on the Cloud Lifecycle Manager

  • MariaDB Database.  The MariaDB database contains most of the data needed to restore services. While the MariaDB database only allows for an incomplete recovery of ESX data, for other services it allows full recovery. Logging data in Elasticsearch is not backed up. Swift objects are not backed up because of the redundant nature of Swift.

  • Swift Rings.  Swift rings are backed up so that you can recover more quickly even though Swift can rebuild the rings without this data. However automatically rebuilding the rings is slower than restoring via a backup.

The following services will be effectively backed up. In other words, the data needed to restore the services is backed up. The critical data that will be backed up are the databases and the configuration-related files. Note the data that is not backed up per service:

  • Ceilometer. However, there is no backup of metrics data

  • Cinder. However, there is no backup of the volumes

  • Glance. However, there is no backup of the images

  • Heat

  • Horizon

  • Keystone

  • Neutron

  • Nova. However, there is no backup of the images

  • Swift. However, there is no backup of the objects. Swift has its own high availability/redundancy. Swift rings are backed up. Although Swift will rebuild the rings itself, restoring from backup is faster.

  • Operations Console

  • Monasca. However, there is no backup of the metrics

14.3.1 Limitations Edit source

The following limitations apply to backups created by the Freezer backup and restore service in SUSE OpenStack Cloud:

  • Recovery of the following services (or cloud topologies) will be partially backed up. They will need additional data (other than the data stored in MariaDB) to return to fully functional.

    • ESX Cloud

    • Network services - LBaaS and VPNaaS

  • Logging data (that is, log files).

  • VMs and volumes are not currently backed up.

14.4 Enabling Default Backups of the Control Plane to an SSH Target Edit source

This topic describes how you can set up an external server as a backup server in case you lose access to your cloud servers that store the default backups.

14.4.1 Default Backup and Restore Edit source

As part of the installation procedure in SUSE OpenStack Cloud, automatic backup/restore jobs are set up to back up to Swift via the Freezer scheduler component of the backup and restore service. The backup jobs perform scheduled backups of SUSE OpenStack Cloud control plane data (files/directories/db). The restore jobs can be used to restore appropriate control plane data. Additional automatic jobs can be added to backup/restore from the secure shell (SSH) server that you set up/designate. It is recommended that you set up SSH backups so that in the event that you lose all of your control plane nodes at once, you have a backup on remote servers that you can use to restore the control plane nodes. Note that you do not have to restore from the SSH location if only one or two control plane nodes are lost. In that case, they can be recovered from the data on the remaining control plane node following the restore procedures in Section 13.2.2.2, “Recovering the Control Plane”. That document also explains how to recover using your remote server SSH backups.

While control plane backups to Swift are set up automatically, you must use the following procedure to set up SSH backups.

14.4.2 Setting up SSH backups Edit source

By default, during SUSE OpenStack Cloud 8 deployment, backup jobs are automatically deployed to Swift, the MySQL database, the Cloud Lifecycle Manager, and Swift rings. Restore jobs are also deployed for convenience. It is more secure to store those backups also outside of the SUSE OpenStack Cloud infrastructure. If you provide all the values required in the following file, jobs will also be deployed to backup and restore to/from an SSH server of your choice:

~/openstack/my_cloud/config/freezer/ssh_credentials.yml

14.4.3 Backing up your SUSE OpenStack Cloud control plane to an SSH server Edit source

You must provide the following connection information to this server:

  • The SSH server's IP address

  • The SSH server's port to connect to (usually port 22). You may want to confirm how to open the port on the SUSE OpenStack Cloud firewall.

  • The user to connect to the SSH server as

  • The SSH private key authorized to connect to that user (see below for details of how to set up one if it is not already done)

  • The directory where you wish to store the backup on that server

14.4.4 Setting up SSH for backups before deployment Edit source

Before running the configuration processor, edit the following file:

~/openstack/my_cloud/config/freezer/ssh_credentials.yml

All parameters are mandatory. Take care in providing the SSH private key.

14.4.5 Preparing the server that will store the backup Edit source

In this example, the information is as follows:

  • IP: 192.168.100.42

  • Port: 22

  • User: backupuser

  • Target directory for backups: /mnt/backups/

Please replace these values to meet your own requirements, as appropriate.

  1. Connect to the server:

    ardana > ssh -p 22 root@192.168.100.42
  2. Create the user:

    tux > sudo useradd backupuser --create-home --home-dir /mnt/backups/
  3. Switch to that user:

    su backupuser
  4. Create the SSH keypair:

    backupuser > ssh-keygen -t rsa
    # Just leave the default for the first question and do not set any passphrase
    > Generating public/private rsa key pair.
    > Enter file in which to save the key (/mnt/backups//.ssh/id_rsa):
    > Created directory '/mnt/backups//.ssh'.
    > Enter passphrase (empty for no passphrase):
    > Enter same passphrase again:
    > Your identification has been saved in /mnt/backups//.ssh/id_rsa
    > Your public key has been saved in /mnt/backups//.ssh/id_rsa.pub
    > The key fingerprint is:
    > a9:08:ae:ee:3c:57:62:31:d2:52:77:a7:4e:37:d1:28 backupuser@padawan-ccp-c0-m1-mgmt
    > The key's randomart image is:
    > +---[RSA 2048]----+
    > |          o      |
    > |   . . E + .     |
    > |  o . . + .      |
    > | o +   o +       |
    > |  + o o S .      |
    > | . + o o         |
    > |  o + .          |
    > |.o .             |
    > |++o              |
    > +-----------------+
  5. Add the public key to the list of the keys authorized to connect to that user on this server:

    backupuser > cat /mnt/backups/.ssh/id_rsa.pub >> /mnt/backups/.ssh/authorized_keys
  6. View the private key. This is what you will use for the backup configuration:

    backupuser > cat /mnt/backups/.ssh/id_rsa
    -----BEGIN RSA PRIVATE KEY-----
    MIIEogIBAAKCAQEAvjwKu6f940IVGHpUj3ffl3eKXACgVr3L5s9UJnb15+zV3K5L
    BZuor8MLvwtskSkgdXNrpPZhNCsWSkryJff5I335Jhr/e5o03Yy+RqIMrJAIa0X5
    ...
    iBKVKGPhOnn4ve3dDqy3q7fS5sivTqCrpaYtByJmPrcJNjb2K7VMLNvgLamK/AbL
    qpSTZjicKZCCl+J2+8lrKAaDWqWtIjSUs29kCL78QmaPOgEvfsw=
    -----END RSA PRIVATE KEY-----

Your server is now ready to receive backups. If you wish, you can check our advice on how to secure it in Section 14.4.8, “Securing your SSH backup server”.

14.4.6 Setting up SSH for backups after deployment Edit source

If you already deployed your cloud and forgot to configure SSH backups, or if you wish to modify the settings for where the backups are stored, follow the following instructions:

  1. Edit the following file:

    ~/openstack/my_cloud/config/freezer/ssh_credentials.yml

    Please be advised that all parameters are mandatory, and take care in providing the SSH private key.

  2. Run the following commands:

    ardana > cd ~/openstack
    ardana > git add -A
    ardana > git commit -m "My config"
    ardana > cd ~/openstack/ardana/ansible/
    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml

    This will deploy the SSH key and configure SSH backup and restore jobs for you. It may take some time before the backups occur.

14.4.7 Opening ports in the cloud firewall Edit source

There is a strict policy of firewalling deployed with SUSE OpenStack Cloud. If you use a non-standard SSH port, you may need to specifically open it by using the following process:

  1. When creating your model, edit the following file:

    ~/openstack/my_cloud/definition/data/firewall_rules.yml
  2. You must add a new element in the firewall-rules list, such as:

    - name: BACKUP
      # network-groups is a list of all the network group names
      # that the rules apply to
      network-groups:
      - MANAGEMENT
      rules:
      - type: allow
        # range of remote addresses in CIDR format that this
        # rule applies to
        remote-ip-prefix:
        port-range-min:
        port-range-max:
        # protocol must be one of: null, tcp, udp or icmp
        protocol: tcp

14.4.8 Securing your SSH backup server Edit source

You can do the following to harden an SSH server (these techniques are well documented on the internet):

  • Disable root login

  • Move SSH to a non-default port (that is, something other than 22)

  • Disable password login (only allow RSA keys)

  • Disable SSH v1

  • Authorize Secure File Transfer Protocol (SFTP) only for that user (disable SSH shell)

  • Firewall SSH traffic to ensure it comes from the SUSE OpenStack Cloud address range

  • Install a Fail2Ban solution

  • Restrict users that are allowed to SSH

Remove the key pair generated earlier on the backup server: the only thing needed is the .ssh/authorized_keys. You can remove the .ssh/id_rsa and .ssh/id_rsa.pub files. Be sure to save a backup of them somewhere.

14.4.9 Finish Firewall Configuration Edit source

Run the following commands to finish configuring the firewall.

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts _freezer_manage_jobs.yml

14.4.10 General tips Edit source

  • Take care when sizing the directory that will receive the backup.

  • Monitor the space left on that directory.

  • Keep the system up to date on that server.

14.5 Changing Default Jobs Edit source

The procedure to make changes to jobs created by default in SUSE OpenStack Cloud is to edit the model file, my_cloud/config/freezer/jobs.yml and then re-run the _freezer-manage-jobs.yml playbook. (Note that the backup/restore component is called "Freezer" so you may see commands by that name.)

  1. Open jobs.yml in an editor, then change and save the file:

    ardana > cd ~/openstack
    ardana > nano my_cloud/config/freezer/jobs.yml
  2. Commit the file to the local git repository:

    ardana > git add -A
    ardana > git commit -m "Backup job changes"
  3. Next, run the configuration processor followed by the ready-deployment playbooks:

    ardana > cd ~/openstack/ardana/ansible/
    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
  4. Run _freezer_manage_jobs.yml:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts _freezer_manage_jobs.yml

14.6 Backup/Restore Via the Horizon UI Edit source

A number of backup and restore tasks can be performed using the Horizon UI. This topic lists the available tasks, the access requirements, and limitations of using Horizon for backup and restore.

14.6.1 Accessing the UI Edit source

User name

The only supported user in this version is "ardana_backup". The login credentials are available in backup.osrc located at ~/backup.osrc/

UI access

To access the Horizon UI, follow the instructions shown in Book “User Guide”, Chapter 3 “Cloud Admin Actions with the Dashboard”. Once logged in as "ardana_backup", navigate to "Disaster Recovery" panel located in the left-hand menu where you should see "Backup and Restore."

14.6.2 Backup and Restore Operations Supported in the UI Edit source

The following Operations are supported via the UI

  • Ability to create new jobs to Backup/Restore files

  • List the freezer jobs that have completed

  • Create sessions to link multiple jobs

  • List the various nodes ( hosts/servers) on which the freezer scheduler and freezer agent are installed

14.6.3 Limitations Edit source

The following limitations apply to Freezer backups in SUSE OpenStack Cloud:

  • The UI for backup and restore is supported only if you log in as "ardana_backup". All other users will see the UI panel but the UI will not work.

  • If Backup/Restore action fails via the UI, you must check the Freezer logs for details of the failure.

  • Job Status and Job Result on the UI and backend (CLI) are not in sync.

  • For a given "Action" the following modes are not supported from the UI:

    • Microsoft SQL Server

    • Cinder

    • Nova

  • There is a known issue which will be fixed in future releases while using Start and End dates and times in creating a job. Please refrain from using those fields.

14.7 Restore from a Specific Backup Edit source

This topic describes how you can get a list of previous backups and how to restore from them.

Warning
Warning

Note that the existing contents of the directory to which you will restore your data (and its children) will be completely overwritten. You must take that into account if there is data in that directory that you want to survive the restore by either copying that data somewhere else or changing the directory to which you will restore.

By default, freezer-agent restores only the latest (most recent) backup. Here is a manual procedure to restore from a list of backups

  1. Obtain the list of backups:

    ardana > freezer backup-list
      [--limit]
      [--offset]

    --limit limit results to this limit.

    --offset return results from this offset.

    ardana > freezer backup-list
    +----------------------------------+-------------+-----------------------------+-------------------------------------------------------------------------+---------------------+-------+
    | Backup ID                        | Backup UUID | Hostname                    | Path                                                                    | Created at          | Level |
    +----------------------------------+-------------+-----------------------------+-------------------------------------------------------------------------+---------------------+-------+
    | 75f8312788fa4e95bf975807905287f8 |             | ardana-qe202-cp1-c1-m3-mgmt | /var/lib/freezer/mount_94e03f120c9e4ae78ad50328d782cea6/.               | 2018-07-06 08:26:00 |     0 |
    | 4229d71c840e4ee1b78680131695a330 |             | ardana-qe202-cp1-c1-m2-mgmt | /var/lib/freezer/mount_77d3c7a76b16435181bcaf41837cc7fe/.               | 2018-07-06 08:26:01 |     0 |
    | 6fe59b58924e43f88729dc0a1fe1290b |             | ardana-qe202-cp1-c1-m1-mgmt | /var/lib/freezer/mount_4705ac61026c4e77b6bf59b7bcfc286a/.               | 2018-07-06 16:38:50 |     0 |
  2. Use the "restore-from-date" option to restore a backup based on data/timestamp. The restore-from-data is an option available in freezer-agent. When using the parameter --restore-from-date, Freezer searches the available backups and selects the nearest older backup relative to the provided date. To use this option, the following parameters of the backup must be provided - storage target details (example, target-name, container-name), backup_name, hostname. Usually these parameters can be obtained from the backup_job.

    For example, take the following simple backup job:

    [default]
     action = backup
     backup_name = mystuffbackup
     storage = local
     container = /home/me/mystorage
     max_level = 7
         path_to_backup = ~/mydata

    Suppose you schedule that every day and you end up with backups that happened at:

    1) 2015-12-10T02:00:00
         2) 2015-12-11T02:00:00
         3) 2015-12-12T02:00:00
         3) 2015-12-13T02:00:00

    Now, if you restore using the following parameters:

    [default]
         action = restore
         backup_name = mystuffbackup
         storage = local
         container = /home/me/mystorage
         restore_abs_path = ~/mydata_restore_dir
         restore_from_date = 2015-12-11T23:00:00

    The nearest oldest backup will be number 2, taken at 2015-12-11T02:00:00.

14.8 Backup/Restore Scheduler Edit source

14.8.1 Freezer (backup/restore service) Scheduler Overview Edit source

This document explains, through examples, how to set up backup and restore jobs using the backup/restore service scheduler (referred to as Freezer Scheduler).

The scheduler is a long running process that executes the following:

  • Interact with the Freezer API

  • Generate a client_id to register the client on to the API (to identify the node during the next executions)

  • Execute the freezer-agent according the jobs information retrieved from the API

  • Write to the freezer API the outcome of the freezer-agent execution

Note
Note

Freezer API maintains information about jobs in the Elasticsearch Database.

Note
Note

You must run as root to perform any tasks using the Freezer backup/restore service.

14.8.2 Freezer (backup/restore service) Scheduler Client-ID Edit source

In SUSE OpenStack Cloud 8, Freezer Scheduler is automatically installed on the Cloud Lifecycle Manager and controller nodes.

There is a client_id for each node and its corresponds to the hostname. The client_id is created at registration time. The registration is done automatically when the scheduler executes any request to the API.

The following command lists all the freezer scheduler clients:

ardana > freezer client-list

Here is an example:

ardana > freezer client-list
+--------------------------------+----------------------------------+--------------------------------+-------------+
| Client ID                      | uuid                             | hostname                       | description |
+--------------------------------+----------------------------------+--------------------------------+-------------+
| ardana-qe202-cp1-c1-m3-mgmt    | 7869340f2efc4fb9b29e94397385ac39 | ardana-qe202-cp1-c1-m3-mgmt    |             |
| ardana-qe202-cp1-c1-m2-mgmt    | 18041c2b12054802bdaf8cc458abc35d | ardana-qe202-cp1-c1-m2-mgmt    |             |
| ardana-qe202-cp1-c1-m1-mgmt    | 884045a72026425dbcea754806d1022d | ardana-qe202-cp1-c1-m1-mgmt    |             |
| ardana-qe202-cp1-comp0001-mgmt | e404b34e5f7844ed957ca5dd90e6446f | ardana-qe202-cp1-comp0001-mgmt |             |
+--------------------------------+----------------------------------+--------------------------------+-------------+

14.8.3 Creating a Scheduler Job Edit source

  1. Log in to a controller node and create the job.

  2. Source the operating system variables and use the correct client_id. (The client-id corresponds to the node where the backup files/directory/database resides.) In SUSE OpenStack Cloud the sourcing of the variable should be done like this when you need to use ardana_backup user and backup tenant (used for infrastructure backup): Note that when you perform these actions you must be running as root. The following command will provide the necessary credentials to run the job.

    ardana > source ~/backup.osrc

    And with the following when you need to use admin user and admin tenant. The following file will contain the admin user credentials. These are not for jobs that were created automatically; they are only used for jobs created manually to be created/executed under the admin account. Jobs created automatically use the credentials stored in the backup.osrc file noted above.

    source ~/service.osrc
      {
                            "job_actions": [
                            {
                            "freezer_action": {
                            "action": "backup",
                            "mode": "fs",
                            "backup_name": "backup1",
                            "path_to_backup": "/home/user/tmp",
                            "container": "tmp_backups"
                            },
                            "max_retries": 3,
                            "max_retries_interval": 60
                            }
                            ],
                            "job_schedule": {
                            "schedule_interval": "24 hours"
                            },
                            "description": "backup for tmp dir"
                            }
  3. Upload it into the api using the correct client_id:

    ardana > freezer job-create -C CLIENT-ID --file FREEZER-FILE
    Note
    Note

    Freezer file examples can be found in Section 14.8.6, “Example Backup Job File”.

  4. The status of the jobs can be checked with:

    ardana > freezer job-list -C CLIENT-ID
  5. If no scheduling information is provided, the job will be executed as soon as possible so its status will go into a "running" state, then "completed".

You can find information about the scheduling and backup-execution in /var/log/freezer/freezer-scheduler.log and /var/log/freezer-api/freezer-api.log, respectively.

Note
Note

Recurring jobs never go into a "completed" state, as they go back into "scheduled" state.

14.8.4 Restore from a Different Node Edit source

The scheduler can be used to restore from a different node using the hostname parameter that you see in the JSON below. Here is an example conf file.

{
                    "job_actions": [
                    {
                    "freezer_action": {
                    "action": "restore",
                    "restore_abs_path": "/var/lib/mysql",
                    "hostname": "test_machine_1",
                    "backup_name": "freezer-db-mysql",
                    "container": "freezer_backup_devstack_1"
                    },
                    "max_retries": 5,
                    "max_retries_interval": 60,
                    "mandatory": true
                    }
                    ],
                    "description": "mysql test restore"
                    }

Create the job like so:

ardana > freezer job-create -C CLIENT-ID --file job-restore-mysql.conf

14.8.5 Differential Backup and Restore Edit source

The difference is in the use of the parameter always_level: 1. We also specify a different container, so it is easier to spot the files created in the Swift container:

ardana > swift list freezer_backup_devstack_1_alwayslevel

14.8.6 Example Backup Job File Edit source

Here is a sample backup file:

{
                "job_actions": [
                {
                "freezer_action": {
                "mode" : "mysql",
                "mysql_conf" : "/etc/mysql/debian.cnf",
                "path_to_backup": "/var/lib/mysql/",
                "backup_name": "freezer-db-mysql",
                "snapshot": true,
                "always_level": 1,
                "max_priority": true,
                "remove_older_than": 90,
                "container": "freezer_backup_devstack_1_alwayslevel"
                },
                "max_retries": 5,
                "max_retries_interval": 60,
                "mandatory": true
                }
                ],
                "job_schedule" : {
                },
                "description": "mysql backup"
                }

To create the job:

ardana > freezer job-create -C client_node_1 --file job-backup.conf

14.8.7 Example Restore Job File Edit source

Here is an example of job-restore.conf

{
                    "job_actions": [
                    {
                    "freezer_action": {
                    "action": "restore",
                    "restore_abs_path": "/var/lib/mysql",
                    "hostname": "test_machine_1",
                    "backup_name": "freezer-db-mysql",
                    "container": "freezer_backup_devstack_1_alwayslevel"
                    },
                    "max_retries": 5,
                    "max_retries_interval": 60,
                    "mandatory": true
                    }
                    ],
                    "description": "mysql test restore"
                    }

To create the job:

ardana > freezer job-create -C client_node_1 --file job-restore.conf

14.9 Backup/Restore Agent Edit source

This topic describes how to configure backup jobs and restore jobs.

14.9.1 Introduction Edit source

The backup/restore service agent (Freezer Agent) is a tool that is used to manually back up and restore your data. It can be run from any place you want to take a backup (or do a restore) because all SUSE OpenStack Cloud nodes have the freezer-agent installed on them. To use it, you should run as root. The agent runs in conjunction with the Section 14.8, “Backup/Restore Scheduler”. The following explains their relationship:

  • The backup/restore scheduler (openstack-freezer-scheduler, also see Section 14.8, “Backup/Restore Scheduler”) takes JSON-style config files, and can run them automatically according to a schedule in the job_schedule field of the scheduler's JSON config file. It takes anything you pass in via the job_actions field and translates those requirements into an INI-style config file. Then it runs freezer-agent. As a user, you could also run the freezer agent using freezer-agent --config file.ini, which is exactly how the scheduler runs it.

  • The agent (freezer-agent) actually performs the jobs. Whenever any backup or restore action happens, the agent is the one doing the actual work. It can be run directly by the user, as noted above, or by the scheduler. It accepts either command-line flags (such as --action backup) or INI-style config files.

    Note
    Note

    You can run freezer-agent --help to view a definitive list of all possible flags that can be used (with the transform rules mentioned) in these configuration files.

For SUSE OpenStack Cloud 8, you must follow these steps to perform backups:

  1. Define what you want to back up.

  2. Define a mode for that backup. The following modes are available:

    • fs (filesystem) (default)

    • mysql

    • sqlserver

    Note
    Note

    It is recommended that you use snapshots if the mode is mysql or sqlserver.

  3. Define whether to use a snapshot in the file system for the backup:

    • In Unix systems LVM is used (when available).

    • In Windows systems virtual shadow copies are used.

  4. Define a storage media in a job from the following list:

    • Swift (requires OpenStack credentials)(default)

    • Local (no credentials required)

    • SSH (no credentials required) (not implemented on Windows)

14.9.2 Basic Configuration for Backups Edit source

There are several mandatory parameters you need to specify in order to execute a backup. Note storage is optional:

  • action (backup by default)

  • mode (fs by default)

  • path-to-backup

  • backup-name

  • container (Swift container or local path)

  • storage is not mandatory. It is Swift by default.

For SUSE OpenStack Cloud 8, you can create a backup using only mandatory values, as in the following example:

ardana > freezer-agent --action backup --mode fs --storage swift --path-to-backup /home/user/tmp --container tmp_backups --backup-name backup1

Running the above command from the command line will cause this backup to execute once. To create a configuration file for this same backup, in case you want to run it manually another time, create a configuration file like the one below. Note that in the config file, the parameter names such as backup-name will use underscores instead of dashes. Thus backup-name as used in the CLI will be backup_name when used in the config file. Note also that where you use -- in the CLI, such as --mode, you do not use the -- in the config file.

[default]
                action = backup
                mode = fs
                backup_name = backup1
                path_to_backup = /home/user/tmp
                container = tmp_backups

A configuration file similar to the one above will be generated if you create a JSON configuration file for automated jobs to be run by the scheduler. Instructions on how to do that are found on the Section 14.8, “Backup/Restore Scheduler” page.

14.9.3 Restoring your Data Edit source

For SUSE OpenStack Cloud 8, you must do the following in order to restore data after a backup:

  1. Select a backup to restore.

  2. Define a mode for the restore: The following modes are available:

    • fs (filesystem) (default)

    • mysql

    • sqlserver

  3. If the restore involves an application (such as MariaDB) remember to shut down the application or service and start it again after the restore.

14.9.4 Basic Configuration for Restoring Edit source

To restore from a backup, note that in some cases you must stop the service (for instance, MariaDB) before the restore.

There are several parameters that are required and there are some optional parameters used to execute a restore:

  • action (backup by default)

  • mode (fs by default)

  • restore-abs-path

  • backup-name

  • container (Swift container or local path)

  • restore-from-host

  • restore-from-date (optional)

  • storage is not mandatory. It is Swift by default

You can create a restore using mandatory values, as in the following example:

ardana > freezer-agent --action restore --mode fs --storage swift --restore-abs-path /home/user/tmp --container tmp_backups --backup-name backup1 --restore-from-host ubuntu

To create a configuration file for this same restore, the file would look like the one below. Note that in the config file, the parameter names such as backup-name will use underscores instead of dashes. Thus backup-name as used in the CLI will be backup_name when used in the config file. Note also that where you use -- in the CLI, such as --mode, you do not use the -- in the config file. This is the same format as used above for backup configuration.

  {
                    "job_actions": [
                    {
                    "freezer_action": {
                    "action": "restore",
                    "mode": "fs",
                    "backup_name": "backup1",
                    "restore_abs_path": "/home/user/tmp",
                    "container": "tmp_backups",
                    "hostname": "ubuntu"
                    },
                    "max_retries": 3,
                    "max_retries_interval": 60
                    }
                    ],
                    "description": "backup for tmp dir"
                    }

14.10 Backup and Restore Limitations Edit source

The following limitations apply to backups created by the Freezer backup and restore service in SUSE OpenStack Cloud:

  • Recovery of the following services (or cloud topologies) will be partially supported as they need additional data (other than MariaDB) to return to fully functional.

    • ESX Cloud

    • Network services - LBaaS and VPNaaS

  • Logging data (that is, log files)

14.11 Disabling Backup/Restore before Deployment Edit source

Backups are enabled by default. Therefore, you must take action if you want backups to be disabled for any reason. This topic explains how to disable default backup jobs before completing the installation of your cloud.

Warning
Warning

You should make modifications in the ~/openstack/my_cloud/ directory before running the configuration processor and ready-deployment steps.

14.11.1 Disable backups before installation: Edit source

To disable deployment of the Freezer backup and restore service, remove the following lines in control_plane.yml:

  • freezer-agent

  • freezer-api

Note
Note

This action is required even if you already removed Freezer lines from your model (control_plane.yml).

14.11.2 Deploy Freezer but disable backup/restore job creation: Edit source

It is also possible to allow Freezer deployment yet prevent the lifecycle manager from creating automatic backup jobs. By default, the lifecycle manager deployment automatically creates jobs for the backup and restore of the following:

  • Lifecycle-manager node

  • MySQL database

  • Swift rings

Before running the configuration processor, you can prevent Freezer from automatically creating backup and restore jobs by changing the variables freezer_create_backup_jobs and freezer_create_restore_jobs to false in:

~/openstack/my_cloud/config/freezer/activate_jobs.yml

Alternatively, you can disable the creation of those jobs while launching the deployment process, as follows:

ardana > ansible-playbook -i hosts/verb_hosts site.yml -e '{ "freezer_create_backup_jobs": false }' -e '{ "freezer_create_restore_jobs": false }'

When using these options, the Freezer infrastructure will still be deployed but will not execute any backups.

14.11.3 Disable backup and restore jobs for a specific service Edit source

To manage which jobs will be enabled, set the appropriate paramters in the jobs.yml freezer jobs configuration file:

~/openstack/my_cloud/config/freezer/jobs.yml

You can completely disable the backup of a component by changing the enabled field that corresponds to that service to false in jobs.yml.

You can specify where a job will store its backup by setting store_in_swift, store_in_ssh, store_in_local to true or false. Note that these are not mutually exclusive. You can set true for all of these backup targets. Setting SSH, Swift, and local to true will cause one backup job (and one restore job) per storage target to be created.

Note also that even if store_in_ssh is set to true, the SSH backup job will not be created unless SSH credentials are provided in /openstack/my_cloud/config/freezer/ssh_credentials.yml.

When setting store_in_local to true, the backup job will store backups on the server executing the backup. This option is useful, for example, if you plan to mount an NFS share and want your backup stored on it. You need to provide the path where the backup will be stored by setting the local_storage_base_dir parameter.

By default, one backup job per storage medium per component will be created. A corresponding restore job for each of those backup jobs will also be created by default. These jobs can be used to quickly restore the corresponding backup. To disable the creation of these restore jobs, change also_create_restore_job to false.

14.11.4 Activating and deactivating jobs after cloud deployment Edit source

  1. Make modifications similar to those discussed above in /openstack/my_cloud/config/freezer/jobs.yml.

  2. Commit modifications to the git repo

    ardana > git add -A
    ardana > git commit -m "A message that explains what modifications have been made"
  3. Run the configuration processor

    ardana > cd ~/openstack/ardana/ansible/
    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
  4. Run the ready deployment playbook. (This will update the scratch/... directories with all of the above modifications).

    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
  5. Change directories to scratch

    ardana > cd ~/scratch/ansible/next/ardana/ansible
  6. Run _freezer_manage_jobs.yml

    ardana > ansible-playbook -i hosts/verb_hosts _freezer_manage_jobs.yml

14.12 Enabling, Disabling and Restoring Backup/Restore Services Edit source

14.12.1 Stop, Start and Restart the Backup Services Edit source

To stop the Freezer backup and restore service globally, launch the following playbook from the Cloud Lifecycle Manager (this will stop all freezer-api and all freezer-agent running on your clusters):

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts freezer-stop.yml

To start the Freezer backup and restore service globally, launch the following playbook from the Cloud Lifecycle Manager:

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts freezer-start.yml

To restart the Freezer backup and restore services use the ansible playbooks from above.

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts freezer-stop.yml
ardana > ansible-playbook -i hosts/verb_hosts freezer-start.yml

It is possible to target only specific nodes using the ansible --limit parameter.

14.12.2 Manually Edit source

For the freezer-agent:

  1. Connect to the concerned host.

  2. Run the following command to stop the freezer agent:

    tux > sudo systemctl stop openstack-freezer-scheduler

    or run the following command to start the freezer-agent:

    tux > sudo systemctl start openstack-freezer-scheduler

    or run the following command to restart the freezer-agent:

    tux > sudo systemctl restart openstack-freezer-scheduler

For the freezer-api:

  1. Connect to the concerned host.

  2. Run the following commands to stop the freezer-api:

    tux > sudo rm /etc/apache2/vhosts.d/freezer-modwsgi.conf
    tux > sudo systemctl reload apache2

    or run the following commands to start the freezer-api:

    tux > sudo ln -s /etc/apache2/vhosts.d/freezer-modwsgi.vhost /etc/apache2/vhosts.d/freezer-modwsgi.conf
    tux > sudo systemctl reload apache2

14.13 Backing up and Restoring Audit Logs Edit source

To enable backup of the audit log directory, follow these steps. Before performing the following steps, run through Section 12.2.7.2, “Enable Audit Logging” .

  • First, from the Cloud Lifecycle Manager node, run the following playbook:

    tux > sudo ansible-playbook -i hosts/verb_hosts _freezer_manage_jobs.yml

    This will create a job to back up the audit log directory on any node where that directory exists. In order to limit this to only specific nodes, use the --limit option of Ansible

In order to restore the logs, follow one of the following procedures:

To restore from the node that made the backup to the same node directly in the audit log directory (for example, the folder has been deleted):

  1. Connect to the node

  2. Source OpenStack credentials

    ardana > source ~/backup.osrc
  3. List pre-configured jobs

    ardana > freezer job-list -C `hostname`
  4. Note the id corresponding to the job: "Ardana Default: Audit log restore from ..."

  5. Schedule the restore

    ardana > freezer job-start JOB-ID

To restore the backup in another directory, or from another host,

  1. Connect to the node

  2. Source the OpenStack credentials

    ardana > source ~/backup.osrc
  3. Choose from where you will restore (from Swift or from and SSH backup)

    1. Swift: Create a restore config file (for example, restore.ini) with the following content to restore from a swift backup (make sure to fill in <value> )

      [default]
      action = restore
      backup_name = freezer_audit_log_backup
      container = freezer_audit_backup
      log_file = /freezer-agent/freezer-agent.log
      restore_abs_path = PATH TO THE DIRECTORY WHERE YOU WANT TO RESTORE
      hostname = HOSTNAME OF THE HOST YOU WANT TO RESTORE THE BACKUP FROM
    2. Or: Create a restore configuration file (for example, restore.ini) with the following content to restore from an SSH backup (make sure to fill in VALUE) SSH information is available in openstack/my_cloud/config/freezer/ssh_credentials.yml

      [default]
      action = restore
      storage = ssh
      backup_name = freezer_audit_log_backup
      log_file = /freezer-agent/freezer-agent.log
      ssh_key = /etc/freezer/ssh_key
      restore_abs_path = PATH TO THE DIRECTORY WHERE YOU WANT TO RESTORE
      hostname = HOSTNAME OF THE HOST YOU WANT TO RESTORE THE BACKUP FROM
      ssh_host = YOUR SSH BACKUP HOST
      ssh_port = YOUR SSH BACKUP PORT
      ssh_username = YOUR SSH BACKUP USERNAME
      container = YOUR SSH BACKUP BASEDIR/freezer_audit_backup
  4. Run the freezer-agent to restore

    freezer-agent --config restore.ini
Print this page