14 Backup and Restore

The following limitations apply to backups created by the Freezer backup and restore service in HPE Helion OpenStack:

As part of the installation procedure in HPE Helion OpenStack, automatic backup/restore jobs are set up to back up to Swift via the Freezer scheduler component of the backup and restore service. The backup jobs perform scheduled backups of HPE Helion OpenStack control plane data (files/directories/db). The restore jobs can be used to restore appropriate control plane data. Additional automatic jobs can be added to backup/restore from the secure shell (SSH) server that you set up/designate. It is recommended that you set up SSH backups so that in the event that you lose all of your control plane nodes at once, you have a backup on remote servers that you can use to restore the control plane nodes. Note that you do not have to restore from the SSH location if only one or two control plane nodes are lost. In that case, they can be recovered from the data on the remaining control plane node following the restore procedures in Section 13.2.2.2, “Recovering the Control Plane”. That document also explains how to recover using your remote server SSH backups.

While control plane backups to Swift are set up automatically, you must use the following procedure to set up SSH backups.

By default, during HPE Helion OpenStack 8 deployment, backup jobs are automatically deployed to Swift, the MySQL database, the Cloud Lifecycle Manager, and Swift rings. Restore jobs are also deployed for convenience. It is more secure to store those backups also outside of the HPE Helion OpenStack infrastructure. If you provide all the values required in the following file, jobs will also be deployed to backup and restore to/from an SSH server of your choice:

~/openstack/my_cloud/config/freezer/ssh_credentials.yml

You must provide the following connection information to this server:

Before running the configuration processor, edit the following file:

~/openstack/my_cloud/config/freezer/ssh_credentials.yml

All parameters are mandatory. Take care in providing the SSH private key.

In this example, the information is as follows:

Please replace these values to meet your own requirements, as appropriate.

Your server is now ready to receive backups. If you wish, you can check our advice on how to secure it in Section 14.4.8, “Securing your SSH backup server”.

If you already deployed your cloud and forgot to configure SSH backups, or if you wish to modify the settings for where the backups are stored, follow the following instructions:

There is a strict policy of firewalling deployed with HPE Helion OpenStack. If you use a non-standard SSH port, you may need to specifically open it by using the following process:

You can do the following to harden an SSH server (these techniques are well documented on the internet):

Remove the key pair generated earlier on the backup server: the only thing needed is the .ssh/authorized_keys. You can remove the .ssh/id_rsa and .ssh/id_rsa.pub files. Be sure to save a backup of them somewhere.

Run the following commands to finish configuring the firewall.

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts _freezer_manage_jobs.yml

The following Operations are supported via the UI

The following limitations apply to Freezer backups in HPE Helion OpenStack:

This document explains, through examples, how to set up backup and restore jobs using the backup/restore service scheduler (referred to as Freezer Scheduler).

The scheduler is a long running process that executes the following:

Note

Freezer API maintains information about jobs in the Elasticsearch Database.

Note

You must run as root to perform any tasks using the Freezer backup/restore service.

In HPE Helion OpenStack 8, Freezer Scheduler is automatically installed on the Cloud Lifecycle Manager and controller nodes.

There is a client_id for each node and its corresponds to the hostname. The client_id is created at registration time. The registration is done automatically when the scheduler executes any request to the API.

The following command lists all the freezer scheduler clients:

ardana > freezer client-list

Here is an example:

ardana > freezer client-list
+--------------------------------+----------------------------------+--------------------------------+-------------+
| Client ID                      | uuid                             | hostname                       | description |
+--------------------------------+----------------------------------+--------------------------------+-------------+
| ardana-qe202-cp1-c1-m3-mgmt    | 7869340f2efc4fb9b29e94397385ac39 | ardana-qe202-cp1-c1-m3-mgmt    |             |
| ardana-qe202-cp1-c1-m2-mgmt    | 18041c2b12054802bdaf8cc458abc35d | ardana-qe202-cp1-c1-m2-mgmt    |             |
| ardana-qe202-cp1-c1-m1-mgmt    | 884045a72026425dbcea754806d1022d | ardana-qe202-cp1-c1-m1-mgmt    |             |
| ardana-qe202-cp1-comp0001-mgmt | e404b34e5f7844ed957ca5dd90e6446f | ardana-qe202-cp1-comp0001-mgmt |             |
+--------------------------------+----------------------------------+--------------------------------+-------------+

You can find information about the scheduling and backup-execution in /var/log/freezer/freezer-scheduler.log and /var/log/freezer-api/freezer-api.log, respectively.

Note

Recurring jobs never go into a "completed" state, as they go back into "scheduled" state.

The scheduler can be used to restore from a different node using the hostname parameter that you see in the JSON below. Here is an example conf file.

{
                    "job_actions": [
                    {
                    "freezer_action": {
                    "action": "restore",
                    "restore_abs_path": "/var/lib/mysql",
                    "hostname": "test_machine_1",
                    "backup_name": "freezer-db-mysql",
                    "container": "freezer_backup_devstack_1"
                    },
                    "max_retries": 5,
                    "max_retries_interval": 60,
                    "mandatory": true
                    }
                    ],
                    "description": "mysql test restore"
                    }

Create the job like so:

ardana > freezer job-create -C CLIENT-ID --file job-restore-mysql.conf

The difference is in the use of the parameter always_level: 1. We also specify a different container, so it is easier to spot the files created in the Swift container:

ardana > swift list freezer_backup_devstack_1_alwayslevel

Here is a sample backup file:

{
                "job_actions": [
                {
                "freezer_action": {
                "mode" : "mysql",
                "mysql_conf" : "/etc/mysql/debian.cnf",
                "path_to_backup": "/var/lib/mysql/",
                "backup_name": "freezer-db-mysql",
                "snapshot": true,
                "always_level": 1,
                "max_priority": true,
                "remove_older_than": 90,
                "container": "freezer_backup_devstack_1_alwayslevel"
                },
                "max_retries": 5,
                "max_retries_interval": 60,
                "mandatory": true
                }
                ],
                "job_schedule" : {
                },
                "description": "mysql backup"
                }

To create the job:

ardana > freezer job-create -C client_node_1 --file job-backup.conf

Here is an example of job-restore.conf

{
                    "job_actions": [
                    {
                    "freezer_action": {
                    "action": "restore",
                    "restore_abs_path": "/var/lib/mysql",
                    "hostname": "test_machine_1",
                    "backup_name": "freezer-db-mysql",
                    "container": "freezer_backup_devstack_1_alwayslevel"
                    },
                    "max_retries": 5,
                    "max_retries_interval": 60,
                    "mandatory": true
                    }
                    ],
                    "description": "mysql test restore"
                    }

To create the job:

ardana > freezer job-create -C client_node_1 --file job-restore.conf

The backup/restore service agent (Freezer Agent) is a tool that is used to manually back up and restore your data. It can be run from any place you want to take a backup (or do a restore) because all HPE Helion OpenStack nodes have the freezer-agent installed on them. To use it, you should run as root. The agent runs in conjunction with the Section 14.8, “Backup/Restore Scheduler”. The following explains their relationship:

For HPE Helion OpenStack 8, you must follow these steps to perform backups:

There are several mandatory parameters you need to specify in order to execute a backup. Note storage is optional:

For HPE Helion OpenStack 8, you can create a backup using only mandatory values, as in the following example:

ardana > freezer-agent --action backup --mode fs --storage swift --path-to-backup /home/user/tmp --container tmp_backups --backup-name backup1

Running the above command from the command line will cause this backup to execute once. To create a configuration file for this same backup, in case you want to run it manually another time, create a configuration file like the one below. Note that in the config file, the parameter names such as backup-name will use underscores instead of dashes. Thus backup-name as used in the CLI will be backup_name when used in the config file. Note also that where you use -- in the CLI, such as --mode, you do not use the -- in the config file.

[default]
                action = backup
                mode = fs
                backup_name = backup1
                path_to_backup = /home/user/tmp
                container = tmp_backups

A configuration file similar to the one above will be generated if you create a JSON configuration file for automated jobs to be run by the scheduler. Instructions on how to do that are found on the Section 14.8, “Backup/Restore Scheduler” page.

For HPE Helion OpenStack 8, you must do the following in order to restore data after a backup:

To restore from a backup, note that in some cases you must stop the service (for instance, MariaDB) before the restore.

There are several parameters that are required and there are some optional parameters used to execute a restore:

You can create a restore using mandatory values, as in the following example:

ardana > freezer-agent --action restore --mode fs --storage swift --restore-abs-path /home/user/tmp --container tmp_backups --backup-name backup1 --restore-from-host ubuntu

To create a configuration file for this same restore, the file would look like the one below. Note that in the config file, the parameter names such as backup-name will use underscores instead of dashes. Thus backup-name as used in the CLI will be backup_name when used in the config file. Note also that where you use -- in the CLI, such as --mode, you do not use the -- in the config file. This is the same format as used above for backup configuration.

  {
                    "job_actions": [
                    {
                    "freezer_action": {
                    "action": "restore",
                    "mode": "fs",
                    "backup_name": "backup1",
                    "restore_abs_path": "/home/user/tmp",
                    "container": "tmp_backups",
                    "hostname": "ubuntu"
                    },
                    "max_retries": 3,
                    "max_retries_interval": 60
                    }
                    ],
                    "description": "backup for tmp dir"
                    }

To disable deployment of the Freezer backup and restore service, remove the following lines in control_plane.yml:

Note

This action is required even if you already removed Freezer lines from your model (control_plane.yml).

It is also possible to allow Freezer deployment yet prevent the lifecycle manager from creating automatic backup jobs. By default, the lifecycle manager deployment automatically creates jobs for the backup and restore of the following:

Before running the configuration processor, you can prevent Freezer from automatically creating backup and restore jobs by changing the variables freezer_create_backup_jobs and freezer_create_restore_jobs to false in:

~/openstack/my_cloud/config/freezer/activate_jobs.yml

Alternatively, you can disable the creation of those jobs while launching the deployment process, as follows:

ardana > ansible-playbook -i hosts/verb_hosts site.yml -e '{ "freezer_create_backup_jobs": false }' -e '{ "freezer_create_restore_jobs": false }'

When using these options, the Freezer infrastructure will still be deployed but will not execute any backups.

To manage which jobs will be enabled, set the appropriate paramters in the jobs.yml freezer jobs configuration file:

~/openstack/my_cloud/config/freezer/jobs.yml

You can completely disable the backup of a component by changing the enabled field that corresponds to that service to false in jobs.yml.

You can specify where a job will store its backup by setting store_in_swift, store_in_ssh, store_in_local to true or false. Note that these are not mutually exclusive. You can set true for all of these backup targets. Setting SSH, Swift, and local to true will cause one backup job (and one restore job) per storage target to be created.

Note also that even if store_in_ssh is set to true, the SSH backup job will not be created unless SSH credentials are provided in /openstack/my_cloud/config/freezer/ssh_credentials.yml.

When setting store_in_local to true, the backup job will store backups on the server executing the backup. This option is useful, for example, if you plan to mount an NFS share and want your backup stored on it. You need to provide the path where the backup will be stored by setting the local_storage_base_dir parameter.

By default, one backup job per storage medium per component will be created. A corresponding restore job for each of those backup jobs will also be created by default. These jobs can be used to quickly restore the corresponding backup. To disable the creation of these restore jobs, change also_create_restore_job to false.

To stop the Freezer backup and restore service globally, launch the following playbook from the Cloud Lifecycle Manager (this will stop all freezer-api and all freezer-agent running on your clusters):

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts freezer-stop.yml

To start the Freezer backup and restore service globally, launch the following playbook from the Cloud Lifecycle Manager:

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts freezer-start.yml

To restart the Freezer backup and restore services use the ansible playbooks from above.

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts freezer-stop.yml
ardana > ansible-playbook -i hosts/verb_hosts freezer-start.yml

It is possible to target only specific nodes using the ansible --limit parameter.

For the freezer-agent:

For the freezer-api: