Applies to SUSE OpenStack Cloud 9

17 Backup and Restore #

The following sections cover backup and restore operations. Before installing your cloud, there are several things you must do so that you achieve the backup and recovery results you need. SUSE OpenStack Cloud comes with playbooks and procedures to recover the control plane from various disaster scenarios.

As of SUSE OpenStack Cloud 9, Freezer (a distributed backup restore and disaster recovery service) is no longer supported; backup and restore are manual operations.

Consider Section 17.2, “Enabling Backups to a Remote Server” in case you lose cloud servers that back up and restore services.

The following features are supported:

File system backup using a point-in-time snapshot.
Strong encryption: AES-256-CFB.
MariaDB database backup with LVM snapshot.
Restoring your data from a previous backup.
Low storage requirement: backups are stored as compressed files.
Flexible backup (both incremental and differential).
Data is archived in GNU Tar format for file-based incremental backup and restore.
When a key is provided, Open SSL is used to encrypt data (AES-256-CFB).

17.1 Manual Backup Overview #

This section covers manual backup and some restore processes. Full documentation for restore operations is in Section 15.2, “Unplanned System Maintenance”.To back up outside the cluster, refer to Section 17.2, “Enabling Backups to a Remote Server”. Backups of the following types of resources are covered:

Cloud Lifecycle Manager Data. All important information on the Cloud Lifecycle Manager
MariaDB database that is part of the Control Plane. The MariaDB database contains most of the data needed to restore services. MariaDB supports full back up and recovery for all services. Logging data in Elasticsearch is not backed up. swift objects are not backed up because of the redundant nature of swift.
swift Rings used in the swift storage deployment. swift rings are backed up so that you can recover more quickly than rebuilding with swift. swift can rebuild the rings without this backup data, but automatically rebuilding the rings is slower than restoring from a backup.
Audit Logs. Audit Logs are backed up to provide retrospective information and statistical data for performance and security purposes.

The following services will be backed up. Specifically, the data needed to restore the services is backed up. This includes databases and configuration-related files.

Important

Data content for some services is not backed up, as indicated below.

ceilometer. There is no backup of metrics data.
cinder. There is no backup of the volumes.
glance. There is no backup of the images.
heat
horizon
keystone
neutron
nova. There is no backup of the images.
swift. There is no backup of the objects. swift has its own high availability and redundancy. swift rings are backed up. Although swift can rebuild the rings itself, restoring from backup is faster.
Operations Console
monasca. There is no backup of the metrics.

17.2 Enabling Backups to a Remote Server #

We recommend that you set up a remote server to store your backups, so that you can restore the control plane nodes. This may be necessary if you lose all of your control plane nodes at the same time.

Important

A remote backup server must be set up before proceeding.

You do not have to restore from the remote server if only one or two control plane nodes are lost. In that case, the control planes can be recovered from the data on a remaining control plane node following the restore procedures in Section 15.2.3.2, “Recovering the Control Plane”.

17.2.1 Securing your SSH backup server #

You can do the following to harden an SSH server:

Disable root login
Move SSH to a non-default port (the default SSH port is 22)
Disable password login (only allow RSA keys)
Disable SSH v1
Authorize Secure File Transfer Protocol (SFTP) only for the designated backup user (disable SSH shell)
Firewall SSH traffic to ensure it comes from the SUSE OpenStack Cloud address range
Install a Fail2Ban solution
Restrict users who are allowed to SSH
Additional suggestions are available online

Remove the key pair generated earlier on the backup server; the only thing needed is .ssh/authorized_keys. You can remove the .ssh/id_rsa and .ssh/id_rsa.pub files. Be sure to save a backup of them.

17.2.2 General tips #

Provide adequate space in the directory that is used for backup.
Monitor the space left on that directory.
Keep the system up to date on that server.

17.3 Manual Backup and Restore Procedures #

Each backup requires the following steps:

Create a snapshot.
Mount the snapshot.
Generate a TAR archive and save it.
Unmount and delete the snapshot.

17.3.1 Cloud Lifecycle Manager Data Backup #

The following procedure is used for each of the seven BACKUP_TARGETS (list below). Incremental backup instructions follow the full backup procedure. For both full and incremental backups, the last step of the procedure is to unmount and delete the snapshot after the TAR archive has been created and saved. A new snapshot must be created every time a backup is created.

Procedure 17.1: Manual Backup Setup #

Create a snapshot on the Cloud Lifecycle Manager in (ardana-vg), the location where all Cloud Lifecycle Manager data is stored.
```
ardana > sudo lvcreate --size 2G --snapshot --permission r \
--name lvm_clm_snapshot /dev/ardana-vg/root
```
Note
If you have stored additional data or files in your ardana-vg directory, you may need more space than the 2G indicated for the size parameter. In this situation, create a preliminary TAR archive with the tar command on the directory before creating a snapshot. Set the size snapshot parameter larger than the size of the archive.

Mount the snapshot

ardana > sudo mkdir /var/tmp/clm_snapshot
ardana > sudo mount -o ro /dev/ardana-vg/lvm_clm_snapshot /var/tmp/clm_snapshot

Generate a TAR archive (does not apply to incremental backups) with an appropriate BACKUP_TAR_ARCHIVE_NAME.tar.gz backup file for each of the following BACKUP_TARGETS.
Backup Targets
- home
- ssh
- shadow
- passwd
- group
The backup TAR archive should contain only the necessary data; nothing extra. Some of the archives will be stored as directories, others as files. The backup commands are slightly different for each type.
If the BACKUP_TARGET is a directory, then that directory must be appended to /var/tmp/clm_snapshot/TARGET_DIR. If the BACKUP_TARGET is a file, then its parent directory must be appended to /var/tmp/clm_snapshot/.
In the commands that follow, replace BACKUP_TARGET with the appropriate BACKUP_PATH (replacement table is below).
```
ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --file BACKUP_TAR_ARCHIVE_NAME.tar.gz -C \
/var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIR
```
- If BACKUP_TARGET is a directory, replace TARGET_DIR with BACKUP_PATH.
  For example, where BACKUP_PATH=/etc/ssh/ (a directory):
```
ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --file ssh.tar.gz -C /var/tmp/clm_snapshot/etc/ssh .
```
- If BACKUP_TARGET is a file (not a directory), replace TARGET_DIR with the parent directory of BACKUP_PATH.
  For example, where BACKUP_PATH=/etc/passwd (a file):
```
ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --file passwd.tar.gz -C /var/tmp/clm_snapshot/etc/passwd
```

Save the TAR archive to the remote server.

ardana > scp TAR_ARCHIVE USER@REMOTE_SERVER

Use the following commands to unmount and delete a snapshot.

ardana > sudo umount -l -f /var/tmp/clm_snapshot; rm -rf /var/tmp/clm_snapshot
ardana > sudo lvremove -f /dev/ardana-vg/lvm_clm_snapshot

The table below shows Cloud Lifecycle Manager backup_targets and their respective backup_paths.

Table 17.1: Cloud Lifecycle Manager Backup Paths #

backup_name	backup_path
home_backup	/var/lib/ardana (file)
etc_ssh_backup	/etc/ssh/ (directory)
shadow_backup	/etc/shadow (file)
passwd_backup	/etc/passwd (file)
group_backup	/etc/group (file)

17.3.1.1 Cloud Lifecycle Manager Incremental Backup #

Incremental backups require a meta file. If you use the incremental backup option, a meta file must be included in the tar command in the initial backup and whenever you do an incremental backup. A copy of the original meta file should be stored in each backup. The meta file is used to determine the incremental changes from the previous backup, so it is rewritten with each incremental backup.

Versions are useful for incremental backup because they provide a way to differentiate between each backup. Versions are included in the tar command.

Every incremental backup requires creating and mounting a separate snapshot. After the TAR archive is created, the snapshot is unmounted and deleted.

To prepare for incremental backup, follow the steps in Procedure 17.1, “Manual Backup Setup” with the following differences in the commands for generating a tar archive.

First time full backup

ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --listed-incremental=PATH_TO_YOUR_META \
--file BACKUP_TAR_ARCHIVE_NAME.tar.gz -C \
/var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIR

For example, where BACKUP_PATH=/etc/ssh/

ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--listed-incremental=mysshMeta --file ssh.tar.gz -C \
/var/tmp/clm_snapshot/etc/ssh .

Incremental backup

ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --listed-incremental=PATH_TO_YOUR_META\
--file BACKUP_TAR_ARCHIVE_NAME_VERSION.tar.gz -C \
/var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIR

For example, where BACKUP_PATH=/etc/ssh/:

ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--listed-incremental=mysshMeta --file \
ssh_v1.tar.gz -C \
/var/tmp/clm_snapshot/etc/ssh .

After creating an incremental backup, use the following commands to unmount and delete a snapshot.

ardana > sudo umount -l -f /var/tmp/clm_snapshot; rm -rf /var/tmp/clm_snapshot
ardana > sudo lvremove -f /dev/ardana-vg/lvm_clm_snapshot

17.3.1.2 Encryption #

When a key is provided, Open SSL is used to encrypt data (AES-256-CFB). Backup files can be encrypted with the following command:

ardana > sudo openssl enc -aes-256-cfb -pass file:ENCRYPT_PASS_FILE_PATH -in \
YOUR_BACKUP_TAR_ARCHIVE_NAME.tar.gz -out YOUR_BACKUP_TAR_ARCHIVE_NAME.tar.gz.enc

For example, using the ssh.tar.gz generated above:

ardana > sudo openssl enc  -aes-256-cfb -pass file:myEncFile -in ssh.tar.gz  -out ssh.tar.gz.enc

17.3.2 MariaDB Database Backup #

When backing up MariaDB, the following process must be performed on all nodes in the cluster. It is similar to the backup procedure above for the Cloud Lifecycle Manager (see Procedure 17.1, “Manual Backup Setup”). The difference is the addition of SQL commands, which are run with the create_db_snapshot.yml playbook.

Create the create_db_snapshot.yml file in ~/scratch/ansible/next/ardana/ansible/ on the deployer with the following content:

- hosts: FND-MDB
vars:
 - snapshot_name: lvm_mysql_snapshot
 - lvm_target: /dev/ardana-vg/mysql

 tasks:
 - name: Cleanup old snapshots
   become: yes
   shell: |
    lvremove -f /dev/ardana-vg/{{ snapshot_name }}
   ignore_errors: True

 - name: Create snapshot
   become: yes
   shell: |
    lvcreate --size 2G --snapshot --permission r --name {{ snapshot_name }} {{ lvm_target }}
   register: snapshot_st
   ignore_errors: True

 - fail:
     msg: "Fail to create snapshot on  {{ lvm_target }}"
   when: snapshot_st.rc != 0

Note

Verify the validity of the lvm_target variable (which refers to the actual database LVM volume) before proceeding with the backup.

Doing the MariaDB backup

We recommend storing the MariaDB version with your backup. The following command saves the MariaDB version as MARIADB_VER.
```
mysql -V | grep -Eo '(\S+?)-MariaDB' > MARIADB_VER
```
Open a MariaDB client session on all controllers.
Run the command to spread read lock on all controllers and keep the MariaDB session open.
```
>> FLUSH TABLES WITH READ LOCK;
```

Open a new terminal and run the create_db_snapshot.yml playbook created above.

ardana > cd ~/scratch/ansible/next/ardana/ansible/
ardana > ansible-playbook -i hosts/verb_hosts create_db_snapshot.yml

Go back to the open MariaDB session and run the command to flush the lock on all controllers.
```
>> UNLOCK TABLES;
```

Mount the snapshot

dbnode>> mkdir /var/tmp/mysql_snapshot
dbnode>> sudo mount -o ro /dev/ardana-vg/lvm_mysql_snapshot  /var/tmp/mysql_snapshot

On each database node, generate a TAR archive with an appropriate BACKUP_TAR_ARCHIVE_NAME.tar.gz backup file for the BACKUP_TARGET.
The backup_name is mysql_backup and the backup_path (BACKUP_TARGET) is /var/lib/mysql/.
```
dbnode>> sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--file mydb.tar.gz /var/tmp/mysql_snapshot/var/lib/mysql .
```

Unmount and delete the MariaDB snapshot on each database node.

dbnode>> sudo  umount -l -f /var/tmp/mysql_snapshot; \
sudo rm -rf /var/tmp/mysql_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_mysql_snapshot

17.3.2.1 Incremental MariaDB Database Backup #

Versions are useful for incremental backup because they provide a way to differentiate between each backup. Versions are included in the tar command.

To prepare for incremental backup, follow the steps in the previous section except for the tar commands. Incremental backup tar commands must have additional information.

First time MariaDB database full backup

dbnode>> sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --listed-incremental=PATH_TO_YOUR_DB_META \
--file mydb.tar.gz -C /var/tmp/mysql_snapshot/var/lib/mysql .

For example, where BACKUP_PATH=/var/lib/mysql/:

dbnode>> sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--listed-incremental=mydbMeta --file mydb.tar.gz -C \
/var/tmp/mysql_snapshot/var/lib/mysql .

Incremental MariaDB database backup

dbnode>> sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --listed-incremental=PATH_TO_YOUR_META\
--file BACKUP_TAR_ARCHIVE_NAME_VERSION.tar.gz -C \
/var/tmp/clm_snapshotTARGET_DIR

For example, where BACKUP_PATH=/var/lib/mysql/:

dbnode>> sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--listed-incremental=mydbMeta --file \
mydb_v1.tar.gz -C /var/tmp/mysql_snapshot/var/lib/mysql .

After creating and saving the TAR archive, unmount and delete the snapshot.

dbnode>> sudo  umount -l -f /var/tmp/mysql_snapshot; \
sudo rm -rf /var/tmp/mysql_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_mysql_snapshot

17.3.2.2 MariaDB Database Encryption #

Encrypt your MariaDB database backup following the instructions in Section 17.3.1.2, “Encryption”
Upload your BACKUP_TARGET.tar.gz to your preferred remote server.

17.3.3 swift Ring Backup #

The following procedure is used to back up swift rings. It is similar to the Cloud Lifecycle Manager backup (see Procedure 17.1, “Manual Backup Setup”).

Important

The steps must be performed only on the building server (For more information, see Section 18.6.2.4, “Identifying the Swift Ring Building Server”.).

The backup_name is swift_builder_dir_backup and the backup_path is /etc/swiftlm/.

Create a snapshot

ardana > sudo lvcreate --size 2G --snapshot --permission r \
--name lvm_root_snapshot /dev/ardana-vg/root

Mount the snapshot

ardana > mkdir /var/tmp/root_snapshot; sudo mount -o ro \
/dev/ardana-vg/lvm_root_snapshot /var/tmp/root_snapshot

Create the TAR archive

ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--file swring.tar.gz -C /var/tmp/root_snapshot/etc/swiftlm .

Upload your swring.tar.gz TAR archive to your preferred remote server.

Unmount and delete the snapshot

ardana > sudo umount -l -f /var/tmp/root_snapshot; sudo rm -rf \
/var/tmp/root_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_root_snapshot

17.3.4 Audit Log Backup and Restore #

17.3.4.1 Audit Log Backup #

The following procedure is used to back up Audit Logs. It is similar to the Cloud Lifecycle Manager backup (see Procedure 17.1, “Manual Backup Setup”). The steps must be performed on all nodes; there will be a backup TAR archive for each node. Before performing the following steps, run through Section 13.2.7.2, “Enable Audit Logging” .

The backup_name is audit_log_backup and the backup_path is /var/audit.

Create a snapshot

ardana > sudo lvcreate --size 2G --snapshot --permission r --name \
lvm_root_snapshot /dev/ardana-vg/root

Mount the snapshot

ardana > mkdir /var/tmp/root_snapshot; sudo mount -o ro \
/dev/ardana-vg/lvm_root_snapshot /var/tmp/root_snapshot

Create the TAR archive

ardana > sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--file audit.tar.gz -C /var/tmp/root_snapshot/var/audit .

Upload your audit.tar.gz TAR archive to your preferred remote server.

Unmount and delete a snapshot

ardana > sudo umount -l -f /var/tmp/root_snapshot; sudo rm -rf \
/var/tmp/root_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_root_snapshot

17.3.4.2 Audit Logs Restore #

Restore the Audit Logs backup with the following commands

Retrieve the Audit Logs TAR archive

Extract the TAR archive to the proper backup location

ardana > sudo tar -z --incremental --extract --ignore-zeros \
--warning=none --overwrite --directory /var/audit/  -f audit.tar.gz

17.4 Full Disaster Recovery Test #

Full Disaster Recovery Test

17.4.1 High Level View of the Recovery Process #

Back up the control plane using the manual backup procedure
Backup the Cassandra Database
Re-install Controller 1 with the SUSE OpenStack Cloud ISO
Use manual restore steps to recover deployment data (and model)
Re-install SUSE OpenStack Cloud on Controllers 1, 2, 3
Recover the backup of the MariaDB database
Recover the Cassandra Database
Verify testing

17.4.2 Description of the testing environment #

The testing environment is similar to the Entry Scale model.

It uses five servers: three Control Nodes and two Compute Nodes.

The controller node has three disks. The first is reserved for the system; the others are used for swift.

Note

For this Disaster Recovery test, data has been saved on disks 2 and 3 of the swift controllers, which allows for swift objects to be restored the recovery. If these disks were also wiped, swift data would be lost, but the procedure would not change. The only difference is that glance images would be lost and would have to be uploaded again.

Unless specified otherwise, all commands should be executed on controller 1, which is also the deployer node.

17.4.3 Pre-Disaster testing #

In order to validate the procedure after recovery, we need to create some workloads.

Source the service credential file
```
ardana > source ~/service.osrc
```

Copy an image to the platform and create a glance image with it. In this example, Cirros is used

ardana > openstack image create --disk-format raw --container-format \
bare --public --file ~/cirros-0.3.5-x86_64-disk.img cirros

Create a network

ardana > openstack network create test_net

Create a subnet

ardana > openstack subnet create 07c35d11-13f9-41d4-8289-fa92147b1d44 192.168.42.0/24 --name test_subnet

Create some instances

ardana > openstack server create server_1 --image 411a0363-7f4b-4bbc-889c-b9614e2da52e --flavor m1.small --nic net-id=07c35d11-13f9-41d4-8289-fa92147b1d44
ardana > openstack server create server_2 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44
ardana > openstack server create server_3 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44
ardana > openstack server create server_4 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44
ardana > openstack server create server_5 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44
ardana > openstack server list

Create containers and objects

ardana > openstack object create container_1 ~/service.osrc
var/lib/ardana/service.osrc

ardana > openstack object create container_1 ~/backup.osrc
swift upload container_1 ~/backup.osrc

ardana > openstack object list container_1
var/lib/ardana/backup.osrc
var/lib/ardana/service.osrc

17.4.4 Preparation of the test backup server #

17.4.4.1 Preparation to store backups #

In this example, backups are stored on the server 192.168.69.132

Connect to the backup server

Create the user

root # useradd BACKUPUSER --create-home --home-dir /mnt/backups/

Switch to that user
```
root # su BACKUPUSER
```

Create the SSH keypair

backupuser > ssh-keygen -t rsa
> # Leave the default for the first question and do not set any passphrase
> Generating public/private rsa key pair.
> Enter file in which to save the key (/mnt/backups//.ssh/id_rsa):
> Created directory '/mnt/backups//.ssh'.
> Enter passphrase (empty for no passphrase):
> Enter same passphrase again:
> Your identification has been saved in /mnt/backups//.ssh/id_rsa
> Your public key has been saved in /mnt/backups//.ssh/id_rsa.pub
> The key fingerprint is:
> a9:08:ae:ee:3c:57:62:31:d2:52:77:a7:4e:37:d1:28 backupuser@padawan-ccp-c0-m1-mgmt
> The key's randomart image is:
> +---[RSA 2048]----+
> |          o      |
> |   . . E + .     |
> |  o . . + .      |
> | o +   o +       |
> |  + o o S .      |
> | . + o o         |
> |  o + .          |
> |.o .             |
> |++o              |
> +-----------------+

Add the public key to the list of the keys authorized to connect to that user on this server
```
backupuser > cat /mnt/backups/.ssh/id_rsa.pub >> /mnt/backups/.ssh/authorized_keys
```

Print the private key. This will be used for the backup configuration (ssh_credentials.yml file)

backupuser > cat /mnt/backups/.ssh/id_rsa

> -----BEGIN RSA PRIVATE KEY-----
> MIIEogIBAAKCAQEAvjwKu6f940IVGHpUj3ffl3eKXACgVr3L5s9UJnb15+zV3K5L
> BZuor8MLvwtskSkgdXNrpPZhNCsWSkryJff5I335Jhr/e5o03Yy+RqIMrJAIa0X5
> ...
> ...
> ...
> iBKVKGPhOnn4ve3dDqy3q7fS5sivTqCrpaYtByJmPrcJNjb2K7VMLNvgLamK/AbL
> qpSTZjicKZCCl+J2+8lrKAaDWqWtIjSUs29kCL78QmaPOgEvfsw=
> -----END RSA PRIVATE KEY-----

17.4.4.2 Preparation to store Cassandra backups #

In this example, backups will be stored on the server 192.168.69.132, in the /mnt/backups/cassandra_backups/ directory.

Create a directory on the backup server to store Cassandra backups.
```
backupuser > mkdir /mnt/backups/cassandra_backups
```
Copy the private SSH key from the backup server to all controller nodes.
Replace CONTROLLER with each control node e.g. doc-cp1-c1-m1-mgmt, doc-cp1-c1-m2-mgmt etc
Log in to each controller node and copy the private SSH key to .ssh directory of the root user.
```
ardana >  sudo cp /var/lib/ardana/.ssh/id_rsa_backup /root/.ssh/
```
Verify that you can SSH to the backup server as backupuser using the private key.
```
root # ssh -i ~/.ssh/id_rsa_backup backupuser@192.168.69.132
```

17.4.5 Perform Backups for disaster recovery test #

17.4.5.1 Execute backup of Cassandra #

Create the following cassandra-backup-extserver.sh script on all controller nodes.

root # cat > ~/cassandra-backup-extserver.sh << EOF
#!/bin/sh

# backup user
BACKUP_USER=backupuser
# backup server
BACKUP_SERVER=192.168.69.132
# backup directory
BACKUP_DIR=/mnt/backups/cassandra_backups/

# Setup variables
DATA_DIR=/var/cassandra/data/data
NODETOOL=/usr/bin/nodetool

# example: cassandra-snp-2018-06-26-1003
SNAPSHOT_NAME=cassandra-snp-\$(date +%F-%H%M)
HOST_NAME=\$(/bin/hostname)_

# Take a snapshot of Cassandra database
\$NODETOOL snapshot -t \$SNAPSHOT_NAME monasca

# Collect a list of directories that make up the snapshot
SNAPSHOT_DIR_LIST=\$(find \$DATA_DIR -type d -name \$SNAPSHOT_NAME)
for d in \$SNAPSHOT_DIR_LIST
  do
    # copy snapshot directories to external server
    rsync -avR -e "ssh -i /root/.ssh/id_rsa_backup" \$d \$BACKUP_USER@\$BACKUP_SERVER:\$BACKUP_DIR/\$HOST_NAME\$SNAPSHOT_NAME
  done

\$NODETOOL clearsnapshot monasca
EOF

root # chmod +x ~/cassandra-backup-extserver.sh

Execute following steps on all the controller nodes

Note

The /usr/local/sbin/cassandra-backup-extserver.sh script should be executed on all three controller nodes at the same time (within seconds of each other) for a successful backup.

Edit the /usr/local/sbin/cassandra-backup-extserver.sh script
Set BACKUP_USER and BACKUP_SERVER to the desired backup user (for example, backupuser) and desired backup server (for example, 192.168.68.132), respectively.
```
BACKUP_USER=backupuser
BACKUP_SERVER=192.168.69.132
BACKUP_DIR=/mnt/backups/cassandra_backups/
```

Execute ~/cassandra-backup-extserver.sh on on all controller nodes which are also Cassandra nodes.

root # ~/cassandra-backup-extserver.sh

Requested creating snapshot(s) for [monasca] with snapshot name [cassandra-snp-2018-06-28-0251] and options {skipFlush=false}
Snapshot directory: cassandra-snp-2018-06-28-0251
sending incremental file list
created directory /mnt/backups/cassandra_backups//doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251
/var/
/var/cassandra/
/var/cassandra/data/
/var/cassandra/data/data/
/var/cassandra/data/data/monasca/

...
...
...

/var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-72-big-Summary.db
/var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-72-big-TOC.txt
/var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/schema.cql
sent 173,691 bytes  received 531 bytes  116,148.00 bytes/sec
total size is 171,378  speedup is 0.98
Requested clearing snapshot(s) for [monasca]

Verify the Cassandra backup directory on the backup server.

backupuser > ls -alt /mnt/backups/cassandra_backups
total 16
drwxr-xr-x 4 backupuser users 4096 Jun 28 03:06 .
drwxr-xr-x 3 backupuser users 4096 Jun 28 03:06 doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306
drwxr-xr-x 3 backupuser users 4096 Jun 28 02:51 doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251
drwxr-xr-x 8 backupuser users 4096 Jun 27 20:56 ..

$backupuser@backupserver> du -shx /mnt/backups/cassandra_backups/*
6.2G    /mnt/backups/cassandra_backups/doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251
6.3G    /mnt/backups/cassandra_backups/doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306

17.4.5.2 Execute backup of SUSE OpenStack Cloud #

Back up the Cloud Lifecycle Manager using the procedure at Section 17.3.1, “Cloud Lifecycle Manager Data Backup”
Back up the MariaDB database using the procedure at Section 17.3.2, “MariaDB Database Backup”
Back up swift rings using the procedure at Section 17.3.3, “swift Ring Backup”

17.4.5.2.1 Restore the first controller #

Log in to the Cloud Lifecycle Manager.
Retrieve the Cloud Lifecycle Manager backups that were created with Section 17.3.1, “Cloud Lifecycle Manager Data Backup”. There are multiple backups; directories are handled differently than files.

Extract the TAR archives for each of the seven locations.

ardana > sudo tar -z --incremental --extract --ignore-zeros \
--warning=none --overwrite --directory RESTORE_TARGET \
-f BACKUP_TARGET.tar.gz

For example, with a directory such as BACKUP_TARGET=/etc/ssh/

ardana > sudo tar -z --incremental --extract --ignore-zeros \
--warning=none --overwrite --directory /etc/ssh/ -f ssh.tar.gz

With a file such as BACKUP_TARGET=/etc/passwd

ardana > sudo tar -z --incremental --extract --ignore-zeros --warning=none --overwrite --directory /etc/ -f passwd.tar.gz

17.4.5.2.2 Re-deployment of controllers 1, 2 and 3 #

Change back to the default ardana user.

Run the cobbler-deploy.yml playbook.

ardana > cd ~/openstack/ardana/ansible
ardana > ansible-playbook -i hosts/localhost cobbler-deploy.xml

Run the bm-reimage.yml playbook limited to the second and third controllers.
```
ardana > ansible-playbook -i hosts/localhost bm-reimage.yml -e nodelist=controller2,controller3
```
The names of controller2 and controller3. Use the bm-power-status.yml playbook to check the cobbler names of these nodes.

Run the site.yml playbook limited to the three controllers and localhost—in this example, doc-cp1-c1-m1-mgmt, doc-cp1-c1-m2-mgmt, doc-cp1-c1-m3-mgmt, and localhost

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts site.yml --limit \
doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhost

17.4.5.2.3 Restore Databases #

17.4.5.2.3.1 Restore MariaDB database #

Log in to the first controller node.
Retrieve the MariaDB backup that was created with Section 17.3.2, “MariaDB Database Backup”.

Create a temporary directory and extract the TAR archive (for example, mydb.tar.gz).

ardana > mkdir /tmp/mysql_restore; sudo tar -z --incremental \
--extract --ignore-zeros --warning=none --overwrite --directory /tmp/mysql_restore/ \
-f mydb.tar.gz

Verify that the files have been restored on the controller.

ardana > sudo du -shx /tmp/mysql_restore/*
16K     /tmp/mysql_restore/aria_log.00000001
4.0K    /tmp/mysql_restore/aria_log_control
3.4M    /tmp/mysql_restore/barbican
8.0K    /tmp/mysql_restore/ceilometer
4.2M    /tmp/mysql_restore/cinder
2.9M    /tmp/mysql_restore/designate
129M    /tmp/mysql_restore/galera.cache
2.1M    /tmp/mysql_restore/glance
4.0K    /tmp/mysql_restore/grastate.dat
4.0K    /tmp/mysql_restore/gvwstate.dat
2.6M    /tmp/mysql_restore/heat
752K    /tmp/mysql_restore/horizon
4.0K    /tmp/mysql_restore/ib_buffer_pool
76M     /tmp/mysql_restore/ibdata1
128M    /tmp/mysql_restore/ib_logfile0
128M    /tmp/mysql_restore/ib_logfile1
12M     /tmp/mysql_restore/ibtmp1
16K     /tmp/mysql_restore/innobackup.backup.log
313M    /tmp/mysql_restore/keystone
716K    /tmp/mysql_restore/magnum
12M     /tmp/mysql_restore/mon
8.3M    /tmp/mysql_restore/monasca_transform
0       /tmp/mysql_restore/multi-master.info
11M     /tmp/mysql_restore/mysql
4.0K    /tmp/mysql_restore/mysql_upgrade_info
14M     /tmp/mysql_restore/nova
4.4M    /tmp/mysql_restore/nova_api
14M     /tmp/mysql_restore/nova_cell0
3.6M    /tmp/mysql_restore/octavia
208K    /tmp/mysql_restore/opsconsole
38M     /tmp/mysql_restore/ovs_neutron
8.0K    /tmp/mysql_restore/performance_schema
24K     /tmp/mysql_restore/tc.log
4.0K    /tmp/mysql_restore/test
8.0K    /tmp/mysql_restore/winchester
4.0K    /tmp/mysql_restore/xtrabackup_galera_info

Stop SUSE OpenStack Cloud services on the three controllers (using the hostnames of the controllers in your configuration).

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts ardana-stop.yml --limit \
doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhost

Delete the files in the mysql directory and copy the restored backup to that directory.
```
root # cd /var/lib/mysql/
root # rm -rf ./*
root # cp -pr /tmp/mysql_restore/* ./
```
Switch back to the ardana user when the copy is finished.

17.4.5.2.3.2 Restore Cassandra database #

Create a script called cassandra-restore-extserver.sh on all controller nodes

root # cat > ~/cassandra-restore-extserver.sh << EOF
#!/bin/sh

# backup user
BACKUP_USER=backupuser
# backup server
BACKUP_SERVER=192.168.69.132
# backup directory
BACKUP_DIR=/mnt/backups/cassandra_backups/

# Setup variables
DATA_DIR=/var/cassandra/data/data
NODETOOL=/usr/bin/nodetool

HOST_NAME=\$(/bin/hostname)_

#Get snapshot name from command line.
if [ -z "\$*"  ]
then
  echo "usage \$0 <snapshot to restore>"
  exit 1
fi
SNAPSHOT_NAME=\$1

# restore
rsync -av -e "ssh -i /root/.ssh/id_rsa_backup" \$BACKUP_USER@\$BACKUP_SERVER:\$BACKUP_DIR/\$HOST_NAME\$SNAPSHOT_NAME/ /

# set ownership of newley restored files
chown -R cassandra:cassandra \$DATA_DIR/monasca/*

# Get a list of snapshot directories that have files to be restored.
RESTORE_LIST=\$(find \$DATA_DIR -type d -name \$SNAPSHOT_NAME)

# use RESTORE_LIST to move snapshot files back into place of database.
for d in \$RESTORE_LIST
do
  cd \$d
  mv * ../..
  KEYSPACE=\$(pwd | rev | cut -d '/' -f4 | rev)
  TABLE_NAME=\$(pwd | rev | cut -d '/' -f3 |rev | cut -d '-' -f1)
  \$NODETOOL refresh \$KEYSPACE \$TABLE_NAME
done
cd
# Cleanup snapshot directories
\$NODETOOL clearsnapshot \$KEYSPACE
EOF

root # chmod +x ~/cassandra-restore-extserver.sh

Execute following steps on all the controller nodes.

Edit the ~/cassandra-restore-extserver.sh script.
Set BACKUP_USER,BACKUP_SERVER to the desired backup user (for example, backupuser) and the desired backup server (for example, 192.168.68.132), respectively.
```
BACKUP_USER=backupuser
BACKUP_SERVER=192.168.69.132
BACKUP_DIR=/mnt/backups/cassandra_backups/
```

Execute ~/cassandra-restore-extserver.sh SNAPSHOT_NAME

Find SNAPSHOT_NAME from listing of /mnt/backups/cassandra_backups. All the directories have the format HOST_SNAPSHOT_NAME.

ardana > ls -alt /mnt/backups/cassandra_backups
total 16
drwxr-xr-x 4 backupuser users 4096 Jun 28 03:06 .
drwxr-xr-x 3 backupuser users 4096 Jun 28 03:06 doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306

root # ~/cassandra-restore-extserver.sh cassandra-snp-2018-06-28-0306

receiving incremental file list
./
var/
var/cassandra/
var/cassandra/data/
var/cassandra/data/data/
var/cassandra/data/data/monasca/
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/manifest.json
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-37-big-CompressionInfo.db
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-37-big-Data.db
...
...
...
/usr/bin/nodetool clearsnapshot monasca

17.4.5.2.3.3 Restart SUSE OpenStack Cloud services #

Restart the MariaDB database
```
ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts galera-bootstrap.yml
```
On the deployer node, execute the galera-bootstrap.yml playbook which will determine the log sequence number, bootstrap the main node, and start the database cluster.
If this process fails to recover the database cluster, refer to Section 15.2.3.1.2, “Recovering the MariaDB Database”.

Restart SUSE OpenStack Cloud services on the three controllers as in the following example.

ardana > ansible-playbook -i hosts/verb_hosts ardana-start.yml \
--limit doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhost

Reconfigure SUSE OpenStack Cloud

ardana > ansible-playbook -i hosts/verb_hosts ardana-reconfigure.yml

17.4.5.2.4 Post restore testing #

Source the service credential file
```
ardana > source ~/service.osrc
```

swift

ardana > openstack container list
container_1
volumebackups

ardana > openstack object list container_1
var/lib/ardana/backup.osrc
var/lib/ardana/service.osrc

ardana > openstack object save container_1 /tmp/backup.osrc

neutron

ardana > openstack network list
+--------------------------------------+---------------------+--------------------------------------+
| ID                                   | Name                | Subnets                              |
+--------------------------------------+---------------------+--------------------------------------+
| 07c35d11-13f9-41d4-8289-fa92147b1d44 | test-net             | 02d5ca3b-1133-4a74-a9ab-1f1dc2853ec8|
+--------------------------------------+---------------------+--------------------------------------+

glance

ardana > openstack image list
+--------------------------------------+----------------------+--------+
| ID                                   | Name                 | Status |
+--------------------------------------+----------------------+--------+
| 411a0363-7f4b-4bbc-889c-b9614e2da52e | cirros-0.4.0-x86_64  | active |
+--------------------------------------+----------------------+--------+
ardana > openstack image save --file /tmp/cirros f751c39b-f1e3-4f02-8332-3886826889ba
ardana > ls -lah /tmp/cirros
-rw-r--r-- 1 ardana ardana 12716032 Jul  2 20:52 /tmp/cirros

nova

ardana > openstack server list

ardana > openstack server create server_6 --image 411a0363-7f4b-4bbc-889c-b9614e2da52e  --flavor m1.small --nic net-id=07c35d11-13f9-41d4-8289-fa92147b1d44
+-------------------------------------+------------------------------------------------------------+
| Field                               | Value                                                      |
+-------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                     |
| OS-EXT-AZ:availability_zone         |                                                            |
| OS-EXT-SRV-ATTR:host                | None                                                       |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                       |
| OS-EXT-SRV-ATTR:instance_name       |                                                            |
| OS-EXT-STS:power_state              | NOSTATE                                                    |
| OS-EXT-STS:task_state               | scheduling                                                 |
| OS-EXT-STS:vm_state                 | building                                                   |
| OS-SRV-USG:launched_at              | None                                                       |
| OS-SRV-USG:terminated_at            | None                                                       |
| accessIPv4                          |                                                            |
| accessIPv6                          |                                                            |
| addresses                           |                                                            |
| adminPass                           | iJBoBaj53oUd                                               |
| config_drive                        |                                                            |
| created                             | 2018-07-02T21:02:01Z                                       |
| flavor                              | m1.small (2)                                               |
| hostId                              |                                                            |
| id                                  | ce7689ff-23bf-4fe9-b2a9-922d4aa9412c                       |
| image                               | cirros-0.4.0-x86_64 (f751c39b-f1e3-4f02-8332-3886826889ba) |
| key_name                            | None                                                       |
| name                                | server_6                                                   |
| progress                            | 0                                                          |
| project_id                          | cca416004124432592b2949a5c5d9949                           |
| properties                          |                                                            |
| security_groups                     | name='default'                                             |
| status                              | BUILD                                                      |
| updated                             | 2018-07-02T21:02:01Z                                       |
| user_id                             | 8cb1168776d24390b44c3aaa0720b532                           |
| volumes_attached                    |                                                            |
+-------------------------------------+------------------------------------------------------------+

ardana > openstack server list
+--------------------------------------+----------+--------+---------------------------------+---------------------+-----------+
| ID                                   | Name     | Status | Networks                        | Image               | Flavor    |
+--------------------------------------+----------+--------+---------------------------------+---------------------+-----------+
| ce7689ff-23bf-4fe9-b2a9-922d4aa9412c | server_6 | ACTIVE | n1=1.1.1.8                      | cirros-0.4.0-x86_64 | m1.small  |

ardana > openstack server delete ce7689ff-23bf-4fe9-b2a9-922d4aa9412c