17 Backup and Restore #
The following sections cover backup and restore operations. Before installing your cloud, there are several things you must do so that you achieve the backup and recovery results you need. SUSE OpenStack Cloud comes with playbooks and procedures to recover the control plane from various disaster scenarios.
As of SUSE OpenStack Cloud 9, Freezer (a distributed backup restore and disaster recovery service) is no longer supported; backup and restore are manual operations.
Consider Section 17.2, “Enabling Backups to a Remote Server” in case you lose cloud servers that back up and restore services.
The following features are supported:
File system backup using a point-in-time snapshot.
Strong encryption: AES-256-CFB.
MariaDB database backup with LVM snapshot.
Restoring your data from a previous backup.
Low storage requirement: backups are stored as compressed files.
Flexible backup (both incremental and differential).
Data is archived in GNU Tar format for file-based incremental backup and restore.
When a key is provided, Open SSL is used to encrypt data (AES-256-CFB).
17.1 Manual Backup Overview #
This section covers manual backup and some restore processes. Full documentation for restore operations is in Section 15.2, “Unplanned System Maintenance”.To back up outside the cluster, refer to Section 17.2, “Enabling Backups to a Remote Server”. Backups of the following types of resources are covered:
Cloud Lifecycle Manager Data. All important information on the Cloud Lifecycle Manager
MariaDB database that is part of the Control Plane. The MariaDB database contains most of the data needed to restore services. MariaDB supports full back up and recovery for all services. Logging data in Elasticsearch is not backed up. swift objects are not backed up because of the redundant nature of swift.
swift Rings used in the swift storage deployment. swift rings are backed up so that you can recover more quickly than rebuilding with swift. swift can rebuild the rings without this backup data, but automatically rebuilding the rings is slower than restoring from a backup.
Audit Logs. Audit Logs are backed up to provide retrospective information and statistical data for performance and security purposes.
The following services will be backed up. Specifically, the data needed to restore the services is backed up. This includes databases and configuration-related files.
Data content for some services is not backed up, as indicated below.
ceilometer. There is no backup of metrics data.
cinder. There is no backup of the volumes.
glance. There is no backup of the images.
heat
horizon
keystone
neutron
nova. There is no backup of the images.
swift. There is no backup of the objects. swift has its own high availability and redundancy. swift rings are backed up. Although swift can rebuild the rings itself, restoring from backup is faster.
Operations Console
monasca. There is no backup of the metrics.
17.2 Enabling Backups to a Remote Server #
We recommend that you set up a remote server to store your backups, so that you can restore the control plane nodes. This may be necessary if you lose all of your control plane nodes at the same time.
A remote backup server must be set up before proceeding.
You do not have to restore from the remote server if only one or two control plane nodes are lost. In that case, the control planes can be recovered from the data on a remaining control plane node following the restore procedures in Section 15.2.3.2, “Recovering the Control Plane”.
17.2.1 Securing your SSH backup server #
You can do the following to harden an SSH server:
Disable root login
Move SSH to a non-default port (the default SSH port is 22)
Disable password login (only allow RSA keys)
Disable SSH v1
Authorize Secure File Transfer Protocol (SFTP) only for the designated backup user (disable SSH shell)
Firewall SSH traffic to ensure it comes from the SUSE OpenStack Cloud address range
Install a Fail2Ban solution
Restrict users who are allowed to SSH
Additional suggestions are available online
Remove the key pair generated earlier on the backup server; the only thing
needed is .ssh/authorized_keys
. You can remove the
.ssh/id_rsa
and .ssh/id_rsa.pub
files. Be sure to save a backup of them.
17.2.2 General tips #
Provide adequate space in the directory that is used for backup.
Monitor the space left on that directory.
Keep the system up to date on that server.
17.3 Manual Backup and Restore Procedures #
Each backup requires the following steps:
Create a snapshot.
Mount the snapshot.
Generate a TAR archive and save it.
Unmount and delete the snapshot.
17.3.1 Cloud Lifecycle Manager Data Backup #
The following procedure is used for each of the seven BACKUP_TARGETS (list below). Incremental backup instructions follow the full backup procedure. For both full and incremental backups, the last step of the procedure is to unmount and delete the snapshot after the TAR archive has been created and saved. A new snapshot must be created every time a backup is created.
Create a snapshot on the Cloud Lifecycle Manager in (
ardana-vg
), the location where all Cloud Lifecycle Manager data is stored.ardana >
sudo lvcreate --size 2G --snapshot --permission r \ --name lvm_clm_snapshot /dev/ardana-vg/rootNoteIf you have stored additional data or files in your
ardana-vg
directory, you may need more space than the 2G indicated for thesize
parameter. In this situation, create a preliminary TAR archive with thetar
command on the directory before creating a snapshot. Set thesize
snapshot parameter larger than the size of the archive.Mount the snapshot
ardana >
sudo mkdir /var/tmp/clm_snapshotardana >
sudo mount -o ro /dev/ardana-vg/lvm_clm_snapshot /var/tmp/clm_snapshotGenerate a TAR archive (does not apply to incremental backups) with an appropriate
BACKUP_TAR_ARCHIVE_NAME.tar.gz
backup file for each of the following BACKUP_TARGETS.Backup Targets
home
ssh
shadow
passwd
group
The backup TAR archive should contain only the necessary data; nothing extra. Some of the archives will be stored as directories, others as files. The backup commands are slightly different for each type.
If the BACKUP_TARGET is a directory, then that directory must be appended to
/var/tmp/clm_snapshot/
TARGET_DIR. If the BACKUP_TARGET is a file, then its parent directory must be appended to/var/tmp/clm_snapshot/
.In the commands that follow, replace BACKUP_TARGET with the appropriate BACKUP_PATH (replacement table is below).
ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek \ --ignore-failed-read --file BACKUP_TAR_ARCHIVE_NAME.tar.gz -C \ /var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIRIf BACKUP_TARGET is a directory, replace TARGET_DIR with BACKUP_PATH.
For example, where BACKUP_PATH=
/etc/ssh/
(a directory):ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek \ --ignore-failed-read --file ssh.tar.gz -C /var/tmp/clm_snapshot/etc/ssh .If BACKUP_TARGET is a file (not a directory), replace TARGET_DIR with the parent directory of BACKUP_PATH.
For example, where BACKUP_PATH=
/etc/passwd
(a file):ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek \ --ignore-failed-read --file passwd.tar.gz -C /var/tmp/clm_snapshot/etc/passwd
Save the TAR archive to the remote server.
ardana >
scp TAR_ARCHIVE USER@REMOTE_SERVERUse the following commands to unmount and delete a snapshot.
ardana >
sudo umount -l -f /var/tmp/clm_snapshot; rm -rf /var/tmp/clm_snapshotardana >
sudo lvremove -f /dev/ardana-vg/lvm_clm_snapshot
The table below shows Cloud Lifecycle Manager backup_targets
and their
respective backup_paths
.
backup_name |
backup_path |
---|---|
home_backup |
/var/lib/ardana (file) |
etc_ssh_backup |
/etc/ssh/ (directory) |
shadow_backup |
/etc/shadow (file) |
passwd_backup |
/etc/passwd (file) |
group_backup |
/etc/group (file) |
17.3.1.1 Cloud Lifecycle Manager Incremental Backup #
Incremental backups require a meta
file. If you use the
incremental backup option, a meta file must be included in the
tar
command in the initial backup and whenever you do an
incremental backup. A copy of the original meta
file
should be stored in each backup. The meta
file is used
to determine the incremental changes from the previous backup, so it is
rewritten with each incremental backup.
Versions are useful for incremental backup because they provide a way
to differentiate between each backup. Versions are included in the
tar
command.
Every incremental backup requires creating and mounting a separate snapshot. After the TAR archive is created, the snapshot is unmounted and deleted.
To prepare for incremental backup, follow the steps in Procedure 17.1, “Manual Backup Setup” with the following differences in the
commands for generating a tar
archive.
First time full backup
ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek \ --ignore-failed-read --listed-incremental=PATH_TO_YOUR_META \ --file BACKUP_TAR_ARCHIVE_NAME.tar.gz -C \ /var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIRFor example, where BACKUP_PATH=
/etc/ssh/
ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --listed-incremental=mysshMeta --file ssh.tar.gz -C \ /var/tmp/clm_snapshot/etc/ssh .Incremental backup
ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek \ --ignore-failed-read --listed-incremental=PATH_TO_YOUR_META\ --file BACKUP_TAR_ARCHIVE_NAME_VERSION.tar.gz -C \ /var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIRFor example, where BACKUP_PATH=
/etc/ssh/
:ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --listed-incremental=mysshMeta --file \ ssh_v1.tar.gz -C \ /var/tmp/clm_snapshot/etc/ssh .
After creating an incremental backup, use the following commands to unmount and delete a snapshot.
ardana >
sudo umount -l -f /var/tmp/clm_snapshot; rm -rf /var/tmp/clm_snapshotardana >
sudo lvremove -f /dev/ardana-vg/lvm_clm_snapshot
17.3.1.2 Encryption #
When a key is provided, Open SSL is used to encrypt data (AES-256-CFB). Backup files can be encrypted with the following command:
ardana >
sudo openssl enc -aes-256-cfb -pass file:ENCRYPT_PASS_FILE_PATH -in \
YOUR_BACKUP_TAR_ARCHIVE_NAME.tar.gz -out YOUR_BACKUP_TAR_ARCHIVE_NAME.tar.gz.enc
For example, using the ssh.tar.gz
generated above:
ardana >
sudo openssl enc -aes-256-cfb -pass file:myEncFile -in ssh.tar.gz -out ssh.tar.gz.enc
17.3.2 MariaDB Database Backup #
When backing up MariaDB, the following process must be performed on
all nodes in the cluster. It is similar to
the backup procedure above for the Cloud Lifecycle Manager (see Procedure 17.1, “Manual Backup Setup”). The difference is the addition of SQL
commands, which are run with the create_db_snapshot.yml
playbook.
Create the create_db_snapshot.yml
file in
~/scratch/ansible/next/ardana/ansible/
on the deployer
with the following content:
- hosts: FND-MDB vars: - snapshot_name: lvm_mysql_snapshot - lvm_target: /dev/ardana-vg/mysql tasks: - name: Cleanup old snapshots become: yes shell: | lvremove -f /dev/ardana-vg/{{ snapshot_name }} ignore_errors: True - name: Create snapshot become: yes shell: | lvcreate --size 2G --snapshot --permission r --name {{ snapshot_name }} {{ lvm_target }} register: snapshot_st ignore_errors: True - fail: msg: "Fail to create snapshot on {{ lvm_target }}" when: snapshot_st.rc != 0
Verify the validity of the lvm_target
variable
(which refers to the actual database LVM volume) before proceeding with
the backup.
Doing the MariaDB backup
We recommend storing the MariaDB version with your backup. The following command saves the MariaDB version as MARIADB_VER.
mysql -V | grep -Eo '(\S+?)-MariaDB' > MARIADB_VER
Open a MariaDB client session on all controllers.
Run the command to spread
read lock
on all controllers and keep the MariaDB session open.>> FLUSH TABLES WITH READ LOCK;
Open a new terminal and run the
create_db_snapshot.yml
playbook created above.ardana >
cd ~/scratch/ansible/next/ardana/ansible/ardana >
ansible-playbook -i hosts/verb_hosts create_db_snapshot.ymlGo back to the open MariaDB session and run the command to flush the lock on all controllers.
>> UNLOCK TABLES;
Mount the snapshot
dbnode>> mkdir /var/tmp/mysql_snapshot dbnode>> sudo mount -o ro /dev/ardana-vg/lvm_mysql_snapshot /var/tmp/mysql_snapshot
On each database node, generate a TAR archive with an appropriate
BACKUP_TAR_ARCHIVE_NAME.tar.gz
backup file for the BACKUP_TARGET.The
backup_name
ismysql_backup
and thebackup_path
(BACKUP_TARGET) is/var/lib/mysql/
.dbnode>> sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --file mydb.tar.gz /var/tmp/mysql_snapshot/var/lib/mysql .
Unmount and delete the MariaDB snapshot on each database node.
dbnode>> sudo umount -l -f /var/tmp/mysql_snapshot; \ sudo rm -rf /var/tmp/mysql_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_mysql_snapshot
17.3.2.1 Incremental MariaDB Database Backup #
Incremental backups require a meta
file. If you use the
incremental backup option, a meta file must be included in the
tar
command in the initial backup and whenever you do an
incremental backup. A copy of the original meta
file
should be stored in each backup. The meta
file is used
to determine the incremental changes from the previous backup, so it is
rewritten with each incremental backup.
Versions are useful for incremental backup because they provide a way
to differentiate between each backup. Versions are included in the
tar
command.
To prepare for incremental backup, follow the steps in the previous section
except for the tar
commands. Incremental backup
tar
commands must have additional information.
First time MariaDB database full backup
dbnode>> sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek \ --ignore-failed-read --listed-incremental=PATH_TO_YOUR_DB_META \ --file mydb.tar.gz -C /var/tmp/mysql_snapshot/var/lib/mysql .
For example, where BACKUP_PATH=
/var/lib/mysql/
:dbnode>> sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --listed-incremental=mydbMeta --file mydb.tar.gz -C \ /var/tmp/mysql_snapshot/var/lib/mysql .
Incremental MariaDB database backup
dbnode>> sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek \ --ignore-failed-read --listed-incremental=PATH_TO_YOUR_META\ --file BACKUP_TAR_ARCHIVE_NAME_VERSION.tar.gz -C \ /var/tmp/clm_snapshotTARGET_DIR
For example, where BACKUP_PATH=
/var/lib/mysql/
:dbnode>> sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --listed-incremental=mydbMeta --file \ mydb_v1.tar.gz -C /var/tmp/mysql_snapshot/var/lib/mysql .
After creating and saving the TAR archive, unmount and delete the snapshot.
dbnode>> sudo umount -l -f /var/tmp/mysql_snapshot; \ sudo rm -rf /var/tmp/mysql_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_mysql_snapshot
17.3.2.2 MariaDB Database Encryption #
Encrypt your MariaDB database backup following the instructions in Section 17.3.1.2, “Encryption”
Upload your BACKUP_TARGET.tar.gz to your preferred remote server.
17.3.3 swift Ring Backup #
The following procedure is used to back up swift rings. It is similar to the Cloud Lifecycle Manager backup (see Procedure 17.1, “Manual Backup Setup”).
The steps must be performed only on the building server (For more information, see Section 18.6.2.4, “Identifying the Swift Ring Building Server”.).
The backup_name
is
swift_builder_dir_backup
and the
backup_path
is /etc/swiftlm/
.
Create a snapshot
ardana >
sudo lvcreate --size 2G --snapshot --permission r \ --name lvm_root_snapshot /dev/ardana-vg/rootMount the snapshot
ardana >
mkdir /var/tmp/root_snapshot; sudo mount -o ro \ /dev/ardana-vg/lvm_root_snapshot /var/tmp/root_snapshotCreate the TAR archive
ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --file swring.tar.gz -C /var/tmp/root_snapshot/etc/swiftlm .Upload your
swring.tar.gz
TAR archive to your preferred remote server.Unmount and delete the snapshot
ardana >
sudo umount -l -f /var/tmp/root_snapshot; sudo rm -rf \ /var/tmp/root_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_root_snapshot
17.3.4 Audit Log Backup and Restore #
17.3.4.1 Audit Log Backup #
The following procedure is used to back up Audit Logs. It is similar to the Cloud Lifecycle Manager backup (see Procedure 17.1, “Manual Backup Setup”). The steps must be performed on all nodes; there will be a backup TAR archive for each node. Before performing the following steps, run through Section 13.2.7.2, “Enable Audit Logging” .
The backup_name
is
audit_log_backup
and the
backup_path
is /var/audit
.
Create a snapshot
ardana >
sudo lvcreate --size 2G --snapshot --permission r --name \ lvm_root_snapshot /dev/ardana-vg/rootMount the snapshot
ardana >
mkdir /var/tmp/root_snapshot; sudo mount -o ro \ /dev/ardana-vg/lvm_root_snapshot /var/tmp/root_snapshotCreate the TAR archive
ardana >
sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --file audit.tar.gz -C /var/tmp/root_snapshot/var/audit .Upload your
audit.tar.gz
TAR archive to your preferred remote server.Unmount and delete a snapshot
ardana >
sudo umount -l -f /var/tmp/root_snapshot; sudo rm -rf \ /var/tmp/root_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_root_snapshot
17.3.4.2 Audit Logs Restore #
Restore the Audit Logs backup with the following commands
Retrieve the Audit Logs TAR archive
Extract the TAR archive to the proper backup location
ardana >
sudo tar -z --incremental --extract --ignore-zeros \ --warning=none --overwrite --directory /var/audit/ -f audit.tar.gz
17.4 Full Disaster Recovery Test #
Full Disaster Recovery Test
17.4.1 High Level View of the Recovery Process #
Back up the control plane using the manual backup procedure
Backup the Cassandra Database
Re-install Controller 1 with the SUSE OpenStack Cloud ISO
Use manual restore steps to recover deployment data (and model)
Re-install SUSE OpenStack Cloud on Controllers 1, 2, 3
Recover the backup of the MariaDB database
Recover the Cassandra Database
Verify testing
17.4.2 Description of the testing environment #
The testing environment is similar to the Entry Scale model.
It uses five servers: three Control Nodes and two Compute Nodes.
The controller node has three disks. The first is reserved for the system; the others are used for swift.
For this Disaster Recovery test, data has been saved on disks 2 and 3 of the swift controllers, which allows for swift objects to be restored the recovery. If these disks were also wiped, swift data would be lost, but the procedure would not change. The only difference is that glance images would be lost and would have to be uploaded again.
Unless specified otherwise, all commands should be executed on controller 1, which is also the deployer node.
17.4.3 Pre-Disaster testing #
In order to validate the procedure after recovery, we need to create some workloads.
Source the service credential file
ardana >
source ~/service.osrcCopy an image to the platform and create a glance image with it. In this example, Cirros is used
ardana >
openstack image create --disk-format raw --container-format \ bare --public --file ~/cirros-0.3.5-x86_64-disk.img cirrosCreate a network
ardana >
openstack network create test_netCreate a subnet
ardana >
openstack subnet create 07c35d11-13f9-41d4-8289-fa92147b1d44 192.168.42.0/24 --name test_subnetCreate some instances
ardana >
openstack server create server_1 --image 411a0363-7f4b-4bbc-889c-b9614e2da52e --flavor m1.small --nic net-id=07c35d11-13f9-41d4-8289-fa92147b1d44ardana >
openstack server create server_2 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44ardana >
openstack server create server_3 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44ardana >
openstack server create server_4 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44ardana >
openstack server create server_5 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44ardana >
openstack server listCreate containers and objects
ardana >
openstack object create container_1 ~/service.osrc var/lib/ardana/service.osrcardana >
openstack object create container_1 ~/backup.osrc swift upload container_1 ~/backup.osrcardana >
openstack object list container_1 var/lib/ardana/backup.osrc var/lib/ardana/service.osrc
17.4.4 Preparation of the test backup server #
17.4.4.1 Preparation to store backups #
In this example, backups are stored on the server
192.168.69.132
Connect to the backup server
Create the user
root #
useradd BACKUPUSER --create-home --home-dir /mnt/backups/Switch to that user
root #
su BACKUPUSERCreate the SSH keypair
backupuser >
ssh-keygen -t rsa > # Leave the default for the first question and do not set any passphrase > Generating public/private rsa key pair. > Enter file in which to save the key (/mnt/backups//.ssh/id_rsa): > Created directory '/mnt/backups//.ssh'. > Enter passphrase (empty for no passphrase): > Enter same passphrase again: > Your identification has been saved in /mnt/backups//.ssh/id_rsa > Your public key has been saved in /mnt/backups//.ssh/id_rsa.pub > The key fingerprint is: > a9:08:ae:ee:3c:57:62:31:d2:52:77:a7:4e:37:d1:28 backupuser@padawan-ccp-c0-m1-mgmt > The key's randomart image is: > +---[RSA 2048]----+ > | o | > | . . E + . | > | o . . + . | > | o + o + | > | + o o S . | > | . + o o | > | o + . | > |.o . | > |++o | > +-----------------+Add the public key to the list of the keys authorized to connect to that user on this server
backupuser >
cat /mnt/backups/.ssh/id_rsa.pub >> /mnt/backups/.ssh/authorized_keysPrint the private key. This will be used for the backup configuration (ssh_credentials.yml file)
backupuser >
cat /mnt/backups/.ssh/id_rsa > -----BEGIN RSA PRIVATE KEY----- > MIIEogIBAAKCAQEAvjwKu6f940IVGHpUj3ffl3eKXACgVr3L5s9UJnb15+zV3K5L > BZuor8MLvwtskSkgdXNrpPZhNCsWSkryJff5I335Jhr/e5o03Yy+RqIMrJAIa0X5 > ... > ... > ... > iBKVKGPhOnn4ve3dDqy3q7fS5sivTqCrpaYtByJmPrcJNjb2K7VMLNvgLamK/AbL > qpSTZjicKZCCl+J2+8lrKAaDWqWtIjSUs29kCL78QmaPOgEvfsw= > -----END RSA PRIVATE KEY-----
17.4.4.2 Preparation to store Cassandra backups #
In this example, backups will be stored on the server
192.168.69.132
, in the
/mnt/backups/cassandra_backups/
directory.
Create a directory on the backup server to store Cassandra backups.
backupuser >
mkdir /mnt/backups/cassandra_backupsCopy the private SSH key from the backup server to all controller nodes.
Replace CONTROLLER with each control node e.g. doc-cp1-c1-m1-mgmt, doc-cp1-c1-m2-mgmt etc
Log in to each controller node and copy the private SSH key to
.ssh
directory of the root user.ardana >
sudo cp /var/lib/ardana/.ssh/id_rsa_backup /root/.ssh/Verify that you can SSH to the backup server as backupuser using the private key.
root #
ssh -i ~/.ssh/id_rsa_backup backupuser@192.168.69.132
17.4.5 Perform Backups for disaster recovery test #
17.4.5.1 Execute backup of Cassandra #
Create the following cassandra-backup-extserver.sh
script on all controller nodes.
root #
cat > ~/cassandra-backup-extserver.sh << EOF
#!/bin/sh
# backup user
BACKUP_USER=backupuser
# backup server
BACKUP_SERVER=192.168.69.132
# backup directory
BACKUP_DIR=/mnt/backups/cassandra_backups/
# Setup variables
DATA_DIR=/var/cassandra/data/data
NODETOOL=/usr/bin/nodetool
# example: cassandra-snp-2018-06-26-1003
SNAPSHOT_NAME=cassandra-snp-\$(date +%F-%H%M)
HOST_NAME=\$(/bin/hostname)_
# Take a snapshot of Cassandra database
\$NODETOOL snapshot -t \$SNAPSHOT_NAME monasca
# Collect a list of directories that make up the snapshot
SNAPSHOT_DIR_LIST=\$(find \$DATA_DIR -type d -name \$SNAPSHOT_NAME)
for d in \$SNAPSHOT_DIR_LIST
do
# copy snapshot directories to external server
rsync -avR -e "ssh -i /root/.ssh/id_rsa_backup" \$d \$BACKUP_USER@\$BACKUP_SERVER:\$BACKUP_DIR/\$HOST_NAME\$SNAPSHOT_NAME
done
\$NODETOOL clearsnapshot monasca
EOF
root #
chmod +x ~/cassandra-backup-extserver.sh
Execute following steps on all the controller nodes
The /usr/local/sbin/cassandra-backup-extserver.sh
script should be executed on all three controller nodes at the same time
(within seconds of each other) for a successful backup.
Edit the
/usr/local/sbin/cassandra-backup-extserver.sh
scriptSet BACKUP_USER and BACKUP_SERVER to the desired backup user (for example,
backupuser
) and desired backup server (for example,192.168.68.132
), respectively.BACKUP_USER=backupuser BACKUP_SERVER=192.168.69.132 BACKUP_DIR=/mnt/backups/cassandra_backups/
Execute
~/cassandra-backup-extserver.sh
on on all controller nodes which are also Cassandra nodes.root #
~/cassandra-backup-extserver.sh Requested creating snapshot(s) for [monasca] with snapshot name [cassandra-snp-2018-06-28-0251] and options {skipFlush=false} Snapshot directory: cassandra-snp-2018-06-28-0251 sending incremental file list created directory /mnt/backups/cassandra_backups//doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251 /var/ /var/cassandra/ /var/cassandra/data/ /var/cassandra/data/data/ /var/cassandra/data/data/monasca/ ... ... ... /var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-72-big-Summary.db /var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-72-big-TOC.txt /var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/schema.cql sent 173,691 bytes received 531 bytes 116,148.00 bytes/sec total size is 171,378 speedup is 0.98 Requested clearing snapshot(s) for [monasca]Verify the Cassandra backup directory on the backup server.
backupuser >
ls -alt /mnt/backups/cassandra_backups total 16 drwxr-xr-x 4 backupuser users 4096 Jun 28 03:06 . drwxr-xr-x 3 backupuser users 4096 Jun 28 03:06 doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306 drwxr-xr-x 3 backupuser users 4096 Jun 28 02:51 doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251 drwxr-xr-x 8 backupuser users 4096 Jun 27 20:56 .. $backupuser@backupserver> du -shx /mnt/backups/cassandra_backups/* 6.2G /mnt/backups/cassandra_backups/doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251 6.3G /mnt/backups/cassandra_backups/doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306
17.4.5.2 Execute backup of SUSE OpenStack Cloud #
Back up the Cloud Lifecycle Manager using the procedure at Section 17.3.1, “Cloud Lifecycle Manager Data Backup”
Back up the MariaDB database using the procedure at Section 17.3.2, “MariaDB Database Backup”
Back up swift rings using the procedure at Section 17.3.3, “swift Ring Backup”
17.4.5.2.1 Restore the first controller #
Log in to the Cloud Lifecycle Manager.
Retrieve the Cloud Lifecycle Manager backups that were created with Section 17.3.1, “Cloud Lifecycle Manager Data Backup”. There are multiple backups; directories are handled differently than files.
Extract the TAR archives for each of the seven locations.
ardana >
sudo tar -z --incremental --extract --ignore-zeros \ --warning=none --overwrite --directory RESTORE_TARGET \ -f BACKUP_TARGET.tar.gzFor example, with a directory such as BACKUP_TARGET=
/etc/ssh/
ardana >
sudo tar -z --incremental --extract --ignore-zeros \ --warning=none --overwrite --directory /etc/ssh/ -f ssh.tar.gzWith a file such as BACKUP_TARGET=/etc/passwd
ardana >
sudo tar -z --incremental --extract --ignore-zeros --warning=none --overwrite --directory /etc/ -f passwd.tar.gz
17.4.5.2.2 Re-deployment of controllers 1, 2 and 3 #
Change back to the default ardana user.
Run the
cobbler-deploy.yml
playbook.ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost cobbler-deploy.xmlRun the
bm-reimage.yml
playbook limited to the second and third controllers.ardana >
ansible-playbook -i hosts/localhost bm-reimage.yml -e nodelist=controller2,controller3The names of controller2 and controller3. Use the
bm-power-status.yml
playbook to check the cobbler names of these nodes.Run the
site.yml
playbook limited to the three controllers and localhost—in this example,doc-cp1-c1-m1-mgmt
,doc-cp1-c1-m2-mgmt
,doc-cp1-c1-m3-mgmt
, andlocalhost
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts site.yml --limit \ doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhost
17.4.5.2.3 Restore Databases #
17.4.5.2.3.1 Restore MariaDB database #
Log in to the first controller node.
Retrieve the MariaDB backup that was created with Section 17.3.2, “MariaDB Database Backup”.
Create a temporary directory and extract the TAR archive (for example,
mydb.tar.gz
).ardana >
mkdir /tmp/mysql_restore; sudo tar -z --incremental \ --extract --ignore-zeros --warning=none --overwrite --directory /tmp/mysql_restore/ \ -f mydb.tar.gzVerify that the files have been restored on the controller.
ardana >
sudo du -shx /tmp/mysql_restore/* 16K /tmp/mysql_restore/aria_log.00000001 4.0K /tmp/mysql_restore/aria_log_control 3.4M /tmp/mysql_restore/barbican 8.0K /tmp/mysql_restore/ceilometer 4.2M /tmp/mysql_restore/cinder 2.9M /tmp/mysql_restore/designate 129M /tmp/mysql_restore/galera.cache 2.1M /tmp/mysql_restore/glance 4.0K /tmp/mysql_restore/grastate.dat 4.0K /tmp/mysql_restore/gvwstate.dat 2.6M /tmp/mysql_restore/heat 752K /tmp/mysql_restore/horizon 4.0K /tmp/mysql_restore/ib_buffer_pool 76M /tmp/mysql_restore/ibdata1 128M /tmp/mysql_restore/ib_logfile0 128M /tmp/mysql_restore/ib_logfile1 12M /tmp/mysql_restore/ibtmp1 16K /tmp/mysql_restore/innobackup.backup.log 313M /tmp/mysql_restore/keystone 716K /tmp/mysql_restore/magnum 12M /tmp/mysql_restore/mon 8.3M /tmp/mysql_restore/monasca_transform 0 /tmp/mysql_restore/multi-master.info 11M /tmp/mysql_restore/mysql 4.0K /tmp/mysql_restore/mysql_upgrade_info 14M /tmp/mysql_restore/nova 4.4M /tmp/mysql_restore/nova_api 14M /tmp/mysql_restore/nova_cell0 3.6M /tmp/mysql_restore/octavia 208K /tmp/mysql_restore/opsconsole 38M /tmp/mysql_restore/ovs_neutron 8.0K /tmp/mysql_restore/performance_schema 24K /tmp/mysql_restore/tc.log 4.0K /tmp/mysql_restore/test 8.0K /tmp/mysql_restore/winchester 4.0K /tmp/mysql_restore/xtrabackup_galera_infoStop SUSE OpenStack Cloud services on the three controllers (using the hostnames of the controllers in your configuration).
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts ardana-stop.yml --limit \ doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhostDelete the files in the
mysql
directory and copy the restored backup to that directory.root #
cd /var/lib/mysql/root #
rm -rf ./*root #
cp -pr /tmp/mysql_restore/* ./Switch back to the
ardana
user when the copy is finished.
17.4.5.2.3.2 Restore Cassandra database #
Create a script called
cassandra-restore-extserver.sh
on all controller
nodes
root #
cat > ~/cassandra-restore-extserver.sh << EOF
#!/bin/sh
# backup user
BACKUP_USER=backupuser
# backup server
BACKUP_SERVER=192.168.69.132
# backup directory
BACKUP_DIR=/mnt/backups/cassandra_backups/
# Setup variables
DATA_DIR=/var/cassandra/data/data
NODETOOL=/usr/bin/nodetool
HOST_NAME=\$(/bin/hostname)_
#Get snapshot name from command line.
if [ -z "\$*" ]
then
echo "usage \$0 <snapshot to restore>"
exit 1
fi
SNAPSHOT_NAME=\$1
# restore
rsync -av -e "ssh -i /root/.ssh/id_rsa_backup" \$BACKUP_USER@\$BACKUP_SERVER:\$BACKUP_DIR/\$HOST_NAME\$SNAPSHOT_NAME/ /
# set ownership of newley restored files
chown -R cassandra:cassandra \$DATA_DIR/monasca/*
# Get a list of snapshot directories that have files to be restored.
RESTORE_LIST=\$(find \$DATA_DIR -type d -name \$SNAPSHOT_NAME)
# use RESTORE_LIST to move snapshot files back into place of database.
for d in \$RESTORE_LIST
do
cd \$d
mv * ../..
KEYSPACE=\$(pwd | rev | cut -d '/' -f4 | rev)
TABLE_NAME=\$(pwd | rev | cut -d '/' -f3 |rev | cut -d '-' -f1)
\$NODETOOL refresh \$KEYSPACE \$TABLE_NAME
done
cd
# Cleanup snapshot directories
\$NODETOOL clearsnapshot \$KEYSPACE
EOF
root #
chmod +x ~/cassandra-restore-extserver.sh
Execute following steps on all the controller nodes.
Edit the
~/cassandra-restore-extserver.sh
script.Set BACKUP_USER,BACKUP_SERVER to the desired backup user (for example,
backupuser
) and the desired backup server (for example,192.168.68.132
), respectively.BACKUP_USER=backupuser BACKUP_SERVER=192.168.69.132 BACKUP_DIR=/mnt/backups/cassandra_backups/
Execute
~/cassandra-restore-extserver.sh
SNAPSHOT_NAMEFind SNAPSHOT_NAME from listing of /mnt/backups/cassandra_backups. All the directories have the format HOST_SNAPSHOT_NAME.
ardana >
ls -alt /mnt/backups/cassandra_backups total 16 drwxr-xr-x 4 backupuser users 4096 Jun 28 03:06 . drwxr-xr-x 3 backupuser users 4096 Jun 28 03:06 doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306root #
~/cassandra-restore-extserver.sh cassandra-snp-2018-06-28-0306 receiving incremental file list ./ var/ var/cassandra/ var/cassandra/data/ var/cassandra/data/data/ var/cassandra/data/data/monasca/ var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/ var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/ var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/ var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/manifest.json var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-37-big-CompressionInfo.db var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-37-big-Data.db ... ... ... /usr/bin/nodetool clearsnapshot monasca
17.4.5.2.3.3 Restart SUSE OpenStack Cloud services #
Restart the MariaDB database
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts galera-bootstrap.ymlOn the deployer node, execute the
galera-bootstrap.yml
playbook which will determine the log sequence number, bootstrap the main node, and start the database cluster.If this process fails to recover the database cluster, refer to Section 15.2.3.1.2, “Recovering the MariaDB Database”.
Restart SUSE OpenStack Cloud services on the three controllers as in the following example.
ardana >
ansible-playbook -i hosts/verb_hosts ardana-start.yml \ --limit doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhostReconfigure SUSE OpenStack Cloud
ardana >
ansible-playbook -i hosts/verb_hosts ardana-reconfigure.yml
17.4.5.2.4 Post restore testing #
Source the service credential file
ardana >
source ~/service.osrcswift
ardana >
openstack container list container_1 volumebackupsardana >
openstack object list container_1 var/lib/ardana/backup.osrc var/lib/ardana/service.osrcardana >
openstack object save container_1 /tmp/backup.osrcneutron
ardana >
openstack network list +--------------------------------------+---------------------+--------------------------------------+ | ID | Name | Subnets | +--------------------------------------+---------------------+--------------------------------------+ | 07c35d11-13f9-41d4-8289-fa92147b1d44 | test-net | 02d5ca3b-1133-4a74-a9ab-1f1dc2853ec8| +--------------------------------------+---------------------+--------------------------------------+glance
ardana >
openstack image list +--------------------------------------+----------------------+--------+ | ID | Name | Status | +--------------------------------------+----------------------+--------+ | 411a0363-7f4b-4bbc-889c-b9614e2da52e | cirros-0.4.0-x86_64 | active | +--------------------------------------+----------------------+--------+ardana >
openstack image save --file /tmp/cirros f751c39b-f1e3-4f02-8332-3886826889baardana >
ls -lah /tmp/cirros -rw-r--r-- 1 ardana ardana 12716032 Jul 2 20:52 /tmp/cirrosnova
ardana >
openstack server listardana >
openstack server create server_6 --image 411a0363-7f4b-4bbc-889c-b9614e2da52e --flavor m1.small --nic net-id=07c35d11-13f9-41d4-8289-fa92147b1d44 +-------------------------------------+------------------------------------------------------------+ | Field | Value | +-------------------------------------+------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | iJBoBaj53oUd | | config_drive | | | created | 2018-07-02T21:02:01Z | | flavor | m1.small (2) | | hostId | | | id | ce7689ff-23bf-4fe9-b2a9-922d4aa9412c | | image | cirros-0.4.0-x86_64 (f751c39b-f1e3-4f02-8332-3886826889ba) | | key_name | None | | name | server_6 | | progress | 0 | | project_id | cca416004124432592b2949a5c5d9949 | | properties | | | security_groups | name='default' | | status | BUILD | | updated | 2018-07-02T21:02:01Z | | user_id | 8cb1168776d24390b44c3aaa0720b532 | | volumes_attached | | +-------------------------------------+------------------------------------------------------------+ardana >
openstack server list +--------------------------------------+----------+--------+---------------------------------+---------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------+--------+---------------------------------+---------------------+-----------+ | ce7689ff-23bf-4fe9-b2a9-922d4aa9412c | server_6 | ACTIVE | n1=1.1.1.8 | cirros-0.4.0-x86_64 | m1.small |ardana >
openstack server delete ce7689ff-23bf-4fe9-b2a9-922d4aa9412c