The following sections cover backup and restore operations. Before installing your cloud, there are several things you must do so that you achieve the backup and recovery results you need. SUSE OpenStack Cloud comes with playbooks and procedures to recover the control plane from various disaster scenarios.
As of SUSE OpenStack Cloud 9, Freezer (a distributed backup restore and disaster recovery service) is no longer supported; backup and restore are manual operations.
Consider Section 17.2, “Enabling Backups to a Remote Server” in case you lose cloud servers that back up and restore services.
The following features are supported:
File system backup using a point-in-time snapshot.
Strong encryption: AES-256-CFB.
MariaDB database backup with LVM snapshot.
Restoring your data from a previous backup.
Low storage requirement: backups are stored as compressed files.
Flexible backup (both incremental and differential).
Data is archived in GNU Tar format for file-based incremental backup and restore.
When a key is provided, Open SSL is used to encrypt data (AES-256-CFB).
This section covers manual backup and some restore processes. Full documentation for restore operations is in Section 15.2, “Unplanned System Maintenance”.To back up outside the cluster, refer to Section 17.2, “Enabling Backups to a Remote Server”. Backups of the following types of resources are covered:
Cloud Lifecycle Manager Data. All important information on the Cloud Lifecycle Manager
MariaDB database that is part of the Control Plane. The MariaDB database contains most of the data needed to restore services. MariaDB supports full back up and recovery for all services. Logging data in Elasticsearch is not backed up. swift objects are not backed up because of the redundant nature of swift.
swift Rings used in the swift storage deployment. swift rings are backed up so that you can recover more quickly than rebuilding with swift. swift can rebuild the rings without this backup data, but automatically rebuilding the rings is slower than restoring from a backup.
Audit Logs. Audit Logs are backed up to provide retrospective information and statistical data for performance and security purposes.
The following services will be backed up. Specifically, the data needed to restore the services is backed up. This includes databases and configuration-related files.
Data content for some services is not backed up, as indicated below.
ceilometer. There is no backup of metrics data.
cinder. There is no backup of the volumes.
glance. There is no backup of the images.
heat
horizon
keystone
neutron
nova. There is no backup of the images.
swift. There is no backup of the objects. swift has its own high availability and redundancy. swift rings are backed up. Although swift can rebuild the rings itself, restoring from backup is faster.
Operations Console
monasca. There is no backup of the metrics.
We recommend that you set up a remote server to store your backups, so that you can restore the control plane nodes. This may be necessary if you lose all of your control plane nodes at the same time.
A remote backup server must be set up before proceeding.
You do not have to restore from the remote server if only one or two control plane nodes are lost. In that case, the control planes can be recovered from the data on a remaining control plane node following the restore procedures in Section 15.2.3.2, “Recovering the Control Plane”.
You can do the following to harden an SSH server:
Disable root login
Move SSH to a non-default port (the default SSH port is 22)
Disable password login (only allow RSA keys)
Disable SSH v1
Authorize Secure File Transfer Protocol (SFTP) only for the designated backup user (disable SSH shell)
Firewall SSH traffic to ensure it comes from the SUSE OpenStack Cloud address range
Install a Fail2Ban solution
Restrict users who are allowed to SSH
Additional suggestions are available online
Remove the key pair generated earlier on the backup server; the only thing
needed is .ssh/authorized_keys
. You can remove the
.ssh/id_rsa
and .ssh/id_rsa.pub
files. Be sure to save a backup of them.
Provide adequate space in the directory that is used for backup.
Monitor the space left on that directory.
Keep the system up to date on that server.
Each backup requires the following steps:
Create a snapshot.
Mount the snapshot.
Generate a TAR archive and save it.
Unmount and delete the snapshot.
The following procedure is used for each of the seven BACKUP_TARGETS (list below). Incremental backup instructions follow the full backup procedure. For both full and incremental backups, the last step of the procedure is to unmount and delete the snapshot after the TAR archive has been created and saved. A new snapshot must be created every time a backup is created.
Create a snapshot on the Cloud Lifecycle Manager in (ardana-vg
), the
location where all Cloud Lifecycle Manager data is stored.
ardana >
sudo lvcreate --size 2G --snapshot --permission r \
--name lvm_clm_snapshot /dev/ardana-vg/root
If you have stored additional data or files in your
ardana-vg
directory, you may need more space than the
2G indicated for the size
parameter. In this
situation, create a preliminary TAR archive with the
tar
command on the directory before creating a
snapshot. Set the size
snapshot parameter larger than
the size of the archive.
Mount the snapshot
ardana >
sudo mkdir /var/tmp/clm_snapshotardana >
sudo mount -o ro /dev/ardana-vg/lvm_clm_snapshot /var/tmp/clm_snapshot
Generate a TAR archive (does not apply to incremental backups) with an
appropriate
BACKUP_TAR_ARCHIVE_NAME.tar.gz
backup file for each of the following BACKUP_TARGETS.
Backup Targets
home
ssh
shadow
passwd
group
The backup TAR archive should contain only the necessary data; nothing extra. Some of the archives will be stored as directories, others as files. The backup commands are slightly different for each type.
If the BACKUP_TARGET
is a directory, then that directory must be appended to
/var/tmp/clm_snapshot/
TARGET_DIR. If the BACKUP_TARGET
is a file, then its parent directory must be appended to /var/tmp/clm_snapshot/
.
In the commands that follow, replace BACKUP_TARGET with the appropriate BACKUP_PATH (replacement table is below).
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --file BACKUP_TAR_ARCHIVE_NAME.tar.gz -C \
/var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIR
If BACKUP_TARGET is a directory, replace TARGET_DIR with BACKUP_PATH.
For example, where
BACKUP_PATH=/etc/ssh/
(a
directory):
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --file ssh.tar.gz -C /var/tmp/clm_snapshot/etc/ssh .
If BACKUP_TARGET is a file (not a directory), replace TARGET_DIR with the parent directory of BACKUP_PATH.
For example, where
BACKUP_PATH=/etc/passwd
(a file):
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --file passwd.tar.gz -C /var/tmp/clm_snapshot/etc/passwd
Save the TAR archive to the remote server.
ardana >
scp TAR_ARCHIVE USER@REMOTE_SERVER
Use the following commands to unmount and delete a snapshot.
ardana >
sudo umount -l -f /var/tmp/clm_snapshot; rm -rf /var/tmp/clm_snapshotardana >
sudo lvremove -f /dev/ardana-vg/lvm_clm_snapshot
The table below shows Cloud Lifecycle Manager backup_targets
and their
respective backup_paths
.
backup_name |
backup_path |
---|---|
home_backup |
/var/lib/ardana (file) |
etc_ssh_backup |
/etc/ssh/ (directory) |
shadow_backup |
/etc/shadow (file) |
passwd_backup |
/etc/passwd (file) |
group_backup |
/etc/group (file) |
Incremental backups require a meta
file. If you use the
incremental backup option, a meta file must be included in the
tar
command in the initial backup and whenever you do an
incremental backup. A copy of the original meta
file
should be stored in each backup. The meta
file is used
to determine the incremental changes from the previous backup, so it is
rewritten with each incremental backup.
Versions are useful for incremental backup because they provide a way
to differentiate between each backup. Versions are included in the
tar
command.
Every incremental backup requires creating and mounting a separate snapshot. After the TAR archive is created, the snapshot is unmounted and deleted.
To prepare for incremental backup, follow the steps in Procedure 17.1, “Manual Backup Setup” with the following differences in the
commands for generating a tar
archive.
First time full backup
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --listed-incremental=PATH_TO_YOUR_META \
--file BACKUP_TAR_ARCHIVE_NAME.tar.gz -C \
/var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIR
For example, where
BACKUP_PATH=/etc/ssh/
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--listed-incremental=mysshMeta --file ssh.tar.gz -C \
/var/tmp/clm_snapshot/etc/ssh .
Incremental backup
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --listed-incremental=PATH_TO_YOUR_META\
--file BACKUP_TAR_ARCHIVE_NAME_VERSION.tar.gz -C \
/var/tmp/clm_snapshotTARGET_DIR|BACKUP_TARGET_WITHOUT_LEADING_DIR
For example, where
BACKUP_PATH=/etc/ssh/
:
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--listed-incremental=mysshMeta --file \
ssh_v1.tar.gz -C \
/var/tmp/clm_snapshot/etc/ssh .
After creating an incremental backup, use the following commands to unmount and delete a snapshot.
ardana >
sudo umount -l -f /var/tmp/clm_snapshot; rm -rf /var/tmp/clm_snapshotardana >
sudo lvremove -f /dev/ardana-vg/lvm_clm_snapshot
When a key is provided, Open SSL is used to encrypt data (AES-256-CFB). Backup files can be encrypted with the following command:
ardana >
sudo openssl enc -aes-256-cfb -pass file:ENCRYPT_PASS_FILE_PATH -in \
YOUR_BACKUP_TAR_ARCHIVE_NAME.tar.gz -out YOUR_BACKUP_TAR_ARCHIVE_NAME.tar.gz.enc
For example, using the ssh.tar.gz
generated above:
ardana >
sudo openssl enc -aes-256-cfb -pass file:myEncFile -in ssh.tar.gz -out ssh.tar.gz.enc
When backing up MariaDB, the following process must be performed on
all nodes in the cluster. It is similar to
the backup procedure above for the Cloud Lifecycle Manager (see Procedure 17.1, “Manual Backup Setup”). The difference is the addition of SQL
commands, which are run with the create_db_snapshot.yml
playbook.
Create the create_db_snapshot.yml
file in
~/scratch/ansible/next/ardana/ansible/
on the deployer
with the following content:
- hosts: FND-MDB vars: - snapshot_name: lvm_mysql_snapshot - lvm_target: /dev/ardana-vg/mysql tasks: - name: Cleanup old snapshots become: yes shell: | lvremove -f /dev/ardana-vg/{{ snapshot_name }} ignore_errors: True - name: Create snapshot become: yes shell: | lvcreate --size 2G --snapshot --permission r --name {{ snapshot_name }} {{ lvm_target }} register: snapshot_st ignore_errors: True - fail: msg: "Fail to create snapshot on {{ lvm_target }}" when: snapshot_st.rc != 0
Verify the validity of the lvm_target
variable
(which refers to the actual database LVM volume) before proceeding with
the backup.
Doing the MariaDB backup
We recommend storing the MariaDB version with your backup. The following command saves the MariaDB version as MARIADB_VER.
mysql -V | grep -Eo '(\S+?)-MariaDB' > MARIADB_VER
Open a MariaDB client session on all controllers.
Run the command to spread read lock
on all controllers
and keep the MariaDB session open.
>> FLUSH TABLES WITH READ LOCK;
Open a new terminal and run the
create_db_snapshot.yml
playbook created above.
ardana >
cd ~/scratch/ansible/next/ardana/ansible/ardana >
ansible-playbook -i hosts/verb_hosts create_db_snapshot.yml
Go back to the open MariaDB session and run the command to flush the lock on all controllers.
>> UNLOCK TABLES;
Mount the snapshot
dbnode>> mkdir /var/tmp/mysql_snapshot dbnode>> sudo mount -o ro /dev/ardana-vg/lvm_mysql_snapshot /var/tmp/mysql_snapshot
On each database node, generate a TAR archive with an appropriate
BACKUP_TAR_ARCHIVE_NAME.tar.gz
backup file for the BACKUP_TARGET.
The backup_name
is mysql_backup
and
the backup_path
(BACKUP_TARGET) is
/var/lib/mysql/
.
dbnode>> sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --file mydb.tar.gz /var/tmp/mysql_snapshot/var/lib/mysql .
Unmount and delete the MariaDB snapshot on each database node.
dbnode>> sudo umount -l -f /var/tmp/mysql_snapshot; \ sudo rm -rf /var/tmp/mysql_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_mysql_snapshot
Incremental backups require a meta
file. If you use the
incremental backup option, a meta file must be included in the
tar
command in the initial backup and whenever you do an
incremental backup. A copy of the original meta
file
should be stored in each backup. The meta
file is used
to determine the incremental changes from the previous backup, so it is
rewritten with each incremental backup.
Versions are useful for incremental backup because they provide a way
to differentiate between each backup. Versions are included in the
tar
command.
To prepare for incremental backup, follow the steps in the previous section
except for the tar
commands. Incremental backup
tar
commands must have additional information.
First time MariaDB database full backup
dbnode>> sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek \
--ignore-failed-read --listed-incremental=PATH_TO_YOUR_DB_META \
--file mydb.tar.gz -C /var/tmp/mysql_snapshot/var/lib/mysql .
For example, where
BACKUP_PATH=/var/lib/mysql/
:
dbnode>> sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--listed-incremental=mydbMeta --file mydb.tar.gz -C \
/var/tmp/mysql_snapshot/var/lib/mysql .
Incremental MariaDB database backup
dbnode>> sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek \ --ignore-failed-read --listed-incremental=PATH_TO_YOUR_META\ --file BACKUP_TAR_ARCHIVE_NAME_VERSION.tar.gz -C \ /var/tmp/clm_snapshotTARGET_DIR
For example, where
BACKUP_PATH=/var/lib/mysql/
:
dbnode>> sudo tar --create -z --warning=none --no-check-device \ --one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \ --listed-incremental=mydbMeta --file \ mydb_v1.tar.gz -C /var/tmp/mysql_snapshot/var/lib/mysql .
After creating and saving the TAR archive, unmount and delete the snapshot.
dbnode>> sudo umount -l -f /var/tmp/mysql_snapshot; \ sudo rm -rf /var/tmp/mysql_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_mysql_snapshot
Encrypt your MariaDB database backup following the instructions in Section 17.3.1.2, “Encryption”
Upload your BACKUP_TARGET.tar.gz to your preferred remote server.
The following procedure is used to back up swift rings. It is similar to the Cloud Lifecycle Manager backup (see Procedure 17.1, “Manual Backup Setup”).
The steps must be performed only on the building server (For more information, see Section 18.6.2.4, “Identifying the Swift Ring Building Server”.).
The backup_name
is
swift_builder_dir_backup
and the
backup_path
is /etc/swiftlm/
.
Create a snapshot
ardana >
sudo lvcreate --size 2G --snapshot --permission r \
--name lvm_root_snapshot /dev/ardana-vg/root
Mount the snapshot
ardana >
mkdir /var/tmp/root_snapshot; sudo mount -o ro \
/dev/ardana-vg/lvm_root_snapshot /var/tmp/root_snapshot
Create the TAR archive
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--file swring.tar.gz -C /var/tmp/root_snapshot/etc/swiftlm .
Upload your swring.tar.gz
TAR archive to your preferred remote server.
Unmount and delete the snapshot
ardana >
sudo umount -l -f /var/tmp/root_snapshot; sudo rm -rf \
/var/tmp/root_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_root_snapshot
The following procedure is used to back up Audit Logs. It is similar to the Cloud Lifecycle Manager backup (see Procedure 17.1, “Manual Backup Setup”). The steps must be performed on all nodes; there will be a backup TAR archive for each node. Before performing the following steps, run through Section 13.2.7.2, “Enable Audit Logging” .
The backup_name
is
audit_log_backup
and the
backup_path
is /var/audit
.
Create a snapshot
ardana >
sudo lvcreate --size 2G --snapshot --permission r --name \
lvm_root_snapshot /dev/ardana-vg/root
Mount the snapshot
ardana >
mkdir /var/tmp/root_snapshot; sudo mount -o ro \
/dev/ardana-vg/lvm_root_snapshot /var/tmp/root_snapshot
Create the TAR archive
ardana >
sudo tar --create -z --warning=none --no-check-device \
--one-file-system --preserve-permissions --same-owner --seek --ignore-failed-read \
--file audit.tar.gz -C /var/tmp/root_snapshot/var/audit .
Upload your audit.tar.gz
TAR archive to your
preferred remote server.
Unmount and delete a snapshot
ardana >
sudo umount -l -f /var/tmp/root_snapshot; sudo rm -rf \
/var/tmp/root_snapshot; sudo lvremove -f /dev/ardana-vg/lvm_root_snapshot
Restore the Audit Logs backup with the following commands
Retrieve the Audit Logs TAR archive
Extract the TAR archive to the proper backup location
ardana >
sudo tar -z --incremental --extract --ignore-zeros \
--warning=none --overwrite --directory /var/audit/ -f audit.tar.gz
Full Disaster Recovery Test
Back up the control plane using the manual backup procedure
Backup the Cassandra Database
Re-install Controller 1 with the SUSE OpenStack Cloud ISO
Use manual restore steps to recover deployment data (and model)
Re-install SUSE OpenStack Cloud on Controllers 1, 2, 3
Recover the backup of the MariaDB database
Recover the Cassandra Database
Verify testing
The testing environment is similar to the Entry Scale model.
It uses five servers: three Control Nodes and two Compute Nodes.
The controller node has three disks. The first is reserved for the system; the others are used for swift.
For this Disaster Recovery test, data has been saved on disks 2 and 3 of the swift controllers, which allows for swift objects to be restored the recovery. If these disks were also wiped, swift data would be lost, but the procedure would not change. The only difference is that glance images would be lost and would have to be uploaded again.
Unless specified otherwise, all commands should be executed on controller 1, which is also the deployer node.
In order to validate the procedure after recovery, we need to create some workloads.
Source the service credential file
ardana >
source ~/service.osrc
Copy an image to the platform and create a glance image with it. In this example, Cirros is used
ardana >
openstack image create --disk-format raw --container-format \
bare --public --file ~/cirros-0.3.5-x86_64-disk.img cirros
Create a network
ardana >
openstack network create test_net
Create a subnet
ardana >
openstack subnet create 07c35d11-13f9-41d4-8289-fa92147b1d44 192.168.42.0/24 --name test_subnet
Create some instances
ardana >
openstack server create server_1 --image 411a0363-7f4b-4bbc-889c-b9614e2da52e --flavor m1.small --nic net-id=07c35d11-13f9-41d4-8289-fa92147b1d44ardana >
openstack server create server_2 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44ardana >
openstack server create server_3 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44ardana >
openstack server create server_4 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44ardana >
openstack server create server_5 --image 411a03...e2da52e --flavor m1.small --nic net-id=07c35d...147b1d44ardana >
openstack server list
Create containers and objects
ardana >
openstack object create container_1 ~/service.osrc var/lib/ardana/service.osrcardana >
openstack object create container_1 ~/backup.osrc swift upload container_1 ~/backup.osrcardana >
openstack object list container_1 var/lib/ardana/backup.osrc var/lib/ardana/service.osrc
In this example, backups are stored on the server
192.168.69.132
Connect to the backup server
Create the user
root #
useradd BACKUPUSER --create-home --home-dir /mnt/backups/
Switch to that user
root #
su BACKUPUSER
Create the SSH keypair
backupuser >
ssh-keygen -t rsa
> # Leave the default for the first question and do not set any passphrase
> Generating public/private rsa key pair.
> Enter file in which to save the key (/mnt/backups//.ssh/id_rsa):
> Created directory '/mnt/backups//.ssh'.
> Enter passphrase (empty for no passphrase):
> Enter same passphrase again:
> Your identification has been saved in /mnt/backups//.ssh/id_rsa
> Your public key has been saved in /mnt/backups//.ssh/id_rsa.pub
> The key fingerprint is:
> a9:08:ae:ee:3c:57:62:31:d2:52:77:a7:4e:37:d1:28 backupuser@padawan-ccp-c0-m1-mgmt
> The key's randomart image is:
> +---[RSA 2048]----+
> | o |
> | . . E + . |
> | o . . + . |
> | o + o + |
> | + o o S . |
> | . + o o |
> | o + . |
> |.o . |
> |++o |
> +-----------------+
Add the public key to the list of the keys authorized to connect to that user on this server
backupuser >
cat /mnt/backups/.ssh/id_rsa.pub >> /mnt/backups/.ssh/authorized_keys
Print the private key. This will be used for the backup configuration (ssh_credentials.yml file)
backupuser >
cat /mnt/backups/.ssh/id_rsa
> -----BEGIN RSA PRIVATE KEY-----
> MIIEogIBAAKCAQEAvjwKu6f940IVGHpUj3ffl3eKXACgVr3L5s9UJnb15+zV3K5L
> BZuor8MLvwtskSkgdXNrpPZhNCsWSkryJff5I335Jhr/e5o03Yy+RqIMrJAIa0X5
> ...
> ...
> ...
> iBKVKGPhOnn4ve3dDqy3q7fS5sivTqCrpaYtByJmPrcJNjb2K7VMLNvgLamK/AbL
> qpSTZjicKZCCl+J2+8lrKAaDWqWtIjSUs29kCL78QmaPOgEvfsw=
> -----END RSA PRIVATE KEY-----
In this example, backups will be stored on the server
192.168.69.132
, in the
/mnt/backups/cassandra_backups/
directory.
Create a directory on the backup server to store Cassandra backups.
backupuser >
mkdir /mnt/backups/cassandra_backups
Copy the private SSH key from the backup server to all controller nodes.
Replace CONTROLLER with each control node e.g. doc-cp1-c1-m1-mgmt, doc-cp1-c1-m2-mgmt etc
Log in to each controller node and copy the private SSH key to
.ssh
directory of the root user.
ardana >
sudo cp /var/lib/ardana/.ssh/id_rsa_backup /root/.ssh/
Verify that you can SSH to the backup server as backupuser using the private key.
root #
ssh -i ~/.ssh/id_rsa_backup backupuser@192.168.69.132
Create the following cassandra-backup-extserver.sh
script on all controller nodes.
root #
cat > ~/cassandra-backup-extserver.sh << EOF
#!/bin/sh
# backup user
BACKUP_USER=backupuser
# backup server
BACKUP_SERVER=192.168.69.132
# backup directory
BACKUP_DIR=/mnt/backups/cassandra_backups/
# Setup variables
DATA_DIR=/var/cassandra/data/data
NODETOOL=/usr/bin/nodetool
# example: cassandra-snp-2018-06-26-1003
SNAPSHOT_NAME=cassandra-snp-\$(date +%F-%H%M)
HOST_NAME=\$(/bin/hostname)_
# Take a snapshot of Cassandra database
\$NODETOOL snapshot -t \$SNAPSHOT_NAME monasca
# Collect a list of directories that make up the snapshot
SNAPSHOT_DIR_LIST=\$(find \$DATA_DIR -type d -name \$SNAPSHOT_NAME)
for d in \$SNAPSHOT_DIR_LIST
do
# copy snapshot directories to external server
rsync -avR -e "ssh -i /root/.ssh/id_rsa_backup" \$d \$BACKUP_USER@\$BACKUP_SERVER:\$BACKUP_DIR/\$HOST_NAME\$SNAPSHOT_NAME
done
\$NODETOOL clearsnapshot monasca
EOF
root #
chmod +x ~/cassandra-backup-extserver.sh
Execute following steps on all the controller nodes
The /usr/local/sbin/cassandra-backup-extserver.sh
script should be executed on all three controller nodes at the same time
(within seconds of each other) for a successful backup.
Edit the
/usr/local/sbin/cassandra-backup-extserver.sh
script
Set BACKUP_USER and
BACKUP_SERVER to the desired backup user (for
example, backupuser
) and
desired backup server (for example, 192.168.68.132
),
respectively.
BACKUP_USER=backupuser BACKUP_SERVER=192.168.69.132 BACKUP_DIR=/mnt/backups/cassandra_backups/
Execute ~/cassandra-backup-extserver.sh
on on all
controller nodes which are also Cassandra nodes.
root #
~/cassandra-backup-extserver.sh
Requested creating snapshot(s) for [monasca] with snapshot name [cassandra-snp-2018-06-28-0251] and options {skipFlush=false}
Snapshot directory: cassandra-snp-2018-06-28-0251
sending incremental file list
created directory /mnt/backups/cassandra_backups//doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251
/var/
/var/cassandra/
/var/cassandra/data/
/var/cassandra/data/data/
/var/cassandra/data/data/monasca/
...
...
...
/var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-72-big-Summary.db
/var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-72-big-TOC.txt
/var/cassandra/data/data/monasca/measurements-e29033d0488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/schema.cql
sent 173,691 bytes received 531 bytes 116,148.00 bytes/sec
total size is 171,378 speedup is 0.98
Requested clearing snapshot(s) for [monasca]
Verify the Cassandra backup directory on the backup server.
backupuser >
ls -alt /mnt/backups/cassandra_backups
total 16
drwxr-xr-x 4 backupuser users 4096 Jun 28 03:06 .
drwxr-xr-x 3 backupuser users 4096 Jun 28 03:06 doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306
drwxr-xr-x 3 backupuser users 4096 Jun 28 02:51 doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251
drwxr-xr-x 8 backupuser users 4096 Jun 27 20:56 ..
$backupuser@backupserver> du -shx /mnt/backups/cassandra_backups/*
6.2G /mnt/backups/cassandra_backups/doc-cp1-c1-m1-mgmt_cassandra-snp-2018-06-28-0251
6.3G /mnt/backups/cassandra_backups/doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306
Back up the Cloud Lifecycle Manager using the procedure at Section 17.3.1, “Cloud Lifecycle Manager Data Backup”
Back up the MariaDB database using the procedure at Section 17.3.2, “MariaDB Database Backup”
Back up swift rings using the procedure at Section 17.3.3, “swift Ring Backup”
Log in to the Cloud Lifecycle Manager.
Retrieve the Cloud Lifecycle Manager backups that were created with Section 17.3.1, “Cloud Lifecycle Manager Data Backup”. There are multiple backups; directories are handled differently than files.
Extract the TAR archives for each of the seven locations.
ardana >
sudo tar -z --incremental --extract --ignore-zeros \
--warning=none --overwrite --directory RESTORE_TARGET \
-f BACKUP_TARGET.tar.gz
For example, with a directory such as
BACKUP_TARGET=/etc/ssh/
ardana >
sudo tar -z --incremental --extract --ignore-zeros \
--warning=none --overwrite --directory /etc/ssh/ -f ssh.tar.gz
With a file such as BACKUP_TARGET=/etc/passwd
ardana >
sudo tar -z --incremental --extract --ignore-zeros --warning=none --overwrite --directory /etc/ -f passwd.tar.gz
Change back to the default ardana user.
Run the cobbler-deploy.yml
playbook.
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost cobbler-deploy.xml
Run the bm-reimage.yml
playbook limited to the
second and third controllers.
ardana >
ansible-playbook -i hosts/localhost bm-reimage.yml -e nodelist=controller2,controller3
The names of controller2 and controller3. Use the
bm-power-status.yml
playbook to check the cobbler
names of these nodes.
Run the site.yml
playbook limited to the three
controllers and localhost—in this example,
doc-cp1-c1-m1-mgmt
,
doc-cp1-c1-m2-mgmt
,
doc-cp1-c1-m3-mgmt
, and localhost
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts site.yml --limit \ doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhost
Log in to the first controller node.
Retrieve the MariaDB backup that was created with Section 17.3.2, “MariaDB Database Backup”.
Create a temporary directory and extract the TAR archive (for example,
mydb.tar.gz
).
ardana >
mkdir /tmp/mysql_restore; sudo tar -z --incremental \
--extract --ignore-zeros --warning=none --overwrite --directory /tmp/mysql_restore/ \
-f mydb.tar.gz
Verify that the files have been restored on the controller.
ardana >
sudo du -shx /tmp/mysql_restore/*
16K /tmp/mysql_restore/aria_log.00000001
4.0K /tmp/mysql_restore/aria_log_control
3.4M /tmp/mysql_restore/barbican
8.0K /tmp/mysql_restore/ceilometer
4.2M /tmp/mysql_restore/cinder
2.9M /tmp/mysql_restore/designate
129M /tmp/mysql_restore/galera.cache
2.1M /tmp/mysql_restore/glance
4.0K /tmp/mysql_restore/grastate.dat
4.0K /tmp/mysql_restore/gvwstate.dat
2.6M /tmp/mysql_restore/heat
752K /tmp/mysql_restore/horizon
4.0K /tmp/mysql_restore/ib_buffer_pool
76M /tmp/mysql_restore/ibdata1
128M /tmp/mysql_restore/ib_logfile0
128M /tmp/mysql_restore/ib_logfile1
12M /tmp/mysql_restore/ibtmp1
16K /tmp/mysql_restore/innobackup.backup.log
313M /tmp/mysql_restore/keystone
716K /tmp/mysql_restore/magnum
12M /tmp/mysql_restore/mon
8.3M /tmp/mysql_restore/monasca_transform
0 /tmp/mysql_restore/multi-master.info
11M /tmp/mysql_restore/mysql
4.0K /tmp/mysql_restore/mysql_upgrade_info
14M /tmp/mysql_restore/nova
4.4M /tmp/mysql_restore/nova_api
14M /tmp/mysql_restore/nova_cell0
3.6M /tmp/mysql_restore/octavia
208K /tmp/mysql_restore/opsconsole
38M /tmp/mysql_restore/ovs_neutron
8.0K /tmp/mysql_restore/performance_schema
24K /tmp/mysql_restore/tc.log
4.0K /tmp/mysql_restore/test
8.0K /tmp/mysql_restore/winchester
4.0K /tmp/mysql_restore/xtrabackup_galera_info
Stop SUSE OpenStack Cloud services on the three controllers (using the hostnames of the controllers in your configuration).
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts ardana-stop.yml --limit \ doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhost
Delete the files in the mysql
directory and copy
the restored backup to that directory.
root #
cd /var/lib/mysql/root #
rm -rf ./*root #
cp -pr /tmp/mysql_restore/* ./
Switch back to the ardana
user when the copy is finished.
Create a script called
cassandra-restore-extserver.sh
on all controller
nodes
root #
cat > ~/cassandra-restore-extserver.sh << EOF
#!/bin/sh
# backup user
BACKUP_USER=backupuser
# backup server
BACKUP_SERVER=192.168.69.132
# backup directory
BACKUP_DIR=/mnt/backups/cassandra_backups/
# Setup variables
DATA_DIR=/var/cassandra/data/data
NODETOOL=/usr/bin/nodetool
HOST_NAME=\$(/bin/hostname)_
#Get snapshot name from command line.
if [ -z "\$*" ]
then
echo "usage \$0 <snapshot to restore>"
exit 1
fi
SNAPSHOT_NAME=\$1
# restore
rsync -av -e "ssh -i /root/.ssh/id_rsa_backup" \$BACKUP_USER@\$BACKUP_SERVER:\$BACKUP_DIR/\$HOST_NAME\$SNAPSHOT_NAME/ /
# set ownership of newley restored files
chown -R cassandra:cassandra \$DATA_DIR/monasca/*
# Get a list of snapshot directories that have files to be restored.
RESTORE_LIST=\$(find \$DATA_DIR -type d -name \$SNAPSHOT_NAME)
# use RESTORE_LIST to move snapshot files back into place of database.
for d in \$RESTORE_LIST
do
cd \$d
mv * ../..
KEYSPACE=\$(pwd | rev | cut -d '/' -f4 | rev)
TABLE_NAME=\$(pwd | rev | cut -d '/' -f3 |rev | cut -d '-' -f1)
\$NODETOOL refresh \$KEYSPACE \$TABLE_NAME
done
cd
# Cleanup snapshot directories
\$NODETOOL clearsnapshot \$KEYSPACE
EOF
root #
chmod +x ~/cassandra-restore-extserver.sh
Execute following steps on all the controller nodes.
Edit the ~/cassandra-restore-extserver.sh
script.
Set
BACKUP_USER,BACKUP_SERVER
to the desired backup user (for example,
backupuser
) and the
desired backup server (for example, 192.168.68.132
),
respectively.
BACKUP_USER=backupuser BACKUP_SERVER=192.168.69.132 BACKUP_DIR=/mnt/backups/cassandra_backups/
Execute ~/cassandra-restore-extserver.sh
SNAPSHOT_NAME
Find SNAPSHOT_NAME from listing of /mnt/backups/cassandra_backups. All the directories have the format HOST_SNAPSHOT_NAME.
ardana >
ls -alt /mnt/backups/cassandra_backups
total 16
drwxr-xr-x 4 backupuser users 4096 Jun 28 03:06 .
drwxr-xr-x 3 backupuser users 4096 Jun 28 03:06 doc-cp1-c1-m2-mgmt_cassandra-snp-2018-06-28-0306
root #
~/cassandra-restore-extserver.sh cassandra-snp-2018-06-28-0306
receiving incremental file list
./
var/
var/cassandra/
var/cassandra/data/
var/cassandra/data/data/
var/cassandra/data/data/monasca/
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/manifest.json
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-37-big-CompressionInfo.db
var/cassandra/data/data/monasca/alarm_state_history-e6bbdc20488d11e8bdabc32666406af1/snapshots/cassandra-snp-2018-06-28-0306/mc-37-big-Data.db
...
...
...
/usr/bin/nodetool clearsnapshot monasca
Restart the MariaDB database
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts galera-bootstrap.yml
On the deployer node, execute the
galera-bootstrap.yml
playbook which will determine
the log sequence number, bootstrap the main node, and start the
database cluster.
If this process fails to recover the database cluster, refer to Section 15.2.3.1.2, “Recovering the MariaDB Database”.
Restart SUSE OpenStack Cloud services on the three controllers as in the following example.
ardana >
ansible-playbook -i hosts/verb_hosts ardana-start.yml \
--limit doc-cp1-c1-m1-mgmt,doc-cp1-c1-m2-mgmt,doc-cp1-c1-m3-mgmt,localhost
Reconfigure SUSE OpenStack Cloud
ardana >
ansible-playbook -i hosts/verb_hosts ardana-reconfigure.yml
Source the service credential file
ardana >
source ~/service.osrc
swift
ardana >
openstack container list container_1 volumebackupsardana >
openstack object list container_1 var/lib/ardana/backup.osrc var/lib/ardana/service.osrcardana >
openstack object save container_1 /tmp/backup.osrc
neutron
ardana >
openstack network list
+--------------------------------------+---------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+---------------------+--------------------------------------+
| 07c35d11-13f9-41d4-8289-fa92147b1d44 | test-net | 02d5ca3b-1133-4a74-a9ab-1f1dc2853ec8|
+--------------------------------------+---------------------+--------------------------------------+
glance
ardana >
openstack image list +--------------------------------------+----------------------+--------+ | ID | Name | Status | +--------------------------------------+----------------------+--------+ | 411a0363-7f4b-4bbc-889c-b9614e2da52e | cirros-0.4.0-x86_64 | active | +--------------------------------------+----------------------+--------+ardana >
openstack image save --file /tmp/cirros f751c39b-f1e3-4f02-8332-3886826889baardana >
ls -lah /tmp/cirros -rw-r--r-- 1 ardana ardana 12716032 Jul 2 20:52 /tmp/cirros
nova
ardana >
openstack server listardana >
openstack server create server_6 --image 411a0363-7f4b-4bbc-889c-b9614e2da52e --flavor m1.small --nic net-id=07c35d11-13f9-41d4-8289-fa92147b1d44 +-------------------------------------+------------------------------------------------------------+ | Field | Value | +-------------------------------------+------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | iJBoBaj53oUd | | config_drive | | | created | 2018-07-02T21:02:01Z | | flavor | m1.small (2) | | hostId | | | id | ce7689ff-23bf-4fe9-b2a9-922d4aa9412c | | image | cirros-0.4.0-x86_64 (f751c39b-f1e3-4f02-8332-3886826889ba) | | key_name | None | | name | server_6 | | progress | 0 | | project_id | cca416004124432592b2949a5c5d9949 | | properties | | | security_groups | name='default' | | status | BUILD | | updated | 2018-07-02T21:02:01Z | | user_id | 8cb1168776d24390b44c3aaa0720b532 | | volumes_attached | | +-------------------------------------+------------------------------------------------------------+ardana >
openstack server list +--------------------------------------+----------+--------+---------------------------------+---------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------+--------+---------------------------------+---------------------+-----------+ | ce7689ff-23bf-4fe9-b2a9-922d4aa9412c | server_6 | ACTIVE | n1=1.1.1.8 | cirros-0.4.0-x86_64 | m1.small |ardana >
openstack server delete ce7689ff-23bf-4fe9-b2a9-922d4aa9412c