Backup and Restore

Back up your SUSE Manager installation regularly, to prevent data loss. Because SUSE Manager relies on a database as well as the installed program and configurations, it is important to back up all components of your installation. This chapter contains information on the files you need to back up, and introduces the smdba tool to manage database backups. It also contains information about restoring from your backups in the case of a system failure.

Regardless of the backup method you use, you must have available at least three times the amount of space your current installation uses. Running out of space can result in backups failing, so check this often.

1. Back up SUSE Manager

The most comprehensive method for backing up your SUSE Manager installation is to back up the relevant files and directories. This can save you time in administering your backup, and can be faster to reinstall and re-synchronize in the case of failure. However, this method requires significant disk space and could take a long time to perform the backup.

If you want to back up only the required files and directories, use the following list. To make this process simpler, and more comprehensive, we recommend backing up the entire /etc and /root directories, not just the ones specified here. Some files only exist if you are actually using the related SUSE Manager feature.

  • /etc/cobbler/

  • /etc/dhcp.conf

  • /etc/fstab and any ISO mount points you require.

    If your UUID has changed, ensure you have updated the fstab entries accordingly.

  • /etc/rhn/

  • /etc/salt

  • /etc/sudoers

  • /etc/sysconfig/rhn/

  • /root/.gnupg/

  • /root/.ssh

    This file exists if you are using an SSH tunnel or SSH push. You also need to have saved a copy of the id-susemanager key.

  • /root/ssl-build/

  • /srv/formula_metadata

  • /srv/pillar

  • /srv/salt

  • /srv/susemanager

  • /srv/tftpboot/

  • /srv/www/cobbler

  • /srv/www/htdocs/pub/

  • /srv/www/os-images

  • /var/cache/rhn

  • /var/cache/salt

  • /var/lib/cobbler/

  • /var/lib/cobbler/templates/ (before version 4.0 it is /var/lib/rhn/kickstarts/)

  • /var/lib/Kiwi

  • /var/lib/rhn/

  • /var/run/pgsql/

  • /var/lib/salt/

  • /var/run/salt/

  • /var/spacewalk/

  • Any directories containing custom data such as scripts, Kickstart or AutoYaST profiles, and custom RPMs.

You also need to back up your database, which you can do with the smdba tool.
Procedure: Restoring from a Manual Backup

  1. Re-deploy the SUSE Manager Server with mgradm install podman.

  2. Re-synchronize your SUSE Manager repositories using either the SUSE Manager Web UI, or with the mgr-sync tool at the command prompt. You can choose to re-register your product, or skip the registration and SSL certificate generation sections.

  3. Re-install the /var/lib/containers/storage/volumes/root/ssl-build/rhn-org-httpd-ssl-key-pair-MACHINE_NAME-VER-REL.noarch.rpm package.

  4. Schedule the re-creation of search indexes next time the rhn-search service is started. This command produces only debug messages, it does not produce error messages:

    rhn-search cleanindex

  5. Check whether you need to restore /var/lib/containers/storage/volumes/var-spacewalk/_data/packages/. If /var/lib/containers/storage/volumes/var-spacewalk/_data/packages/ was not in your backup, you need to restore it. If the source repository is available, you can restore `/var/lib/containers/storage/volumes/var-spacewalk/_data/packages/ with a complete channel synchronization:

    mgr-sync refresh --refresh-channels

2. Administering the Database with smdba

The smdba tool is used for managing a local PostgreSQL database. It allows you to back up and restore your database, and manage backups. It can also be used to check the status of your database, and perform administration tasks, such as restarting. For more information about the smdba tool, see [reference:cli-smdba].

The smdba tool works with local PostgreSQL databases only, it does not work with remotely accessed databases, or Oracle databases.

To access a shell inside the Server container, run mgrctl term on the container host. From there, one can run the CLI tools as usual.

The smdba tool requires sudo access, to execute system changes. Ensure you have enabled sudo access for the admin user before you begin, by checking the /etc/sudoers file for this line:

admin   ALL=(postgres) /usr/bin/smdba

Check the runtime status of your database with:

smdba db-status

This command returns either online or offline, for example:

Checking database core...       online

Starting and stopping the database can be performed with:

smdba db-start

And:

smdba db-stop

3. Database Backup with smdba

The smdba tool performs a continuous archiving backup. This backup method combines a log of every change made to the database during the current session, with a series of more traditional backup files. When a crash occurs, the database state is first restored from the most recent backup file on disk, then the log of the current session is replayed exactly, to bring the database back to a current state. A continuous archiving backup with smdba is performed with the database running, so there is no need for downtime.

This method of backing up is stable and generally creates consistent snapshots, however it can take up a lot of storage space. Ensure you have at least three times the current database size of space available for backups. You can check your current database size by navigating to /var/lib/pgsql/ and running df -h.

The smdba tool also manages your archives, keeping only the most recent backup, and the current archive of logs. The log files can only be a maximum file size of 16 MB, so a new log file is created when the files reach this size. Every time you create a new backup, previous backups are purged to release disk space. We recommend you use cron to schedule your smdba backups to ensure that your storage is managed effectively, and you always have a backup ready in case of failure.

3.1. Perform a Manual Database Backup

The smdba tool can be run directly from the command line. We recommend you run a manual database backup immediately after installation, or if you have made any significant changes to your configuration.

When smdba is run for the first time, or if you have changed the location of the backup, it needs to restart your database before performing the archive. This results in a small amount of downtime. Regular database backups do not require any downtime.
Procedure: Performing a Manual Database Backup

  1. Allocate permanent storage space for your backup. This example uses a directory located at /var/spacewalk/. This becomes a permanent target for your backup, so ensure it remains accessible by your server at all times.

  2. In your backup location, create a directory for the backup:

    As root:

    install -d -o postgres -g postgres -m 700 /var/spacewalk/db-backup

  3. Ensure you have the correct permissions set on the backup location:

    chown postgres:postgres /var/spacewalk/db-backup

  4. To create a backup for the first time, run the smdba backup-hot command with the enable option set. This creates the backup in the specified directory, and, if necessary, restart the database:

    smdba backup-hot --enable=on --backup-dir=/var/spacewalk/db-backup

    This command produces debug messages and finishes sucessfully with the output:

    INFO: Finished

  5. Check that the backup files exist in the /var/spacewalk/db-backup directory, to ensure that your backup has been successful.

3.2. Scheduling Automatic Backups

You do not need to shut down your system to perform a database backup with smdba. However, because it is a large operation, database performance can slow down while the backup is running. We recommend you schedule regular database backups for a low-traffic period, to minimize disruption.

Ensure you have at least three times the current database size of space available for backups. You can check your current database size by navigating to /var/lib/pgsql/ and running df -h.
Procedure: Scheduling Automatic Backups

  1. Create a directory for the backup, and set the appropriate permissions (as root):

    install -d -m 700 -o postgres -g postgres /var/spacewalk/db-backup

  2. Open /etc/cron.d/db-backup-mgr, or create it if it does not exist, and add the following line to create the cron job:

    0 2 * * * root /usr/bin/smdba backup-hot --enable=on --backup-dir=/var/spacewalk/db-backup

  3. Check the backup directory regularly to ensure the backups are working as expected.

4. Restore from Backup

The smdba tool can be used to restore from backup in the case of failure.

Procedure: Restoring from Backup

  1. Shut down the database:

    smdba db-stop

  2. Start the restore process and wait for it to complete:

    smdba backup-restore start

  3. Restart the database:

    smdba db-start

  4. Check if there are differences between the RPMs and the database.

    spacewalk-data-fsck

5. Archive Log Settings

Archive logging allows the database management tool smdba to perform hot backups. In SUSE Manager with an embedded database, archive logging is enabled by default.

PostgreSQL maintains a limited number of archive logs. Using the default configuration, approximately 64 files with a size of 16 MiB are stored.

Creating a user and syncing the channels:

  • SLES12-SP2-Pool-x86_64

  • SLES12-SP2-Updates-x86_64

  • SLE-Manager-Tools12-Pool-x86_64-SP2

  • SLE-Manager-Tools12-Updates-x86_64-SP2

PostgreSQL generates an additional roughly 1 GB of data. So it is important to think about a backup strategy and create a backups in a regular way.

Archive logs are stored at /var/lib/pgsql/data/pg_xlog/ (postgresql).

6. Retrieve an Overview of Occupied Database Space

Database administrators may use the subcommand space-overview to get a report about occupied table spaces, for example:

smdba space-overview

outputs:

SUSE Manager Database Control. Version 1.5.2
Copyright (c) 2012 by SUSE Linux Products GmbH

Tablespace  | Size (Mb) | Avail (Mb) | Use %
------------+-----------+------------+------
postgres    | 7         | 49168      | 0.013
susemanager | 776       | 48399      | 1.602

The smdba command is available for PostgreSQL. For a more detailed report, use the space-tables subcommand. It lists the table and its size, for example:

smdba space-tables

outputs:

SUSE Manager Database Control. Version 1.5.2
Copyright (c) 2012 by SUSE Linux Products GmbH

Table                                 | Size
--------------------------------------+-----------
public.all_primary_keys               | 0 bytes
public.all_tab_columns                | 0 bytes
public.allserverkeywordsincereboot    | 0 bytes
public.dblink_pkey_results            | 0 bytes
public.dual                           | 8192 bytes
public.evr_t                          | 0 bytes
public.log                            | 32 kB
...

7. Move the Database

It is possible to move the database to another location. For example, if database storage space is running low.

Procedure: Moving the Database

  1. The default storage location for SUSE Manager is /var/lib/pgsql/. If you would like to move it, for example to /storage/postgres/, proceed as follows.

  2. At the command prompt, as root, stop the running database:

    rcpostgresql stop

    Shut down the running spacewalk services:

    spacewalk-service stop

  3. Copy the current working directory structure with cp using the -a, --archive option. For example:

    cp --archive /var/lib/pgsql/ /storage/postgres/

    This command copies the contents of /var/lib/pgsql/ to /storage/postgres/pgsql/.

    The contents of the /var/lib/pgsql directory needs to remain the same, otherwise the SUSE Manager database may malfunction. You also should ensure that there is enough available disk space.

  4. Mount the new database directory:

    mount /storage/postgres/pgsql

  5. Make sure ownership is postgres:postgres and not root:root by changing to the new directory and running these commands:

    cd /storage/postgres/pgsql/
ls -l

    Outputs:

    total 8
drwxr-x---  4 postgres postgres   47 Jun  2 14:35 ./

  6. Add the new database mount location to your servers fstab by editing etc/fstab.

  7. Start the database with:

    rcpostgresql start

  8. Start the spacewalk services:

    spacewalk-service start

8. Recover From a Crashed Root Partition

If your root partition has crashed, you can restart the SUSE Manager Server with some additional steps. This section assumes you have setup your server using separate partitions for the database and for channels, mounted at /var/lib/pgsql and /var/spacewalk/.

After a new installation of a system most users and groups get different IDs. Most backup systems store the names instead of the IDs and will restore the files with the correct ownership and permissions. But if you mount existing partitions, you must align the ownership to the new system.
Procedure: Recovering from a Crashed Root Partition

  1. Install SUSE Manager. Do not mount the /var/spacewalk and /var/lib/pgsql partitions. Wait for the installation to complete before going on to the next step.

  2. Shut down the database:

    rcpostgresql stop

  3. Shut down the services:

    spacewalk-service stop

  4. Mount /var/spacewalk and /var/lib/pgsql partitions.

  5. Restore the directories listed in Back up SUSE Manager.

  6. Start the database:

    rcpostgresql start

  7. Start the spacewalk services:

    spacewalk-service start

SUSE Manager should now operate normally without loss of your database or synchronized channels.

9. Database Connection Information

You can set information for connecting to the SUSE Manager database by adding or editing these variable in /etc/rhn/rhn.conf:

db_backend = postgresql
db_user = susemanager
db_password = susemanager
db_name = susemanager
db_host = localhost
db_port = 5432
db_ssl_enabled =