Backup and Restore

Back up your SUSE Multi-Linux Manager installation regularly, to prevent data loss. Because SUSE Multi-Linux Manager relies on a database as well as the installed program and configurations, it is important to back up all components of your installation. This chapter contains information on the files you need to back up, and introduces the smdba tool to manage database backups. It also contains information about restoring from your backups in the case of a system failure.

Regardless of the backup method you use, you must have available at least three times the amount of space your current installation uses. Running out of space can result in backups failing, so check this often.

The following section describes backing up data within the SUSE Multi-Linux Manager container.

1. Back up SUSE Multi-Linux Manager

The most comprehensive method for backing up your SUSE Multi-Linux Manager installation is to back up the relevant files and directories. This can save you time in administering your backup, and can be faster to reinstall and re-synchronize in the case of failure. However, this method requires significant disk space and could take a long time to perform the backup.

If you want to back up only the required files and directories, use the following list. To make this process simpler, and more comprehensive, we recommend backing up the entire /etc and /root directories within the container, not just the ones specified here. Some files only exist if you are actually using the related SUSE Multi-Linux Manager feature.

  • /etc/cobbler/

  • /etc/dhcp.conf

  • /etc/fstab and any ISO mount points you require.

    If your UUID has changed, ensure you have updated the fstab entries accordingly.

  • /etc/rhn/

  • /etc/salt

  • /etc/sudoers

  • /etc/sysconfig/rhn/

  • /root/.gnupg/

  • /root/.ssh

    This file exists if you are using an SSH tunnel or SSH push. You also need to have saved a copy of the id-susemanager key.

  • /root/ssl-build/

  • /srv/formula_metadata

  • /srv/pillar

  • /srv/salt

  • /srv/susemanager

  • /srv/tftpboot/

  • /srv/www/cobbler

  • /srv/www/htdocs/pub/

  • /srv/www/os-images

  • /var/cache/rhn

  • /var/cache/salt

  • /var/lib/cobbler/

  • /var/lib/cobbler/templates/ (before version 4.0 it is /var/lib/rhn/kickstarts/)

  • /var/lib/Kiwi

  • /var/lib/rhn/

  • /var/run/pgsql/

  • /var/lib/salt/

  • /var/run/salt/

  • /var/spacewalk/

  • Any directories containing custom data such as scripts, Kickstart or AutoYaST profiles, and custom RPMs.

You also need to back up your database, which you can do with the smdba tool.
Procedure: Restoring from a Manual Backup

  1. On the container host, as root, re-deploy the SUSE Multi-Linux Manager Server with mgradm install podman.

  2. Re-synchronize your SUSE Multi-Linux Manager repositories using either the SUSE Multi-Linux Manager Web UI, or with the mgr-sync tool at the command prompt in the container. You can choose to re-register your product, or skip the registration and SSL certificate generation sections.

  3. In the container, re-install the /root/ssl-build/rhn-org-httpd-ssl-key-pair-MACHINE_NAME-VER-REL.noarch.rpm package.

  4. Schedule the re-creation of search indexes next time the rhn-search service is started. This command produces only debug messages, it does not produce error messages. In the container, enter:

    rhn-search cleanindex

  5. On the container host, check whether you need to restore /var/lib/containers/storage/volumes/var-spacewalk/_data/packages/. If /var/lib/containers/storage/volumes/var-spacewalk/_data/packages/ was not in your backup, you need to restore it. If the source repository is available, you can restore `/var/lib/containers/storage/volumes/var-spacewalk/_data/packages/ with a complete channel synchronization:

    mgrctl exec -ti -- mgr-sync refresh --refresh-channels

2. Administering the Database with smdba

The smdba tool is used for managing a local PostgreSQL database. It allows you to back up and restore your database, and manage backups. It can also be used to check the status of your database, and perform administration tasks, such as restarting. For more information about the smdba tool, see Database CLI Tool.

The smdba tool works with local PostgreSQL databases only, it does not work with remotely accessed databases, or Oracle databases.

To access a shell inside the Server container, run mgrctl term on the container host. From there, one can run the CLI tools as usual.

Check the runtime status of the database with:

smdba db-status

This command returns either online or offline, for example:

Checking database core...       online

Starting and stopping the database can be performed with:

smdba db-start

And:

smdba db-stop

3. Database Backup with smdba

The smdba tool performs a continuous archiving backup. This backup method combines a log of every change made to the database during the current session, with a series of more traditional backup files. When a crash occurs, the database state is first restored from the most recent backup file on disk, then the log of the current session is replayed exactly, to bring the database back to a current state. A continuous archiving backup with smdba is performed with the database running, so there is no need for downtime.

This method of backing up is stable and generally creates consistent snapshots, however it can take up a lot of storage space. Ensure you have at least three times the current database size of space available for backups. You can check your current database size by navigating to /var/lib/pgsql/ and running df -h.

The smdba tool also manages your archives, keeping only the most recent backup, and the current archive of logs. The log files can only be a maximum file size of 16 MB, so a new log file is created when the files reach this size. Every time you create a new backup, previous backups are purged to release disk space. We recommend you use systemd timer on host system to schedule your smdba backups to ensure that your storage is managed effectively, and you always have a backup ready in case of failure.

3.1. Perform a Manual Database Backup

To access a shell inside the Server container, run mgrctl term on the container host. From there, run command line tools as usual.

The smdba tool can be run in the container shell. We recommend you run a manual database backup immediately after installation, or if you have made any significant changes to your configuration.

When smdba is run for the first time, or if you have changed the location of the backup, it needs to restart your database before performing the archive. This results in a small amount of downtime. Regular database backups do not require any downtime.
Procedure: Performing a Manual Database Backup

  1. Allocate permanent storage space for your backup. This example, in the container, uses a directory located at /var/spacewalk/. This becomes a permanent target for your backup, so ensure it remains accessible by your server at all times. This directory is on persistent storage, and for more information, see Persistent Container Volumes.

  2. At the command prompt of the SUSE Multi-Linux Manager container host, as root, enter the server container:

    mgrctl term

  3. Inside the container, execute the following steps:

    1. In your backup location, create a directory for the backup:

      As root:

      install -d -o postgres -g postgres -m 700 /var/spacewalk/db-backup

    2. To create a backup for the first time, run the smdba backup-hot command with the enable option set. This creates the backup in the specified directory, and, if necessary, restart the database:

      smdba backup-hot --enable=on --backup-dir=/var/spacewalk/db-backup

      This command produces debug messages and finishes sucessfully with the output:

      INFO: Finished

    3. Check that the backup files exist in the /var/spacewalk/db-backup directory, to ensure that your backup has been successful.

3.2. Scheduling Automatic Backups

You do not need to shut down your system to perform a database backup with smdba. However, because it is a large operation, database performance can slow down while the backup is running. We recommend you schedule regular database backups for a low-traffic period, to minimize disruption.

Ensure you have at least three times the current database size of space available for backups. You can check your current database size by running on the container host:

mgrctl exec "df -h /var/lib/pgsql"
Procedure: Creating a directory for the backup (do it only once)

  1. At the command prompt of the SUSE Multi-Linux Manager container host, as root, enter the server container:

    mgrctl term

  2. Inside the container, execute the following steps:

    1. In your backup location, create a directory for the backup. As root, enter:

      install -d -o postgres -g postgres -m 700 /var/spacewalk/db-backup
Procedure: Scheduling Automatic Backups

  1. On the container host, as root, create a system unit file at /etc/systemd/system/server-db-backup.service with the content:

    [Unit]
Description="Server backup service"

[Service]
ExecStart=mgrctl exec "smdba backup-hot --enable=on --backup-dir=/var/spacewalk/db-backup"

  2. Create a systemd timer file at /etc/systemd/system/server-db-backup.timer with the content:

    [Unit]
Description="Run server database backup relative to activation time"

[Timer]
OnCalendar=Mon..Sun *-*-* 2:00:00
Unit=server-db-backup.service

[Install]
WantedBy=multi-user.target

  3. Check that all configuration are correct:

    systemd-analyze verify /etc/systemd/system/server-db-backup.*

  4. Enable the timer:

    systemctl enable --now server-db-backup.timer

4. Restore from Backup

The smdba tool can be used to restore from backup in the case of failure.

Procedure: Restoring from Backup

  1. At the command prompt of the SUSE Multi-Linux Manager container host, as root, enter the server container:

    mgrctl term

  2. Inside the container, execute the following steps:

    1. Shut down the database:

      smdba db-stop

    2. Start the restore process and wait for it to complete:

      smdba backup-restore start

    3. Restart the database:

      smdba db-start

    4. Check if there are differences between the RPMs and the database.

      spacewalk-data-fsck

5. Archive Log Settings

Archive logging allows the database management tool smdba to perform hot backups. In SUSE Multi-Linux Manager with an embedded database, archive logging is enabled by default.

PostgreSQL maintains a limited number of archive logs. Using the default configuration, approximately 64 files with a size of 16 MiB are stored.

Creating a user and syncing the channels:

  • SLES12-SP2-Pool-x86_64

  • SLES12-SP2-Updates-x86_64

  • SLE-Manager-Tools12-Pool-x86_64-SP2

  • SLE-Manager-Tools12-Updates-x86_64-SP2

PostgreSQL generates an additional roughly 1 GB of data. So it is important to think about a backup strategy and create a backups in a regular way.

Archive logs are stored, inside the container, at /var/lib/pgsql/data/pg_xlog/ (postgresql).

6. Retrieve an Overview of Occupied Database Space

Database administrators may use the subcommand space-overview to get a report about occupied table spaces, for example:

mgrctl exec smdba space-overview

outputs:

SUSE Manager Database Control. Version 1.5.2
Copyright (c) 2012 by SUSE Linux Products GmbH

Tablespace  | Size (Mb) | Avail (Mb) | Use %
------------+-----------+------------+------
postgres    | 7         | 49168      | 0.013
susemanager | 776       | 48399      | 1.602

The smdba command is available for PostgreSQL. For a more detailed report, use the space-tables subcommand. It lists the table and its size, for example:

mgrctl exec smdba space-tables

outputs:

SUSE Manager Database Control. Version 1.5.2
Copyright (c) 2012 by SUSE Linux Products GmbH

Table                                 | Size
--------------------------------------+-----------
public.all_primary_keys               | 0 bytes
public.all_tab_columns                | 0 bytes
public.allserverkeywordsincereboot    | 0 bytes
public.dblink_pkey_results            | 0 bytes
public.dual                           | 8192 bytes
public.evr_t                          | 0 bytes
public.log                            | 32 kB
...

7. Database Connection Information

Inside the container, you can set information for connecting to the SUSE Multi-Linux Manager database by adding or editing these variable in /etc/rhn/rhn.conf:

db_backend = postgresql
db_user = susemanager
db_password = susemanager
db_name = susemanager
db_host = localhost
db_port = 5432
db_ssl_enabled =