SUSE Multi-Linux Manager Server Migration to a Containerized Environment

1. Requirements and considerations

1.1. General requirements

To migrate a SUSE Multi-Linux Manager 4.3 Server to a container, you require a new machine with SL Micro 6.1 or SUSE Linux Enterprise Server 15 SP7 and mgradm installed.
An in-place migration from SUSE Multi-Linux Manager 4.3 to 5.1 is not supported, regardless of whether the chosen host operating system is SL Micro 6.1 or SUSE Linux Enterprise Server 15 SP7.
Before migrating from SUSE Multi-Linux Manager 4.3 to 5.1, any existing traditional clients including the traditional proxies must be migrated to Salt. For more information about migrating traditional SUSE Multi-Linux Manager 4.3 clients to Salt clients, see Migrate Traditional Clients to Salt Clients.
Traditional contact protocol is no longer supported in SUSE Multi-Linux Manager 5.0 and later.

This guide only covers the migration from SUSE Multi-Linux Manager 4.3 to 5.1.

Migrating an existing SUSE Multi-Linux Manager 5.1 instance to the same version while switching the host operating system from SL Micro 6.1 to SUSE Linux Enterprise Server 15 SP7, or vice versa, is not handled by the mgradm migrate command.

1.2. Hostnames

Current migration procedure does not include functionality for renaming hostnames. As a result, fully qualified domain name (FQDN) of the new server will remain the same as that of the old server.
The IP address must remain unchanged to ensure that the clients can contact the server.

After the migration, it is necessary to manually update the DHCP and DNS records to point to the new server.

1.3. GPG keys

Self-trusted GPG keys are not migrated.
GPG keys that are trusted in the RPM database only are not migrated. Thus, synchronizing channels with spacewalk-repo-sync can fail.
The administrator must migrate these keys manually from the 4.3 installation to the container host after the actual server migration.
Procedure: Manual migration of the 4.3 GPG keys to the new server
1. Copy the keys from the 4.3 server to the container host of the new server.
2. Later, add each key to the migrated server with the command mgradm gpg add <PATH_TO_KEY_FILE>.

1.4. SSL certificates

During a migration the server SSL certificate and CA chain are copied from the source server, meaning that only the database certificates are required.

Starting with SUSE Multi-Linux Manager 5.1, the PostgreSQL database runs in a separate container that requires its own SSL certificate during migration. This is why database certificates are needed even when the server certificate and CA chain are copied from the source.

Third-party certificates that are already in use on your SUSE Multi-Linux Manager 4.3 installation will be copied automatically during the migration. No additional configuration is required for most scenarios.

When migrating, SUSE Multi-Linux Manager uses self-signed certificates by default. To use certificates from an external PKI (certificate authority), you can provide all certificate files when running mgradm migrate using the --ssl* flags.

Table 1. Quick-reference: which flags are needed for your migration
Migration situation	Flags needed
Self-signed CA, everything unchanged	None
New server certificate (same CA)	`--ssl-server-cert`, `--ssl-server-key`
New database certificate (same CA)	`--ssl-db-cert`, `--ssl-db-key`
New server and database certificates (same CA)	`--ssl-server-cert`, `--ssl-server-key`, `--ssl-db-cert`, `--ssl-db-key`
New server certificate + new CA	`--ssl-server-cert`, `--ssl-server-key`, `--ssl-ca-root`, `--ssl-ca-intermediate`
New database certificate + new CA	`--ssl-db-cert`, `--ssl-db-key`, `--ssl-db-ca-root`, `--ssl-db-ca-intermediate`
Changing CA only (same cert files, different CA chain)	`--ssl-ca-root`, `--ssl-ca-intermediate`, `--ssl-db-ca-root`, `--ssl-db-ca-intermediate`
Full replacement — new certs and CA for both components	See "All certificates from an external PKI" below

Any --ssl* flag not specified uses the corresponding value from the source server. Only provide flags for the files that differ from your current setup.

1.4.1. Flag reference

The following --ssl* flags are supported by mgradm migrate:

Flag Description

Flag	Description
`--ssl-ca-root`	Path to the root CA certificate that signed the server certificate (defaults to source server CA)
`--ssl-ca-intermediate`	Path to an intermediate CA certificate in the chain (can be specified multiple times; defaults to source server)
`--ssl-server-cert`	Path to the server certificate (defaults to source server certificate)
`--ssl-server-key`	Path to the server private key (defaults to source server key)
`--ssl-db-ca-root`	Path to the root CA certificate that signed the database certificate
`--ssl-db-ca-intermediate`	Path to an intermediate CA certificate in the database certificate chain (can be specified multiple times)
`--ssl-db-cert`	Path to the database certificate
`--ssl-db-key`	Path to the database private key

--ssl-ca-root

Path to the root CA certificate that signed the server certificate (defaults to source server CA)

--ssl-ca-intermediate

Path to an intermediate CA certificate in the chain (can be specified multiple times; defaults to source server)

--ssl-server-cert

Path to the server certificate (defaults to source server certificate)

--ssl-server-key

Path to the server private key (defaults to source server key)

--ssl-db-ca-root

Path to the root CA certificate that signed the database certificate

--ssl-db-ca-intermediate

Path to an intermediate CA certificate in the database certificate chain (can be specified multiple times)

--ssl-db-cert

Path to the database certificate

--ssl-db-key

Path to the database private key

The database certificate chain is completely separate from the server certificate chain. The --ssl-db-* flags must be provided even when using the same CA, as the database container requires its own certificates generated during migration.

1.4.2. Certificate requirements

The following requirements apply to all certificate files used with mgradm migrate:

All certificate files must be in PEM format. If your certificates are in DER format, convert them first.
The hostname in the X509v3 Subject Alternative Name section of the certificate must match the fully qualified hostname of the machine the certificates are deployed on.
Database certificates require reportdb and db as Subject Alternative Name (SAN).
Supported key types are RSA and EC (Elliptic Curve).
When a CA chain is used, concatenate the certificates with the server or database certificate first, followed by all intermediate CA certificates in order. The root CA certificate should be provided in its own file. To combine files, you can use the following command:
```
cat server.pem intermediate-ca1.pem intermediate-ca2.pem > combined-server.pem
```

For full details on certificate preparation, see Import SSL Certificates.

1.4.3. Scenario: All certificates from an external PKI

This scenario covers migrating a server where both the web and database components use certificates from a third-party CA. The organization must gather the following certificate files:

Root CA certificate for the web component
Intermediate CA certificate for the web component (if applicable)
Server certificate for the web component
Private key for the web component
Root CA certificate for the database component
Intermediate CA certificate for the database component (if applicable)
Server certificate for the database component (with reportdb and db as SAN)
Private key for the database component

In most deployments, a single CA serves both the server and database components. In that case, the same certificate files are used for both --ssl- and --ssl-db- flags. The example below uses separate file names for clarity.

Listing 1. Example migration command with all --ssl* flags

mgradm migrate podman <oldserver.fqdn> \
  --ssl-ca-root rootCAweb.pem \
  --ssl-ca-intermediate intermediateCAweb.pem \
  --ssl-server-cert webcert.pem \
  --ssl-server-key webkey.key \
  --ssl-db-ca-root rootCAdb.pem \
  --ssl-db-ca-intermediate intermediateCAdb.pem \
  --ssl-db-cert dbcert.pem \
  --ssl-db-key dbkey.key

Client deployment of external CA certificates

After a successful migration with external PKI certificates, deploy the CA certificate to all clients so they can verify the new server certificates:

mgrctl exec -- 'salt -b 50 * state.apply certs'

1.4.4. Scenario: Mixed external and self-signed certificates

When migrating with a mix of external and self-signed certificates, only replace the CA in a certificate chain when the server or database certificate and its private key are also being replaced in that same chain. For example, when replacing both the server certificate and key with external PKI versions, also provide the corresponding CA certificates. However, when only replacing database certificates, the self-signed CA password is still needed if the original CA is being reused.

For the complete procedure, see the SCC article: SUSE Multi-Linux Manager Server migration fails due to unknown (or lost) CA password.

2. Migration

2.1. Prepare SUSE Multi-Linux Manager 5.1 server host

Do not pre-install SUSE Multi-Linux Manager on the prepared SL Micro 6.1 or SUSE Linux Enterprise Server 15 SP7 system.

The migration process is designed to perform the server installation automatically. Running mgradm install and then mgradm migrate is not supported and will lead to an unsupported system state.

In the following steps, we are only preparing the host system, not installing the actual SUSE Multi-Linux Manager 5.1 Server.

You can use VM images based on SL Micro 6.1 as a migration target. In such a scenario, you can prepare the host system as described in

However, at the end the last step is executing the command mgradm migrate <FQDN> instead of mgradm install <FQDN>.

2.1.1. Prepare SL Micro 6.1 Host

2.1.1.1. Download the installation media

Procedure: Downloading the installation media

Locate the SL Micro 6.1 installation media at https://www.suse.com/download/sle-micro/, and download the appropriate media file.

Prepare a DVD or USB flash drive with the downloaded .iso image for installation.

2.1.1.2. Install SL Micro 6.1

Procedure: Installing SL Micro 6.1

Insert the DVD or USB flash drive (USB disk or key) containing the installation image for SLE Micro 6.1.

Boot or reboot your system.

Use the arrow keys to select Installation.

Adjust Keyboard and language.

Click the checkbox to accept the license agreement.

Click Next to continue.

Select the registration method. For this example, we will register the server with SUSE Customer Center.

The SUSE Multi-Linux Manager 5.1 containers are installed as extensions. Depending on the specific extension needed from the list below, additional SUSE Customer Center registration codes will be required for each.

SUSE Multi-Linux Manager 5.1 Server

SUSE Multi-Linux Manager 5.1 Proxy

SUSE Multi-Linux Manager 5.1 Retail Branch Server

The SL Micro 6.1 entitlement is included within the SUSE Multi-Linux Manager entitlement, so it does not require a separate registration code.

Enter your SUSE Customer Center email address.

Enter your registration code for SL Micro 6.1.

Click Next to continue.

To install a proxy, select the SUSE Multi-Linux Manager 5.1 Proxy extension; to install a server, select the SUSE Multi-Linux Manager 5.1 Server extension Checkbox.

Click Next to continue.

Enter your SUSE Multi-Linux Manager 5.1 extension registration code.

Click Next to continue.

On the NTP Configuration page click Next.

On the Authentication for the System page enter a password for the root user. Click Next.

On the Installation Settings page click Install.

This concludes installation of SL Micro 6.1 and SUSE Multi-Linux Manager 5.1 as an extension. For more information about preparing your machines (virtual or physical), see the SL Micro Deployment Guide.

2.1.1.3. OPTIONAL: Registration from the command line

If you added SUSE Multi-Linux Manager 5.1 as an extension during SL Micro 6.1 installation then you can skip this procedure. However, optionally you may skip registration during SL Micro 6.1 installation by selecting the Skip Registration button. This section provides steps on registering your products after SL Micro 6.1 installation.

The following steps register a SUSE Multi-Linux Manager 5.1 extension with the x86-64 architecture and thus require a registration code for the x86-64 architecture. To register ARM or s390x architectures use the correct registration code.

Procedure: Registering from the command line

List available extensions with the following command:
transactional-update --quiet register --list-extensions
From the list of available extensions, select the one you wish to install:
If installing the Server, use your SUSE Multi-Linux Manager Server Extension 5.1 x86_64 registration code with following command:
transactional-update register -p Multi-Linux-Manager-Server/5.1/x86_64 -r <reg_code>
If installing the Proxy, use your SUSE Multi-Linux Manager Proxy Extension 5.1 x86_64 registration code with following command:
transactional-update register -p Multi-Linux-Manager-Proxy/5.1/x86_64 -r <reg_code>
Reboot.

2.1.1.4. Update the system

Procedure: Updating the system

Log in as root.
Run transactional-update:
transactional-update
Reboot.

SL Micro is designed to update itself automatically by default and will reboot after applying updates. However, this behavior is not desirable for the SUSE Multi-Linux Manager environment. To prevent automatic updates on your server, SUSE Multi-Linux Manager disables the transactional-update timer during the bootstrap process.

If you prefer the SL Micro default behavior, enable the timer by running the following command:

systemctl enable --now transactional-update.timer

2.1.2. Prepare SUSE Linux Enterprise Server 15 SP7 host

Alternatively, you can deploy SUSE Multi-Linux Manager on SUSE Linux Enterprise Server 15 SP7.

The following procedures describe the main steps of the installation process.

2.1.2.1. Install SUSE Multi-Linux Manager extensions on SUSE Linux Enterprise Server

Procedure: Installing SUSE Multi-Linux Manager Extensions on SUSE Linux Enterprise Server

Locate and download SUSE Linux Enterprise Server 15 SP7 .iso at https://www.suse.com/download/sles/.

Make sure that you have registration codes both for the host operating system (SUSE Linux Enterprise Server 15 SP7) and extensions

Start the installation of SUSE Linux Enterprise Server 15 SP7.

On the Language, keyboard and product selection select the product to install.

On the License agreement read the agreement and check I Agree to the License Terms.

Select the registration method. For this example, we will register the server with SUSE Customer Center.

Enter your SUSE Customer Center email address.

Enter your registration code for SUSE Linux Enterprise Server 15 SP7.

Click Next to continue.

Please note that for SUSE Linux Enterprise Server 15 SP7, you are required to have a valid SUSE Linux Enterprise Server subscription and corresponding registration code, which you must provide on this screen. You will be required to enter the SUSE Multi-Linux Manager Extension registration code below.

In the screen Extensions and Modules Selection check the following:

Select the SUSE Multi-Linux Manager Server Extension to install the Server, or the SUSE Multi-Linux Manager Proxy Extension to install the Proxy.

Basesystem Module

Containers Module

Click Next to continue.

Enter your SUSE Multi-Linux Manager 5.1 extension registration code.

Click Next to continue.

Complete the installation.

When the installation completes, log in to the newly installed server as root.
Update the System (optional, if the system was not set to download updates during install):
zypper up
Reboot.

2.1.2.2. OPTIONAL: Registration from the command line

If you added SUSE Multi-Linux Manager 5.1 as an extension during SUSE Linux Enterprise Server installation then you can skip this procedure.

However, optionally you may skip registration during SUSE Linux Enterprise Server installation by selecting the Skip Registration button. This section provides steps on registering your products after SUSE Linux Enterprise Server installation.

The following steps register a SUSE Multi-Linux Manager 5.1 extension with the x86-64 architecture and thus require a registration code for the x86-64 architecture.

To register ARM or s390x architectures use the correct registration code.

Procedure: Registering from the command line

List available extensions with the following command:
SUSEConnect --list-extensions
From the list of available extensions, select the one you wish to install:
If installing the Server, use your SUSE Multi-Linux Manager Server Extension 5.1 x86_64 registration code.

For example for SUSE Linux Enterprise 15 SP7, use the following commands:
SUSEConnect -r <regcode>
SUSEConnect -p sle-module-containers/15.7/x86_64
SUSEConnect -p Multi-Linux-Manager-Server-SLE/5.1/x86_64 -r <regcode>
If installing the Proxy, use your SUSE Multi-Linux Manager Proxy Extension 5.1 x86_64 registration code with the following command:
SUSEConnect -p Multi-Linux-Manager-Proxy-SLE/5.1/x86_64 -r <regcode>

2.1.2.3. Install and enable `podman`

Procedure: Installing podman

Log in as root and install the product package.
On the server:
zypper in podman
zypper in -t product Multi-Linux-Manager-Server-SLE
On the proxies:
zypper in podman
zypper in -t product Multi-Linux-Manager-Proxy-SLE
Make sure that package podman is installed. Additionally, on the server mgradm and mgradm-bash-completion or on the proxies, mgrpxy and mgrpxy-bash-completion also need to be installed.
Start the Podman service by rebooting the system, or running a command:
systemctl enable --now podman.service

2.2. SSH connection preparation

This step ensures that the new SUSE Multi-Linux Manager 5.1 Server can connect to the existing 4.3 Server over SSH without requiring a password. It involves generating and configuring SSH keys, setting up an SSH agent, and copying the public key to the old server. This setup is required for the migration process to run without manual intervention.

Procedure: Preparing the SSH connection

Ensure that for root an SSH key exists on the new 5.1 server. If a key does not exist, create it with:
ssh-keygen -t rsa
The SSH configuration and agent should be ready on the new server for a connection to the 4.3 server that does not prompt for a password.
eval $(ssh-agent); ssh-add
To establish a connection that does not prompt for a password, the migration script relies on an SSH agent running on the new server. If the agent is not active yet, initiate it by running eval $(ssh-agent). Then add the SSH key to the running agent with ssh-add followed by the path to the private key. You will be prompted to enter the password for the private key during this process.
Copy the public SSH key to the SUSE Multi-Linux Manager 4.3 Server (<oldserver.fqdn>) with ssh-copy-id. Replace <oldserver.fqdn> with the FQDN of the 4.3 server:
ssh-copy-id <old server.fqdn>
The SSH key will be copied into the old server’s ~/.ssh/authorized_keys file. For more information, see the ssh-copy-id manpage.
Establish an SSH connection from the new server to the old SUSE Multi-Linux Manager Server to check that no password is needed. Also there must not by any problem with the host fingerprint. In case of trouble, remove old fingerprints from the ~/.ssh/known_hosts file. Then try again. The fingerprint will be stored in the local ~/.ssh/known_hosts file.

2.3. Perform the migration

When planning your migration from SUSE Manager 4.3 to SUSE Multi-Linux Manager 5.1, ensure that your target instance meets or exceeds the specifications of the old setup.

This includes, but is not limited to, memory (RAM), CPU Cores, Storage, and Network Bandwidth.

SUSE Multi-Linux Manager server hosts that are hardened for security may restrict execution of files from the /tmp folder. In such cases, as a workaround, export the TMPDIR environment variable to another existing path before running mgradm.

For example:

export TMPDIR=/path/to/other/tmp

In SUSE Multi-Linux Manager updates, tools will be changed to make this workaround unnecessary.

When migrating from SUSE Manager 4.3, you will be prompted for the Password for the CA key to generate. It is essential to enter the same CA password that was used in your SUSE Manager 4.3 installation.

Entering the wrong password will result in a failure to generate the database certificate and will cause the migration to abort with the following error:

Error: cannot configure db container: Cannot generate database certificate: CA validation failed!

Make sure the correct CA password is available before starting the migration process.

Procedure: Performing the migration

This step is optional. If custom persistent storage is required for your infrastructure, use the mgr-storage-server tool. For more information about mgr-storage-server, see installation-and-upgrade:hardware-requirements.adoc#install-hardware-requirements-storage.
Execute the following command to migrate and set up a new SUSE Multi-Linux Manager 5.1 server. Replace <oldserver.fqdn> with the FQDN of the 4.3 server:

Make sure to upgrade your 4.3 server and apply all available updates before starting the migration process. Additionally, remove any unnecessary channels to help reduce the overall migration time.

The migration can take a very long time depending on the amount of data that needs to be replicated. To reduce downtime it is possible to run the migration multiple times in a process of initial replication, re-replication, or final replication and switch over while all the services on the old 4.3 server can stay up and running.

Only during the final migration, the processes on the old 4.3 server need to be stopped.

For all non-final replications add the parameter --prepare to prevent the automatic stopping of services on the old 4.3 server.

Re-running a failed migration creates multiple large backup directories, which accumulate and consume significant space, since each unsuccessful migration creates a new backup directory inside /var/lib/containers/storage/volumes.

These backups are not automatically removed by the migration tool. If a migration fails, you must manually inspect and clean up old backup directories to ensure sufficient space is available for the next attempt.

Failure to manage this space may result in a failure of the next migration attempt.

After the migration and validation of everything, var-pgsql-backup volume can be removed.

The --prepare step can be run multiple times only as long as a final migration has not been started.

Once the final migration is initiated and reaches the database upgrade stage, you must not run the --prepare command again, as it can lead to data corruption.
mgradm migrate podman <oldserver.fqdn> --prepare

Procedure: Final migration

Stop the SUSE Manager services on 4.3 Server:
spacewalk-service stop
Stop the PostgreSQL service on 4.3 Server:
systemctl stop postgresql
Perform the final migration on SUSE Multi-Linux Manager 5.1 Server
mgradm migrate podman <oldserver.fqdn>
Migrate trusted SSL CA certificates.

2.3.1. Migration of the certificates

Trusted SSL CA certificates that were installed as part of an RPM and stored on SUSE Multi-Linux Manager 4.3 in the /usr/share/pki/trust/anchors/ directory will not be migrated. Because SUSE does not install RPM packages in the container, the administrator must migrate these certificate files manually from the SUSE Manager 4.3 server after the migration.

Procedure: Migrating the certificates

Copy the file from the SUSE Manager 4.3 Server to the new SUSE Multi-Linux Manager 5.1 Server. For example, as /local/ca.file.
Copy the file into the container with:
mgrctl cp /local/ca.file server:/etc/pki/trust/anchors/

After successfully running the mgradm migrate command, the Salt setup on all clients will still point to the old 4.3 server.

To redirect them to the 5.1 server, it is required to rename the new server at the infrastructure level (DHCP and DNS) to use the same FQDN and IP address as 4.3 server.

Adjusting the IP address can be avoided if the latest version of the minion is installed on the clients, as the newer version can automatically re-connect with the server using only the FQDN.

3. Client tools rebranding

SUSE Multi-Linux Manager 5.1 introduces a rebranded set of client tools for all supported operating systems. This transition is seamless, and users performing a new product synchronization should only notice the updated channel names.

Channels named SUSE Manager Client Tools for XYZ, used by clients previously registered with SUSE Multi-Linux Manager 4.3 or 5.0, are no longer available in version 5.1 and will no longer receive updates in 5.1.

Although the legacy channels remain assigned to existing clients after migration, the corresponding repositories have been removed.

To ensure continued updates, users must:

Mirror the new SUSE Multi-Linux Manager Client Tools for XYZ channels for the relevant products and assign them to the appropriate clients.
Unassign the outdated SUSE Manager Client Tools for XYZ channels.

This also means that any CLM projects based on the old client tools must be adjusted accordingly.

For example workflow, see Switch to new client tools channels.