Migrate SSL Certificate Authorities

Use this procedure to replace the root certificate authority (CA) of a SUSE Multi-Linux Manager server, for example to replace an older self-signed CA that does not mark the X509v3 Basic Constraints extension as critical. For more information, see administration:ssl-certs-imported.adoc#ssl-certs-verify-ca-basic-constraints.

On Podman installations, the mgradm ssl commands help perform the rotation. They generate or import the new CA, update the server and database secrets, and restart the services.

The migration is done in two phases:

With mgradm ssl add-ca, you add the new CA to the trusted bundle so that both the old and new CA are trusted, and distribute it to all clients and proxies.
After all clients and proxies trust the new CA, mgradm ssl rotate switches the server to a certificate signed by the new CA and drops the old one.

Also consider any machine that is not managed by the server but still accesses it through the API or a browser, as it needs the new CA too. These machines are not detected by the trust check in Confirm the Clients Trust the New CA, so track and verify them yourself.

Do not switch the server certificate before all clients and proxies trust the new CA. A client that does not trust the new CA can no longer connect, and you have to deploy the CA to it manually.

The certificates are stored in Podman secrets, which the container reads at startup. The server CA bundle, certificate, and key are in uyuni-ca, uyuni-cert, and uyuni-key; the database equivalents are uyuni-db-ca, uyuni-db-cert, and uyuni-db-key. On Kubernetes, the root CAs are read from the uyuni-ca and db-ca config maps, while the certificates are read from the uyuni-cert and db-cert secrets. The mgradm ssl commands manage these secrets on Podman; on Kubernetes, you update the config maps and secrets manually as described below.

1. Add the New CA to the Trust Bundle

On Podman, add the new CA with mgradm ssl add-ca. The command generates a new self-signed root CA, or imports a third-party one, appends it to the uyuni-ca and uyuni-db-ca secrets without changing the served certificate, restarts the services, and prints the SHA-256 fingerprint of the new CA.

To generate a new self-signed CA, run the following command on the container host. Replace the subject values with the ones for your organization. The server’s fully qualified domain name is detected automatically, or you can pass it as an argument:

mgradm ssl add-ca --ssl-password CA_PASSWORD \
  --ssl-country COUNTRY --ssl-state STATE --ssl-city CITY \
  --ssl-org ORGANIZATION --ssl-ou "ORGANIZATION UNIT"

If --ssl-password is omitted, the command prompts for it. To use a third-party root CA instead, pass its certificate and any intermediate certificates in place of the generation options. Repeat --ssl-ca-intermediate for each intermediate CA:

mgradm ssl add-ca --ssl-ca-root /root/new-ca.pem \
  --ssl-ca-intermediate /root/intermediate-ca.pem

On Kubernetes, append the new root CA certificate to the uyuni-ca and db-ca config maps manually:

kubectl get configmap -n uyuni-server uyuni-ca -o jsonpath='{.data.ca\.crt}' > combined.crt && \
echo "" >> combined.crt && \
cat ca.crt >> combined.crt && \
kubectl create configmap uyuni-ca --from-file=ca.crt=combined.crt --dry-run=client -o json | \
kubectl patch -n uyuni-server configmap uyuni-ca --patch-file /dev/stdin  -o yaml && \
rm combined.crt

You also need to run a similar command for db-ca.

Once the server trusts the new CA, distribute it to every client and proxy. Deploy the new CA to all Salt clients by applying the highstate, as described in administration:ssl-certs-imported.adoc#ssl-certs-import-deploy-root-ca.

On each SUSE Multi-Linux Manager Proxy, make it trust the new CA without a full redeployment. The proxy reads its trusted CA from the uyuni-ca Podman secret, so update that secret directly. After mgradm ssl add-ca, the server publishes the combined CA bundle, which contains both the old and new root CAs, at http://SERVER_FQDN/pub/RHN-ORG-TRUSTED-SSL-CERT. Download it on the proxy host, replace the secret, and restart the proxy:

curl -o /tmp/proxy-ca.pem http://SERVER_FQDN/pub/RHN-ORG-TRUSTED-SSL-CERT
podman secret create --replace uyuni-ca /tmp/proxy-ca.pem
systemctl restart uyuni-proxy-pod

Also make sure the new CA is deployed to any other machine that is not a reachable Salt minion but still accesses the server, such as proxies running on Kubernetes or unmanaged systems.

2. Confirm the Clients Trust the New CA

Before switching, confirm that the clients trust the new CA. On Podman, run mgradm ssl rotate --check-only to query the Salt minions and report which ones already trust the new CA and which do not, without performing the rotation:

mgradm ssl rotate --check-only

This check is best-effort: it only covers Salt minions that respond at the time of the check. Offline minions and non-minion systems such as proxies running on Kubernetes or unmanaged machines are not covered and must be verified manually.

To verify one of those manually, compare fingerprints. The mgradm ssl add-ca command already reported the SHA-256 fingerprint of the new CA. You can also recompute it from the new CA certificate. For a self-signed CA, read the staged certificate from the server:

mgrctl exec -- openssl x509 -in /root/ssl-build-rotation/RHN-ORG-TRUSTED-SSL-CERT \
  -noout -fingerprint -sha256

For a third-party CA, run the same command against the root CA file you passed to mgradm ssl add-ca.

The client trust file /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT contains several certificates. List the fingerprints of all of them on a client and confirm that the new CA fingerprint is present:

while openssl x509 -noout -fingerprint -sha256; do :; done < /etc/pki/trust/anchors/RHN-ORG-TRUSTED-SSL-CERT

3. Switch to the New CA

Only continue after all clients and proxies trust the new CA.

On Podman, switch to the new CA with mgradm ssl rotate. The command promotes the CA staged by mgradm ssl add-ca, issues a new server certificate signed by it, replaces the server and database certificates, keys, and CA secrets, and restarts the services. Before switching, it checks over Salt that the clients trust the new CA and refuses to continue if some do not, unless you pass --force. Like mgradm ssl rotate --check-only, this check only covers Salt minions that respond, so an offline minion that does not report does not block the rotation; verify such minions manually beforehand.

For a self-signed CA, run the following command. Use the fully qualified domain name of your SUSE Multi-Linux Manager server, or allow it to be detected automatically, and repeat --ssl-cname for additional aliases:

mgradm ssl rotate --ssl-password CA_PASSWORD

To install a third-party server certificate signed by the new CA, provide the certificate, key, and root CA instead. The database reuses the server values unless you pass the matching --ssl-db-* options:

mgradm ssl rotate --ssl-ca-root /root/new-ca.pem \
  --ssl-ca-intermediate /root/intermediate-ca.pem \
  --ssl-server-cert /root/server.crt \
  --ssl-server-key /root/server.key

The mgradm ssl add-ca and mgradm ssl rotate commands are only available on Podman. On Kubernetes, replace the uyuni-cert and db-cert secrets with the new server certificate and key, update the uyuni-ca and db-ca config maps to contain only the new CA, then restart the server.

3.1. Rotate a Compromised CA Immediately

If the current CA is compromised, you cannot wait for the overlap period in which both the old and new CA are trusted. Pass --emergency to mgradm ssl rotate to generate a new CA together with a matching server certificate and drop the old CA in a single step, without the two-phase overlap:

mgradm ssl rotate --emergency --ssl-password CA_PASSWORD

An emergency rotation drops the old CA immediately, so any client or proxy that does not yet trust the new CA can no longer connect until you deploy the new CA to it manually. After an emergency rotation, distribute the new CA to all clients and proxies as described in Add the New CA to the Trust Bundle, and update the proxies as described below.

4. Update Proxies

In Add the New CA to the Trust Bundle, you already made each proxy trust the new CA by updating its uyuni-ca Podman secret. Now that the server uses a certificate signed by the new CA, every proxy also needs its own certificate signed by it. For each proxy, regenerate the proxy configuration and redeploy the proxy:

Proxy Deployment.