Install and Configure a SUSE RMT Server

1 What is an RMT server? #

This article describes how to run an RMT server on SUSE Linux Enterprise Server 16.1. Install RMT directly during the installation of SUSE Linux Enterprise Server or install it on a running system. After the packages are installed, use Ansible to do an initial configuration.

Warning: RMT server conflicts with installation server

Configuring a host to be an RMT server installs and configures the NGINX Web server, listening on port 80. However, configuring a host to be an installation server automatically installs the Apache Web server, which also listens on port 80. If both use the same IP address, the service that starts second cannot bind the port and does not start.

Do not try to enable both these functions on the same server. It is not possible for a single server to host both simultaneously.

2 RMT server storage and disk space requirements #

The amount of storage your RMT server requires depends on several variables: the number of repositories and architectures that you mirror and the number of products that are enabled. As a general guide, 1.5 times the total size of all enabled repositories is sufficient. The total is about 200 GB per SUSE Linux Enterprise release, including all extensions.

Downloaded packages are stored in /var/lib/rmt/public/repo/. Plan the size of the file system that contains /var/lib/rmt/ accordingly.

The rmt-server package also installs the symbolic link /usr/share/rmt/public/repo/, which points to /var/lib/rmt/public/repo/. Because the link is part of the package and RMT writes only to /var/lib/rmt/public/repo/, mirroring also works on systems where /usr is read-only, such as immutable systems.

Important: RMT mirroring fails when /tmp runs out of space

When RMT mirrors repositories from SUSE Customer Center, it writes large temporary files in /tmp. If /tmp is too small, or is part of the /root file system, and this file system is small in size, /tmp quickly runs out of space, resulting in a mirroring failure. For more information, see https://support.scc.suse.com/s/kb/RMT-mirroring-fails-when-tmp-runs-out-of-disk-space/.

3 Installing RMT during system installation #

Install RMT while installing SUSE Linux Enterprise Server with Agama by adding the rmt-server package to the installation profile.

RMT is not available as a software pattern, and the Agama Web interface only lets you select patterns. To install RMT during installation, add the rmt-server package to the software section of an Agama installation profile:

"software": {
  "packages": [
    "rmt-server"
  ]
}

The rmt-server package is available from the SUSE Linux Enterprise Server product repository, so no additional repository is required. If you install with the Agama Web interface instead, install RMT after the installation as described in Section 4, “Installing RMT on a running SLES system”.

Immediately after the SLES installation, run zypper patch to get the latest RMT updates. SUSE continuously releases maintenance updates for RMT.

4 Installing RMT on a running SLES system #

Install RMT on an active SUSE Linux Enterprise Server using the zypper package manager.

For RMT installation, run the following zypper command:

> sudo zypper install rmt-server

On an immutable system, install RMT with the transactional-update command and reboot the system to switch to the new snapshot:

> sudo transactional-update pkg install rmt-server
> sudo reboot

5 Upgrading an RMT server from SUSE Linux Enterprise 15 to SLES 16.1 #

After you upgrade an existing RMT server from SUSE Linux Enterprise 15 SP7 to SUSE Linux Enterprise Server 16.1, repair two platform changes. Until you do, RMT remains unavailable.

Repair the following two platform changes:

Network stack replacement: SLES 16.1 removes the legacy wicked networking framework and standardizes on NetworkManager as the sole network control stack.
Mandatory access control framework: SLES 16.1 replaces AppArmor with SELinux in enforcing mode by default.

Upgrade the host itself using the distribution migration system, as described in Upgrading to SUSE Linux Enterprise Server 16.1. This section covers only the steps specific to a host running RMT.

Important: A successful upgrade can still leave RMT unavailable

The distribution migration system reports success for the distribution upgrade, not for the services running on the host. Even when every suse-migration-* unit finishes with Result=success and /var/log/distro_migration.exitcode contains 0, RMT does not start until you complete the steps below.

5.1 Preparing for the upgrade #

Back up the RMT configuration file and the RMT database to a location outside the host:
```
# cp /etc/rmt.conf /root/rmt.conf.backup
# mysqldump --single-transaction rmt > /root/rmt.sql
```
Record the ownership of the MariaDB data directory. On SUSE Linux Enterprise 15, as it belongs to the group root, which breaks MariaDB after the upgrade:
```
# ls -ld /var/lib/mysql
drwx------ 1 mysql root 442 /var/lib/mysql
```

Note: Registered clients follow the RMT server address

The upgrade replaces wicked with NetworkManager, which changes the DHCP client identifier. Consequently, a host using DHCP is likely to receive a new IP address. Because clients registered against this RMT server reach it through the URL stored in their Zypper service definitions, assign the server a stable IP address or host name before upgrading.

5.2 Completing the upgrade #

Check which services are running. RMT depends on MariaDB through Requires=mariadb.service, so a failed database keeps RMT inactive:
```
# systemctl --failed
# systemctl is-active mariadb nginx rmt-server
```
Correct the group ownership of the MariaDB data directory:
```
# chgrp -R mysql /var/lib/mysql
```
On SUSE Linux Enterprise Server 16.1, mariadb.service combines StateDirectory=mysql with User=mysql and Group=mysql. systemd therefore adjusts the ownership of /var/lib/mysql on every start, and the mysql:root ownership inherited from SUSE Linux Enterprise 15 makes that fail:
```
mariadb.service: Failed to set up special execution directory in /var/lib: Permission denied
mariadb.service: Control process exited, code=exited, status=238/STATE_DIRECTORY
```
The error is reported as a permission problem rather than an ownership one, which makes it easy to mistake for a file system or SELinux fault.
Start MariaDB:
```
# systemctl start mariadb
```
Allow nginx to connect to the RMT back-end:
```
# setsebool -P httpd_can_network_connect on
```
SUSE Linux Enterprise Server 16.1 uses SELinux in enforcing mode instead of AppArmor. The httpd_can_network_connect boolean is off by default, which prevents nginx from opening a connection to the RMT back-end on 127.0.0.1:4224. Without this setting, nginx answers every request that it proxies to RMT with 502.
Verify that RMT answers correctly:
```
# curl -o /dev/null -w '%{http_code}\n' http://localhost/connect/subscriptions/products
401
# curl -o /dev/null -w '%{http_code}\n' http://localhost/repo/
200
```
401 is the expected answer on the registration endpoint: it shows that the RMT application is reachable and requires authentication. A 502 means that the previous step is incomplete.
Restart the host and repeat the checks. Confirming the changes across a reboot is essential: The SELinux boolean only persists when it is set with -P, and MariaDB may not be independently enabled because rmt-server.service manages it via Requires=.

Note: RMT 2 is upgraded to RMT 3

SUSE Linux Enterprise 15 SP7 offers both RMT 2 and RMT 3, while SUSE Linux Enterprise Server 16.1 ships only RMT 3. A host still running RMT 2 therefore moves to RMT 3 as part of the distribution upgrade.

No action is required. /etc/rmt.conf and the contents of the database are preserved, and rmt-server-migration.service updates the database schema before rmt-server.service starts. RMT 3 differs from RMT 2 in the Ruby and Rails versions it runs on, not in its configuration file or its rmt-cli commands.

Confirm the version after the upgrade:

# rmt-cli version

Warning: Mirrored repositories keep working while registration is broken

nginx serves /repo/ directly from disk, so clients continue to download packages even when the RMT application is unreachable and no client can register. Check the registration endpoint, not just the repository paths.

6 Deploying RMT on top of the Kubernetes cluster #

This section describes how to deploy RMT on a Kubernetes cluster. It uses Helm as the package manager to interact with the Kubernetes cluster. For more information about using Helm, see https://helm.sh/docs/intro/using_helm/.

6.1 Prerequisites #

Running Kubernetes cluster
helm command configured to interact with the cluster
A storage class for the persistent volumes of RMT and MariaDB. The example in Section 6.3, “The values.yaml file” uses the local-path storage class, which is only available in Rancher workloads. On other clusters, install the local-path storage provisioner:
```
> kubectl apply -f \
https://raw.githubusercontent.com/rancher/local-path-provisioner/\
v0.0.26/deploy/local-path-storage.yaml
```
Alternatively, set app.storage.class and db.storage.class to a storage class that already exists in your cluster.

6.2 Application components #

Each component of the RMT application is deployed in its own container. RMT consists of the following components:

RMT server: Containerized version of the RMT application server with the ability to pass its configuration via Helm values. Storage is done on a volume that is allocated to the Kubernetes cluster. Adjust the size of the storage depending on the number of repositories you need to mirror.
MariaDB: The database back-end for RMT. No post-installation task is needed since RMT creates the required database and tables at start-up. If passwords are not specified in the values.yaml file, they are generated automatically.
nginx: A Web server configured for RMT routes. Having a properly configured Web server allows you to target your Ingress traffic (for RMT) to this nginx service directly. You do not need to configure Ingress for RMT-specific path handling, as nginx handles path routing automatically.

6.3 The `values.yaml` file #

The RMT chart includes the values.yaml file where all parameters are documented and their default values are defined. You can override these values by providing your own values file, for example:

> cat << EOF > rmt-config.yaml
---
app:
  storage:
    class: local-path1
  scc:
    enabled: false
    username: "UXXXXXXX"
    password: "PASSXXXX"
    products_enable:
      - SLES/15.3/x86_64
      - sle-module-python2/15.3/x86_64
    products_disable:
      - sle-module-legacy/15.3/x86_64
      - sle-module-cap-tools/15.3/x86_64
db:
  storage:
    class: local-path2
ingress:
  enabled: true
  hosts:
    - host: chart-example.local
      paths:
        - path: "/"
          pathType: Prefix
  tls:
    - secretName: rmt-cert
      hosts:
        - chart-example.local
EOF

1 2	The storage class for the RMT and MariaDB persistent volumes. The storage class must exist in the cluster. For more information, see Section 6.1, “Prerequisites”.

And to install RMT, run:

> helm install rmtsle oci://registry.suse.com/suse/rmt-helm -f rmt-config.yaml

Required values #

Key: app.scc.password

Type: string

Default: nil

Description: SUSE Customer Center proxy password. The password string needs to be put inside quotes. If the quote character " is part of the string, it has to be escaped with \.

Key: app.scc.username

Type: string

Default: nil

Description: SUSE Customer Center proxy user name. The user name string needs to be put inside quotes. If the quote character " is part of the string, it has to be escaped with \.

Key: app.scc.products_enable

Type: list

Default: []

Description: list of products to enable for mirroring

Key: app.scc.products_disable

Type: list

Default: []

Description: list of products to disable for mirroring

Key: app.storage.class

Type: string

Default: ""

Description: Kubernetes storageclass.

Key: db.storage.class

Type: string

Default: ""

Description: Kubernetes storageclass.

Key: ingress.enabled

Type: bool

Default: false

Description: Ingress Enabled

Key: ingress.hosts[0]

Type: object

Default: {"host":"chart-example.local","paths":[{"path":"/","pathType":"Prefix"}]}

Description: DNS name at which the RMT service will be accessible from clients

Key: ingress.tls[0].hosts[0]

Type: string

Default: "chart-example.local"

Description: DNS name at which the RMT service will be accessible from clients

Key: ingress.tls[0].secretName

Type: string

Default: "rmt-cert"

Description: TLS Ingress Certificate

7 Configuring RMT with Ansible #

Configure RMT with Ansible as described in the following procedure. The ansible-rmt-server package provides pre-built playbooks and roles for automated RMT configuration.

Install the ansible-rmt-server package:
- On a mutable system:
```
# zypper in ansible-rmt-server
```
- On an immutable system, also install the packages that the playbook requires, then reboot the system to switch to the new snapshot. If you skip these packages, the playbook installs them into a new snapshot that is not active until the next reboot, and the services that the playbook starts are missing.
```
# transactional-update pkg install ansible-rmt-server mariadb nginx python3-PyMySQL
# reboot
```
Navigate to the Ansible configuration directory:
```
# cd /usr/share/ansible/rmt
```
Create the configuration file based on the provided example:
- On a mutable system, create group_vars/all.yml:
```
# cp group_vars/all.yml.example group_vars/all.yml
```
- On an immutable system, /usr is read-only. Create the configuration file in a writable location, for example, /root/rmt-vars.yml, and make it readable only by root, because it contains credentials:
```
# install -m 600 group_vars/all.yml.example /root/rmt-vars.yml
```
Edit the configuration file and configure the following variables:
rmt_db_username
Database user name for RMT (default: rmt)
rmt_db_password
Secure password for the RMT database user
RMT_ORGANIZATION_USERNAME
Your SUSE Customer Center organization credentials. To retrieve your credentials, log in to Mirroring credentials.
RMT_ORGANIZATION_PASSWORD
Your SUSE Customer Center organization password
Example configuration:
```
---
rmt_db_username: rmt
rmt_db_password: "your_secure_password"

rmt:
  RMT_ORGANIZATION_USERNAME: "your_scc_username"
  RMT_ORGANIZATION_PASSWORD: "your_scc_password"
```
Run the Ansible playbook to configure RMT:
On an immutable system, pass the configuration file that you created in /root as an extra variable:
```
# ansible-playbook site.yml -e @/root/rmt-vars.yml
```
The playbook performs the following tasks:
- Install required dependencies (MariaDB, nginx, Python modules)
- Configure and start the MariaDB database service
- Create the RMT database and database user
- Generate SSL certificates for HTTPS
- Configure nginx virtual hosts
- Create /etc/rmt.conf configuration file
- Run database migrations
- Start and enable the RMT server service
On systems with SELinux in enforcing mode, allow nginx to connect to the RMT back-end service:
```
# setsebool -P httpd_can_network_connect 1
```

Verify the installation by checking the API health status:

# curl -s http://localhost/api/health/status
{"state":"online"}

Tip: SSL certificate locations

The Ansible playbook generates SSL certificates at the following locations:

CA certificate: /etc/rmt/ssl/rmt-ca.crt
CA private key: /etc/rmt/ssl/rmt-ca.key
Server certificate: /etc/rmt/ssl/rmt-server.crt
Server private key: /etc/rmt/ssl/rmt-server.key

8 Enabling SLP announcements #

Enable SLP (Service Location Protocol) announcements so that clients can discover the RMT service on the network.

RMT includes the SLP service description file /etc/slp.reg.d/rmt-server.reg. To enable SLP announcements of the RMT service, follow these steps:

If firewalld is running:

Open relevant ports:

> sudo firewall-cmd --permanent --add-port=427/tcp
    success

      > sudo firewall-cmd --permanent --add-port=427/udp
      success

Reload the firewalld configuration:
```
> sudo firewall-cmd --reload
```

Verify that the SLP server is installed and install it if it is not:
```
> sudo zypper install openslp-server
```

Enable and start the SLP service:

> sudo  systemctl enable --now slpd.service

9 Accessing SUSE Customer Center from behind a firewall #

Allow the RMT server to reach SUSE Customer Center when it is located behind a firewall that does not support host name allowlisting.

If the RMT server is behind a firewall and cannot access SUSE Customer Center directly, you need to allow connections to the following domains on both ports 80 and 443 on the firewall.

scc.suse.com
updates.suse.com
installer-updates.suse.com

If the firewall you are using does not support allowing connections from host names—such as firewalld—allow connections to the corresponding IP addresses below.

Resolve the IP address of the domain, for example, installer-updates.suse.com.
```
> nslookup installer-updates.suse.com
[...]
Address: 152.199.22.115
```

Add firewall rules for ports 80 and 443 for each discovered IP address.

Port 80:

> sudo firewall-cmd --permanent --zone=public \
--add-rich-rule='rule family="ipv4" source address="152.199.22.115" \
 port protocol="tcp" port="80" accept'

Port 443:

> sudo firewall-cmd --permanent --zone=public \      --add-rich-rule='rule family="ipv4" source address="152.199.22.115" \
port protocol="tcp" port="443" accept'

Reload the firewall configuration.
```
> sudo firewall-cmd --reload
```

10 Main configuration file `/etc/rmt.conf` #

This section documents the configuration options available in the main RMT configuration file, /etc/rmt.conf.

The Ansible playbooks provided in the ansible-rmt-server package perform the initial configuration. You only need to specify the proxy configuration manually. The remaining configuration parameters are documented below for reference.

10.1 Mirroring settings #

The mirroring section lets you adjust mirroring behavior.

mirror_src: Decides whether to mirror source RPM packages (architecture is src).
dedup_method: Creates hard links during mirroring when set to hardlink. If the file system does not support hard links, it can be set to copy instead. Possible values: hardlink, copy.

10.2 HTTP client settings #

The http_client section defines the global HTTP connection settings of RMT.

verbose: Enables additional debug output to the systemd journal.
proxy: The proxy server URL including the protocol and the port number. For example: http://proxy_url:8080.
noproxy: A list of domains that should not go through the proxy, separated by commas. For example: localhost.mylocaldomain.
proxy_auth: This setting determines the proxy authentication mechanism. Possible values are: none, basic, digest, gssnegotiate, ntlm, digest_ie, ntlm_wb.
proxy_user: The proxy server user name.
proxy_password: The proxy server password.
low_speed_limit: Lower speed limit when a download is aborted in bytes per second.
low_speed_time: Time until a download is aborted, when download speed is below low_speed_limit.

10.3 Settings for accessing SUSE repositories #

The scc section contains your mirroring credentials for contacting SUSE Customer Center. To obtain your mirroring credentials, see Mirroring credentials.

Valid configuration keys for the section are:

username: Mirroring credentials user name.
password: Mirroring credentials password.

10.4 Web server settings #

The web_server section lets you tune the performance of your RMT server.

min_threads

Specifies the minimum number of threads that an RMT server worker should spawn.

Acceptable values: integer greater than or equal to 1.

max_threads

Specifies the maximum number of threads that an RMT server worker should spawn.

Acceptable values: integer greater than or equal to 1.

workers

Specifies the number of Web workers for RMT.

Acceptable values: integer greater than or equal to 1.

11 Managing TLS/SSL certificates #

Regenerate the TLS/SSL certificates of your RMT server using the Ansible playbook when they expire.

11.1 SSL certificates and HTTPS #

By default access to API endpoints consumed by SUSEConnect is limited to HTTPS only. nginx is configured to use an SSL certificate and a private key from the following locations:

Certificate: /etc/rmt/ssl/rmt-server.crt
Private key: /etc/rmt/ssl/rmt-server.key

The Ansible playbook generates a custom certificate authority which is used to sign HTTPS certificates. To register, client machines must trust the certificate authority:

For registrations during installation from the media or with YaST Registration module, a message prompts you to trust the server certificate.
For registering a client system on the command line, use the rmt-client-setup script.

11.2 Regenerating HTTPS certificates #

HTTPS certificates must be regenerated before they expire. No additional actions are required on the client machines registered to the RMT server if only HTTPS certificates are regenerated.

Note: Certificates generated by the playbook are self-signed

The playbook generates a self-signed server certificate that carries neither a common name nor subject alternative names, and that is not signed by the CA certificate in /etc/rmt/ssl/rmt-ca.crt. Clients that verify host names reject such a certificate. If your clients verify the server certificate, issue one yourself with a common name, the required subject alternative names and a signature from your CA, then store it as /etc/rmt/ssl/rmt-server.crt together with its key in /etc/rmt/ssl/rmt-server.key.

Warning: Overwriting existing certificates

Any subsequent run of the playbook overwrites the existing server certificates.

Stop nginx and rmt-server services.

# systemctl stop nginx
# systemctl stop rmt-server

Remove previously generated certificates.
```
# rm /etc/rmt/ssl/rmt-server.*
```

Regenerate certificates using the Ansible playbook:

# cd /usr/share/ansible/rmt
# ansible-playbook site.yml

11.3 Regenerating CA certificates and HTTPS certificates #

CA certificates can be regenerated after they have expired or in case of security issues.

Warning: Import CA certificate on all clients

The newly generated CA certificate must be imported on all clients registered to the RMT server. If the certificate is not imported, clients cannot establish a secure connection to the RMT server, and all subsequent package updates and registration requests will fail with SSL/TLS verification errors.

To import the certificate, run the rmt-client-setup script on the client machines.

Stop nginx service. rmt-server services.
```
# systemctl stop nginx
```
Stop rmt-server service.
```
# systemctl stop rmt-server
```

Remove previously generated CA and HTTPS certificates.

# rm /etc/rmt/ssl/rmt-ca.*
  rm /etc/rmt/ssl/rmt-server.*

Regenerate certificates using the Ansible playbook:

# cd /usr/share/ansible/rmt
# ansible-playbook site.yml

12 For more information #

To learn more about RMT, see the following resource:

RMT source code and issue tracker on GitHub.

13 Legal Notice #

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”.

For SUSE trademarks, see https://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.

Installing and Configuring an RMT Server

1 What is an RMT server? #

2 RMT server storage and disk space requirements #

3 Installing RMT during system installation #

4 Installing RMT on a running SLES system #

5 Upgrading an RMT server from SUSE Linux Enterprise 15 to SLES 16.1 #

5.1 Preparing for the upgrade #

5.2 Completing the upgrade #

6 Deploying RMT on top of the Kubernetes cluster #

6.1 Prerequisites #

6.2 Application components #

6.3 The values.yaml file #

7 Configuring RMT with Ansible #

8 Enabling SLP announcements #

9 Accessing SUSE Customer Center from behind a firewall #

10 Main configuration file /etc/rmt.conf #

10.1 Mirroring settings #

10.2 HTTP client settings #

10.3 Settings for accessing SUSE repositories #

10.4 Web server settings #

11 Managing TLS/SSL certificates #

11.1 SSL certificates and HTTPS #

11.2 Regenerating HTTPS certificates #

11.3 Regenerating CA certificates and HTTPS certificates #

12 For more information #

13 Legal Notice #

6.3 The `values.yaml` file #

10 Main configuration file `/etc/rmt.conf` #