29 Image preparation for connected environments #

Edge Image Builder is used to create the image for the management cluster, in this document we cover the minimal configuration necessary to set up the management cluster.

Edge Image Builder runs inside a container, so a container runtime is required such as Podman or Rancher Desktop. For this guide, we assume podman is available.

Also, as a prerequisite to deploy a highly available management cluster, you need to reserve three IPs in your network:

apiVIP for the API VIP Address (used to access the Kubernetes API server).
ingressVIP for the Ingress VIP Address (consumed, for example, by the Rancher UI).
metal3VIP for the Metal3 VIP Address.

29.1 Directory structure #

When running EIB, a directory is mounted from the host, so the first thing to do is to create a directory structure to be used by EIB to store the configuration files and the image itself. This directory has the following structure:

eib
├── mgmt-cluster.yaml
├── network
│ └── mgmt-cluster-node1.yaml
├── os-files
│ └── var
│   └── lib
│     └── rancher
│       └── rke2
│         └── server
│           └── manifests
│             └── rke2-ingress-config.yaml
├── kubernetes
│ ├── manifests
│ │ ├── neuvector-namespace.yaml
│ │ ├── ingress-l2-adv.yaml
│ │ └── ingress-ippool.yaml
│ ├── helm
│ │ └── values
│ │     ├── rancher.yaml
│ │     ├── neuvector.yaml
│ │     ├── longhorn.yaml
│ │     ├── metal3.yaml
│ │     └── certmanager.yaml
│ └── config
│     └── server.yaml
├── custom
│ ├── scripts
│ │ ├── 99-register.sh
│ │ ├── 99-mgmt-setup.sh
│ │ └── 99-alias.sh
│ └── files
│     ├── rancher.sh
│     ├── mgmt-stack-setup.service
│     ├── metal3.sh
│     └── basic-setup.sh
└── base-images

Note

The image SL-Micro.x86_64-6.2-Base-SelfInstall-GM.install.iso must be downloaded from the SUSE Customer Center or the SUSE Download page, and it must be located under the base-images folder.

You should check the SHA256 checksum of the image to ensure it has not been tampered with. The checksum can be found in the same location where the image was downloaded.

An example of the directory structure can be found in the SUSE Telco Cloud GitHub repository under the "telco-examples" folder.

29.2 Management cluster definition file #

The mgmt-cluster.yaml file is the main definition file for the management cluster. It contains the following information:

apiVersion: 1.3
image:
  imageType: iso
  arch: x86_64
  baseImage: SL-Micro.x86_64-6.2-Base-SelfInstall-GM.install.iso
  outputImageName: eib-mgmt-cluster-image.iso
operatingSystem:
  isoConfiguration:
    installDevice: /dev/sda
  users:
  - username: root
    encryptedPassword: $ROOT_PASSWORD
  packages:
    packageList:
    - jq
    - open-iscsi
    sccRegistrationCode: $SCC_REGISTRATION_CODE
kubernetes:
  version: v1.35.3+rke2r3
  helm:
    charts:
      - name: cert-manager
        repositoryName: jetstack
        version: 1.20.1
        targetNamespace: cert-manager
        valuesFile: certmanager.yaml
        createNamespace: true
        installationNamespace: kube-system
      - name: suse-storage
        releaseName: longhorn
        version: 1.11.1
        repositoryName: rancher-application-collection
        targetNamespace: longhorn-system
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: longhorn.yaml
      - name: metallb
        version: 306.0.2+up0.15.3
        targetNamespace: metallb-system
        createNamespace: true
        repositoryName: suse-edge-charts
        installationNamespace: kube-system
      - name: metal3
        version: 306.0.26+up0.15.0
        repositoryName: suse-edge-charts
        targetNamespace: metal3-system
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: metal3.yaml
      - name: rancher-turtles-providers
        version: 306.0.6+up0.26.1
        repositoryName: suse-edge-charts
        targetNamespace: cattle-turtles-system
        createNamespace: true
        installationNamespace: kube-system
      - name: neuvector-crd
        version: 109.0.1+up2.8.13
        repositoryName: rancher-charts
        targetNamespace: neuvector
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: neuvector.yaml
      - name: neuvector
        version: 109.0.1+up2.8.13
        repositoryName: rancher-charts
        targetNamespace: neuvector
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: neuvector.yaml
      - name: rancher
        version: 2.14.1
        repositoryName: rancher-prime
        targetNamespace: cattle-system
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: rancher.yaml
    repositories:
      - name: jetstack
        url: https://charts.jetstack.io
      - name: rancher-charts
        url: https://charts.rancher.io/
      - name: suse-edge-charts
        url: oci://registry.suse.com/edge/charts
      - name: rancher-prime
        url: https://charts.rancher.com/server-charts/prime
      - name: rancher-application-collection
        url: oci://dp.apps.rancher.io/charts
        authentication:
          username: $\{APPS.RANCHER.IO_USERNAME\}
          password: $\{APPS.RANCHER.IO_ACCESS_TOKEN\}
  network:
    apiHost: $API_HOST
    apiVIP: $API_VIP
  nodes:
    - hostname: mgmt-cluster-node1
      initializer: true
      type: server
#   - hostname: mgmt-cluster-node2
#     type: server
#   - hostname: mgmt-cluster-node3
#     type: server

To explain the fields and values in the mgmt-cluster.yaml definition file, we have divided it into the following sections.

Image section (definition file):

image:
  imageType: iso
  arch: x86_64
  baseImage: SL-Micro.x86_64-6.2-Base-SelfInstall-GM.install.iso
  outputImageName: eib-mgmt-cluster-image.iso

where the baseImage is the original image you downloaded from the SUSE Customer Center or the SUSE Download page. outputImageName is the name of the new image that will be used to provision the management cluster.

Operating system section (definition file):

operatingSystem:
  isoConfiguration:
    installDevice: /dev/sda
  users:
  - username: root
    encryptedPassword: $ROOT_PASSWORD
  packages:
    packageList:
    - jq
    sccRegistrationCode: $SCC_REGISTRATION_CODE

where the installDevice is the device to be used to install the operating system, the username and encryptedPassword are the credentials to be used to access the system, the packageList is the list of packages to be installed (jq is required internally during the installation process), and the sccRegistrationCode is the registration code used to get the packages and dependencies at build time and can be obtained from the SUSE Customer Center. The encrypted password can be generated using the openssl command as follows:

openssl passwd -6 MyPassword!123

This outputs something similar to:

$6$UrXB1sAGs46DOiSq$HSwi9GFJLCorm0J53nF2Sq8YEoyINhHcObHzX2R8h13mswUIsMwzx4eUzn/rRx0QPV4JIb0eWCoNrxGiKH4R31

Kubernetes section (definition file):

kubernetes:
  version: v1.35.3+rke2r3
  helm:
    charts:
      - name: cert-manager
        repositoryName: jetstack
        version: 1.20.1
        targetNamespace: cert-manager
        valuesFile: certmanager.yaml
        createNamespace: true
        installationNamespace: kube-system
      - name: suse-storage
        releaseName: longhorn
        version: 1.11.1
        repositoryName: rancher-application-collection
        targetNamespace: longhorn-system
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: longhorn.yaml
      - name: metal3
        version: 306.0.26+up0.15.0
        repositoryName: suse-edge-charts
        targetNamespace: metal3-system
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: metal3.yaml
      - name: metallb
        version: 306.0.2+up0.15.3
        targetNamespace: metallb-system
        createNamespace: true
        repositoryName: suse-edge-charts
        installationNamespace: kube-system
      - name: rancher-turtles-providers
        version: 306.0.6+up0.26.1
        repositoryName: suse-edge-charts
        targetNamespace: cattle-turtles-system
        createNamespace: true
        installationNamespace: kube-system
      - name: neuvector-crd
        version: 109.0.1+up2.8.13
        repositoryName: rancher-charts
        targetNamespace: neuvector
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: neuvector.yaml
      - name: neuvector
        version: 109.0.1+up2.8.13
        repositoryName: rancher-charts
        targetNamespace: neuvector
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: neuvector.yaml
      - name: rancher
        version: 2.14.1
        repositoryName: rancher-prime
        targetNamespace: cattle-system
        createNamespace: true
        installationNamespace: kube-system
        valuesFile: rancher.yaml
    repositories:
      - name: jetstack
        url: https://charts.jetstack.io
      - name: rancher-charts
        url: https://charts.rancher.io/
      - name: suse-edge-charts
        url: oci://registry.suse.com/edge/charts
      - name: rancher-prime
        url: https://charts.rancher.com/server-charts/prime
      - name: rancher-application-collection
        url: oci://dp.apps.rancher.io/charts
        authentication:
          username: $\{APPS.RANCHER.IO_USERNAME\}
          password: $\{APPS.RANCHER.IO_ACCESS_TOKEN\}
    network:
      apiHost: $API_HOST
      apiVIP: $API_VIP
    nodes:
    - hostname: mgmt-cluster-node1
      initializer: true
      type: server
#   - hostname: mgmt-cluster-node2
#     type: server
#   - hostname: mgmt-cluster-node3
#     type: server

The helm section contains the list of Helm charts to be installed, the repositories to be used, and the version configuration for all of them.

The network section contains the configuration for the network, like the apiHost and apiVIP to be used by the RKE2 component. The apiVIP should be an IP address that is not used in the network and should not be part of the DHCP pool (in case we use DHCP). Also, when we use the apiVIP in a multi-node cluster, it is used to access the Kubernetes API server. The apiHost is the name resolution to apiVIP to be used by the RKE2 component.

The nodes section contains the list of nodes to be used in the cluster. In this example, a single-node cluster is being used, but it can be extended to a multi-node cluster by adding more nodes to the list (by uncommenting the lines).

Note

The names of the nodes must be unique in the cluster.
Optionally, use the initializer field to specify the bootstrap host, otherwise it will be the first node in the list.
The names of the nodes must be the same as the host names defined in the Network Folder (Section 29.5, “Networking folder”) when network configuration is required.

29.3 Custom folder #

The custom folder contains the following subfolders:

...
├── custom
│ ├── scripts
│ │ ├── 99-register.sh
│ │ ├── 99-mgmt-setup.sh
│ │ └── 99-alias.sh
│ └── files
│     ├── rancher.sh
│     ├── mgmt-stack-setup.service
│     ├── metal3.sh
│     └── basic-setup.sh
...

The custom/files folder contains the configuration files to be used by the management cluster.
The custom/scripts folder contains the scripts to be used by the management cluster.

The custom/files folder contains the following files:

basic-setup.sh: contains configuration parameters for Metal³, Rancher and MetalLB. Only modify this file if you want to change the namespaces to be used.

#!/bin/bash
# Pre-requisites. Cluster already running
export KUBECTL="/var/lib/rancher/rke2/bin/kubectl"
export KUBECONFIG="/etc/rancher/rke2/rke2.yaml"

##################
# METAL3 DETAILS #
##################
export METAL3_CHART_TARGETNAMESPACE="metal3-system"

###########
# METALLB #
###########
export METALLBNAMESPACE="metallb-system"

###########
# RANCHER #
###########
export RANCHER_CHART_TARGETNAMESPACE="cattle-system"
export RANCHER_FINALPASSWORD="adminadminadmin"

die(){
  echo ${1} 1>&2
  exit ${2}
}

metal3.sh: contains the configuration for the Metal³ component to be used (no modifications needed). In future versions, this script will be replaced to use instead Rancher Turtles to make it easy.

#!/bin/bash
set -euo pipefail

BASEDIR="$(dirname "$0")"
source ${BASEDIR}/basic-setup.sh

METAL3LOCKNAMESPACE="default"
METAL3LOCKCMNAME="metal3-lock"

trap 'catch $? $LINENO' EXIT

catch() {
  if [ "$1" != "0" ]; then
    echo "Error $1 occurred on $2"
    ${KUBECTL} delete configmap ${METAL3LOCKCMNAME} -n ${METAL3LOCKNAMESPACE}
  fi
}

# Get or create the lock to run all those steps just in a single node
# As the first node is created WAY before the others, this should be enough
# TODO: Investigate if leases is better
if [ $(${KUBECTL} get cm -n ${METAL3LOCKNAMESPACE} ${METAL3LOCKCMNAME} -o name | wc -l) -lt 1 ]; then
  ${KUBECTL} create configmap ${METAL3LOCKCMNAME} -n ${METAL3LOCKNAMESPACE} --from-literal foo=bar
else
  exit 0
fi

# Wait for metal3
while ! ${KUBECTL} wait --for condition=ready -n ${METAL3_CHART_TARGETNAMESPACE} $(${KUBECTL} get pods -n ${METAL3_CHART_TARGETNAMESPACE} -l app.kubernetes.io/name=metal3-ironic -o name) --timeout=10s; do sleep 2 ; done

# Get the ironic IP
IRONICIP=$(${KUBECTL} get cm -n ${METAL3_CHART_TARGETNAMESPACE} ironic -o jsonpath='{.data.IRONIC_IP}')

# If LoadBalancer, use metallb, else it is NodePort
if [ $(${KUBECTL} get svc -n ${METAL3_CHART_TARGETNAMESPACE} metal3-metal3-ironic -o jsonpath='{.spec.type}') == "LoadBalancer" ]; then
  # Wait for metallb
  while ! ${KUBECTL} wait --for condition=ready -n ${METALLBNAMESPACE} $(${KUBECTL} get pods -n ${METALLBNAMESPACE} -l app.kubernetes.io/component=controller -o name) --timeout=10s; do sleep 2 ; done

  # Do not create the ippool if already created
  ${KUBECTL} get ipaddresspool -n ${METALLBNAMESPACE} ironic-ip-pool -o name || cat <<-EOF | ${KUBECTL} apply -f -
  apiVersion: metallb.io/v1beta1
  kind: IPAddressPool
  metadata:
    name: ironic-ip-pool
    namespace: ${METALLBNAMESPACE}
  spec:
    addresses:
    - ${IRONICIP}/32
    serviceAllocation:
      priority: 100
      serviceSelectors:
      - matchExpressions:
        - {key: app.kubernetes.io/name, operator: In, values: [metal3-ironic]}
	EOF

  # Same for L2 Advs
  ${KUBECTL} get L2Advertisement -n ${METALLBNAMESPACE} ironic-ip-pool-l2-adv -o name || cat <<-EOF | ${KUBECTL} apply -f -
  apiVersion: metallb.io/v1beta1
  kind: L2Advertisement
  metadata:
    name: ironic-ip-pool-l2-adv
    namespace: ${METALLBNAMESPACE}
  spec:
    ipAddressPools:
    - ironic-ip-pool
	EOF
fi

# Clean up the lock cm

${KUBECTL} delete configmap ${METAL3LOCKCMNAME} -n ${METAL3LOCKNAMESPACE}

rancher.sh: contains the configuration for the Rancher component to be used (no modifications needed).

#!/bin/bash
set -euo pipefail

BASEDIR="$(dirname "$0")"
source ${BASEDIR}/basic-setup.sh

RANCHERLOCKNAMESPACE="default"
RANCHERLOCKCMNAME="rancher-lock"

if [ -z "${RANCHER_FINALPASSWORD}" ]; then
  # If there is no final password, then finish the setup right away
  exit 0
fi

trap 'catch $? $LINENO' EXIT

catch() {
  if [ "$1" != "0" ]; then
    echo "Error $1 occurred on $2"
    ${KUBECTL} delete configmap ${RANCHERLOCKCMNAME} -n ${RANCHERLOCKNAMESPACE}
  fi
}

# Get or create the lock to run all those steps just in a single node
# As the first node is created WAY before the others, this should be enough
# TODO: Investigate if leases is better
if [ $(${KUBECTL} get cm -n ${RANCHERLOCKNAMESPACE} ${RANCHERLOCKCMNAME} -o name | wc -l) -lt 1 ]; then
  ${KUBECTL} create configmap ${RANCHERLOCKCMNAME} -n ${RANCHERLOCKNAMESPACE} --from-literal foo=bar
else
  exit 0
fi

# Wait for rancher to be deployed
while ! ${KUBECTL} wait --for condition=ready -n ${RANCHER_CHART_TARGETNAMESPACE} $(${KUBECTL} get pods -n ${RANCHER_CHART_TARGETNAMESPACE} -l app=rancher -o name) --timeout=10s; do sleep 2 ; done
until ${KUBECTL} get ingress -n ${RANCHER_CHART_TARGETNAMESPACE} rancher > /dev/null 2>&1; do sleep 10; done

RANCHERBOOTSTRAPPASSWORD=$(${KUBECTL} get secret -n ${RANCHER_CHART_TARGETNAMESPACE} bootstrap-secret -o jsonpath='{.data.bootstrapPassword}' | base64 -d)
RANCHERHOSTNAME=$(${KUBECTL} get ingress -n ${RANCHER_CHART_TARGETNAMESPACE} rancher -o jsonpath='{.spec.rules[0].host}')

# Skip the whole process if things have been set already
if [ -z $(${KUBECTL} get settings.management.cattle.io first-login -ojsonpath='{.value}') ]; then
  # Add the protocol
  RANCHERHOSTNAME="https://${RANCHERHOSTNAME}"
  TOKEN=""
  while [ -z "${TOKEN}" ]; do
    # Get token
    sleep 2
    TOKEN=$(curl -sk -X POST ${RANCHERHOSTNAME}/v3-public/localProviders/local?action=login -H 'content-type: application/json' -d "{\"username\":\"admin\",\"password\":\"${RANCHERBOOTSTRAPPASSWORD}\"}" | jq -r .token)
  done

  # Set password
  curl -sk ${RANCHERHOSTNAME}/v3/users?action=changepassword -H 'content-type: application/json' -H "Authorization: Bearer $TOKEN" -d "{\"currentPassword\":\"${RANCHERBOOTSTRAPPASSWORD}\",\"newPassword\":\"${RANCHER_FINALPASSWORD}\"}"

  # Create a temporary API token (ttl=60 minutes)
  APITOKEN=$(curl -sk ${RANCHERHOSTNAME}/v3/token -H 'content-type: application/json' -H "Authorization: Bearer ${TOKEN}" -d '{"type":"token","description":"automation","ttl":3600000}' | jq -r .token)

  curl -sk ${RANCHERHOSTNAME}/v3/settings/server-url -H 'content-type: application/json' -H "Authorization: Bearer ${APITOKEN}" -X PUT -d "{\"name\":\"server-url\",\"value\":\"${RANCHERHOSTNAME}\"}"
  curl -sk ${RANCHERHOSTNAME}/v3/settings/telemetry-opt -X PUT -H 'content-type: application/json' -H 'accept: application/json' -H "Authorization: Bearer ${APITOKEN}" -d '{"value":"out"}'
fi

# Clean up the lock cm
${KUBECTL} delete configmap ${RANCHERLOCKCMNAME} -n ${RANCHERLOCKNAMESPACE}

mgmt-stack-setup.service: contains the configuration to create the systemd service to run the scripts during the first boot (no modifications needed).

[Unit]
Description=Setup Management stack components
Wants=network-online.target
# It requires rke2 or k3s running, but it will not fail if those services are not present
After=network.target network-online.target rke2-server.service k3s.service
# At least, the basic-setup.sh one needs to be present
ConditionPathExists=/opt/mgmt/bin/basic-setup.sh

[Service]
User=root
Type=forking
# Metal3 can take A LOT to download the IPA image
TimeoutStartSec=1800

ExecStartPre=/bin/sh -c "echo 'Setting up Management components...'"
# Scripts are executed in StartPre because Start can only run a single one
ExecStartPre=/opt/mgmt/bin/rancher.sh
ExecStartPre=/opt/mgmt/bin/metal3.sh
ExecStart=/bin/sh -c "echo 'Finished setting up Management components'"
RemainAfterExit=yes
KillMode=process
# Disable & delete everything
ExecStartPost=rm -f /opt/mgmt/bin/rancher.sh
ExecStartPost=rm -f /opt/mgmt/bin/metal3.sh
ExecStartPost=rm -f /opt/mgmt/bin/basic-setup.sh
ExecStartPost=/bin/sh -c "systemctl disable mgmt-stack-setup.service"
ExecStartPost=rm -f /etc/systemd/system/mgmt-stack-setup.service

[Install]
WantedBy=multi-user.target

The custom/scripts folder contains the following files:

99-alias.sh script: contains the alias to be used by the management cluster to load the kubeconfig file at first boot (no modifications needed).

#!/bin/bash
echo "alias k=kubectl" >> /etc/profile.local
echo "alias kubectl=/var/lib/rancher/rke2/bin/kubectl" >> /etc/profile.local
echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml" >> /etc/profile.local

99-mgmt-setup.sh script: contains the configuration to copy the scripts during the first boot (no modifications needed).

#!/bin/bash

# Copy the scripts from combustion to the final location
mkdir -p /opt/mgmt/bin/
for script in basic-setup.sh rancher.sh metal3.sh; do
	cp ${script} /opt/mgmt/bin/
done

# Copy the systemd unit file and enable it at boot
cp mgmt-stack-setup.service /etc/systemd/system/mgmt-stack-setup.service
systemctl enable mgmt-stack-setup.service

99-register.sh script: contains the configuration to register the system using the SCC registration code. The ${SCC_ACCOUNT_EMAIL} and ${SCC_REGISTRATION_CODE} have to be set properly to register the system with your account.

#!/bin/bash
set -euo pipefail

# Registration https://www.suse.com/support/kb/doc/?id=000018564
if ! which SUSEConnect > /dev/null 2>&1; then
	zypper --non-interactive install suseconnect-ng
fi
SUSEConnect --email "${SCC_ACCOUNT_EMAIL}" --url "https://scc.suse.com" --regcode "${SCC_REGISTRATION_CODE}"

29.4 Kubernetes folder #

The kubernetes folder contains the following subfolders:

...
├── kubernetes
│ ├── manifests
│ │ ├── rke2-ingress-config.yaml
│ │ ├── neuvector-namespace.yaml
│ │ ├── ingress-l2-adv.yaml
│ │ └── ingress-ippool.yaml
│ ├── helm
│ │ └── values
│ │     ├── rancher.yaml
│ │     ├── neuvector.yaml
│ │     ├── metal3.yaml
│ │     └── certmanager.yaml
│ └── config
│     └── server.yaml
...

The kubernetes/config folder contains the following files:

server.yaml: The CNI plug-in installed by default is Cilium, so you do not need to create this folder and file to set that. Just in case you need to customize the CNI plug-in, you can use the server.yaml file under the kubernetes/config folder. It contains the following information:
```
cni:
- multus
- cilium
ingress-controller: traefik
write-kubeconfig-mode: '0644'
selinux: true
system-default-registry: registry.rancher.com
```

Note

This is an optional file to define certain Kubernetes customization, like the CNI plug-ins to be used or many options you can check in the official documentation.

Warning

The Traefik ingress provider integrated into RKE2/K3s is the only ingress controller supported in SUSE Telco Cloud 3.6 release, being still possible to temporarily run Ingress-NGINX alongside Traefik in order to support complex ingress migration scenarios, but only after SUSE Telco Cloud Management and/or Downstream clusters have been upgraded to version 3.6 and for the time required to perform that migration. Since Traefik is not yet the default ingress controller in RKE2 (it will be from RKE2 v1.36 onwards), it must be explicitly "requested" from the RKE2 server configuration file, resulting in the need to include the kubernetes/config/server.yaml file in all the EIB self-install iso images generated to provision SUSE Telco Cloud 3.6 Management clusters' nodes.

RKE2 Ingress NGINX to Traefik Migration guide provides details on the ingress migration paths available once the Traefik ingress controller replaces the discontinued Ingress-NGINX.

The os-files/var/lib/rancher/rke2/server/manifests folder contains the following file:

rke2-ingress-config.yaml: contains the configuration to create the Ingress service for the management cluster (no modifications needed).

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ingressClass:
      isDefaultClass: true
    ports:
      web:
        hostPort: null    # disallow hostPort
        exposedPort: 80
      websecure:
        hostPort: null    # disallow hostPort
        exposedPort: 443
    service:
      enabled: true
      type: LoadBalancer
      spec:
        externalTrafficPolicy: Local
        allocateLoadBalancerNodePorts: false  # k8s GA from 1.24; supported by MetalLB
    providers:
      kubernetesIngressNginx:  # this provider allows traefik to "understand" most of the ingress-nginx annotations
        enabled: true
        ingressClass: "rke2-ingress-nginx-migration"
        controllerClass: "rke2.cattle.io/ingress-nginx-migration"

Note

The HelmChartConfig must be included via os-files to the /var/lib/rancher/rke2/server/manifests directory, not via kubernetes/manifests as described in previous releases.

The kubernetes/manifests folder contains the following files:

neuvector-namespace.yaml: contains the configuration to create the NeuVector namespace (no modifications needed).

apiVersion: v1
kind: Namespace
metadata:
  labels:
    pod-security.kubernetes.io/enforce: privileged
  name: neuvector

ingress-l2-adv.yaml: contains the configuration to create the L2Advertisement for the MetalLB component (no modifications needed).

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: ingress-l2-adv
  namespace: metallb-system
spec:
  ipAddressPools:
    - ingress-ippool

ingress-ippool.yaml: contains the configuration to create the MetalLB IPAddressPool object for the rke2-traefik component. The ${INGRESS_VIP} has to be set properly to define the exposed IP address reserved to be used to reach the (internal) rke2-traefik service.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: ingress-ippool
  namespace: metallb-system
spec:
  addresses:
    - ${INGRESS_VIP}/32
  serviceAllocation:
    priority: 100
    serviceSelectors:
      - matchExpressions:
          - {key: app.kubernetes.io/name, operator: In, values: [rke2-traefik]}

The kubernetes/helm/values folder contains the following files:

rancher.yaml: contains the configuration to create the Rancher component. The ${INGRESS_VIP} must be set properly to define the IP address to be consumed by the Rancher component. The URL to access the Rancher component will be https://rancher-${INGRESS_VIP}.sslip.io.
```
hostname: rancher-${INGRESS_VIP}.sslip.io
bootstrapPassword: "foobar"
replicas: 1
global:
  cattle:
    systemDefaultRegistry: "registry.rancher.com"
```

neuvector.yaml: contains the configuration to create the NeuVector component (no modifications needed).

controller:
  replicas: 1
  ranchersso:
    enabled: true
manager:
  enabled: false
cve:
  scanner:
    enabled: false
    replicas: 1
k3s:
  enabled: true
crdwebhook:
  enabled: false
registry: "registry.rancher.com"
global:
  cattle:
    systemDefaultRegistry: "registry.rancher.com"

longhorn.yaml: contains the configuration to create the Longhorn component. To make sure the necessary container images are downloaded at boot time, please add your Rancher Application Collection credentials.

privateRegistry:
  createSecret: true
  registryUrl: dp.apps.rancher.io
  registryUser: $\{APPS.RANCHER.IO_USERNAME\}
  registryPasswd: $\{APPS.RANCHER.IO_ACCESS_TOKEN\}
  registrySecret: rancher-app-collection

metal3.yaml: contains the configuration to create the Metal³ component. The ${METAL3_VIP} must be set properly to define the IP address to be consumed by the Metal³ component.
```
global:
  ironicIP: ${METAL3_VIP}
  enable_vmedia_tls: false
  # trustedCA: tls-ca-bundle  # Optional: Uncomment and set to ConfigMap name for additional trusted CAs
metal3-ironic:
  ipa:
    useHauler: true
  global:
    predictableNicNames: "true"
  persistence:
    ironic:
      size: "5Gi"
```
Important
The metal3-ironic.ipa.useHauler value must be set to true (boolean value) in air-gapped environments. This instructs Metal³ to retrieve the Ironic Python Agent (IPA) image from the embedded artifact registry instead of attempting to download it from the internet. The IPA image must also be included in the embeddedArtifactRegistry.images list as shown in the definition file example above.

Note

The Media Server is an optional feature included in Metal³ (by default is disabled). To use the Metal3 feature, you need to configure it on the previous manifest. To use the Metal³ media server, specify the following variable:

add the enable_metal3_media_server to true to enable the media server feature in the global section.
include the following configuration about the media server where ${MEDIA_VOLUME_PATH} is the path to the media volume in the media (e.g /home/metal3/bmh-image-cache)
```
metal3-media:
  mediaVolume:
    hostPath: ${MEDIA_VOLUME_PATH}
```

An external media server can be used to store the images, and in the case you want to use it with TLS, you will need to provide the trusted CA bundle for Metal³.

Prepare the CA bundle:

First, prepare your CA bundle file containing all the certificates needed by Metal³. This includes your external media server’s CA certificate(s).

+ Optional - System CA Bundle: If Metal³ also needs to trust public CAs (for example, when accessing external HTTPS resources), you can include the system CA bundle. Extract it from a container image and concatenate it with your custom certificates:

# Extract system CAs from a container image
podman run --rm registry.suse.com/bci/bci-base:latest cat /etc/ssl/certs/ca-certificates.crt > system-cas.pem

# Concatenate system CAs with your custom CA
cat system-cas.pem your-custom-ca.crt > ca-bundle.pem

+ Or, if you only need your custom CA:

cp your-custom-ca.crt ca-bundle.pem

If you include the system CA bundle, you become responsible for keeping it up-to-date. System CAs in container images may become outdated as certificates expire or are revoked. Periodically refresh the bundle by re-extracting from an updated container image.

Create the ConfigMap:

You can create the ConfigMap in two ways:

Using a manifest file (recommended for EIB): Create the file kubernetes/manifests/metal3-cacert-configmap.yaml and include the CA bundle content inline. You need to properly indent each line of the certificate with 4 spaces:

apiVersion: v1
kind: Namespace
metadata:
  name: metal3-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: tls-ca-bundle
  namespace: metal3-system
data:
  ca-bundle.pem: |
    -----BEGIN CERTIFICATE-----
    MIIDXTCCAkWgAwIBAgIJAKJ... (your certificate content here)
    ...
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIEkjCCA3qgAwIBAgIQCgF... (additional certificates if any)
    ...
    -----END CERTIFICATE-----

To fill this file automatically from your ca-bundle.pem, you can use:

cat > kubernetes/manifests/metal3-cacert-configmap.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: metal3-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: tls-ca-bundle
  namespace: metal3-system
data:
  ca-bundle.pem: |
$(sed 's/^/    /' ca-bundle.pem)
EOF

Using kubectl directly (for manual deployment): If you’re not using EIB and want to create the ConfigMap directly on the cluster:
```
kubectl -n metal3-system create configmap tls-ca-bundle --from-file=ca-bundle.pem=./ca-bundle.pem
```
- Set the global.trustedCA value in the metal3.yaml file to reference the ConfigMap name:
```
global:
  trustedCA: tls-ca-bundle
```

certmanager.yaml: contains the configuration to create the Cert-Manager component (no modifications needed).
```
installCRDs: true
```

29.5 Networking folder #

The network folder contains as many files as nodes in the management cluster. In our case, we have only one node, so we have only one file called mgmt-cluster-node1.yaml. The name of the file must match the host name defined in the mgmt-cluster.yaml definition file into the network/node section described above.

If you need to customize the networking configuration, for example, to use a specific static IP address (DHCP-less scenario), you can use the mgmt-cluster-node1.yaml file under the network folder. It contains the following information:

${MGMT_GATEWAY}: The gateway IP address.
${MGMT_DNS}: The DNS server IP address.
${MGMT_MAC}: The MAC address of the network interface.
${MGMT_NODE_IP}: The IP address of the management cluster.

routes:
  config:
  - destination: 0.0.0.0/0
    metric: 100
    next-hop-address: ${MGMT_GATEWAY}
    next-hop-interface: eth0
    table-id: 254
dns-resolver:
  config:
    server:
    - ${MGMT_DNS}
    - 8.8.8.8
interfaces:
- name: eth0
  type: ethernet
  state: up
  mac-address: ${MGMT_MAC}
  ipv4:
    address:
    - ip: ${MGMT_NODE_IP}
      prefix-length: 24
    dhcp: false
    enabled: true
  ipv6:
    enabled: false

If you want to use DHCP to get the IP address, you can use the following configuration (the MAC address must be set properly using the ${MGMT_MAC} variable):

## This is an example of a dhcp network configuration for a management cluster
interfaces:
- name: eth0
  type: ethernet
  state: up
  mac-address: ${MGMT_MAC}
  ipv4:
    dhcp: true
    enabled: true
  ipv6:
    enabled: false

Note

Depending on the number of nodes in the management cluster, you can create more files like mgmt-cluster-node2.yaml, mgmt-cluster-node3.yaml, etc. to configure the rest of the nodes.
The routes section is used to define the routing table for the management cluster.