3 How do I deploy SUSE AI Factory with Elemental 3? #
SUSE Elemental 3 builds one bootable image that contains the operating system, RKE2 and the SUSE AI Factory Helm charts. Every cluster node boots from this image, installs itself, and joins the Kubernetes cluster. Rancher Prime and SUSE AI Factory Operator are deployed automatically, so the manual steps described in Chapter 1, How do I prepare for the deployment? and Chapter 2, How do I deploy SUSE AI Factory? are not needed.
3.1 How does the SUSE Elemental 3 deployment work? #
SUSE Elemental 3 uses a release manifest that pins a tested set of components: the operating system image, the RKE2 version, Helm charts and system extensions. SUSE AI Factory publishes its own release manifest that extends the SUSE Elemental core platform with Rancher Prime, cert-manager, the NVIDIA GPU Operator, SUSE Storage, the local path provisioner and SUSE AI Factory Operator.
The deployment consists of the following steps:
Create a configuration directory that describes the desired state of the nodes.
Run the
elemental3 customizecommand to build a RAW disk image or ISO installation media. With SUSE Elemental 3.1 or later, you can also build a qcow2 image.Boot every node from the image. On the first boot, Ignition applies the operating system configuration, RKE2 creates the cluster, and the Helm charts are installed.
Verify the deployment and continue with the post-deployment steps.
The operating system is immutable and image-based.
It has no package manager, and the /usr file system is always read-only.
Additional software, such as the open-iscsi tools required by SUSE Storage, is delivered as systemd system extensions listed in the release manifest.
The configuration files in this chapter are examples. SUSE Elemental 3 supports many more customizations than described here. For complete configuration directories of a management cluster and a downstream cluster, see Section 3.13, “Example deployments”.
3.1.1 Which SUSE Elemental version do I need? #
This section describes SUSE Elemental 3.0, which supports bare-metal servers and virtual machines with a static list of nodes.
Use the 3.0.0 release or later.
Pre-release 3.0 versions use a different configuration format, for example, a top-level kubernetes.yaml file instead of kubernetes/cluster.yaml, and do not work with the examples in this chapter.
The following features require SUSE Elemental 3.1 or later and are marked in the text:
qcow2 images and per-node qcow2 overlays built directly by
elemental3 customizethe
apiVIPModeoption and external load balancers, required on cloud providersnode roles provided at boot time with
runtime.env, required on cloud providersthe
--airgapand--cacheoptions ofelemental3 customize
3.2 Prerequisites #
A customization host with the
x86_64architecture, Podman and access to the following registries:registry.suse.com,ghcr.io,dp.apps.rancher.io,charts.rancher.com,charts.jetstack.ioandhelm.ngc.nvidia.com. The image build downloads several gigabytes, so reserve at least 50 GB of free disk space.Credentials for SUSE Application Collection. The SUSE Storage and local path provisioner charts are pulled from SUSE Application Collection.
Optionally, a SUSE Registry registration code and an NVIDIA NGC API key. SUSE AI Factory Operator uses them to access applications and blueprints right after the deployment.
Cluster nodes, bare-metal or virtual, that boot with UEFI. The images built by SUSE Elemental are EFI-only. We recommend at least 100 GB of disk space for each node. The image itself can be smaller, because the system partition grows to fill the disk on the first boot, as described in Section 3.5, “Configuring the installation and disk image (
install.yaml)”.The planned network values:
the API virtual IP address (VIP) and an optional host name for the Kubernetes API,
the host name for the Rancher Prime UI and a DNS record for it,
host names and MAC addresses of the nodes if you use the static node list described in Section 3.9.1, “Static node list”. On the first boot, each node identifies its network configuration, host name and role by its MAC address.
3.3 Creating the configuration directory #
The configuration directory holds every file that elemental3 customize reads.
The following example shows the files used for a SUSE AI Factory cluster:
aif-elemental/ ├── release.yaml 1 ├── install.yaml 2 ├── butane.yaml 3 ├── kubernetes/ │ ├── cluster.yaml 4 │ ├── config/ │ │ ├── server.yaml 5 │ │ └── agent.yaml 6 │ ├── helm/ │ │ └── values/ 7 │ │ ├── rancher.yaml │ │ └── aif-operator.yaml │ └── manifests/ 8 │ └── appco-pull-secret-priority.yaml ├── network/ 9 └── custom/ 10 ├── files/ └── scripts/
Release manifest and enabled components. Required. | |
Bootloader, kernel command line, image size and installation device. Required. | |
Users, SSH keys, files and systemd units applied by Ignition on the first boot. | |
Nodes, API VIP and load-balancing mode, extra charts and manifests. | |
RKE2 configuration for control-plane ( | |
Helm values files referenced from | |
Additional Kubernetes manifests applied to the cluster. | |
Static network configuration. If the directory is missing, the nodes use DHCP. | |
Custom scripts and files executed on the first boot. |
Create the directory and restrict its permissions, because several files contain credentials:
> mkdir -p aif-elemental/kubernetes/{config,helm/values,manifests}
> chmod 700 aif-elemental3.4 Selecting the release and components (release.yaml) #
The release.yaml file references the SUSE AI Factory release manifest and enables components from it.
Components that are not listed are not installed.
manifestURI: oci://ghcr.io/suse/aif-uc-release-manifest:2.4.0
components:
kubernetes: {}
systemd:
- extension: suse-storage 1
helm:
- chart: rancher 2
valuesFile: rancher.yaml
- chart: gpu-operator 3
- chart: suse-storage 4
credentials:
username: "<APPCO_USERNAME>"
password: "<APPCO_TOKEN>"
- chart: aif-operator
valuesFile: aif-operator.yamlSystem extension with the open-iscsi tools that SUSE Storage needs. Omit it if you use the local path provisioner instead. | |
cert-manager is installed automatically because the | |
Install the NVIDIA GPU Operator only if the nodes have NVIDIA GPUs. | |
Charts from SUSE Application Collection need the |
The SUSE AI Factory release manifest provides the following components:
| Component | Type | Description |
|---|---|---|
| Helm chart | Rancher Prime. Requires the |
| Helm chart | Certificate management. Added as a dependency of |
| Helm chart | NVIDIA GPU Operator configured for precompiled NVIDIA drivers from SUSE Registry. |
| Helm chart | SUSE Storage (Longhorn). Requires the |
| Helm chart | Storage on the local disk of a node. Recommended for single-node clusters. See Important: Enable only one storage provider. |
| Helm chart | SUSE AI Factory Operator and the SUSE AI Factory Rancher Prime UI extension. |
| System extension | open-iscsi tools under |
Both suse-storage and local-path-provisioner configure themselves as the default storage class.
Do not enable them together.
For multi-node clusters, use SUSE Storage. It replicates volumes across nodes, so pods can move between nodes and the data survives the failure of a node.
For single-node clusters, use the local path provisioner. It also works in multi-node clusters, but a volume exists only on the node where it was created. Pods that use the volume are always scheduled on that node, and the data is lost if the node fails.
To review the content of a release manifest, run:
> podman run --rm registry.suse.com/elemental/elemental:3.0 release-info \
oci://ghcr.io/suse/aif-uc-release-manifest:2.4.03.5 Configuring the installation and disk image (install.yaml) #
schema: v0
bootloader: grub
kernelCmdLine: "console=tty0 console=ttyS0,115200" 1
raw:
diskSize: 20G 2
iso:
device: /dev/sda 3
# cryptoPolicy: fips 4Parameters appended to the kernel command line.
On a cloud provider (SUSE Elemental 3.1 or later), add the Ignition platform identifier, for example, | |
Size of the RAW disk image. For qcow2 images (SUSE Elemental 3.1 or later), this is the virtual disk size. | |
Target disk for the installation from ISO media. | |
Enables the FIPS crypto policy. Enable it only if every node is FIPS-ready. |
On the first boot, the system partition grows to fill the whole disk.
Keep diskSize small to speed up copying the image, and resize the target disk to its final size before the first boot, for example, with qemu-img resize for virtual machines.
If you boot the image without resizing the disk, the root file system keeps the size of the image.
3.5.1 Disk layout and data persistence #
SUSE Elemental creates the following partitions:
| Label | File system | Mount point | Purpose |
|---|---|---|---|
| vfat |
| Bootloader, kernel and initrd. |
| ext4 | — | Installation and restore operations. |
| btrfs |
| Operating system and user data. Uses all remaining space. |
| ext4 | — | First-boot configuration. Created only when needed. |
The root subvolume is read-only.
The /etc, /var, /root, /home, /opt, /srv and /usr/local directories are separate read-write subvolumes:
/etcis snapshotted with the operating system. During an upgrade, local changes are merged into the new snapshot, and a rollback also restores/etc./varhas copy-on-write disabled and persists across upgrades and rollbacks. Kubernetes data in/var/lib/rancherand the SUSE Storage data in/var/lib/longhornare stored there./usris replaced on every upgrade. Never store data there.
To use additional disks, for example, a dedicated disk for SUSE Storage or model storage, define the partitions, file systems and mounts in butane.yaml.
Ignition configures them on the first boot.
3.6 Configuring users, SSH access and files (butane.yaml) #
The butane.yaml file contains a Butane configuration.
SUSE Elemental converts it to an Ignition configuration that runs on the first boot of every node.
The following example creates the root user and an unprivileged user with SSH keys, disables SSH logins as root and sets up the root shell for kubectl:
variant: fcos
version: 1.6.0
passwd:
users:
- name: root
password_hash: "<ROOT_PASSWORD_HASH>" 1
- name: suse 2
password_hash: "<USER_PASSWORD_HASH>"
shell: /bin/bash
ssh_authorized_keys:
- "<SSH_PUBLIC_KEY>"
systemd:
units:
- name: sshd.service
enabled: true
storage:
filesystems:
- device: /dev/disk/by-partlabel/SYSTEM 3
format: btrfs
wipe_filesystem: false
path: /home
mount_options:
- subvol=/@/home
files:
- path: /etc/ssh/sshd_config.d/sshd.conf
mode: 384 4
overwrite: true
contents:
inline: |
PermitRootLogin no
AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys.d/ignition
- path: /root/.profile
mode: 416
overwrite: true
contents:
inline: |
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
export PATH=${PATH}:/var/lib/rancher/rke2/binGenerate password hashes with | |
The image does not include | |
| |
Specify file modes as decimal numbers, for example, |
Observe the following rules when writing butane.yaml:
Write files only to read-write paths, such as
/etc,/varor/opt. A write to a read-only path, such as/usr, fails Ignition and the node stops in the emergency shell of the initrd.Store scripts that must persist after the first boot in
/var/lib/elemental.The Ignition
kernelArgumentssection is not supported. UsekernelCmdLineininstall.yamlinstead.Changing
butane.yamlrequires rebuilding the image.
3.6.1 SELinux considerations #
SELinux runs in enforcing mode on the operating system.
The selinux: true option in the RKE2 configuration enables SELinux support for RKE2 and its containers.
For details, see the RKE2 SELinux documentation.
Take the following into account when adding units, scripts or directories:
Files in
/var/lib/elementalhave thevar_lib_tlabel, which systemd is not allowed to execute. A unit withExecStart=/var/lib/elemental/script.shfails with the203/EXECstatus. Run the script through the interpreter instead:ExecStart=/usr/bin/bash /var/lib/elemental/script.shDirectories used by containers need the
container_file_tlabel. For example, the local path provisioner stores data in/opt/local-path-provisioner. Create the directory at boot withmkdir -pZ /opt/local-path-provisionerin a systemd unit ordered beforerke2-server.serviceandrke2-agent.service, so that it gets the label defined by therke2-selinuxpolicy.SELinux labels in
/etcare not merged during upgrades. The directory is relabeled according to the current policy.
3.6.2 Preparing nodes for SUSE Storage #
The suse-storage system extension installs iscsid under /usr, but nothing under /etc.
Without an initiator name and the iscsid configuration, Longhorn volumes are created, but pods cannot attach them.
Add a systemd unit to butane.yaml that runs on every boot after systemd-sysext.service and performs the following tasks:
Creates
/etc/iscsi/initiatorname.iscsiwith a unique name, for example, generated byiscsi-iname.Copies a default
/etc/iscsi/iscsid.confif it does not exist.Starts
iscsidwithsystemctl start --no-block iscsid.service.
To place the default Longhorn disk only on selected nodes, set defaultSettings.createDefaultDiskLabeledNodes: true in a values file for the suse-storage chart, referenced with valuesFile in release.yaml, and add the node.longhorn.io/create-default-disk=true node label in the RKE2 configuration of those nodes.
3.7 Configuring the Kubernetes cluster (kubernetes/) #
3.7.1 RKE2 configuration #
The files in kubernetes/config are standard RKE2 configuration files.
SUSE Elemental adds the address that nodes use to join the cluster, based on the API VIP from kubernetes/cluster.yaml.
The API VIP is provided either by the embedded MetalLB load balancer or by an external load balancer, as described in Section 3.8, “Choosing how to expose the Kubernetes API”.
Example of kubernetes/config/server.yaml:
token: "<RKE2_TOKEN>" 1
selinux: true
write-kubeconfig-group: suse 2
write-kubeconfig-mode: "0640"
ingress-controller: traefik
tls-san:
- api.aif.example.comShared secret for joining the cluster.
Use the same value in | |
Allows the unprivileged user to read |
Example of kubernetes/config/agent.yaml:
token: "<RKE2_TOKEN>"To use a private registry mirror, add a kubernetes/config/registries.yaml file.
It is applied to all nodes.
3.7.2 Helm values #
Place one values file for each chart that has valuesFile set in release.yaml into kubernetes/helm/values.
The values are merged with the defaults from the release manifest.
Example of kubernetes/helm/values/rancher.yaml:
hostname: rancher.aif.example.com
bootstrapPassword: "<RANCHER_BOOTSTRAP_PASSWORD>"The hostname value is an example.
Unless a tool such as external-dns manages your DNS records, create a DNS record for the host name manually that points to the Ingress address, as described in Section 3.8.1, “Using the embedded load balancer (MetalLB)”.
For test deployments without DNS, you can use a host name from a wildcard DNS service such as sslip.io, for example, rancher.192.168.122.11.sslip.io.
Example of kubernetes/helm/values/aif-operator.yaml:
credentials:
applicationCollection:
username: "<APPCO_USERNAME>"
password: "<APPCO_TOKEN>"
suseRegistry:
username: "regcode"
password: "<REGISTRATION_CODE>"
nvidia:
username: "$oauthtoken"
password: "<NGC_API_KEY>"For details about the credentials, see Section 2.3, “Can I provide repository credentials during the SUSE AI Factory installation?”.
Quote every credential in the YAML files.
Unquoted values containing characters such as :, # or " are not parsed correctly.
3.7.3 Additional manifests #
Kubernetes manifests in kubernetes/manifests are applied after the Kubernetes API is available.
Manifests with names ending in -priority.yaml are applied right after the RKE2 core charts, before the Helm charts from release.yaml.
The other manifests are applied only after every Helm chart is installed.
Use the -priority.yaml suffix for manifests that must not depend on the charts, such as pull secrets and HelmChartConfig resources.
3.7.3.1 Image pull secret for SUSE Application Collection #
The credentials block in release.yaml authenticates only the download of the Helm charts.
The container images of SUSE Storage and the local path provisioner are also hosted in SUSE Application Collection and cannot be pulled anonymously.
As described in the SUSE Storage installation guide, the pods pull the images with an image pull secret referenced in the global.imagePullSecrets chart value.
The SUSE AI Factory release manifest already sets global.imagePullSecrets to application-collection for both charts.
You only need to create the secret in the namespace of the chart: longhorn-system for SUSE Storage, or local-path-provisioner for the local path provisioner.
Example of kubernetes/manifests/appco-pull-secret-priority.yaml for SUSE Storage:
apiVersion: v1
kind: Namespace
metadata:
name: longhorn-system 1
---
apiVersion: v1
kind: Secret
metadata:
name: application-collection 2
namespace: longhorn-system
type: kubernetes.io/dockerconfigjson
stringData:
.dockerconfigjson: |
{
"auths": {
"dp.apps.rancher.io": {
"username": "<APPCO_USERNAME>",
"password": "<APPCO_TOKEN>"
}
}
}The namespace does not exist yet when | |
The name must match the |
The -priority.yaml suffix makes sure that the secret exists before the chart is installed.
For the local path provisioner, use local-path-provisioner as the namespace.
3.8 Choosing how to expose the Kubernetes API #
In a multi-node cluster, the Kubernetes API (port 6443) and the RKE2 supervisor (port 9345) are served on a virtual IP address that is shared by the control-plane nodes.
Configure it in the network section of kubernetes/cluster.yaml.
With SUSE Elemental 3.0, setting apiVIP or apiVIP6 always deploys the embedded MetalLB load balancer.
SUSE Elemental 3.1 adds the apiVIPMode option, which selects who provides the address:
| Mode | Load balancer | Use for |
|---|---|---|
| SUSE Elemental deploys MetalLB and the endpoint copier operator, and announces the VIP from one of the control-plane nodes using layer 2 (ARP). | Bare-metal servers and virtual machines on the same layer 2 network. |
| A load balancer that you provide, such as a cloud provider load balancer, forwards the traffic to the control-plane nodes. | Cloud providers and networks where layer 2 announcements are not possible. |
3.8.1 Using the embedded load balancer (MetalLB) #
On bare-metal servers and on-premises virtual machines, no infrastructure component provides load balancer addresses to Kubernetes.
MetalLB fills this gap: it assigns addresses to services of type LoadBalancer and announces them on the local network.
Use the embedded MetalLB in these environments.
On cloud providers, use the load balancers of the provider instead, as described in Section 3.8.2, “Using a cloud load balancer”.
Example of kubernetes/cluster.yaml with the managed VIP:
network:
apiVIP: 192.168.122.10
apiHost: api.aif.example.com
# apiVIPMode: managed 1SUSE Elemental 3.1 or later only.
|
With the embedded load balancer, SUSE Elemental performs the following steps automatically:
Adds the
metallbandendpoint-copier-operatorcharts from the core platform release manifest.Creates an
IPAddressPoolcontaining only the API VIP, anL2Advertisement, and thekubernetes-vipservice of typeLoadBalancerin thedefaultnamespace.Configures the endpoint copier operator to keep the endpoints of the
kubernetes-vipservice in sync with the control-plane nodes, so that the VIP keeps working when a node is down.
The API VIP must be an unused address in the node subnet, outside the DHCP range and outside any other MetalLB address pool.
For IPv6, use apiVIP6.
If you set both apiVIP and apiVIP6, a dual-stack service is created.
The address pool created by SUSE Elemental is reserved for the API VIP.
To give other services, such as the Ingress controller or an application, their own LoadBalancer address, add the following resources as manifests in kubernetes/manifests:
An
IPAddressPoolwith the additional addresses, in themetallb-systemnamespace. SetautoAssign: falseso that addresses are only given to services that request them.An
L2Advertisementthat references the pool.The
metallb.io/loadBalancerIPsannotation on the service, to request a specific address from the pool.
Do not use the -priority.yaml suffix for these manifests, because the MetalLB CRDs exist only after the Helm charts are installed.
For a complete example, see Section 3.13.1, “How is the Ingress configured?”.
The default RKE2 traefik Ingress controller listens on ports 80 and 443 of every node.
Point the Rancher Prime host name to the IP address of a node or to a DNS record with all node addresses.
To publish the Ingress on its own highly available address instead, expose the Traefik service as a LoadBalancer with a MetalLB address, as shown in Section 3.13.1, “How is the Ingress configured?”.
3.8.2 Using a cloud load balancer #
SUSE Elemental 3.0 always deploys MetalLB for the API VIP and does not support external load balancers.
Example of kubernetes/cluster.yaml with an external load balancer:
network:
apiVIP: 10.0.1.100 1
apiHost: api.aif.example.com 2
apiVIPMode: externalPrivate IP address of the internal load balancer. | |
Host name of the load balancer.
Add it to |
Before building the image, create the following resources:
An internal load balancer that forwards TCP ports 6443 and 9345 to every control-plane node. Worker nodes must reach it on both ports.
Optionally, a second load balancer that forwards TCP ports 80 and 443 to the Ingress controller, to make the Rancher Prime UI and SUSE AI Factory applications available outside the cluster.
The load balancer address is stored in the image. If you re-create the load balancer with a different address, rebuild the image.
The Kubernetes API load balancer must exist before the cluster, so you always create it yourself.
For other services of type LoadBalancer, such as the Ingress controller or applications, you can let the cloud provider create the load balancers automatically instead of creating them in advance.
Install the load balancer controller of the provider, for example, the AWS Load Balancer Controller on Amazon Web Services.
The controller watches the services and provisions, updates and deletes a cloud load balancer for each of them.
You can add the controller as an additional Helm chart in kubernetes/cluster.yaml.
It usually needs cloud permissions, for example, an IAM role for the nodes, and provider-specific annotations on the services.
For the requirements, see the documentation of the controller.
If the Ingress load balancer uses the PROXY protocol, configure the Traefik web and websecure entry points to trust it, using a HelmChartConfig for rke2-traefik in kubernetes/manifests.
Otherwise, Traefik rejects the connections.
3.9 Assigning node roles #
Each node needs a host name and a role: control-plane (server) or worker (agent).
Exactly one control-plane node initializes the cluster.
SUSE Elemental supports two methods to assign the roles: a static node list, available in SUSE Elemental 3.0, and roles provided at boot time, which require SUSE Elemental 3.1 or later.
3.9.1 Static node list #
List the nodes in kubernetes/cluster.yaml and configure the network of each node in the network directory.
On the first boot, each node finds its configuration by matching its MAC address and takes the host name and role from the list.
nodes:
- hostname: aif-cp1.example.com
type: server
init: true 1
- hostname: aif-cp2.example.com
type: server
- hostname: aif-cp3.example.com
type: server
- hostname: aif-gpu1.example.com
type: agent
network:
apiVIP: 192.168.122.10
apiHost: api.aif.example.comIf no node has |
For every node, create a nmstate file named after its host name, for example, network/aif-cp1.example.com.yaml:
interfaces:
- name: eth0
type: ethernet
state: up
mac-address: 52:54:00:00:00:01
ipv4:
enabled: true
address:
- ip: 192.168.122.21
prefix-length: 24
ipv6:
enabled: false
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.122.1
next-hop-interface: eth0
dns-resolver:
config:
server:
- 192.168.122.1The files are processed on the first boot by the NetworkManager Configurator (nmc), which matches them to the node by the mac-address value.
The interface name in the file, eth0 in this example, is only a logical name.
If the interface has a different name on the node, for example, a predictable name such as enp1s0, nmc uses the actual name.
Specify the MAC address of every Ethernet interface of the node.
The same image is used for the whole cluster, but adding or removing a node requires rebuilding it.
For network setups that nmstate files cannot express, provide a network/configure-network.sh script instead.
Do not combine nmstate files and the script in the same directory.
3.9.2 Roles provided at boot time #
With SUSE Elemental 3.0, use the static node list.
Omit the nodes list and the network directory, and provide the role of every node when you create it, as Ignition user data.
The image does not depend on the set of nodes, so you can add nodes without rebuilding it.
This method is required on cloud providers and is useful with provisioning tools such as Metal3.
The per-node Ignition configuration must write the following files:
/etc/hostnamewith the host name of the node./var/lib/elemental/runtime.envwith the role:NODETYPE=server # server or agent IS_INIT_NODE=true # true only on the node that initializes the cluster
The following Butane example creates this configuration for the initializing node:
variant: fcos
version: 1.6.0
storage:
files:
- path: /etc/hostname
mode: 420
overwrite: true
contents:
inline: aif-cp1
- path: /var/lib/elemental/runtime.env
mode: 420
overwrite: true
contents:
inline: |
NODETYPE=server
IS_INIT_NODE=trueConvert it to Ignition and pass the result as the user data of the instance:
> butane --strict --pretty aif-cp1.bu > aif-cp1.ignThe per-node configuration is merged with the configuration from butane.yaml that is stored in the image.
You can also use it for other node-specific settings, for example, the node-ip or node labels in /etc/rancher/rke2/config.yaml.d/.
3.10 Building the image #
Run elemental3 customize from the SUSE Elemental container image.
The configuration directory is mounted at /config, and the image is written there.
> sudo podman run --rm -it --network host \
-v $PWD/aif-elemental:/config:Z \
-v $PWD/cache:/cache:Z \
registry.suse.com/elemental/elemental:3.0 \
customize --type raw --output /config/aif.rawThe command takes several minutes, because it downloads the operating system, RKE2 and the Helm charts.
When it finishes, the configuration directory contains the image and its SHA256 checksum, for example, aif.raw and aif.raw.sha256.
Useful options of elemental3 customize:
--type raw|iso|qcow2RAW disk image or ISO installation media. The
qcow2type, a sparse qcow2 disk image, requires SUSE Elemental 3.1 or later.--outputPath of the image. When running in a container, use a path under
/config, otherwise the image stays inside the container.--mode splitStores the first-boot configuration in a separate directory instead of a partition in the image. Use it when another tool, such as Metal3 or Cluster API, delivers the Ignition configuration.
--overlay node1,node2With
--type qcow2, creates a thin qcow2 overlay for each node on top of the image. Requires SUSE Elemental 3.1 or later.--localUses only the artifacts already in the cache and does not download anything.
--cache auto|offline|refresh|offControls the reuse of downloaded artifacts in
/cache. Requires SUSE Elemental 3.1 or later.--airgapEmbeds the Helm charts and container images into the image for deployments without network access. Requires SUSE Elemental 3.1 or later.
3.10.1 Converting the RAW image to qcow2 #
SUSE Elemental 3.0 does not produce qcow2 images.
For hypervisors and cloud platforms that expect the qcow2 format, convert the RAW image with qemu-img:
Convert the image. The
-coption compresses the output, which is useful when uploading the image to a cloud platform:> qemu-img convert -p -f raw -O qcow2 -c aif-elemental/aif.raw aif.qcow2Optionally, grow the virtual disk to the final size of the node disk. The system partition fills the disk on the first boot:
> qemu-img resize aif.qcow2 100GVerify the result:
> qemu-img info aif.qcow2
To boot several virtual machines from one converted image without copying it, create a thin overlay for each node that uses the image as its backing file. Do not modify or boot the base image after creating the overlays:
> qemu-img create -f qcow2 -F qcow2 -b aif.qcow2 aif-cp1.qcow2 100GBuild the qcow2 image and the per-node overlays directly, without converting:
> sudo podman run --rm -it --network host \
-v $PWD/aif-elemental:/config:Z \
<ELEMENTAL_3.1_IMAGE> \
customize --type qcow2 --output /config/aif.qcow2 \
--overlay aif-cp1,aif-cp2,aif-cp3The command creates aif.qcow2 and the aif-cp1.qcow2, aif-cp2.qcow2 and aif-cp3.qcow2 overlays next to it.
The virtual disk size is raw.diskSize from install.yaml.
Keep the base image in the same directory as the overlays, because the overlays reference it by a relative path.
3.11 Booting the cluster nodes #
Boot the node that initializes the cluster first, then the other control-plane nodes, then the worker nodes.
- Bare-metal servers
Boot the ISO media from a USB drive, virtual media of the BMC or the network. The installer writes the image to the disk set in
iso.deviceand reboots. Alternatively, write the RAW image directly to the disk.- Virtual machines
Create a UEFI virtual machine from a qcow2 copy of the image, converted as described in Section 3.10.1, “Converting the RAW image to qcow2”, for example:
> sudo qemu-img convert -p -f raw -O qcow2 aif-elemental/aif.raw /var/lib/libvirt/images/aif-cp1.qcow2 > sudo qemu-img resize /var/lib/libvirt/images/aif-cp1.qcow2 100G > sudo virt-install --name aif-cp1 --ram 16384 --vcpus 4 \ --disk path=/var/lib/libvirt/images/aif-cp1.qcow2,format=qcow2 \ --network network=default,model=virtio,mac=52:54:00:00:00:01 \ --osinfo detect=on,name=sle-unknown --boot uefi --import \ --graphics none --console pty,target_type=serial- Cloud providers (SUSE Elemental 3.1 or later)
Import the RAW image, or the converted qcow2 image if the platform requires it, as a custom image or snapshot, create the load balancers, and create the instances with the per-node Ignition configuration as user data.
3.12 Verifying the deployment #
Log in to a control-plane node and check that all nodes are
Ready:# kubectl get nodesCheck that the Helm charts are installed. Each chart has an install job in the
kube-systemnamespace:# kubectl get helmcharts -n kube-system # kubectl get jobs -n kube-systemCheck that SUSE AI Factory Operator is running:
# kubectl get pods -n aif-operatorOpen the Rancher Prime host name in a browser and log in with the bootstrap password.
If the cluster does not come up, check the following systemd units on the nodes:
# journalctl -b -t ignition # first-boot configuration (initrd)
# journalctl -u k8s-config-installer # RKE2 configuration
# journalctl -u rke2-server # or rke2-agent
# journalctl -u k8s-resource-installer # Helm charts and manifestsAfter the deployment, continue with Chapter 4, Are there any post-deployment steps?.
3.13 Example deployments #
The following examples show complete configuration directories for a typical SUSE AI Factory setup with two highly available clusters:
A management cluster with Rancher Prime and SUSE AI Factory Operator.
A downstream cluster with cert-manager, the NVIDIA GPU Operator and SUSE Storage, which runs the AI workloads.
Both examples use only SUSE Elemental 3.0 features: a static node list, the embedded MetalLB load balancer for the Kubernetes API, and a second MetalLB address for the traefik Ingress controller.
The files in this section, and in the rest of this chapter, are examples that show one possible configuration. Replace the IP addresses, host names, MAC addresses and credentials with your own values.
SUSE Elemental 3 can do much more than shown here. You can add any Helm chart or Kubernetes manifest, configure bonds, VLANs and other network setups with nmstate, add disks, users, files and systemd units with Butane, run custom first-boot scripts, use a private registry, and more. For all options, see the SUSE Elemental 3 documentation.
3.13.1 How is the Ingress configured? #
RKE2 deploys the traefik Ingress controller as a DaemonSet that listens on ports 80 and 443 of every node.
To give the Ingress a single highly available address, both examples expose traefik through a LoadBalancer service with a dedicated MetalLB address:
A
-priority.yamlmanifest with aHelmChartConfigforrke2-traefikenables theLoadBalancerservice and requests the {Ingress} address. It must be applied before RKE2 installs traefik.A second manifest creates the MetalLB
IPAddressPoolandL2Advertisementfor the Ingress address. It is applied after the Helm charts, when the MetalLB CRDs exist. The pool hasautoAssign: false, so the address is only given to the service that requests it.
The Ingress address must be an unused address in the node subnet, different from the API VIP and outside the DHCP range. Create a DNS record for the Rancher Prime host name, or a wildcard record for all applications, that points to the Ingress address.
With SUSE Elemental 3.1 or later and an external load balancer, do not use MetalLB for the Ingress.
If a load balancer controller of the cloud provider is installed, expose traefik as a LoadBalancer service with the annotations of the controller, and the controller creates the load balancer, as described in Section 3.8.2, “Using a cloud load balancer”.
If you create the Ingress load balancer yourself, pin traefik to the nodes behind the load balancer with a nodeSelector, and if the load balancer uses the PROXY protocol, add its source network to ports.web.proxyProtocol.trustedIPs and ports.websecure.proxyProtocol.trustedIPs in the HelmChartConfig.
3.13.2 Management cluster with Rancher Prime and SUSE AI Factory Operator #
The management cluster has three control-plane nodes and no worker nodes. It uses the following values:
| Item | Value |
|---|---|
Nodes |
|
API VIP and host name |
|
Ingress address |
|
Rancher Prime host name |
|
mgmt/
├── release.yaml
├── install.yaml
├── butane.yaml
├── network/
│ ├── mgmt1.aif.example.com.yaml
│ ├── mgmt2.aif.example.com.yaml
│ └── mgmt3.aif.example.com.yaml
└── kubernetes/
├── cluster.yaml
├── config/
│ └── server.yaml
├── helm/
│ └── values/
│ ├── rancher.yaml
│ └── aif-operator.yaml
└── manifests/
├── traefik-priority.yaml
└── ingress-ip.yamlrelease.yaml enables only Rancher Prime and SUSE AI Factory Operator.
cert-manager is added automatically as a dependency of Rancher Prime:
manifestURI: oci://ghcr.io/suse/aif-uc-release-manifest:2.4.0
components:
kubernetes: {}
helm:
- chart: rancher
valuesFile: rancher.yaml
- chart: aif-operator
valuesFile: aif-operator.yamlinstall.yaml:
schema: v0
bootloader: grub
kernelCmdLine: "console=tty0 console=ttyS0,115200"
raw:
diskSize: 20Gbutane.yaml creates the users, as described in Section 3.6, “Configuring users, SSH access and files (butane.yaml)”:
variant: fcos
version: 1.6.0
passwd:
users:
- name: root
password_hash: "<ROOT_PASSWORD_HASH>"
- name: suse
password_hash: "<USER_PASSWORD_HASH>"
shell: /bin/bash
ssh_authorized_keys:
- "<SSH_PUBLIC_KEY>"
systemd:
units:
- name: sshd.service
enabled: true
storage:
filesystems:
- device: /dev/disk/by-partlabel/SYSTEM
format: btrfs
wipe_filesystem: false
path: /home
mount_options:
- subvol=/@/home
files:
- path: /etc/ssh/sshd_config.d/sshd.conf
mode: 384
overwrite: true
contents:
inline: |
PermitRootLogin no
AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys.d/ignitionkubernetes/cluster.yaml lists the nodes and the API VIP:
nodes:
- hostname: mgmt1.aif.example.com
type: server
init: true
- hostname: mgmt2.aif.example.com
type: server
- hostname: mgmt3.aif.example.com
type: server
network:
apiVIP: 192.168.122.10
apiHost: api.mgmt.aif.example.comnetwork/mgmt1.aif.example.com.yaml.
Create the same file for mgmt2 and mgmt3 with their MAC and IP addresses:
interfaces:
- name: eth0
type: ethernet
state: up
mac-address: 52:54:00:00:00:01
ipv4:
enabled: true
address:
- ip: 192.168.122.21
prefix-length: 24
ipv6:
enabled: false
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.122.1
next-hop-interface: eth0
dns-resolver:
config:
server:
- 192.168.122.1kubernetes/config/server.yaml:
token: "<RKE2_TOKEN>"
selinux: true
ingress-controller: traefik
write-kubeconfig-group: suse
write-kubeconfig-mode: "0640"
tls-san:
- api.mgmt.aif.example.comkubernetes/helm/values/rancher.yaml runs three Rancher Prime replicas, one on each node:
hostname: rancher.aif.example.com
bootstrapPassword: "<RANCHER_BOOTSTRAP_PASSWORD>"
replicas: 3kubernetes/helm/values/aif-operator.yaml:
credentials:
applicationCollection:
username: "<APPCO_USERNAME>"
password: "<APPCO_TOKEN>"
suseRegistry:
username: "regcode"
password: "<REGISTRATION_CODE>"
nvidia:
username: "$oauthtoken"
password: "<NGC_API_KEY>"kubernetes/manifests/traefik-priority.yaml exposes traefik on the Ingress address:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-traefik
namespace: kube-system
spec:
valuesContent: |-
service:
enabled: true
type: LoadBalancer
annotations:
metallb.io/loadBalancerIPs: 192.168.122.11
spec:
externalTrafficPolicy: Local 1
providers:
kubernetesIngress:
publishedService:
enabled: true 2Preserves the client IP address. MetalLB announces the address from a node that runs a traefik pod. | |
Writes the Ingress address to the status of the |
kubernetes/manifests/ingress-ip.yaml creates the MetalLB pool for the Ingress address:
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: ingress-ip
namespace: metallb-system
spec:
addresses:
- 192.168.122.11/32
autoAssign: false
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: ingress-ip-l2-adv
namespace: metallb-system
spec:
ipAddressPools:
- ingress-ipBuild the image as described in Section 3.10, “Building the image” and boot mgmt1 first, then mgmt2 and mgmt3.
When the cluster is ready, open https://rancher.aif.example.com.
3.13.3 Downstream cluster with NVIDIA GPU Operator and SUSE Storage #
The downstream cluster has three control-plane nodes and two GPU worker nodes. It does not run Rancher Prime. After the deployment, you import it into the Rancher Prime instance of the management cluster.
| Item | Value |
|---|---|
Control-plane nodes |
|
GPU worker nodes |
|
API VIP and host name |
|
Ingress address |
|
downstream/
├── release.yaml
├── install.yaml
├── butane.yaml
├── network/
│ ├── cp1.aif.example.com.yaml
│ ├── ...
│ └── gpu2.aif.example.com.yaml
└── kubernetes/
├── cluster.yaml
├── config/
│ ├── server.yaml
│ └── agent.yaml
└── manifests/
├── appco-pull-secret-priority.yaml
├── traefik-priority.yaml
└── ingress-ip.yamlrelease.yaml enables cert-manager, the NVIDIA GPU Operator and SUSE Storage with its system extension:
manifestURI: oci://ghcr.io/suse/aif-uc-release-manifest:2.4.0
components:
kubernetes: {}
systemd:
- extension: suse-storage
helm:
- chart: cert-manager
- chart: gpu-operator
- chart: suse-storage
credentials:
username: "<APPCO_USERNAME>"
password: "<APPCO_TOKEN>"install.yaml is the same as for the management cluster.
Give the nodes enough disk space, because SUSE Storage stores the volumes in /var/lib/longhorn on the system disk.
butane.yaml creates the same users as for the management cluster and adds the iscsi-prep unit that SUSE Storage needs, as described in Section 3.6.2, “Preparing nodes for SUSE Storage”.
Add the following entries to the systemd.units and storage.files lists:
systemd:
units:
- name: iscsi-prep.service
enabled: true
contents: |
[Unit]
Description=Seed /etc for the open-iscsi system extension and start iscsid
Wants=systemd-sysext.service
After=systemd-sysext.service local-fs.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/bash /var/lib/elemental/iscsi-prep.sh
[Install]
WantedBy=multi-user.target
storage:
files:
- path: /var/lib/elemental/iscsi-prep.sh
mode: 493
overwrite: true
contents:
inline: |
#!/usr/bin/env bash
set -euo pipefail
# Wait until the system extension is merged.
for _ in $(seq 1 30); do
[ -x /usr/sbin/iscsid ] && break
sleep 2
done
[ -x /usr/sbin/iscsid ] || exit 0
mkdir -p /etc/iscsi
if [ ! -s /etc/iscsi/initiatorname.iscsi ]; then
if command -v iscsi-iname >/dev/null 2>&1; then
name="$(iscsi-iname)"
else
name="iqn.2016-04.com.open-iscsi:$(hostname)"
fi
echo "InitiatorName=${name}" > /etc/iscsi/initiatorname.iscsi
chmod 0600 /etc/iscsi/initiatorname.iscsi
fi
if [ ! -s /etc/iscsi/iscsid.conf ]; then
printf '%s\n' \
'iscsid.startup = /bin/systemctl start iscsid.socket' \
'node.startup = manual' \
'node.session.timeo.replacement_timeout = 120' \
> /etc/iscsi/iscsid.conf
chmod 0600 /etc/iscsi/iscsid.conf
fi
systemctl start --no-block iscsid.servicekubernetes/cluster.yaml:
nodes:
- hostname: cp1.aif.example.com
type: server
init: true
- hostname: cp2.aif.example.com
type: server
- hostname: cp3.aif.example.com
type: server
- hostname: gpu1.aif.example.com
type: agent
- hostname: gpu2.aif.example.com
type: agent
network:
apiVIP: 192.168.122.30
apiHost: api.gpu.aif.example.comCreate one nmstate file in network/ for each of the five nodes, as for the management cluster.
kubernetes/config/server.yaml and kubernetes/config/agent.yaml:
# server.yaml
token: "<RKE2_TOKEN>"
selinux: true
ingress-controller: traefik
write-kubeconfig-group: suse
write-kubeconfig-mode: "0640"
tls-san:
- api.gpu.aif.example.com# agent.yaml
token: "<RKE2_TOKEN>"
selinux: truekubernetes/manifests/appco-pull-secret-priority.yaml creates the longhorn-system namespace and the application-collection image pull secret for the SUSE Storage images, as shown in Section 3.7.3.1, “Image pull secret for SUSE Application Collection”.
kubernetes/manifests/traefik-priority.yaml runs traefik on the control-plane nodes only, so the GPU nodes do not serve Ingress traffic:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-traefik
namespace: kube-system
spec:
valuesContent: |-
nodeSelector:
node-role.kubernetes.io/control-plane: "true"
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: CriticalAddonsOnly
operator: Exists
service:
enabled: true
type: LoadBalancer
annotations:
metallb.io/loadBalancerIPs: 192.168.122.31
spec:
externalTrafficPolicy: Local
providers:
kubernetesIngress:
publishedService:
enabled: truekubernetes/manifests/ingress-ip.yaml is the same as for the management cluster, with the 192.168.122.31/32 address.
Build the image and boot cp1 first, then the other nodes.
When the cluster is ready, import it into Rancher Prime on the management cluster as described in Section 1.6, “Connect the downstream cluster to Rancher Prime”, and assign the GPU nodes as described in Section 1.7, “Assign GPU resources”.
For a test cluster, you can use the local path provisioner instead of SUSE Storage. Volumes are stored on the node of the pod and are lost with that node.
In
release.yaml, remove thesuse-storageextension and chart, and add the chart without a values file:- chart: local-path-provisioner credentials: username: "<APPCO_USERNAME>" password: "<APPCO_TOKEN>"Remove the
iscsi-prepunit and script frombutane.yaml.In
kubernetes/manifests/appco-pull-secret-priority.yaml, change the namespace fromlonghorn-systemtolocal-path-provisioner, as described in Section 3.7.3.1, “Image pull secret for SUSE Application Collection”.Create the
/opt/local-path-provisionerdirectory with the SELinux label for containers, as described in Section 3.6.1, “SELinux considerations”.