4 Preparing the cluster for AI Library #

In this section, you will provision the underlying infrastructure for your Rancher management server in an air gapped environment. You will also set up the private container image registry that must be available to your Rancher node(s).

In this section, you will provision the underlying infrastructure for your Rancher management server in an air-gapped environment. You will also set up the private container image registry that must be available to your Rancher node(s). The procedures below focus on installing Rancher in the RKE2 cluster. To install the Rancher management server on a high-availability SUSE Rancher Prime: RKE2 cluster, we recommend setting up the following infrastructure:

These nodes must be in the same region or data center. You may place these servers in separate availability zones.

4.1.2.3 Set up the load balancer #

 

You will also need to set up a load balancer to direct traffic to the Rancher replica on both nodes. That will prevent the outage of any single node from taking down communications to the Rancher management server. When Kubernetes gets set up in a later step, the RKE2 tool will deploy an NGINX Ingress controller. This controller will listen on ports 80 and 443 of the worker nodes, answering traffic destined for specific hostnames. When Rancher is installed (also in a later step), the Rancher system creates an Ingress resource. That Ingress tells the NGINX Ingress controller to listen for traffic destined for the Rancher host name. The NGINX Ingress controller, when receiving traffic destined for the Rancher host name, will forward that traffic to the running Rancher pods in the cluster. For your implementation, consider if you want or need to use a Layer-4 or Layer-7 load balancer:

• A layer-4 load balancer is the simpler of the two choices, in which you are forwarding TCP traffic to your nodes. We recommend configuring your load balancer as a Layer 4 balancer, forwarding traffic on ports TCP/80 and TCP/443 to the Rancher management cluster nodes. The Ingress controller on the cluster will redirect HTTP traffic to HTTPS and terminate SSL/TLS on port TCP/443. The Ingress controller will forward traffic on port TCP/80 to the Ingress pod in the Rancher deployment.

• A layer-7 load balancer is a bit more complicated but can offer features that you may want. For instance, a layer-7 load balancer is capable of handling TLS termination at the load balancer, as opposed to Rancher doing TLS termination itself. This can be beneficial to centralize your TLS termination in your infrastructure. Layer-7 load balancing also allows your load balancer to make decisions based on HTTP attributes such as cookies—capabilities that a layer-4 load balancer cannot handle. If you decide to terminate the SSL/TLS traffic on a layer-7 load balancer, you will need to use the --set tls=external option when installing Rancher in a later step. For more information, refer to the Rancher Helm chart options.

For an example showing how to set up an NGINX load balancer, refer to this page. For a how-to guide for setting up an Amazon ELB Network Load Balancer, refer to this page.

Important

Do not use this load balancer (that is, the local cluster Ingress) to load balance applications other than Rancher following installation. Sharing this Ingress with other applications may result in WebSocket errors to Rancher following Ingress configuration reloads for other apps. We recommend dedicating the local cluster to Rancher and no other applications.

This section describes how to set up your private registry so that when you install Rancher, Rancher will pull all the required images from this registry.

This section describes how to set up your private registry so that when you install SUSE Rancher Prime, it will pull all the required images from this registry.

By default, all images used to provision Kubernetes clusters or launch any tools in SUSE Rancher Prime, e.g., monitoring, pipelines or alerts, are pulled from Docker Hub.

In an air-gapped installation of SUSE Rancher Prime, you will need a private registry that is located somewhere accessible by your SUSE Rancher Prime server.

You will then load every image into the registry. Populating the private registry with images is the same process for installing SUSE Rancher Prime with Docker and for installing SUSE Rancher Prime on a Kubernetes cluster.

The following steps populate your private registry.

Prerequisites. These steps expect you to use a Linux workstation that has Internet access, access to your private registry, and at least 20 GB of disk space.

If you use ARM64 hosts, the registry must support manifests. As of April 2020, Amazon Elastic Container Registry does not support manifests.

4.1.3.2 Collect the cert-manager image #

 

Note

Skip this step if you are using your own certificates, or if you are terminating TLS on an external load balancer.

In a Kubernetes Install, if you elect to use the Rancher default self-signed TLS certificates, you must add the cert-manager image to rancher-images.txt as well.

1. Fetch the latest cert-manager Helm chart and parse the template for image details:

   

   Note

   Recent changes to cert-manager require an upgrade. If you are upgrading SUSE Rancher Prime and using a version of cert-manager older than v0.12.0, please see our upgrade documentation.

   helm repo add jetstack https://charts.jetstack.io
   helm repo update
   helm fetch jetstack/cert-manager
   helm template ./cert-manager-version.tgz | \
     awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt

2. Sort the image list and deduplicate it to remove any overlap between the sources:

   > sort -u rancher-images.txt -o rancher-images.txt

{prompt_user}chmod +x rancher-save-images.sh

{prompt_user}./rancher-save-images.sh --image-list ./rancher-images.txt

This section describes how to install a Kubernetes cluster according to our best practices for the SUSE Rancher Prime server environment. This cluster should be dedicated to running only the SUSE Rancher Prime server.

Note

Skip this section if you are installing SUSE Rancher Prime on a single node with Docker.

Rancher can be installed on any Kubernetes cluster, including hosted Kubernetes providers.

The steps to set up an air-gapped Kubernetes cluster on RKE, RKE2, or K3s are shown below.

In this guide, we are assuming you have created your nodes in your air-gapped environment and have a secure Docker private registry on your bastion server.

== 1. Create RKE2 configuration

Create the config.yaml file at /etc/rancher/rke2/config.yaml. This will contain all the configuration options necessary to create a highly available RKE2 cluster.

On the first server the minimum configuration is:

token: my-shared-secret
tls-san:
  - loadbalancer-dns-domain.com

On each other server the configuration file should contain the same token and tell RKE2 to connect to the existing first server:

server: https://ip-of-first-server:9345
token: my-shared-secret
tls-san:
  - loadbalancer-dns-domain.com

For more information, refer to the RKE2 documentation.

Note

RKE2 additionally provides a resolv-conf option for kubelets, which may help with configuring DNS in air-gap networks.

== 2. Create Registry YAML

Create the registries.yaml file at /etc/rancher/rke2/registries.yaml. This will tell RKE2 the necessary details to connect to your private registry.

The registries.yaml file should look like this before plugging in the necessary information:

---
mirrors:
  customreg:
    endpoint:
      - "https://ip-to-server:5000"
configs:
  customreg:
    auth:
      username: xxxxxx # this is the registry username
      password: xxxxxx # this is the registry password
    tls:
      cert_file: <path to the cert file used in the registry>
      key_file:  <path to the key file used in the registry>
      ca_file: <path to the ca file used in the registry>

For more information on private registries configuration file for RKE2, refer to the RKE2 documentation.

== 3. Install RKE2

Rancher needs to be installed on a supported Kubernetes version. To find out which versions of Kubernetes are supported for your Rancher version, refer to the SUSE Rancher Support Matrix.

Download the install script, rke2, rke2-images, and sha256sum archives from the release and upload them into a directory on each server:

mkdir /tmp/rke2-artifacts && cd /tmp/rke2-artifacts/
wget https://github.com/rancher/rke2/releases/download/v1.21.5%2Brke2r2/rke2-images.linux-amd64.tar.zst
wget https://github.com/rancher/rke2/releases/download/v1.21.5%2Brke2r2/rke2.linux-amd64.tar.gz
wget https://github.com/rancher/rke2/releases/download/v1.21.5%2Brke2r2/sha256sum-amd64.txt
curl -sfL https://get.rke2.io --output install.sh

Next, run install.sh using the directory on each server, as in the example below:

INSTALL_RKE2_ARTIFACT_PATH=/tmp/rke2-artifacts sh install.sh

Then enable and start the service on all servers:

` systemctl enable rke2-server.service systemctl start rke2-server.service `

For more information, refer to the RKE2 documentation.

== 4. Save and Start Using the kubeconfig File

When you installed RKE2 on each Rancher server node, a kubeconfig file was created on the node at /etc/rancher/rke2/rke2.yaml. This file contains credentials for full access to the cluster, and you should save this file in a secure location.

To use this kubeconfig file,

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [CERTIFICATE-DATA]
    server: [LOAD-BALANCER-DNS]:6443 # Edit this line
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    password: [PASSWORD]
    username: admin

Result: You can now use kubectl to manage your RKE2 cluster. If you have more than one kubeconfig file, you can specify which one you want to use by passing in the path to the file when using kubectl:

kubectl --kubeconfig ~/.kube/config/rke2.yaml get pods --all-namespaces

For more information about the kubeconfig file, refer to the RKE2 documentation or the official Kubernetes documentation about organizing cluster access using kubeconfig files.

== Note on Upgrading

Upgrading an air-gap environment can be accomplished in the following manner:

This section is about how to deploy SUSE Rancher Prime for your air gapped environment in a high-availability Kubernetes installation. An air gapped environment could be where the SUSE Rancher Prime server will be installed offline, behind a firewall, or behind a proxy.

4.1.5.2 Kubernetes Instructions #

 

SUSE Rancher Prime recommends installing SUSE Rancher Prime on a Kubernetes cluster. A highly available Kubernetes install is comprised of three nodes running the SUSE Rancher Prime server components on a Kubernetes cluster. The persistence layer (etcd) is also replicated on these three nodes, providing redundancy and data duplication in case one of the nodes fails.

4.1.5.2.1 1. Add the Helm Chart Repository #

 

From a system that has access to the internet, fetch the latest Helm chart and copy the resulting manifests to a system that has access to the SUSE Rancher Prime server cluster.

1. If you haven’t already, install helm locally on a workstation that has internet access. Note: Refer to the Helm version requirements to choose a version of Helm to install SUSE Rancher Prime.

2. Use the helm repo add command to add the Helm chart repository that contains charts to install SUSE Rancher Prime.

   helm repo add rancher-prime <helm-chart-repo-url>

3. Fetch the SUSE Rancher Prime chart. This will pull down the chart and save it in the current directory as a .tgz file.

   • To fetch the latest version:

     helm fetch rancher-prime/rancher

   • To fetch a specific version:

     Check to see which version of SUSE Rancher Prime are available.

     helm search repo --versions rancher-prime

     Fetch a specific version by specifying the --version parameter:

     helm fetch rancher-prime/rancher --version=<version>

4.1.5.2.2 2. Choose your SSL Configuration #

 

The SUSE Rancher Prime server is designed to be secure by default and requires SSL/TLS configuration.

When SUSE Rancher Prime is installed on an air gapped Kubernetes cluster, there are two recommended options for the source of the certificate.

Note

If you want terminate SSL/TLS externally, see TLS termination on an External Load Balancer.

Configuration	Chart option	Description	Requires cert-manager
Rancher Generated Self-Signed Certificates	ingress.tls.source=rancher	Use certificates issued by Rancher’s generated CA (self signed) This is the default and does not need to be added when rendering the Helm template.	yes
Certificates from Files	ingress.tls.source=secret	Use your own certificate files by creating Kubernetes Secret(s). This option must be passed when rendering the Rancher Helm template.	no

4.1.5.2.3 Helm Chart Options for Air Gap Installations #

 

When setting up the Rancher Helm template, there are several options in the Helm chart that are designed specifically for air gap installations.

Chart Option	Chart Value	Description
certmanager.version	<version>	Configure proper SUSE Rancher Prime TLS issuer depending of running cert-manager version.
systemDefaultRegistry	<REGISTRY.YOURDOMAIN.COM:PORT>	Configure SUSE Rancher Prime server to always pull from your private registry when provisioning clusters.
useBundledSystemChart	true	Configure SUSE Rancher Prime server to use the packaged copy of Helm system charts. The system charts repository contains all the catalog items required for features such as monitoring, logging, alerting and global DNS. These Helm charts are located in GitHub, but since you are in an air gapped environment, using the charts that are bundled within SUSE Rancher Prime is much easier than setting up a Git mirror.

4.1.5.2.4 3. Fetch the Cert-Manager Chart #

 

Based on the choice your made in 2. Choose your SSL Configuration (Section 4.1.5.2.2, “2. Choose your SSL Configuration”), complete one of the procedures below.

4.1.5.2.4.1 Option A: Default Self-Signed Certificate #

 

By default, SUSE Rancher Prime generates a CA and uses cert-manager to issue the certificate for access to the SUSE Rancher Prime server interface.

Note

Recent changes to cert-manager require an upgrade. If you are upgrading SUSE Rancher Prime and using a version of cert-manager older than v0.11.0, please see our upgrade cert-manager documentation.

4.1.5.2.4.1.1 1. Add the cert-manager Repo #

 

From a system connected to the internet, add the cert-manager repo to Helm:

helm repo add jetstack https://charts.jetstack.io
helm repo update

4.1.5.2.4.1.2 2. Fetch the cert-manager Chart #

 

Fetch the latest cert-manager chart available from the Helm chart repository.

helm fetch jetstack/cert-manager --version v1.11.0

4.1.5.2.4.1.3 3. Retrieve the cert-manager CRDs #

 

Download the required CRD file for cert-manager:

   curl -L -o cert-manager-crd.yaml https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml

4.1.5.2.5 4. Install SUSE Rancher Prime #

 

Copy the fetched charts to a system that has access to the SUSE Rancher Prime server cluster to complete installation.

4.1.5.2.5.1 1. Install cert-manager #

 

Install cert-manager with the same options you would use to install the chart. Remember to set the image.repository option to pull the image from your private registry.

Note

To see options on how to customize the cert-manager install (including for cases where your cluster uses PodSecurityPolicies), see the cert-manager docs.

Example 4.1: Click to expand #

 

If you are using self-signed certificates, install cert-manager:

1. Create the namespace for cert-manager.

    kubectl create namespace cert-manager

2. Create the cert-manager CustomResourceDefinitions (CRDs).

    kubectl apply -f cert-manager-crd.yaml

3. Install cert-manager.

    helm install cert-manager ./cert-manager-v1.11.0.tgz \
        --namespace cert-manager \
        --set image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-controller \
        --set webhook.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-webhook \
        --set cainjector.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-cainjector \
        --set startupapicheck.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-ctl

4.1.5.2.5.2 2. Install Rancher #

 

First, refer to Adding TLS Secrets to publish the certificate files so SUSE Rancher Prime and the ingress controller can use them.

Then, create the namespace for SUSE Rancher Prime using kubectl:

kubectl create namespace cattle-system

Next, install SUSE Rancher Prime, declaring your chosen options. Use the reference table below to replace each placeholder. SUSE Rancher Prime needs to be configured to use the private registry in order to provision any SUSE Rancher Prime launched Kubernetes clusters or SUSE Rancher Prime tools.

Placeholder	Description
<VERSION>	The version number of the output tarball.
<RANCHER.YOURDOMAIN.COM>	The DNS name you pointed at your load balancer.
<REGISTRY.YOURDOMAIN.COM:PORT>	The DNS name for your private registry.
<CERTMANAGER_VERSION>	Cert-manager version running on k8s cluster.

   helm install rancher ./rancher-<VERSION>.tgz \
    --namespace cattle-system \
    --set hostname=<RANCHER.YOURDOMAIN.COM> \
    --set certmanager.version=<CERTMANAGER_VERSION> \
    --set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
    --set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Set a default private registry to be used in Rancher
    --set useBundledSystemChart=true # Use the packaged Rancher system charts

Optional: To install a specific SUSE Rancher Prime version, set the rancherImageTag value, example: --set rancherImageTag=v2.5.8

4.1.5.2.5.3 Option B: Certificates From Files Using Kubernetes Secrets #

 

4.1.5.2.5.3.1 1. Create Secrets #

 

Create Kubernetes secrets from your own certificates for SUSE Rancher Prime to use. The common name for the cert will need to match the host name option in the command below, or the ingress controller will fail to provision the site for SUSE Rancher Prime.

4.1.5.2.5.3.2 2. Install Rancher #

 

Install SUSE Rancher Prime, declaring your chosen options. Use the reference table below to replace each placeholder. SUSE Rancher Prime needs to be configured to use the private registry in order to provision any SUSE Rancher Prime launched Kubernetes clusters or SUSE Rancher Prime tools.

Placeholder	Description
<VERSION>	The version number of the output tarball.
<RANCHER.YOURDOMAIN.COM>	The DNS name you pointed at your load balancer.
<REGISTRY.YOURDOMAIN.COM:PORT>	The DNS name for your private registry.

   helm install rancher ./rancher-<VERSION>.tgz \
    --namespace cattle-system \
    --set hostname=<RANCHER.YOURDOMAIN.COM> \
    --set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
    --set ingress.tls.source=secret \
    --set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Set a default private registry to be used in Rancher
    --set useBundledSystemChart=true # Use the packaged Rancher system charts

If you are using a Private CA signed cert, add --set privateCA=true following --set ingress.tls.source=secret:

   helm install rancher ./rancher-<VERSION>.tgz \
    --namespace cattle-system \
    --set hostname=<RANCHER.YOURDOMAIN.COM> \
    --set rancherImage=<REGISTRY.YOURDOMAIN.COM:PORT>/rancher/rancher \
    --set ingress.tls.source=secret \
    --set privateCA=true \
    --set systemDefaultRegistry=<REGISTRY.YOURDOMAIN.COM:PORT> \ # Set a default private registry to be used in Rancher
    --set useBundledSystemChart=true # Use the packaged Rancher system charts

The installation is complete.

To expose the GPU to the pod correctly, the NVIDIA kernel drivers and the libnvidia-ml library must be correctly installed in the host OS. The NVIDIA Operator can automatically install drivers and libraries on some operating systems. Refer to the NVIDIA documentation for information on supported operating system releases. The installation of the NVIDIA components on your host OS is out of the scope of this documentation, refer to the NVIDIA documentation for instructions.

The following three commands should return a correct output if the kernel driver was correctly installed:

Once the OS is ready and RKE2 is running, install the GPU Operator with the following yaml manifest. Use the kubectl apply -f gpu-operator.yaml command to install the operator.

Caution

The NVIDIA operator restarts containerd with a hangup call which restarts RKE2.

Important

The envvars ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED, ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS and DEVICE_LIST_STRATEGY are required to properly isolate GPU resources as explained in Preventing unprivileged access to GPUs in Kubernetes.

# gpu-operator.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: gpu-operator
  namespace: kube-system
spec:
  repo: https://helm.ngc.nvidia.com/nvidia
  chart: gpu-operator
  version: v25.3.4
  targetNamespace: gpu-operator
  createNamespace: true
  valuesContent: |-
    driver:
      enabled: "false"
    toolkit:
      env:
      - name: CONTAINERD_SOCKET
        value: /run/k3s/containerd/containerd.sock
      - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
        value: "false"
      - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS
        value: "true"
    devicePlugin:
      env:
      - name: DEVICE_LIST_STRATEGY
        value: volume-mounts

After one minute approximately, you can perform the following checks to verify that everything is working as expected:

Important: Version Gate

Available as of October 2024 releases: v1.28.15+rke2r1, v1.29.10+rke2r1, v1.30.6+rke2r1, v1.31.2+rke2r1.

RKE2 will now use PATH to find alternative container runtimes, in addition to checking the default paths used by the container runtime packages. In order to use this feature, you must modify the RKE2 service’s PATH environment variable to add the directories containing the container runtime binaries.

It’s recommended that you modify one of this two environment files:

This example will add the PATH in /etc/default/rke2-server:

Caution

PATH changes should be done with care to avoid placing untrusted binaries in the path of services that run as root.

echo PATH=$PATH >> /etc/default/rke2-server

Caution

The NO_PROXY environment variable is not standardized, and the accepted format of the value can differ between applications. When configuring the NO_PROXY variable for Rancher, the value must adhere to the format expected by Golang.

Specifically, the value should be a comma-delimited string which only contains IP addresses, CIDR notation, domain names, or special DNS labels (e.g. *). For a full description of the expected value format, refer to the upstream Golang documentation.

Result:

Note

You can not re-register a cluster that is currently active in a Rancher setup.

4.3.2.1 Configuring an Imported EKS, AKS or GKE Cluster with Terraform #

 

You should define only the minimum fields that Rancher requires when importing an EKS, AKS or GKE cluster with Terraform. This is important as Rancher will overwrite what was in the cluster configuration with any configuration that the user has provided.

Caution

Even a small difference between the current cluster and a user-provided configuration could have unexpected results.

The minimum configuration fields required by Rancher to import EKS clusters with Terraform using eks_config_v2 are as follows:

• cloud_credential_id

• name

• region

• imported (this field should always be set to true for imported clusters)

Example YAML configuration for imported EKS clusters:

resource "rancher2_cluster" "my-eks-to-import" {
  name        = "my-eks-to-import"
  description = "Terraform EKS Cluster"
  eks_config_v2 {
    cloud_credential_id = rancher2_cloud_credential.aws.id
    name                = var.aws_eks_name
    region              = var.aws_region
    imported            = true
  }
}

You can find additional examples for other cloud providers in the Rancher2 Terraform Provider documentation.

The control that Rancher has to manage a registered cluster depends on the type of cluster.

Warning

When version management is enabled for an imported cluster, upgrading it outside of Rancher may lead to unexpected consequences.

The version management feature for imported RKE2 and K3s clusters can be configured using one of the following options:

You can define the default behavior for newly created clusters or existing ones set to "Global default" by modifying the imported-cluster-version-management setting.

Changes to the global imported-cluster-version-management setting take effect during the cluster’s next reconciliation cycle.

Note

If version management is enabled for a cluster, Rancher will deploy the system-upgrade-controller app, along with the associated Plans and other required Kubernetes resources, to the cluster. If version management is disabled, Rancher will remove these components from the cluster.

Tip

It is a Kubernetes best practice to back up the cluster before upgrading. When upgrading a high-availability K3s cluster with an external database, back up the database in whichever way is recommended by the relational database provider.

The concurrency is the maximum number of nodes that are permitted to be unavailable during an upgrade. If number of unavailable nodes is larger than the concurrency, the upgrade will fail. If an upgrade fails, you may need to repair or remove failed nodes before the upgrade can succeed.

In the RKE2 and K3s documentation, controlplane nodes are called server nodes. These nodes run the Kubernetes master, which maintains the desired state of the cluster. By default, these controlplane nodes have the capability to have workloads scheduled to them by default.

Also in the RKE2 and K3s documentation, nodes with the worker role are called agent nodes. Any workloads or pods that are deployed in the cluster can be scheduled to these nodes by default.

Nodes are upgraded by the system upgrade controller running in the downstream cluster. Based on the cluster configuration, Rancher deploys two plans to upgrade nodes: one for controlplane nodes and one for workers. The system upgrade controller follows the plans and upgrades the nodes.

To enable debug logging on the system upgrade controller deployment, edit the configmap to set the debug environment variable to true. Then restart the system-upgrade-controller pod.

Logs created by the system-upgrade-controller can be viewed by running this command:

kubectl logs -n cattle-system system-upgrade-controller

The current status of the plans can be viewed with this command:

kubectl get plans -A -o yaml

Tip

If the cluster becomes stuck in upgrading, restart the system-upgrade-controller.

To prevent issues when upgrading, the Kubernetes upgrade best practices should be followed.

Rancher supports Authorized Cluster Endpoints (ACE) for registered RKE2 and K3s clusters. This support includes manual steps you will perform on the downstream cluster to enable the ACE. For additional information on the authorized cluster endpoint, click here.

Note: Notes:

• These steps only need to be performed on the control plane nodes of the downstream cluster. You must configure each control plane node individually.

• The following steps will work on both RKE2 and K3s clusters registered in v2.6.x as well as those registered (or imported) from a previous version of Rancher with an upgrade to v2.6.x.

• These steps will alter the configuration of the downstream RKE2 and K3s clusters and deploy the kube-api-authn-webhook. If a future implementation of the ACE requires an update to the kube-api-authn-webhook, then this would also have to be done manually. For more information on this webhook, click here.

4.3.7.1 Manual steps to be taken on the control plane of each downstream cluster to enable ACE: #

 

1. Create a file at /var/lib/rancher/{rke2,k3s}/kube-api-authn-webhook.yaml with the following contents:

    apiVersion: v1
    kind: Config
    clusters:
    ** name: Default
   cluster:
     insecure-skip-tls-verify: true
     server: http://127.0.0.1:6440/v1/authenticate
    users:
    ** name: Default
   user:
     insecure-skip-tls-verify: true
    current-context: webhook
    contexts:
    ** name: webhook
   context:
     user: Default
     cluster: Default

2. Add the following to the configuration file (or create one if it doesn’t exist); note that the default location is /etc/rancher/{rke2,k3s}/config.yaml:

    kube-apiserver-arg:
        - authentication-token-webhook-config-file=/var/lib/rancher/{rke2,k3s}/kube-api-authn-webhook.yaml

3. Run the following commands:

   sudo systemctl stop {rke2,k3s}-server
   sudo systemctl start {rke2,k3s}-server

4. Finally, you must go back to the Rancher UI and edit the imported cluster there to complete the ACE enablement. Click on ⋮ > Edit Config, then click the Networking tab under Cluster Configuration. Finally, click the Enabled button for Authorized Endpoint. Once the ACE is enabled, you then have the option of entering a fully qualified domain name (FQDN) and certificate information.

Note

The FQDN field is optional, and if one is entered, it should point to the downstream cluster. Certificate information is only needed if there is a load balancer in front of the downstream cluster that is using an untrusted certificate. If you have a valid certificate, then nothing needs to be added to the CA Certificates field.

For all types of registered Kubernetes clusters except for RKE2 and K3s Kubernetes clusters, Rancher doesn’t have any information about how the cluster is provisioned or configured.

Therefore, when Rancher registers a cluster, it assumes that several capabilities are disabled by default. Rancher assumes this in order to avoid exposing UI options to the user even when the capabilities are not enabled in the registered cluster.

However, if the cluster has a certain capability, a user of that cluster might still want to select the capability for the cluster in the Rancher UI. In order to do that, the user will need to manually indicate to Rancher that certain capabilities are enabled for the cluster.

By annotating a registered cluster, it is possible to indicate to Rancher that a cluster was given additional capabilities outside of Rancher.

The following annotation indicates Ingress capabilities. Note that the values of non-primitive objects need to be JSON encoded, with quotations escaped.

"capabilities.cattle.io/ingressCapabilities": "[
  {
    "customDefaultBackend":true,
    "ingressProvider":"asdf"
  }
]"

These capabilities can be annotated for the cluster:

All the capabilities and their type definitions can be viewed in the Rancher API view, at [Rancher Server URL]/v3/schemas/capabilities.

To annotate a registered cluster,

Result: The annotation does not give the capabilities to the cluster, but it does indicate to Rancher that the cluster has those capabilities.

This section lists some of the most common errors that may occur when importing a cluster and provides steps to troubleshoot them.

Error: Getting static credential is not allowed because this cluster is set to disable local accounts.

az aks update --resource-group <resource-group> --name <cluster-name> --enable-local-accounts

To distinguish nodes with the GPU support from non-GPU nodes, Kubernetes uses labels. Labels are used for relevant metadata and should not be confused with annotations that provide simple information about a resource. It is possible to manipulate labels with the kubectl command, as well as by tweaking configuration files from the nodes. If an IaC tool such as Terraform is used, labels can be inserted in the node resource configuration files.

To label a single node, use the following command:

> kubectl label node <GPU_NODE_NAME> accelerator=nvidia-gpu

To achieve the same result by tweaking the node.yaml node configuration, add the following content and apply the changes with kubectl apply -f node.yaml:

apiVersion: v1
kind: Node
metadata:
  name: node-name
  labels:
    accelerator: nvidia-gpu

Tip: Labeling multiple nodes

To label multiple nodes, use the following command:

> kubectl label node \
  <GPU_NODE_NAME1> \
  <GPU_NODE_NAME2> ... \
  accelerator=nvidia-gpu

Tip

If Terraform is being used as an IaC tool, you can add labels to a group of nodes by editing the .tf files and adding the following values to a resource:

resource "node_group" "example" {
  labels = {
    "accelerator" = "nvidia-gpu"
  }
}

To check if the labels are correctly applied, use the following command:

> kubectl get nodes --show-labels

The matching between a container and a node is configured by the explicit resource allocation and the use of labels and node selectors. The use cases described below focus on NVIDIA GPUs.

resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1

nodeSelector:
  accelerator: nvidia-gpu

If the GPU is correctly detected, the Ollama container logs this event:

| [...] source=routes.go:1172 msg="Listening on :11434 (version 0.0.0)"                                              │
│ [...] source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama2502346830/runners                       │
│ [...] source=payload.go:44 msg="Dynamic LLM libraries [cuda_v12 cpu cpu_avx cpu_avx2]"                             │
│ [...] source=gpu.go:204 msg="looking for compatible GPUs"                                                          │
│ [...] source=types.go:105 msg="inference compute" id=GPU-c9ad37d0-d304-5d2a-c2e6-d3788cd733a7 library=cuda compute │

Follow the information in this section to perform an air-gapped deployment of SUSE Security.

> tar -I zstd -vcf neuvector_airgap.zst neuvector

> tar -I zstd -vxf neuvector_airgap.zst

>  export REGISTRY=registry.awesome.sauce
for file in $(ls neuvector/images | grep -v txt ); do
     skopeo copy docker-archive:neuvector/images/$file docker://$(echo $file | sed 's/.tar//g' | awk -F_ '{print "'$REGISTRY'/neuvector/"$1":"$2}')
done

> export REGISTRY=registry.awesome.sauce
> export NEU_URL=neuvector.awesome.sauce
> helm upgrade -i neuvector --namespace neuvector neuvector/core \
--create-namespace --set imagePullSecrets=regsecret --set k3s.enabled=true \
--set k3s.runtimePath=/run/k3s/containerd/containerd.sock \
--set manager.ingress.enabled=true --set controller.pvc.enabled=true \
--set controller.pvc.capacity=10Gi --set manager.svc.type=ClusterIP \
--set registry=$REGISTRY --set tag=5.3.2 \
--set controller.image.repository=neuvector/controller \
--set enforcer.image.repository=neuvector/enforcer \
--set manager.image.repository=neuvector/manager \
--set cve.updater.image.repository=neuvector/updater \
--set manager.ingress.host=$NEU_URL

You can deploy SUSE Observability and SUSE AI in two different ways:

This section provides instructions for both scenarios.

To set up SUSE Observability for SUSE AI, you need to meet the following requirements:

For details on how to collect traces and metrics from SUSE AI components and user-developed applications, refer to Monitoring SUSE AI with OpenTelemetry and SUSE Observability. It includes configurations that are essential for full observability.

Important: SUSE Application Collection not instrumented by default

Applications from the SUSE Application Collection are not instrumented by default. If you want to monitor your AI applications, you need to follow the instrumentation guidelines that we provide in the document Monitoring SUSE AI with OpenTelemetry and SUSE Observability.

The following chart shows the high-level steps for the setup procedure. You will first set up the SUSE Observability cluster, then configure the SUSE AI cluster, and finally instrument your applications. Execute the steps in each column from left to right and top to bottom.

Figure 4.1: High-level overview of the SUSE Observability setup #

 

Tip: Setup clusters

You can create and configure Kubernetes clusters for SUSE AI and SUSE Observability as you prefer. If you are using SUSE Rancher Prime, check its documentation. For testing purposes, you can even share one cluster for both deployments. You can skip instructions on setting up a specific cluster if you already have one configured.

The diagram below shows the result of the above steps. There are two clusters represented, one for the SUSE Observability workload and another one for SUSE AI. You may use identical setup or customize it for your environment.

Figure 4.2: Separate clusters for SUSE AI and SUSE Observability #

 

Important

Remember that in multi-cluster setups, it is critical to properly expose your endpoints. Configure TLS, be careful with the configuration, and make sure to provide the right keys and tokens. More details are provided in the respective instructions.

This initial step is identical for both single-cluster and multi-cluster deployments.

Follow the instructions for your deployment scenario.

The difference between deployment scenarios affects the OTEL Collector exporter configuration and the SUSE Observability Agent URL as described in the following list.

Warning: SUSE Observability version 2.6.2 and above

With SUSE Observability version 2.6.2, a change of the standard behavior broke the vLLM monitoring performed by the extension. To fix it, update otel-values.yaml to include the following additions. No changes are required for people using SUSE Observability version 2.6.1 and below.

• Add a new processor.

  config:
    processors:
      ... # same as before
      transform:
        metric_statements:
          - context: metric
            statements:
              - replace_pattern(name, "^vllm:", "vllm_")

• Modify the metrics pipeline to perform the transformation defined above:

  config:
    service:
      pipelines:
        ... # same as before
        metrics:
          receivers: [otlp, spanmetrics, prometheus]
          processors: [transform, memory_limiter, resource, batch]
          exporters: [debug, otlp]

Instrumentation is the act of configuring your applications for telemetry data acquisition. Our stack employs OpenTelemetry standards as a vendor-neutral and open base for our telemetry. For a comprehensive guide on how to set up your instrumentation, please refer to Monitoring SUSE AI with OpenTelemetry and SUSE Observability.

By following the instructions in the document referenced above, you will be able to retrieve all relevant telemetry data from Open WebUI, Ollama, Milvus and vLLM by simply applying specific configuration to their Helm chart values. You can find links for advanced use cases (auto-instrumentation with the OTEL Operator) at the end of the document.