Deploying and Installing SUSE AI in Air-Gapped Environments

1. Use SUSE-AI-mirror-nvidia.sh on a remote host to download the required NVIDIA RPM packages. Transfer the downloaded content to an air-gapped local host and add it as a Zypper repository to install NVIDIA drivers on local GPU nodes.

2. Use SUSE-AI-get-images.sh on a remote host to download Docker images of required SUSE AI components. Transfer them to an air-gapped local host.

3. Use SUSE-AI-load-images.sh to load the transferred Docker images of SUSE AI components into a custom local $Docker image registry.

4. Install AI Library components on the local Kubernetes cluster from the local custom Docker registry.

1. Download the air-gap images tarballs from the Prime Artifacts URL for the SUSE Rancher Prime: RKE2 version, CNI and platform you are using.

   • rke2-images.linux-ARCH.tar.zst, or rke2-images-core.linux-ARCH.tar.gz. This tarball contains the core container images required for RKE2 Prime to function. Zstandard offers better compression ratios and faster decompression speeds compared to gzip.

   • rke2-images-CNI.linux-ARCH.tar.gz. This tarball specifically contains the container images for your Container Network Interface (CNI). The default CNI in SUSE Rancher Prime: RKE2 is Canal.

     • If you are using the default CNI, Canal (--cni=canal), you can use either the rke2-image legacy archive as described above or the rke2-images-core and rke2-images-canal archives.

     • If you are using the alternative CNI, Cilium (--cni=cilium), you must download the rke2-images-core and rke2-images-cilium archives.

     • If using your own CNI (--cni=none), download only the rke2-images-core archive.

   • If enabling the vSphere CPI/CSI charts (--cloud-provider-name=rancher-vsphere), you must also download the rke2-images-vsphere archive.

2. Create a directory named /rke2-artifacts on the node and save the previously downloaded files there.

3. Continue with Section 3.3.4, “Install SUSE Rancher Prime: RKE2”.

1. Add all the required system images to your private registry. A list of images can be obtained from the .txt file corresponding to each tarball referenced above. Alternatively, you can docker load the air-gap image tarballs, then tag and push the loaded images.

2. Install SUSE Rancher Prime: RKE2 using the system-default-registry parameter, or use the containerd registry configuration to use your registry as a mirror for docker.io.

1. Run the installer, where INSTALL_RKE2_ARTIFACT_URL is the Prime Artifacts URL and INSTALL_RKE2_CHANNEL is a release channel you can subscribe to and defaults to stable. In this example, INSTALL_RKE2_CHANNEL="latest" gives you the latest version of RKE2.

   > sudo curl -sfL https://get.rke2.io/ | \
     sudo INSTALL_RKE2_ARTIFACT_URL=PRIME-ARTIFACTS-URL/rke2 \
     INSTALL_RKE2_CHANNEL="latest" sh -

   To specify a version, set the INSTALL_RKE2_VERSION environment variable.

   > sudo curl -sfL https://get.rke2.io/ | \
   sudo INSTALL_RKE2_ARTIFACT_URL=PRIME-ARTIFACTS-URL/rke2 \
     INSTALL_RKE2_VERSION="VERSION" ./install.sh

   This will install the rke2-server service and the rke2 binary onto your machine. Due to its nature, it will fail unless it runs as the root user or through sudo.

2. Enable the rke2-server service.

   > sudo systemctl enable rke2-server.service

3. To pull images from the Rancher Prime registry, set the following value in etc/rancher/rke2/config.yaml:

   > sudo system-default-registry: registry.rancher.com

   This configuration tells RKE2 to use registry.rancher.com as the default location for all container images it needs to deploy within the cluster.

4. Start the service.

   > sudo systemctl start rke2-server.service

5. Follow the logs with the following command:

   > sudo journalctl -u rke2-server -f

1. Run the installer.

   > sudo curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -

   This will install the rke2-agent service and the rke2 binary onto your machine. Due to its nature, it will fail unless it runs as the root user or through sudo.

2. Enable the rke2-agent service.

   > sudo systemctl enable rke2-agent.service

3. Configure the rke2-agent service.

   > sudo mkdir -p /etc/rancher/rke2/
   vim /etc/rancher/rke2/config.yaml

   Content for config.yaml:

   > sudo server: https://SERVER_IP_OR_DNS:9345
   token: TOKEN_FROM_SERVER_NODE

   

   Note

   The rke2 server process listens on port 9345 for new nodes to register. The Kubernetes API is still served on port 6443, as normal.

4. Start the service.

   > sudo systemctl start rke2-agent.service

5. Follow the logs with the following command:

   > sudo journalctl -u rke2-agent -f

Note

The Windows Server Containers feature needs to be enabled for the RKE2 agent to work.

1. Open a new PowerShell window with administrator privileges.

   powershell -Command "Start-Process PowerShell -Verb RunAs"

2. In the new PowerShell window, run the following command to install the containers feature.

   Enable-WindowsOptionalFeature -Online -FeatureName containers –All

   This will require a reboot for the Containers feature to function properly.

1. Download the install script.

   Invoke-WebRequest -Uri https://raw.githubusercontent.com/rancher/rke2/master/install.ps1 -Outfile install.ps1

   This script will download the rke2.exe Windows binary onto your machine.

2. Configure the rke2-agent for Windows.

   > sudo New-Item -Type Directory c:/etc/rancher/rke2 -Force
   Set-Content -Path c:/etc/rancher/rke2/config.yaml -Value @"
   server: https://SERVER_IP_OR_DNS:9345
   token: TOKEN_FROM_SERVER_NODE
   "@

   To learn more about the config.yaml file, refer to the Configuration options documentation.

3. Configure the PATH.

   > sudo $env:PATH+=";c:\var\lib\rancher\rke2\bin;c:\usr\local\bin"
   
   [Environment]::SetEnvironmentVariable(
       "Path",
       [Environment]::GetEnvironmentVariable("Path", [EnvironmentVariableTarget]::Machine) + ";c:\var\lib\rancher\rke2\bin;c:\usr\local\bin",
       [EnvironmentVariableTarget]::Machine)

4. Run the installer.

   > sudo ./install.ps1

5. Start the Windows RKE2 Service.

   > sudo rke2.exe agent service --add

The following steps populate your private registry.

1. Find the required assets for your Rancher version

2. Collect the cert-manager image (unless you are bringing your own certificates or terminating TLS on a load balancer)

3. Save the images to your workstation

4. Populate the private registry

1. lsmod | grep nvidia returns a list of NVIDIA kernel modules. For example:

   nvidia_uvm           2129920  0
   nvidia_drm            131072  0
   nvidia_modeset       1572864  1 nvidia_drm
   video                  77824  1 nvidia_modeset
   nvidia               9965568  2 nvidia_uvm,nvidia_modeset
   ecc                    45056  1 nvidia

2. cat /proc/driver/nvidia/version returns the NVRM and GCC versions of the driver. For example:

   NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  555.42.06
     Release Build  (abuild@host)  Thu Jul 11 12:00:00 UTC 2024
     GCC version:  gcc version 7.5.0 (SUSE Linux)

3. find /usr/ -iname libnvidia-ml.so returns a path to the libnvidia-ml.so library. For example:

   /usr/lib64/libnvidia-ml.so

   This library is used by Kubernetes components to interact with the kernel driver.

1. On the agent nodes of RKE2, run the following command:

   # echo PATH=$PATH:/usr/local/nvidia/toolkit >> /etc/default/rke2-agent

2. On the server nodes of RKE2, run the following command:

   # echo PATH=$PATH:/usr/local/nvidia/toolkit >> /etc/default/rke2-server

1. Assuming the drivers and libnvidia-ml.so library are installed, check if the operator detects them correctly.

   > kubectl get node NODENAME \
     -o jsonpath='{.metadata.labels}' | grep "nvidia.com/gpu.deploy.driver"

   You should see the value pre-installed. If you see true, the drivers are not installed correctly. If the requirements in Section 4.2.1, “Host OS requirements” are met, you may have forgotten to reboot the node after installing all packages.

   You can also check other driver labels:

   > kubectl get node NODENAME \
     -o jsonpath='{.metadata.labels}' | jq | grep "nvidia.com"

   You should see labels specifying driver and GPU, for example, nvidia.com/gpu.machine or nvidia.com/cuda.driver.major.

2. Check if the GPU was added by nvidia-device-plugin-daemonset as an allocatable resource in the node.

   > kubectl get node NODENAME \
     -o jsonpath='{.status.allocatable}' | jq

   You should see "nvidia.com/gpu": followed by the number of GPUs in the node.

3. Check that the container runtime binary was installed by the operator (in particular, by the nvidia-container-toolkit-daemonset):

   > ls /usr/local/nvidia/toolkit/nvidia-container-runtime

4. Verify whether the containerd configuration was updated to include the NVIDIA container runtime.

   > grep nvidia /var/lib/rancher/rke2/agent/etc/containerd/config.toml

5. Run a pod to verify that the GPU resource can successfully be scheduled on a pod and the pod can detect it.

   apiVersion: v1
   kind: Pod
   metadata:
     name: nbody-gpu-benchmark
     namespace: default
   spec:
     restartPolicy: OnFailure
     runtimeClassName: nvidia
     containers:
     - name: cuda-container
       image: nvcr.io/nvidia/k8s/cuda-sample:nbody
       args: ["nbody", "-gpu", "-benchmark"]
       resources:
         limits:
           nvidia.com/gpu: 1
       env:
       - name: NVIDIA_VISIBLE_DEVICES
         value: all
       - name: NVIDIA_DRIVER_CAPABILITIES
         value: compute,utility

1. Click ☰ > Cluster Management.

2. On the Clusters page, click Import Existing.

3. Choose the type of cluster.

4. Use Member Roles to configure user authorization for the cluster. Click Add Member to add users who can access the cluster. Use the Role drop-down list to set permissions for each user.

5. If you are importing a generic Kubernetes cluster in Rancher, perform the following steps for setup:

   1. Click Agent Environment Variables under Cluster Options to set environment variables for the Rancher cluster agent. The environment variables can be set using key-value pairs. If the Rancher agent requires the use of a proxy to communicate with the Rancher server, HTTP_PROXY, HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables can be set using agent environment variables.

   2. Enable Project Network Isolation to ensure the cluster supports Kubernetes NetworkPolicy resources. Users can select the Project Network Isolation option under the Advanced Options drop-down list to do so.

   3. Configure the version management feature for imported RKE2 and K3s clusters.

6. Click Create.

7. The requirements for cluster-admin privileges are shown (see Section 4.3.1, “Prerequisites”), including an example command to fulfill them.

8. Copy the kubectl command to your clipboard and run it on a node where kubeconfig is configured to point to the cluster you want to import. If you are unsure it is configured correctly, run kubectl get nodes to verify before running the command shown in Rancher.

9. If you are using self-signed certificates, you will receive the message certificate signed by unknown authority. To work around this validation, copy the command starting with curl displayed in Rancher to your clipboard. Then run the command on a node where kubeconfig is configured to point to the cluster you want to import.

10. After you finish running the command(s) on your node, click Done.

1. Create a file at /var/lib/rancher/{rke2,k3s}/kube-api-authn-webhook.yaml with the following contents:

   apiVersion: v1
   kind: Config
   clusters:
   - name: Default
     cluster:
       insecure-skip-tls-verify: true
       server: http://127.0.0.1:6440/v1/authenticate
   users:
   - name: Default
     user:
       insecure-skip-tls-verify: true
   current-context: webhook
   contexts:
   - name: webhook
     context:
       user: Default
       cluster: Default

2. Add the following to the configuration file (or create one if it does not exist). Note that the default location is /etc/rancher/{rke2,k3s}/config.yaml:

   kube-apiserver-arg:
     - authentication-token-webhook-config-file=/var/lib/rancher/{rke2,k3s}/kube-api-authn-webhook.yaml

3. Run the following commands:

   > sudo systemctl stop {rke2,k3s}-server
   > sudo systemctl start {rke2,k3s}-server

4. Finally, you must go back to the Rancher UI and edit the imported cluster there to complete the ACE enablement. Click on ⋮ > Edit Config, then click the Networking tab under Cluster Configuration. Finally, click the Enabled button for Authorized Endpoint. Once the ACE is enabled, you then have the option of entering a fully qualified domain name (FQDN) and certificate information.

1. Click ☰ > Cluster Management.

2. On the Clusters page, go to the custom cluster you want to annotate and click ⋮ > Edit Config.

3. Expand the Labels & Annotations section.

4. Click Add Annotation.

5. Add an annotation to the cluster with the format capabilities/<capability>: <value> where value is the cluster capability that will be overridden by the annotation. In this scenario, Rancher is not aware of any capabilities of the cluster until you add the annotation.

6. Click Save.

1. Install SUSE Observability. You can follow the official SUSE Observability air-gapped installation documentation for all installation instructions. Remember to expose your APIs and collector endpoints to your SUSE AI cluster.

   

   Important: Multi-cluster setup

   For multi-cluster setups, you must expose the SUSE Observability API and collector endpoints so that the SUSE AI cluster can reach them. Refer to the guide on exposing SUSE Observability outside of the cluster.

2. Install the SUSE Observability extension. Create a new Helm values file named genai_values.yaml. Before creating the file, review the placeholders below.

   SUSE_OBSERVABILITY_API_URL

   The URL of the SUSE Observability API. For multi-cluster deployments, this is the external URL. For single-cluster deployments, this can be the internal service URL. Example: http://suse-observability-api.your-domain.com

   SUSE_OBSERVABILITY_API_KEY

   The API key from the baseConfig_values.yaml file used during the SUSE Observability installation.

   SUSE_OBSERVABILITY_API_TOKEN_TYPE

   Can be api for a token from the Web UI or service for a Service Token.

   SUSE_OBSERVABILITY_TOKEN

   The API or Service token itself.

   OBSERVED_SERVER_NAME

   The name of the cluster to observe. It must match the name used in the Kubernetes StackPack configuration. Example: suse-ai-cluster.

   Create the genai_values.yaml file with the following content:

   global:
     imagePullSecrets:
     - application-collection 1
     imageRegistry: LOCAL_DOCKER_REGISTRY_URL:5043
   serverUrl: SUSE_OBSERVABILITY_API_URL
   apiKey: SUSE_OBSERVABILITY_API_KEY
   tokenType: SUSE_OBSERVABILITY_API_TOKEN_TYPE
   apiToken: SUSE_OBSERVABILITY_TOKEN
   clusterName: OBSERVED_SERVER_NAME

   1	Instructs Helm to use credentials from the SUSE Application Collection. For instructions on how to configure the image pull secrets for the SUSE Application Collection, refer to the official documentation.

   Run the install command.

   > helm upgrade --install ai-obs \
     charts/suse-ai-observability-extension-X.Y.Z.tgz \
     -f genai_values.yaml --namespace so-extensions --create-namespace

   

   Note: Self-signed certificates not supported

   Self-signed certificates are not supported. Consider running the extension in the same cluster as SUSE Observability and then use the internal Kubernetes address.

   After the installation is complete, a new menu called GenAI is added to the Web interface and also a Kubernetes cron job is created that synchronizes the topology view with the components found in the SUSE AI cluster.

3. Verify SUSE Observability extension. After the installation, you can verify that a new lateral menu appears:

   

   Figure 19: New GenAI Observability menu item #

    

1. Install NVIDIA GPU Operator. Follow the instructions in https://documentation.suse.com/cloudnative/rke2/latest/en/advanced.html#_deploy_nvidia_operator.

2. Install OpenTelemetry collector. Create a secret with your SUSE Observability API key in the namespace where you want to install the collector. Retrieve the API key using the Web UI or from the baseConfig_values.yaml file that you used during the SUSE Observability installation. If the namespace does not exist yet, create it.

   kubectl create namespace observability
   kubectl create secret generic open-telemetry-collector \
     --namespace observability \
     --from-literal=API_KEY='SUSE_OBSERVABILITY_API_KEY'

   Create a new file named otel-values.yaml with the following content.

   global:
     imagePullSecrets:
     - application-collection
   repository: LOCAL_DOCKER_REGISTRY_URL:5043/opentelemetry-collector-k8s
   extraEnvsFrom:
     - secretRef:
         name: open-telemetry-collector
   mode: deployment
   ports:
     metrics:
       enabled: true
   presets:
     kubernetesAttributes:
       enabled: true
       extractAllPodLabels: true
   config:
     receivers:
       prometheus:
         config:
           scrape_configs:
             - job_name: 'gpu-metrics'
               scrape_interval: 10s
               scheme: http
               kubernetes_sd_configs:
                 - role: endpoints
                   namespaces:
                     names:
                       - gpu-operator
             - job_name: 'milvus'
               scrape_interval: 15s
               metrics_path: '/metrics'
    
               static_configs:
                 - targets: ['MILVUS_SERVICE_NAME.SUSE_AI_NAMESPACE.svc.cluster.local:9091'] 1
     exporters:
       otlp:
         endpoint: https://OPEN_TELEMETRY_COLLECTOR_NAME.suse-observability.svc.cluster.local:4317 2
         headers:
           Authorization: "SUSEObservability ${env:API_KEY}"
         tls:
           insecure: true
     processors:
       tail_sampling:
         decision_wait: 10s
         policies:
         - name: rate-limited-composite
           type: composite
           composite:
             max_total_spans_per_second: 500
             policy_order: [errors, slow-traces, rest]
             composite_sub_policy:
             - name: errors
               type: status_code
               status_code:
                 status_codes: [ ERROR ]
             - name: slow-traces
               type: latency
               latency:
                 threshold_ms: 1000
             - name: rest
               type: always_sample
             rate_allocation:
             - policy: errors
               percent: 33
             - policy: slow-traces
               percent: 33
             - policy: rest
               percent: 34
       resource:
         attributes:
         - key: k8s.cluster.name
           action: upsert
           value: CLUSTER_NAME 3
         - key: service.instance.id
           from_attribute: k8s.pod.uid
           action: insert
       filter/dropMissingK8sAttributes:
         error_mode: ignore
         traces:
           span:
             - resource.attributes["k8s.node.name"] == nil
             - resource.attributes["k8s.pod.uid"] == nil
             - resource.attributes["k8s.namespace.name"] == nil
             - resource.attributes["k8s.pod.name"] == nil
     connectors:
       spanmetrics:
         metrics_expiration: 5m
         namespace: otel_span
       routing/traces:
         error_mode: ignore
         table:
         - statement: route()
           pipelines: [traces/sampling, traces/spanmetrics]
     service:
       extensions:
         - health_check
       pipelines:
         traces:
           receivers: [otlp, jaeger]
           processors: [filter/dropMissingK8sAttributes, memory_limiter, resource]
           exporters: [routing/traces]
         traces/spanmetrics:
           receivers: [routing/traces]
           processors: []
           exporters: [spanmetrics]
         traces/sampling:
           receivers: [routing/traces]
           processors: [tail_sampling, batch]
           exporters: [debug, otlp]
         metrics:
           receivers: [otlp, spanmetrics, prometheus]
           processors: [memory_limiter, resource, batch]
           exporters: [debug, otlp]

   1	Configure the Milvus service and namespace for the Prometheus scraper. Because Milvus will be installed in subsequent steps, you can return to this step and edit the endpoint if necessary.
   2	Set the exporter to your exposed SUSE Observability collector. Remember that the value can be distinct, depending on the deployment pattern. For production usage, we recommend using TLS communication.
   3	Replace CLUSTER_NAME with the cluster's name.

   Finally, run the installation command.

   > helm upgrade --install opentelemetry-collector \
     oci://dp.apps.rancher.io/charts/opentelemetry-collector \
     -f otel-values.yaml --namespace observability

   Verify the installation by checking the existence of a new deployment and service in the observability namespace.

3. The GPU metrics scraper that we configure in the OTEL Collector requires custom RBAC rules. Create a file named otel-rbac.yaml with the following content:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: suse-observability-otel-scraper
   rules:
     - apiGroups:
         - ""
       resources:
         - services
         - endpoints
       verbs:
         - list
         - watch
   
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: suse-observability-otel-scraper
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: suse-observability-otel-scraper
   subjects:
     - kind: ServiceAccount
       name: opentelemetry-collector
       namespace: observability

   Then apply the configuration by running the following command.

   > kubectl apply -n gpu-operator -f otel-rbac.yaml

4. Install the SUSE Observability Agent.

   > helm upgrade --install \
     --namespace suse-observability --create-namespace \
     --set-string 'stackstate.apiKey'='YOUR_API_KEY'1 \
     --set-string 'stackstate.cluster.name'='CLUSTER_NAME2' \
     --set-string 'stackstate.url'='http://suse-observability-router.suse-observability.svc.cluster.local:8080/receiver/stsAgent'3 \
     --set 'nodeAgent.skipKubeletTLSVerify'=true suse-observability-agent \
     suse-observability/suse-observability-agent

   1	Retrieve the API key using the Web UI or from the baseConfig_values.yaml file that you used during the SUSE Observability installation.
   2	Replace CLUSTER_NAME with the cluster's name.
   3	Replace with your SUSE Observability server URL.

5. Install SUSE AI. Refer to Section 5, “Installing applications from AI Library” for the complete procedure.

• Install the cert-manager chart.

  > helm upgrade
  --install cert-manager charts/cert-manager-X.Y.Z.tgz \
    -n CERT_MANAGER_NAMESPACE \
    --set crds.enabled=true \
    --set 'global.imagePullSecrets[0].name'=application-collection \
    --set 'global.imageRegistry'=LOCAL_DOCKER_REGISTRY_URL:5043

Tip

Before the installation, you need to get user access to the SUSE Application Collection, create a Kubernetes namespace, and log in to the Helm registry as described in Section 5.1, “Installation procedure”.

1. When installed as part of SUSE AI, Milvus depends on etcd, MinIO and Apache Kafka. Because the Milvus chart uses a non-default configuration, create an override file milvus_custom_overrides.yaml with the following content.

   

   Tip

   As a template, you can use the values.yaml file that is included in the charts/milvus-X.Y.Z.tgz TAR archive.

   global:
     imagePullSecrets:
     - application-collection
     imageRegistry: LOCAL_DOCKER_REGISTRY_URL:5043
   cluster:
     enabled: True
   standalone:
     persistence:
       persistentVolumeClaim:
         storageClassName: "local-path"
   etcd:
     replicaCount: 1
     persistence:
       storageClassName: "local-path"
   minio:
     mode: distributed
     replicas: 4
     rootUser: "admin"
     rootPassword: "adminminio"
     persistence:
       storageClass: "local-path"
     resources:
       requests:
         memory: 1024Mi
   kafka:
     enabled: true
     name: kafka
     replicaCount: 3
     broker:
       enabled: true
     cluster:
       listeners:
         client:
           protocol: 'PLAINTEXT'
         controller:
           protocol: 'PLAINTEXT'
     persistence:
       enabled: true
       annotations: {}
       labels: {}
       existingClaim: ""
       accessModes:
         - ReadWriteOnce
       resources:
         requests:
           storage: 8Gi
       storageClassName: "local-path"
   extraConfigFiles:1
     user.yaml: |+
       trace:
         exporter: jaeger
         sampleFraction: 1
         jaeger:
           url: "http://opentelemetry-collector.observability.svc.cluster.local:14268/api/traces"2

   1	The extraConfigFiles section is optional, required only to receive telemetry data from Open WebUI.
   2	The URL of the OpenTelemetry Collector installed by the user.

   

   Tip

   The above example uses local storage. For production environments, we recommend using an enterprise class storage solution such as SUSE Storage in which case the storageClassName option must be set to longhorn.

2. Install the Milvus Helm chart using the milvus_custom_overrides.yaml override file.

   > helm upgrade --install \
     milvus charts/milvus-X.Y.Z.tgz \
     -n SUSE_AI_NAMESPACE \
     --version X.Y.Z -f milvus_custom_overrides.yaml

Tip

Before the installation, you need to get user access to the SUSE Application Collection, create a Kubernetes namespace, and log in to the Helm registry as described in Section 5.1, “Installation procedure”.

1. Create the ollama_custom_overrides.yaml file to override the values of the parent Helm chart. Refer to Section 5.4.4, “Values for the Ollama Helm chart” for more details.

2. Install the Ollama Helm chart using the ollama-custom-overrides.yaml override file.

   > helm upgrade \
     --install ollama charts/ollama-X.Y.Z.tgz \
     -n SUSE_AI_NAMESPACE \
     -f ollama_custom_overrides.yaml

   

   Important: Downloading AI models

   Ollama normally needs to have an active Internet connection to download AI models. In an air-gapped environment, you must download the models manually and copy them to your local Ollama instance, for example:

   kubectl cp
     PATH_TO_LOCALLY_DOWNLOADED_MODELS/blobs/* \
     OLLAMA_POD_NAME:~/.ollama/models/blobs/

   

   Tip: Hugging Face models

   Models downloaded from Hugging Face need to be converted before they can be used by Ollama. Refer to https://github.com/ollama/ollama/blob/main/docs/import.md for more details.

1. Create the owui_custom_overrides.yaml file to override the values of the parent Helm chart. The file contains URLs for Milvus and Ollama and specifies whether a stand-alone Ollama deployment is used or whether Ollama is installed as part of the Open WebUI installation. Find more details in Section 5.5.4, “Examples of Open WebUI Helm chart override files”. For a list of all installation options with examples, refer to Section 5.5.5, “Values for the Open WebUI Helm chart”.

2. Install the Open WebUI Helm chart using the owui_custom_overrides.yaml override file.

   > helm upgrade --install \
     open-webui charts/open-webui-X.Y.Z.tgz \
     -n SUSE_AI_NAMESPACE \
     --version X.Y.Z -f owui_custom_overrides.yaml