CNCF AI Conformance

The CNCF Kubernetes AI Conformance defines a set of additional capabilities, APIs and configurations that a Kubernetes cluster MUST offer, on top of standard CNCF Kubernetes Conformance, to reliably and efficiently run AI/ML workloads.

This page shows how to meet these requirements using RKE2 and NVIDIA GPU operator v26.3.2. The operator is installed following the GPU Operators documentation. Where a specific component version matters for a given requirement (e.g. a Gateway API implementation or CRD release), it is called out in that section.

Support Dynamic Resource Allocation (DRA)

DRA is a new API that enables more flexible, fine-grained resource requests beyond simple counts and has been generally available (GA) since v1.34.

Verify that all resource.k8s.io/v1 DRA API resources are enabled by running:

kubectl api-resources --api-group=resource.k8s.io

Expected Output:

NAME                     SHORTNAMES   APIVERSION           NAMESPACED   KIND
deviceclasses                         resource.k8s.io/v1   false        DeviceClass
resourceclaims                        resource.k8s.io/v1   true         ResourceClaim
resourceclaimtemplates                resource.k8s.io/v1   true         ResourceClaimTemplate
resourceslices                        resource.k8s.io/v1   false        ResourceSlice

Driver Runtime Management

DRA is the ideal API to verify that GPU drivers and container runtime configurations are correctly installed and maintained on nodes with accelerators. However, because DRA features are still evolving into the Kubernetes core ecosystem, platforms utilize Node Feature Discovery (NFD) and GPU Feature Discovery (GFD) to fulfill this requirement via node metadata attestation.

1. Verifying Base Container Runtime Version

To confirm the base engine handling the container lifecycle, the platform exposes the underlying containerd runtime via the core Node API:

kubectl get nodes -l nvidia.com/gpu.present=true -o jsonpath='{.items[*].status.nodeInfo.containerRuntimeVersion}'

Expected output:

containerd://2.2.3-k3s1

2. Verifying Accelerator Drivers and Runtime Configurations

By querying the node labels, administrators can programmatically verify both the driver installation and the specialized container runtime configuration patch applied to containerd:

kubectl get nodes -l nvidia.com/gpu.present=true -o jsonpath='{.items[*].metadata.labels}'

Expected output:

nvidia.com/cuda.driver-version.full: "595.71.05"
nvidia.com/gpu.deploy.container-toolkit: "true"
nvidia.com/gpu-driver-upgrade-state: "upgrade-done"

gpu.deploy.container-toolkit: "true" explicitly confirms that the container runtime configuration (the toolkit patch to containerd) is successfully injected and active.
gpu-driver-upgrade-state: "upgrade-done" confirms the operator is actively maintaining the lifecycle of these components.

GPU Sharing support

There are different strategies to share a GPU. We are going to use time-slicing because it is a pure software trick handled by the NVIDIA driver and container runtime. Therefore, it is compatible with almost every GPU.

Please refer to the NVIDIA documentation for details about how to apply time-slicing.

If everything worked as expected, you can verify it worked by executing:

kubectl describe node $GPU-NODE-NAME

Expected output (assuming 4 replicas were configured):

Labels:
        nvidia.com/gpu.product=Tesla-T4-SHARED
        nvidia.com/gpu.replicas=4
        nvidia.com/gpu.sharing-strategy=time-slicing

Capcity:
  nvidia.com/gpu:     4

Allocatable:
  nvidia.com/gpu:     4

Maintain API consistency when using vGPUs and GPUs

Maintaining consistency with the API and request mechanisms of physical fractional GPUs ensures that workloads, scheduling, policies, and user expectations remain the same across both physical and virtual backends, enabling portability and avoiding fragmented, vendor-specific resource models.

The NVIDIA GPU operator supports vGPUs and maintain the required API consistency. Please refer to the NVIDIA documentation for details about how to configure the NVIDIA GPU operator to support vGPU.

Support the Gateway API

Gateway API represents the next generation of Kubernetes ingress, load balancing and service mesh APIs.

To enable the Gateway API in RKE2, the cluster must be deployed with Traefik enabled and its KubernetesGateway provider configured, as explained in the Ingress Controller docs.

Verify that all gateway.networking.k8s.io/v1 Gateway API resources are enabled by running:

kubectl api-resources --api-group=gateway.networking.k8s.io/v1

Expected Output:

NAME              SHORTNAMES   APIVERSION                          NAMESPACED   KIND
gatewayclasses    gc           gateway.networking.k8s.io/v1        false        GatewayClass
gateways          gtw          gateway.networking.k8s.io/v1        true         Gateway
grpcroutes                     gateway.networking.k8s.io/v1        true         GRPCRoute
httproutes                     gateway.networking.k8s.io/v1        true         HTTPRoute
referencegrants   refgrant     gateway.networking.k8s.io/v1beta1   true         ReferenceGrant

To verify Traefik is consuming Gateway API resources:

Create a GatewayClass:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: traefik
spec:
  controllerName: traefik.io/gateway-controller

Check the status:

kubectl get gatewayclass traefik -o jsonpath='{.status}'

Expected Output:

"message":"Handled by Traefik controller","observedGeneration":1,"reason":"Handled","status":"True","type":"Accepted"

Advanced Inference Ingress

We use the agentgateway controller to satisfy this requirement. RKE2 can install any Gateway API implementation, and multiple implementations can coexist in the same cluster, so there is no need to remove the default ingress controller (Traefik).

Install agentgateway following its Helm installation documentation, making sure to pass --set inferenceExtension.enabled=true so that the controller can resolve InferencePool backends. This test was performed with agentgateway v1.6.0.

Install the Gateway API Inference Extension CRDs:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.6.0/manifests.yaml

If this was done correctly, the inferencepools CRD should be present:

kubectl get crd | grep inference.networking.k8s.io

Expected output:

inferencepools.inference.networking.k8s.io                 Namespaced   v1(storage)                   2026-10-23T11:31:42Z

The agentgateway GatewayClass should also be accepted:

kubectl get gatewayclass

Expected output:

NAME           CONTROLLER                      ACCEPTED   AGE
agentgateway   agentgateway.dev/agentgateway   True       20m

Create a Gateway that points at the agentgateway GatewayClass. Note that allowedRoutes.namespaces.from is set to All, since the HTTPRoute created later lives in a different namespace than this Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
  namespace: agentgateway-system
spec:
  gatewayClassName: agentgateway
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      name: http
      protocol: HTTP
      port: 80

If it worked, kubectl describe gateway inference-gateway -n agentgateway-system should show:

Conditions:
  Last Transition Time:  2026-10-23T11:50:58Z
  Message:               No errors found
  Observed Generation:   1
  Reason:                Accepted
  Status:                True
  Type:                  Accepted

To verify end-to-end functional routing without requiring GPU hardware, we deploy two replicas of the upstream llm-d inference simulator — the same tool referenced by agentgateway’s own inference-routing documentation. It serves the real OpenAI-compatible API shape and emits real vLLM-format Prometheus metrics (vllm:kv_cache_usage_perc, vllm:request_queue_time_seconds, etc.) derived from genuine request concurrency, so the Endpoint Picker sees the same signal it would from a real vLLM deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-sim-test
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-sim-test
  template:
    metadata:
      labels:
        app: vllm-sim-test
        inference.networking.k8s.io/engine-type: vllm
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: vllm-sim-test
                topologyKey: kubernetes.io/hostname
      containers:
        - name: vllm-sim
          image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2
          args:
            - --model
            - meta-llama/Llama-3.1-8B-Instruct
            - --port
            - "8000"
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-sim-test
  namespace: default
spec:
  selector:
    app: vllm-sim-test
  ports:
    - name: http
      port: 8000
      targetPort: 8000

Next, deploy the InferencePool and the Endpoint Picker (EPP) — the component that selects which backend pod a given request should go to, based on the live metrics advertised by the simulator pods above — and bind them to the inference-gateway Gateway via an HTTPRoute:

helm upgrade -i vllm-sim-test \
  oci://ghcr.io/llm-d/charts/llm-d-router-gateway \
  --version v1.0.0 \
  --set router.modelServers.matchLabels.app=vllm-sim-test \
  --set provider.name=none \
  --set httpRoute.create=true \
  --set httpRoute.inferenceGatewayName=inference-gateway \
  --set httpRoute.inferenceGatewayNamespace=agentgateway-system

This creates an InferencePool named vllm-sim-test, an EPP Deployment and Service, and an HTTPRoute named vllm-sim-test. Confirm the route is fully wired up:

kubectl get httproute vllm-sim-test -n default -o yaml

Expected output (relevant section):

status:
  parents:
  - conditions:
    - reason: Accepted
      status: "True"
      type: Accepted
    - reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs

Finally, send a request through the gateway to confirm the full path — Gateway → HTTPRoute → InferencePool → EPP → pod — works end to end:

kubectl port-forward svc/inference-gateway -n agentgateway-system 8080:80 &

curl -s -i -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"ping"}],"max_tokens":5}'

Expected output:

HTTP/1.1 200 OK
Server: fasthttp
Content-Type: application/json
X-Inference-Pod: vllm-sim-test-6b486c467b-jjtqw
X-Inference-Port: 8000

{"id":"chatcmpl-f87026bb-5c37-55f5-ba72-767f692ce105","created":1792763034,"model":"meta-llama/Llama-3.1-8B-Instruct","usage":{"prompt_tokens":6,"completion_tokens":4,"total_tokens":10},"object":"chat.completion","kv_transfer_params":null,"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"Give a man a "}}]}

The X-Inference-Pod response header confirms the request was routed through the Endpoint Picker to a specific backend pod, demonstrating a working GAIE stack on RKE2.

High Performance Networking

This requirement calls for well-defined mechanisms to manage and expose specialized network resources (e.g. RDMA, RoCE, InfiniBand NICs) with discoverable characteristics, and to let pods attach to multiple network interfaces. The requirement text calls out DRA (e.g. DRANET) as the forward-looking mechanism for this, but does not mandate it; SR-IOV combined with Multus is an established, production-proven mechanism that satisfies the same intent today, and is the approach RKE2 documents and supports.

RKE2 has a dedicated, validated guide for this in the SUSE Telco documentation: Configuring SR-IOV with RKE2. It covers two options:

Option 1 — SR-IOV Network Device Plugin: a ConfigMap-driven DaemonSet that discovers NICs (and vRAN accelerator cards) via PCI vendor/device/driver selectors and exposes them as allocatable Kubernetes resources (e.g. rancher.io/intel_sriov_odu).
Option 2 — SR-IOV Network Operator (recommended for network devices): installed via the sriov-crd and sriov-network-operator Helm charts. The operator labels SR-IOV-capable nodes, and a SriovNetworkNodeState resource exposes each discovered interface’s PCI address, driver, vendor, device ID, and total VF count — satisfying the "characteristics discoverable" part of the requirement. A SriovNetworkNodePolicy configures how many VFs to carve out and under what resource name (e.g. rancher.io/intelnicsDpdk), and an optional SriovNetwork resource auto-generates a Multus NetworkAttachmentDefinition, which is how pods request and attach additional high-performance interfaces beyond the primary pod network.

Verify the operator has discovered SR-IOV-capable interfaces on a node:

kubectl get sriovnetworknodestates -n sriov-network-operator -o yaml

Expected output (relevant section, one entry per discovered interface):

status:
  interfaces:
    - pciAddress: "0000:3b:00.0"
      driver: "mlx5_core"
      vendor: "15b3"
      deviceID: "1017"
      totalvfs: 8

Once a SriovNetworkNodePolicy and SriovNetwork are applied, confirm the resulting NetworkAttachmentDefinition is available for pods to reference:

kubectl get net-attach-def -A

A pod can then request the named SR-IOV resource directly in its spec (e.g. rancher.io/intelnicsDpdk: 1) and attach the corresponding NetworkAttachmentDefinition via the standard k8s.v1.cni.cncf.io/networks annotation, giving it a dedicated high-performance VF alongside its default pod network interface.

See the full guide linked above for complete manifests, hardware discovery steps (lspci), and troubleshooting notes.

Disaggregated Inference

Disaggregated serving (splitting the prefill and decode phases of LLM inference into separately scalable components) is an architecture implemented by the inference-serving framework itself (vLLM, SGLang, llm-d, Dynamo), not a feature of the Kubernetes distribution. What RKE2 needs to demonstrate is that nothing about the platform — scheduling, pod-to-pod networking, multi-container pods — prevents such a topology from being deployed and functioning correctly.

To verify this without requiring GPU hardware, we use the official prefill/decode disaggregation example from the upstream llm-d inference simulator, the same no-GPU simulator used above for the Advanced Inference Ingress requirement. The example defines two Deployments — vllm-sim-p (Prefill) and vllm-sim-d (Decode) — plus a routing sidecar colocated with the decode Pod that forwards the prefill phase to the prefill Service before handling decode itself, modeling the KV-cache handoff that a real disaggregated deployment performs over NIXL/UCX.

Apply the manifest:

kubectl apply -f https://raw.githubusercontent.com/llm-d/llm-d-inference-sim/main/manifests/disaggregation/vllm-sim-pd.yaml

At the time of writing, this manifest references a stale image, ghcr.io/llm-d/llm-d-routing-sidecar:v0.3.1-rc.1, for the routing sidecar container. That project has since been renamed/moved to ghcr.io/llm-d/llm-d-router-disagg-sidecar, and its CLI flags changed along with it. If the decode Pod’s routing-sidecar container is stuck in ImagePullBackOff, patch it to the current image and flags:

kubectl edit deployment vllm-sim-d

Replace the routing-sidecar container’s image and args with:

image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v1.0.0
args:
  - --port=8000
  - --model-server-port=8200
  - --kv-connector=nixlv2
  - --secure-serving=false

(--vllm-port became --model-server-port, --connector became --kv-connector, --secure-proxy became --secure-serving, and the old --v=4 klog flag is no longer supported — the new binary uses --zap-log-level instead.)

Confirm both roles are running as separate Pods:

kubectl get pods -l 'llm-d.ai/role in (prefill,decode)'

Expected output:

NAME                          READY   STATUS    RESTARTS   AGE
vllm-sim-d-666f9dd6bf-tpq55   3/3     Running   0          2m26s
vllm-sim-p-84d6b65c49-nlww2   2/2     Running   0          22m

Port-forward to the decode Service and send a request, telling the sidecar where the prefill Service is via the x-prefiller-host-port header:

kubectl port-forward svc/vllm-sim-d-service 8000:8000 &

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-prefiller-host-port: vllm-sim-p-service:8000" \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"Hello from P/D architecture!"}],"max_tokens":32}'

Expected output:

{"id":"chatcmpl-e2f5c223-cda7-5fd7-83dc-69b21eafc1ab","model":"Qwen/Qwen2.5-0.5B-Instruct","object":"chat.completion","created":1792765604,"usage":{"prompt_tokens":35,"completion_tokens":11,"total_tokens":46,"prompt_tokens_details":{"cached_tokens":0}},"choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"Alas, poor Yorick! I knew him, Horatio: "}}]}

Each Pod’s own logs confirm it only handled its own phase, with distinct request IDs and durations consistent with the simulated prefill/decode workload:

# Prefill pod (port 8000)
I1023 14:26:34.304362 http.go:283 "Received" new HTTP="chat completion request (req id 5d0ef6c1-...)"
I1023 14:26:42.848878 worker.go:74 "Finished processing request" id="5d0ef6c1-..."

# Decode pod (port 8200)
I1023 14:26:42.850396 http.go:283 "Received" new HTTP="chat completion request (req id e2f5c223-...)"
I1023 14:26:44.388055 worker.go:74 "Finished processing request" id="e2f5c223-..."

The prefill phase alone took ~8.5s (consistent with the configured --prefill-time-per-token=200ms across the ~35 prompt tokens) while decode took ~1.5s for the 11 completion tokens, and each phase was served by a distinct Pod on a distinct port — confirming RKE2 can host a genuinely disaggregated prefill/decode topology end to end.

Gang Scheduling

A gang scheduling solution (e.g., Kueue or Volcano) must be available for installation to ensure all-or-nothing scheduling for distributed AI workloads.

We will use Volcano in RKE2 for this verification test.

helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm repo update
helm install volcano volcano-sh/volcano -n volcano-system --create-namespace

The installation creates three deployments in the volcano-system namespace:

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/volcano-admission     1/1     1            1           130m
deployment.apps/volcano-controllers   1/1     1            1           130m
deployment.apps/volcano-scheduler     1/1     1            1           130m

Verification is complete, but we will perform a functional test. The following step creates a gang job with two tasks (each requiring an NVIDIA GPU) on a two-GPU cluster:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: gpu-nbody-gang-job
  namespace: default
spec:
  minAvailable: 2
  schedulerName: volcano

  tasks:
    - name: nbody-task-1
      replicas: 1
      template:
        spec:
          restartPolicy: OnFailure
          runtimeClassName: nvidia
          containers:
            - name: cuda-container-1
              image: nvcr.io/nvidia/k8s/cuda-sample:nbody
              command: ["/bin/bash", "-c"]
              args:
                - "while true; do sleep 5 && cuda-samples/nbody -gpu -benchmark; done"
              resources:
                limits:
                  nvidia.com/gpu: 1

    - name: nbody-task-2
      replicas: 1
      template:
        spec:
          restartPolicy: OnFailure
          runtimeClassName: nvidia
          containers:
            - name: cuda-container-2
              image: nvcr.io/nvidia/k8s/cuda-sample:nbody
              command: ["/bin/bash", "-c"]
              args:
                - "while true; do sleep 5 && cuda-samples/nbody -gpu -benchmark; done"
              resources:
                limits:
                  nvidia.com/gpu: 1

Both pods should be running after a few seconds.

To test gang-scheduling failure, modify the manifest to use minAvailable: 3 and add a third task. Re-submit the job:

    - name: nbody-task-3
      replicas: 1
      template:
        spec:
          restartPolicy: OnFailure
          runtimeClassName: nvidia
          containers:
            - name: cuda-container-3
              image: nvcr.io/nvidia/k8s/cuda-sample:nbody
              command: ["/bin/bash", "-c"]
              args:
                - "while true; do sleep 5 && cuda-samples/nbody -gpu -benchmark; done"
              resources:
                limits:
                  nvidia.com/gpu: 1

Observe that the three pods remain in a Pending status. This demonstrates that gang scheduling is working as expected.

default          gpu-nbody-gang-job-nbody-task-1-0                             0/1     Pending     0          50s
default          gpu-nbody-gang-job-nbody-task-2-0                             0/1     Pending     0          50s
default          gpu-nbody-gang-job-nbody-task-3-0                             0/1     Pending     0          50s

Cluster autoscaler

If the platform provides a cluster autoscaler or an equivalent mechanism, it must be capable of scaling accelerator-specific node groups based on pending pods. Since RKE2 is as a Kubernetes distribution, it does not provide an integrated cluster autoscaler.

For reference, we explain how to use the upstream autoscaler autoscaler with Azure as an example.

Create a Virtual Machine Scale Set (VMSS) with GPU-equipped VMs.

Deploy RKE2 with the following options:

disable-cloud-controller: true # Only in rke2-server
kubelet-arg: # On both rke2-server and rke2-agent
- --cloud-provider=external

Install the Azure CCM:

helm install --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo cloud-provider-azure --generate-name --set cloudControllerManager.imageRepository=mcr.microsoft.com/oss/kubernetes --set cloudControllerManager.imageName=azure-cloud-controller-manager --set cloudNodeManager.imageRepository=mcr.microsoft.com/oss/kubernetes --set cloudNodeManager.imageName=azure-cloud-node-manager --set cloudControllerManager.configureCloudRoutes=false --set cloudControllerManager.allocateNodeCidrs=false

Create the azure.json file and save it to /etc/kubernetes/azure.json. Ensure it contains the following two options:
```
  "useManagedIdentityExtension": false,
  "useInstanceMetadata": true
```
The deployed nodes should include a ProviderID. Verify this with:
```
kubectl get nodes -o yaml | grep ProviderID
```
The ProviderID is retrieved from the instance’s Metadata. Check this with:
```
curl -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2021-02-01"
```
Install the upstream autoscaler.
1. Firstly, create a values.yaml configuration file, specifying the VMSS from Step 1 and other necessary Azure details.
2. Then run the following helm commands:
  helm repo add autoscaler https://kubernetes.github.io/autoscaler helm repo update helm install cluster-autoscaler autoscaler/cluster-autoscaler -f values.yaml
When correctly deployed, the autoscaler monitors for pods requesting a GPU resource. If the cluster cannot satisfy the request, the autoscaler contacts Azure to automatically provision and add a new GPU node to the cluster.

Horizontal pod autoscaler

The ability to scale Pods based custom metrics relevant to AI/ML workloads, is achieved using the HorizontalPodAutoscaler (HPA), which is included by default in Kubernetes.

To demonstrate this requirement, install an Ollama deployment in RKE2. The following manifest is then used for verification:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ollama-hpa
spec:
  scaleTargetRef:
        apiVersion: apps/v1
        kind: Deployment
        name: ollama
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Object
    object:
      describedObject:
        apiVersion: v1
        kind: Namespace
        name: suse-private-ai
      metric:
        name: gpu_utilization
      target:
        type: AverageValue
        averageValue: "70"

Increasing the load on Ollama will raise the GPU utilization to 70%, triggering the deployment of new Ollama pods.

Accelerator Performance Metrics

This requirement mandates a functional accelerator metrics solution that exposes fine-grained performance metrics via a standardized, machine-readable metrics endpoint. This solution must include a core set of metrics for per-accelerator utilization and memory usage.

When the NVIDIA GPU Operator is installed (as described in the GPU Operators documentation), a nvidia-dcgm-exporter DaemonSet and Service are deployed. Query this service to collect required GPU metrics, such as accelerator utilization, memory usage, temperature, power usage, etc.

For example, if you SSH into one cluster node from within the cluster, it will show the metrics exposed using the OpenMetrics text format. The following section details how to deploy Prometheus and Grafana to consume them.

# Get the clusterIP
svcIP=$(kubectl get svc nvidia-dcgm-exporter -n gpu-operator -o jsonpath='{.spec.clusterIP}')
# Get the port
svcPort=$(kubectl get svc nvidia-dcgm-exporter -n gpu-operator -o jsonpath='{.spec.ports[0].port}')
# Output the metrics
curl -sL http://${svcIP}:${svcPort}/metrics

AI Job & Inference Service Metrics

This requirement mandates a system capable of discovering and collecting metrics exposed by workloads in a standardized format.

Prometheus and Grafana fulfill this requirement. First, install them:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-stack prometheus-community/kube-prometheus-stack \
>   --namespace monitoring \
>   --create-namespace

Once installed, create a ServiceMonitor to scrape metrics from workloads. As an example, the following manifest configures Prometheus to collect DCGM metrics from the NVIDIA GPU Operator:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nvidia-dcgm-monitor
  namespace: monitoring
  labels:
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app: nvidia-dcgm-exporter
  namespaceSelector:
    matchNames:
      - gpu-operator
  endpoints:
  - port: gpu-metrics
    path: /metrics
    interval: 15s

After a few minutes, the Grafana dashboard will show the DCGM metrics, such as DCGM_FI_DEV_GPU_UTIL.

Secure accelerator access

This requirement mandates access to accelerators from within containers must be properly isolated and mediated by Kubernetes

The NVIDIA GPU operator in v26.3.2 automatically configures that stack with the proper isolation.

Verify the isolation requirement by running the following three Pods in a cluster with only one GPU:

apiVersion: v1
kind: Pod
metadata:
  name: nbody-gpu-benchmark1
  namespace: default
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-container
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    command: ["/bin/bash", "-c"]
    args:
      - "while true; do sleep 5 && cuda-samples/nbody -gpu -benchmark; done"
    resources:
      limits:
        nvidia.com/gpu: 1
apiVersion: v1
kind: Pod
metadata:
  name: nbody-gpu-benchmark2
  namespace: default
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-container2
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    command: ["/bin/bash", "-c"]
    args:
      - "while true; do sleep 5 && cuda-samples/nbody -gpu -benchmark; done"
    resources:
      limits:
        nvidia.com/gpu: 1
apiVersion: v1
kind: Pod
metadata:
  name: nbody-gpu-benchmark3
  namespace: default
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-container3
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody
    command: ["/bin/bash", "-c"]
    args:
      - "while true; do sleep 5 && cuda-samples/nbody -gpu -benchmark; done"

Expected Results (Isolation Confirmed):

Pod 1 runs successfully and consumes the GPU.
Pod 2 is not scheduled by Kubernetes because the only GPU available in the cluster is already being consumed by Pod 1.
Pod 3 runs but fails to find an available GPU, as seen in the logs.

This outcome demonstrates that accelerator isolation is working correctly.

Workload Sandboxing

This requirement (new in Kubernetes v1.37) calls for a mechanism to run workloads in a sandbox with strong isolation from the host node kernel — isolating process, memory, filesystem, and network access from the host and co-located pods. This is particularly relevant for AI agent workloads that may execute untrusted, model-generated code.

RKE2 supports this via Kata Containers (currently a Technology Preview), which runs pods inside lightweight microVMs instead of sharing the host kernel — providing hardware-level isolation rather than just a user-space syscall filter. The full guide covers three progressively stronger scenarios: standard (non-confidential) Kata, non-confidential GPU passthrough, and confidential computing (CoCo) with GPU passthrough using AMD SEV-SNP or Intel TDX for hardware-encrypted memory.

Install Kata via the kata-deploy Helm chart, delivered as a HelmChart manifest:

# /var/lib/rancher/rke2/server/manifests/kata-deploy.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: kata-deploy
  namespace: kube-system
spec:
  chart: oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy
  targetNamespace: kube-system

Confirm the RuntimeClass was created:

kubectl get runtimeclass

Expected output (name may vary by shim selected, e.g. qemu-runtime-rs):

NAME   HANDLER             AGE
kata   kata-qemu-runtime-rs   2m

Verify the shim is actually running VMs, not falling back to the default runtime, on a node running a Kata pod:

ps aux | grep qemu-system-x86_64

A running qemu-system-x86_64 process confirms the pod is genuinely isolated in a microVM.

Run a test pod with runtimeClassName: kata:

apiVersion: v1
kind: Pod
metadata:
  name: kata-test-pod
spec:
  runtimeClassName: kata
  containers:
    - name: test
      image: alpine:latest
      command: ["sh", "-c", "echo Test PASSED && sleep 3600"]

kubectl logs kata-test-pod

Expected output:

Test PASSED

See the full guide linked above for the GPU passthrough and confidential computing (CoCo) scenarios, including NVIDIA GPU Operator integration with sandboxWorkloads enabled and attestation-related configuration.

Robust CRD and Controller Operation

This requirement mandates the installation and reliable function of at least one complex AI Operator with CRDs. Verification requires confirming that CRDs are registered and that an Admission Webhook rejects invalid configurations.

To verify this requirement, install the Kubeflow Training Operator in RKE2. Since a Helm chart is unavailable, use the following kubectl command as a workaround:

kubectl apply -k "github.com/kubeflow/training-operator/manifests/overlays/standalone?ref=v1.8.0"

Verify that CRDs are installed and the webhook is registered:

$> kubectl get crds | grep kubeflow
mpijobs.kubeflow.org                                       2025-10-24T13:04:27Z
mxjobs.kubeflow.org                                        2025-10-24T13:04:27Z
paddlejobs.kubeflow.org                                    2025-10-24T13:04:28Z
pytorchjobs.kubeflow.org                                   2025-10-24T13:04:28Z
tfjobs.kubeflow.org                                        2025-10-24T13:04:29Z
xgboostjobs.kubeflow.org                                   2025-10-24T13:04:29Z

$> kubectl get validatingwebhookconfigurations
validator.training-operator.kubeflow.org   5          10m

$> kubectl get pods -n kubeflow
NAME                                READY   STATUS    RESTARTS   AGE
training-operator-f7d4b59f6-vdnh9   1/1     Running   0          9m54s

Test the admission webhook’s rejection capability by attempting to apply the following invalid TFJob manifest (missing the required image field):

# saved as invalid-tfjob.yaml
apiVersion: kubeflow.org/v1
kind: TFJob
metadata:
  name: tfjob-invalid-test
spec:
  tfReplicaSpecs:
    Chief:
      replicas: 1
      template:
        spec:
          containers:
            - name: tensorflow
              # INTENTIONAL ERROR: Missing the 'image' field
              # image: tensorflow/tensorflow:latest
              # command: ["/bin/bash", "-c"]
              # args: ["echo 'Chief running'; sleep 10;"]

The Admission Webhook returns the expected error, confirming its function:

Error from server (Forbidden): error when creating "invalid-tfjob.yaml": admission webhook "validator.tfjob.training-operator.kubeflow.org" denied the request: spec.tfReplicaSpecs[Chief].template.spec.containers[0].image: Required value: must be required

Remove the comments in the previous example and re-try the job to see successful deployment.