4 How do I debug issues? #
Because Rancher Prime runs on Kubernetes, kubectl is the primary tool for debugging workload issues.
4.1 Validating SUSE AI Factory Operator and UI extension components #
Before troubleshooting workload issues, verify that SUSE AI Factory Operator and its UI components are properly installed and running.
4.1.1 Validating that SUSE AI Factory Operator is running #
Check that the operator pod is in the Running state and ready in the aif-operator namespace:
> kubectl get pods -n aif-operator
NAME READY STATUS RESTARTS AGE
aif-operator-7bf464987-mnrw5 1/1 Running 0 2dCheck the operator Deployment status:
> kubectl get deployment aif-operator -n aif-operator
NAME READY UP-TO-DATE AVAILABLE AGE
aif-operator 1/1 1 1 108dView the operator logs to verify successful reconciliation without errors:
> kubectl logs -n aif-operator -l app.kubernetes.io/name=aif-operator --tail=504.1.2 Validating that the UI extension is installed #
The operator manages the UI extension lifecycle using the InstallAIExtension Custom Resource (CR).
Verify that the extension CR is present and check its status:
> kubectl get installaiextensions.ai-factory.suse.com -A
NAME PHASE SOURCE EXTENSION VERSION AGE
aif-ui Installed Helm aif-ui 2.2.0 108dAlternatively, use the short name aifext:
> kubectl get aifext -AInspect the detailed status and conditions of the extension installation:
> kubectl describe aifext -n aif-operatorIn the Rancher Prime UI, navigate to › and confirm that SUSE AI Factory is listed as installed and active.
4.2 Inspecting objects created by the operator #
SUSE AI Factory Operator manages custom resources to declare blueprints, AI workloads, settings and UI extensions. It also orchestrates Fleet bundles and Helm releases to deploy applications across clusters.
4.2.1 Listing SUSE AI Factory Custom Resources #
List all available blueprints (both bundled and custom templates):
> kubectl get blueprints.ai-factory.suse.com -AList all AI workload instances deployed across namespaces:
> kubectl get aiworkloads.ai-factory.suse.com -AList configuration settings (registries, GitOps and API access):
> kubectl get settings.ai-factory.suse.com -n aif-operatorList UI extension installation resources:
> kubectl get installaiextensions.ai-factory.suse.com -A4.2.2 Listing Fleet GitOps and Helm deployment objects #
When deploying applications and blueprints, the operator generates GitOps bundles and Helm operations.
List Fleet bundles and their deployment status across clusters:
> kubectl get bundles.fleet.cattle.io -A
> kubectl get bundledeployments.fleet.cattle.io -AList Helm releases and chart definitions:
> helm list -A
> kubectl get helmcharts.helm.cattle.io -A4.2.3 Inspecting operator-created object definitions and status #
Display the full YAML specification of a blueprint:
> kubectl get blueprint <BLUEPRINT_NAME> -o yamlDisplay the full YAML specification and runtime status of an AI workload:
> kubectl get aiworkload <WORKLOAD_NAME> -n <NAMESPACE> -o yamlInspect detailed conditions and events for an AI workload:
> kubectl describe aiworkload <WORKLOAD_NAME> -n <NAMESPACE>Inspect a Fleet bundle deployment state:
> kubectl describe bundle <BUNDLE_NAME> -n <FLEET_NAMESPACE>4.3 Find the namespace and pods #
List all installed pods sorted by their namespace.
> kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
aif-operator aif-operator-7d6b8f6bb5-j6lkk 1/1 Running 30 (79m ago) 33d
cattle-capi-system capi-controller-manager-7654f98879-zrk2v 1/1 Running 82 (79m ago) 115d
cattle-fleet-local-system fleet-agent-598d897c9b-bshfg 1/1 Running 69 (79m ago) 114d
cattle-fleet-system fleet-controller-58f99fb59b-cv2qd 3/3 Running 219 (79m ago) 115d
[...]Filter lines for a specific application and use wider view.
> kubectl get pods -A -o wide | grep -i ollama
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ollama ollama-6ffbcfc6cf-nlrcr 1/1 Running 9 (89m ago) 13d 10.42.0.177 debian <none> <none>kubectl get can output the actual Kubernetes object as YAML or JSON, which is useful for debugging.
Add -o yaml or -o json to the kubectl command line to retrieve more details about an object.
4.4 Get the complete definition of an object #
Display the full YAML specification of a Deployment.
> kubectl get deployment ollama -n ollama -o yamlDisplay the full YAML specification and runtime status of a specific Pod.
> kubectl get pod ollama-6ffbcfc6cf-nlrcr -n ollama -o yamlFind and inspect the configuration of a Service.
> kubectl get svc -A | grep -i ollama
[...]
> kubectl get svc ollama -n ollama -o yamlLocate and inspect the YAML definition of a Secret.
> kubectl get secret -A | grep ollama
[...]
> kubectl get secret sh.helm.release.v1.ollama.v1 -n ollama -o yamlLocate and view the contents of a ConfigMap.
> kubectl get configmap -A | grep ollama
[...]
> kubectl get configmap kube-root-ca.crt -n ollama -o yaml4.5 Describe detailed resource information and related events #
The kubectl describe command displays human-readable details about a resource, including runtime conditions and recent cluster events:
Scheduling issues: Explains why a Pod remains in
Pendingstate (for example, insufficient CPU/memory, missing GPU resources, or unmet node selector and taint constraints).Image pull failures: Highlights why container images cannot be fetched, distinguishing between invalid tags, network timeouts, or missing authentication secrets (
ErrImagePull,ImagePullBackOff).Volume and mount issues: Shows persistent volume claims (PVC) binding status, missing secrets or configmaps required for mounts, and storage attachment errors.
Environment and configuration: Lists resolved environment variables, command arguments, resource limits/requests, and active annotations or labels.
Health checks and probes: Reports readiness, liveness and startup probe failures along with exit codes and error messages from failing check scripts.
Container states and restart reasons: Details whether containers are running, terminated or crashing (
CrashLoopBackOff), including exit codes and time stamps of previous terminations.Events: Displays chronological Kubernetes events related to the object (such as successful assignment, pulling, creation or warning/failure notices from controllers).
> kubectl describe pod ollama-6ffbcfc6cf-nlrcr -n ollama4.6 Inspect logs #
Print the current stdout and stderr logs of a Pod.
> kubectl logs ollama-6ffbcfc6cf-nlrcr -n ollamaStream (follow) logs in real time to monitor ongoing activity.
> kubectl logs -f ollama-6ffbcfc6cf-nlrcr -n ollamaRetrieve logs from a previously crashed or terminated container instance to identify failure causes.
> kubectl logs ollama-6ffbcfc6cf-nlrcr -n ollama --previous4.7 Viewing events #
Events sometimes offer more useful information than logs.
View events for the whole cluster.
> kubectl events -AView events for a specific pod.
> kubectl events -n ollama --for pod/ollama-6ffbcfc6cf-nlrcr4.8 Inspecting Helm charts #
Because Kubernetes applications are commonly Helm-based, the helm command can reveal useful information too.
List all deployed Helm releases across all namespaces.
> helm list -ADisplay the deployment status, revision number, and post-installation notes for a specific release.
> helm status ollama -n ollama
NAME: ollama
LAST DEPLOYED: Mon Sep 21 12:12:05 2026
NAMESPACE: ollama
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
[...]Output the rendered Kubernetes resource manifests generated by the chart.
> helm get manifest ollama -n ollamaDisplay the user-supplied values used when installing or upgrading the release.
> helm get values ollama -n ollamaSave the user-supplied values to a file.
> helm get values ollama -n ollama > ollama-values.yamlSave all values including chart defaults to a file.
> helm get values ollama -n ollama -a > ollama-all-values.yamlRetrieve all information about the release, including notes, custom values, and generated manifests.
> helm get all ollama -n ollama