|Index|Troubleshooting SUSE AI Factory|How do I debug issues?
Applies to SUSE AI Factory

4 How do I debug issues?

Because Rancher Prime runs on Kubernetes, kubectl is the primary tool for debugging workload issues.

4.1 Validating SUSE AI Factory Operator and UI extension components

Before troubleshooting workload issues, verify that SUSE AI Factory Operator and its UI components are properly installed and running.

4.1.1 Validating that SUSE AI Factory Operator is running

Check that the operator pod is in the Running state and ready in the aif-operator namespace:

> kubectl get pods -n aif-operator
NAME                            READY   STATUS    RESTARTS   AGE
aif-operator-7bf464987-mnrw5    1/1     Running   0          2d

Check the operator Deployment status:

> kubectl get deployment aif-operator -n aif-operator
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
aif-operator   1/1     1            1           108d

View the operator logs to verify successful reconciliation without errors:

> kubectl logs -n aif-operator -l app.kubernetes.io/name=aif-operator --tail=50

4.1.2 Validating that the UI extension is installed

The operator manages the UI extension lifecycle using the InstallAIExtension Custom Resource (CR). Verify that the extension CR is present and check its status:

> kubectl get installaiextensions.ai-factory.suse.com -A
NAME     PHASE       SOURCE   EXTENSION   VERSION   AGE
aif-ui   Installed   Helm     aif-ui      2.2.0     108d

Alternatively, use the short name aifext:

> kubectl get aifext -A

Inspect the detailed status and conditions of the extension installation:

> kubectl describe aifext -n aif-operator

In the Rancher Prime UI, navigate to Extensions › Installed Extensions and confirm that SUSE AI Factory is listed as installed and active.

4.2 Inspecting objects created by the operator

SUSE AI Factory Operator manages custom resources to declare blueprints, AI workloads, settings and UI extensions. It also orchestrates Fleet bundles and Helm releases to deploy applications across clusters.

4.2.1 Listing SUSE AI Factory Custom Resources

List all available blueprints (both bundled and custom templates):

> kubectl get blueprints.ai-factory.suse.com -A

List all AI workload instances deployed across namespaces:

> kubectl get aiworkloads.ai-factory.suse.com -A

List configuration settings (registries, GitOps and API access):

> kubectl get settings.ai-factory.suse.com -n aif-operator

List UI extension installation resources:

> kubectl get installaiextensions.ai-factory.suse.com -A

4.2.2 Listing Fleet GitOps and Helm deployment objects

When deploying applications and blueprints, the operator generates GitOps bundles and Helm operations.

List Fleet bundles and their deployment status across clusters:

> kubectl get bundles.fleet.cattle.io -A
> kubectl get bundledeployments.fleet.cattle.io -A

List Helm releases and chart definitions:

> helm list -A
> kubectl get helmcharts.helm.cattle.io -A

4.2.3 Inspecting operator-created object definitions and status

Display the full YAML specification of a blueprint:

> kubectl get blueprint <BLUEPRINT_NAME> -o yaml

Display the full YAML specification and runtime status of an AI workload:

> kubectl get aiworkload <WORKLOAD_NAME> -n <NAMESPACE> -o yaml

Inspect detailed conditions and events for an AI workload:

> kubectl describe aiworkload <WORKLOAD_NAME> -n <NAMESPACE>

Inspect a Fleet bundle deployment state:

> kubectl describe bundle <BUNDLE_NAME> -n <FLEET_NAMESPACE>

4.3 Find the namespace and pods

List all installed pods sorted by their namespace.

> kubectl get pods -A
NAMESPACE                   NAME                                       READY   STATUS    RESTARTS       AGE
aif-operator                aif-operator-7d6b8f6bb5-j6lkk              1/1     Running   30 (79m ago)   33d
cattle-capi-system          capi-controller-manager-7654f98879-zrk2v   1/1     Running   82 (79m ago)   115d
cattle-fleet-local-system   fleet-agent-598d897c9b-bshfg               1/1     Running   69 (79m ago)   114d
cattle-fleet-system         fleet-controller-58f99fb59b-cv2qd          3/3     Running   219 (79m ago)  115d
[...]

Filter lines for a specific application and use wider view.

> kubectl get pods -A -o wide | grep -i ollama
NAMESPACE  NAME                      READY   STATUS   RESTARTS  AGE            IP            NODE     NOMINATED NODE   READINESS GATES
ollama     ollama-6ffbcfc6cf-nlrcr   1/1     Running  9         (89m ago) 13d  10.42.0.177   debian   <none>           <none>
Tip
Tip

kubectl get can output the actual Kubernetes object as YAML or JSON, which is useful for debugging. Add -o yaml or -o json to the kubectl command line to retrieve more details about an object.

4.4 Get the complete definition of an object

Display the full YAML specification of a Deployment.

> kubectl get deployment ollama -n ollama -o yaml

Display the full YAML specification and runtime status of a specific Pod.

> kubectl get pod ollama-6ffbcfc6cf-nlrcr -n ollama -o yaml

Find and inspect the configuration of a Service.

> kubectl get svc -A | grep -i ollama
[...]
> kubectl get svc ollama -n ollama -o yaml

Locate and inspect the YAML definition of a Secret.

> kubectl get secret -A | grep ollama
[...]
> kubectl get secret sh.helm.release.v1.ollama.v1 -n ollama -o yaml

Locate and view the contents of a ConfigMap.

> kubectl get configmap -A | grep ollama
[...]
> kubectl get configmap kube-root-ca.crt -n ollama -o yaml

4.5 Describe detailed resource information and related events

The kubectl describe command displays human-readable details about a resource, including runtime conditions and recent cluster events:

  • Scheduling issues: Explains why a Pod remains in Pending state (for example, insufficient CPU/memory, missing GPU resources, or unmet node selector and taint constraints).

  • Image pull failures: Highlights why container images cannot be fetched, distinguishing between invalid tags, network timeouts, or missing authentication secrets (ErrImagePull, ImagePullBackOff).

  • Volume and mount issues: Shows persistent volume claims (PVC) binding status, missing secrets or configmaps required for mounts, and storage attachment errors.

  • Environment and configuration: Lists resolved environment variables, command arguments, resource limits/requests, and active annotations or labels.

  • Health checks and probes: Reports readiness, liveness and startup probe failures along with exit codes and error messages from failing check scripts.

  • Container states and restart reasons: Details whether containers are running, terminated or crashing (CrashLoopBackOff), including exit codes and time stamps of previous terminations.

  • Events: Displays chronological Kubernetes events related to the object (such as successful assignment, pulling, creation or warning/failure notices from controllers).

> kubectl describe pod ollama-6ffbcfc6cf-nlrcr -n ollama

4.6 Inspect logs

Print the current stdout and stderr logs of a Pod.

> kubectl logs ollama-6ffbcfc6cf-nlrcr -n ollama

Stream (follow) logs in real time to monitor ongoing activity.

> kubectl logs -f ollama-6ffbcfc6cf-nlrcr -n ollama

Retrieve logs from a previously crashed or terminated container instance to identify failure causes.

> kubectl logs ollama-6ffbcfc6cf-nlrcr -n ollama --previous

4.7 Viewing events

Events sometimes offer more useful information than logs.

View events for the whole cluster.

> kubectl events -A

View events for a specific pod.

> kubectl events -n ollama --for pod/ollama-6ffbcfc6cf-nlrcr

4.8 Inspecting Helm charts

Because Kubernetes applications are commonly Helm-based, the helm command can reveal useful information too.

List all deployed Helm releases across all namespaces.

> helm list -A

Display the deployment status, revision number, and post-installation notes for a specific release.

> helm status ollama -n ollama
NAME: ollama
LAST DEPLOYED: Mon Sep 21 12:12:05 2026
NAMESPACE: ollama
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
[...]

Output the rendered Kubernetes resource manifests generated by the chart.

> helm get manifest ollama -n ollama

Display the user-supplied values used when installing or upgrading the release.

> helm get values ollama -n ollama

Save the user-supplied values to a file.

> helm get values ollama -n ollama > ollama-values.yaml

Save all values including chart defaults to a file.

> helm get values ollama -n ollama -a > ollama-all-values.yaml

Retrieve all information about the release, including notes, custom values, and generated manifests.

> helm get all ollama -n ollama