|Index|SUSE AI Factory Applications|Kubeflow

1 Kubeflow

Kubeflow (https://www.kubeflow.org) is a cloud-native platform that orchestrates and simplifies the entire machine learning (ML) lifecycle on Kubernetes. Unlike traditional software with hard-coded rules, ML applications learn to make decisions and predictions directly from data. Kubeflow automates this data-driven process, managing everything from initial development to production deployment.

Warning
Warning: Technology preview

Kubeflow is currently available as a Technology Preview.

1.1 What components does Kubeflow include?

The following table describes all the components of the Kubeflow package.

ComponentChart sourceNamespacePurpose

Dex

SUSE Application Collection dex-idp 0.23.0 (0.24.0 incompatible — see Known Limitations)

kubeflow

OIDC identity provider

oauth2-proxy

SUSE Application Collection oauth2-proxy 10.1.4

kubeflow

OIDC login broker / session cookie

Central Dashboard

local

kubeflow

Web UI shell

Kubeflow Pipelines

local

kubeflow

ML workflow engine (KFP v2)

Notebook Controller

local

kubeflow

Jupyter Notebook CRD controller

Jupyter Web App

local

kubeflow

Notebook creation UI

Katib

local

kubeflow

Hyperparameter tuning

KServe

local

kubeflow

Model serving

Training Operator V1

local

kubeflow

TFJob / PyTorchJob / etc. (kubeflow.org API)

Trainer V2

local

kubeflow

TrainJob / TrainingRuntime / ClusterTrainingRuntime (trainer.kubeflow.org API)

Profiles & KFAM

local

kubeflow

Multi-tenancy / namespace management

Volumes Web App

local

kubeflow

PVC management UI (co-enabled with PVCViewer Controller via volumesWebApp.enabled)

PVCViewer Controller

local

kubeflow

File browser for PVCs (co-enabled with Volumes Web App)

Tensorboard Controller

local

kubeflow

TensorBoard CRD controller

Tensorboards Web App

local

kubeflow

TensorBoard UI

Models Web App

local

kubeflow

Model serving UI (co-enabled with KServe via kserve.enabled)

Model Registry

local

kubeflow

Model versioning and metadata

Admission Webhook

local

kubeflow

PodDefault mutating webhook (always installed — no condition)

Knative Serving

local

knative-serving

Serverless layer required by KServe

Knative Eventing

local

knative-eventing

Optional event-driven pipeline triggers (disabled by default)

MariaDB (KFP)

SUSE Application Collection mariadb 0.1.7

kubeflow

Pipeline + MLMD metadata DB

MariaDB (Katib)

inline

kubeflow

Katib experiment DB

SeaweedFS

inline

kubeflow

Pipeline artifact store (S3-compatible, Docker Hub image — requires internet access)

external-dns

SUSE Application Collection external-dns 1.20.0

kubeflow

Automatic DNS record management for Cloudflare (disabled by default)

NetworkPolicies

local

kubeflow / knative-serving

Deny-all + explicit allow NetworkPolicies (disabled by default)

Monitoring

local

kubeflow

ServiceMonitors + PrometheusRules for Rancher Monitoring (disabled by default)

1.2 What are Kubeflow requirements?

Deploying and running Kubeflow has the following requirements.

1.2.1 Hardware requirements

ProfileCPURAMGPUStorage

Minimal evaluation

8 cores

16 GB

None

100+ GB

Recommended evaluation

8-18 cores

32-64 GB

1 GPU

250+ GB

Production / large language model development

16+ cores

64-128+ GB

2+ GPUs (workload-dependent)

500+ GB

1.2.2 Software requirements

RequirementVersionNotes

Kubernetes

>= 1.31

Tested on RKE2 / K3s

Helm

>= 4.0.1

Required for OCI chart support

cert-manager

1.20.2

—

Istio

mesh ≥ 1.9 (chart 1.3.0)

—

Load Balancer (i.e. metallb)

—

Required when used in conjunction with external-dns and cert-manager. Tested with MetalLB v0.15.3

Default StorageClass

—

Local Path Provisioner (development) or Longhorn (production)

SUSE Application Collection credentials

—

User name + token for dp.apps.rancher.io (application-collection secret)

SUSE Registry credentials

—

User name + token for registry.suse.com (suse-ai-registry secret) — required when sub-charts pull from SUSE Registry

Storage class

All PVCs use the cluster default StorageClass unless global.storageClass is set. For single-node clusters the Local Path Provisioner is sufficient for development. For production, Longhorn is recommended.

Note
Note

Make sure a default StorageClass is set for the cluster before installing. Run kubectl get storageclass and confirm that one has the (default) designation next to its name. Otherwise, designate one, for example:

> kubectl patch storageclass local-path \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Cloudflare API token

Only required if you want automatic DNS record management via external-dns or Let’s Encrypt DNS-01 challenges.include::../common/generic-attributes.adoc[]

1.2.3 What are known Kubeflow limitations?

The current Kubeflow distribution has the following known limitations. Understanding these limitations is crucial for successful deployment and operation.

  • Namespace names, such as kubeflow, knative-serving or istio-system, are hardcoded in most templates. Deploying to a non-standard namespace requires template modifications.

  • Argo workflow executor image pull secrets must be set via the workflow-controller-configmap workflowDefaults field. They cannot be set via Helm values at runtime because the configmap is rendered at pod-create time, not Helm time.

  • knativeServing.enabled must be true for KServe to function. Disabling Knative Serving will cause KServe InferenceService resources to remain in a non-Ready state.

  • Only the Cloudflare provider is tested for external-dns out of the box. Other providers (AWS Route53, Azure, GCP) require additional externaldns.env configuration.

  • SeaweedFS is single-node, non-replicated. Pipeline artifact storage has no HA. A SeaweedFS pod restart causes a brief (~10–60s) S3 outage. For production HA, replace SeaweedFS with an external S3-compatible store.

  • Not all Kubeflow controllers support multiple replicas. Leader election is confirmed for katib-controller (v0.17+), training-operator, kserve-controller-manager, pvcviewer-controller, and model-registry-controller. These are safe to scale via ha-overrides.yaml. It is not confirmed for notebook-controller, profiles-controller, tensorboard-controller, and several KFP background workers. Setting replicaCount > 1 for these controllers causes undefined behavior, such as duplicate reconciliation or data corruption.

  • CRDs are not automatically upgraded by helm upgrade. Kubeflow CRDs are placed in crds/ subdirectories. Helm intentionally skips them on upgrade. After a chart version bump, run:

    > helm show crds oci://registry.suse.com/ai/charts/kubeflow \
      --version <VERSION> | kubectl apply -f -
  • Dex must be v0.23.0 (dex 2.42.0). v0.24.0 (dex 2.44.0) uses Go 1.25, which introduced strict IPv6 URL parsing. Kubernetes API server addresses like [10.43.0.1]:443 are rejected, crashing Dex on startup. The chart pins v0.23.0.

  • SeaweedFS image is pulled from Docker Hub (chrislusf/seaweedfs:4.00). Air-gapped clusters or environments with Docker Hub pull-rate limits will fail to start Kubeflow Pipelines. Mirror the image to a private registry and set pipelines.seaweedfs.image.registry to override.

1.3 How do I get the values list for Kubeflow Helm chart?

Before deploying Kubeflow, it is important to know more about the supported configurations.

  1. (Optional) To get the values for a specific Kubeflow chart version, log in to SUSE Registry and identify the version. To prevent the password from being stored in the shell history, save it to the password.txt file.

    > skopeo login registry.suse.com \
      --username regcode \
      --password-stdin < password.txt
    > skopeo list-tags docker://registry.suse.com/ai/charts/kubeflow
  2. Log in to SUSE Registry where Kubeflow chart is stored. To prevent the password from being stored in the shell history, save it to the password.txt file.

    > helm registry login registry.suse.com \
      --username regcode \
      --password-stdin < password.txt
  3. Display all values of the Kubeflow Helm chart. Omit --version to display the latest chart.

    > helm show values oci://registry.suse.com/ai/charts/kubeflow \
      --version <VERSION>

1.4 How do I install Kubeflow?

You can install Kubeflow on the command line with Helm.

1.4.1 Install using Helm

To install Kubeflow on the command line using Helm, follow these steps:

  1. Log in to the SUSE Application Collection registry. To prevent the password from being stored in the shell history, save it to the paassword.txt file.

    > helm registry login dp.apps.rancher.io \
      --username=<APPCO-REGISTRY-USERNAME> \
      --password-stdin < paassword.txt
  2. Log in to SUSE Registry. To prevent the password from being stored in the shell history, save it to the paassword.txt file.

    > helm registry login registry.suse.com \
      --username=regcode \
      --password-stdin < paassword.txt
  3. Create namespaces.

    for ns in cert-manager istio-system kubeflow knative-serving; do
      kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
    done
    Note
    Note

    kubeflow-user-example-com and all other user namespaces are created by the Kubeflow chart and receive registry pull secrets automatically via ESO — no manual creation needed.

  4. Create image pull secrets.

    Create secrets only in the static system namespaces. ESO propagates them to user namespaces automatically by copying suse-ai-registry and application-collection from the Kubeflow namespace into every user namespace labelled app.kubernetes.io/part-of: kubeflow-profile, including namespaces created dynamically by the profiles controller at runtime.

    1. Create SUSE Application Collection image pull secrets.

      for ns in cert-manager istio-system kubeflow knative-serving; do
        kubectl create secret docker-registry application-collection \
          --docker-server=dp.apps.rancher.io \
          --docker-username=<appco-registry-username> \
          --docker-password=<appco-registry-token> \
          -n "$ns" \
          --dry-run=client -o yaml | kubectl apply -f -
      done
    2. Create SUSE Registry image pull secrets.

      for ns in cert-manager istio-system kubeflow knative-serving; do
        kubectl create secret docker-registry suse-ai-registry \
          --docker-server=registry.suse.com \
          --docker-username=regcode \
          --docker-password=<suse-ai-registry-token> \
          -n "$ns" \
          --dry-run=client -o yaml | kubectl apply -f -
      done
  5. Label namespaces for Helm.

    for ns in kubeflow kubeflow-user-example-com knative-serving; do
      kubectl label namespace "$ns" app.kubernetes.io/managed-by=Helm --overwrite
      kubectl annotate namespace "$ns" \
        meta.helm.sh/release-name=kubeflow \
        meta.helm.sh/release-namespace=kubeflow \
        --overwrite
    done
  6. Install cert-manager.

    > helm upgrade --install cert-manager \
      oci://dp.apps.rancher.io/charts/cert-manager \
      --version 1.20.2 \
      --namespace cert-manager \
      --set crds.enabled=true \
      --set crds.keep=true \
      --set global.imagePullSecrets[0].name=application-collection \
      --wait --timeout 5m
  7. Install Istio.

    > helm upgrade --install istio \
      oci://dp.apps.rancher.io/charts/istio \
      --version 1.3.0 \
      --namespace istio-system \
      --set global.imagePullSecrets[0].name=application-collection \
      --set gateway.enabled=true \
      --force-conflicts \
      --server-side=true \
      --wait --timeout 5m
  8. Install External Secrets Operator (ESO). ESO must be installed before the Kubeflow chart. Its CRDs must be present when Helm applies the ClusterSecretStore and ClusterExternalSecret resources.

    > helm upgrade --install external-secrets-operator \
      oci://dp.apps.rancher.io/charts/external-secrets-operator \
      --version 2.3.0 \
      --namespace kubeflow \
      --set installCRDs=true \
      --set global.imagePullSecrets[0].name=application-collection \
      --wait --timeout 5m
  9. To override default installation values, list them in a values.yaml file. For a complete list of Kubeflow chart values, refer to Section 1.3, “How do I get the values list for Kubeflow Helm chart?”.

  10. Install Kubeflow. To keep the default installation values, omit -f values.yaml.

    > helm upgrade --install kubeflow \
      oci://registry.suse.com/ai/charts/kubeflow \
      --version 0.4.0 \
      -n kubeflow \
      --force-conflicts \ 1
      --server-side=true \
      --wait --timeout 15m

    1

    --force-conflicts is required because cert-manager-cainjector, istiod (pilot-discovery), and the clusterrole-aggregation-controller modify fields (caBundle, webhook failurePolicy, aggregated RBAC rules) that Helm tracks. This flag lets Helm reclaim ownership of those fields on each upgrade.

1.5 How do I install Kubeflow in air-gapped environments?

Installing Kubeflow in an air-gapped environment allows you to deploy and manage machine learning workflows securely without active Internet connectivity. This deployment method uses local image registries and Helm repositories to satisfy all software dependencies, ensuring complete data privacy and network isolation.

1.5.1 Prerequisites

  • A local private container registry mirror

  • Synced container images and Helm charts

  • A running Kubernetes cluster (v1.31 or higher)

1.5.2 How to install Kubeflow in an air-gapped environment?

An air-gapped deployment requires you to host all container images and Helm charts in a local container registry. This process ensures the system can satisfy all software dependencies without contacting external networks.

Before starting the installation, you must mirror the required AI application charts and container images to your local private registry.

1.5.2.1 How to configure local registry credentials?

To access your private registry, authenticate with your local mirror address. To prevent passwords from appearing in the shell history, save your credentials to the mirror-password.txt file.

> helm registry login mirror.corp.example.com \
  --username=<username> \
  --password-stdin < mirror-password.txt

1.5.2.2 How to prepare the Kubernetes namespaces?

Create the required namespaces on the target cluster. Creating namespaces before the deployment allows you to pre-configure the necessary image pull secrets.

> for ns in cert-manager istio-system kubeflow knative-serving; do
  kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
done

+

Note
Note

User namespaces, such as kubeflow-user-example-com, are automatically created by the Kubeflow chart. The External Secrets Operator (ESO) handles registry pull secrets for these dynamic namespaces automatically.

1.5.2.3 How to create local image pull secrets?

Create the pull secrets in all static system namespaces. The cluster uses these credentials to pull container images from your local mirror. ESO propagates these secrets to user namespaces at runtime.

  1. Create SUSE Application Collection secrets pointing to your local registry.

    > for ns in cert-manager istio-system kubeflow knative-serving; do
      kubectl create secret docker-registry application-collection \
        --docker-server=mirror.corp.example.com \
        --docker-username=<username> \
        --docker-password=<token> \
        -n "$ns" \
        --dry-run=client -o yaml | kubectl apply -f -
    done
  2. Create SUSE Registry secrets pointing to your local registry.

    > for ns in cert-manager istio-system kubeflow knative-serving; do
      kubectl create secret docker-registry suse-ai-registry \
        --docker-server=mirror.corp.example.com \
        --docker-username=<username> \
        --docker-password=<token> \
        -n "$ns" \
        --dry-run=client -o yaml | kubectl apply -f -
    done

1.5.2.4 How to label and annotate namespaces?

Configure the namespaces to allow Helm management. Applying these labels prevents resource ownership conflicts during the upgrade process.

> for ns in kubeflow kubeflow-user-example-com knative-serving; do
  kubectl label namespace "$ns" app.kubernetes.io/managed-by=Helm --overwrite
  kubectl annotate namespace "$ns" \
    meta.helm.sh/release-name=kubeflow \
    meta.helm.sh/release-namespace=kubeflow \
    --overwrite
done

1.5.2.5 How to install required infrastructure components?

Install the supporting operators and controllers from your local registry mirror.

  1. Install cert-manager from the local mirror.

    > helm upgrade --install cert-manager \
      oci://mirror.corp.example.com/charts/cert-manager \
      --version 1.20.2 \
      --namespace cert-manager \
      --set crds.enabled=true \
      --set crds.keep=true \
      --set global.imagePullSecrets[0].name=application-collection \
      --wait --timeout 5m
  2. Install Istio from the local mirror.

    > helm upgrade --install istio \
      oci://mirror.corp.example.com/charts/istio \
      --version 1.3.0 \
      --namespace istio-system \
      --set global.imagePullSecrets[0].name=application-collection \
      --set gateway.enabled=true \
      --force-conflicts \
      --server-side=true \
      --wait --timeout 5m
  3. Install External Secrets Operator (ESO) from the local mirror.

    > helm upgrade --install external-secrets-operator \
      oci://mirror.corp.example.com/charts/external-secrets-operator \
      --version 2.3.0 \
      --namespace kubeflow \
      --set installCRDs=true \
      --set global.imagePullSecrets[0].name=application-collection \
      --wait --timeout 5m

1.5.2.6 How to override default registry settings?

You must instruct the Kubeflow chart and all its sub-charts to pull container images from your private registry instead of the default online repositories.

To configure registry overrides, create an airgapped-values.yaml file with the following settings:

# airgapped-values.yaml
global:
  suseApplicationCollectionRegistry: "mirror.corp.example.com"
  suseRegistry: "mirror.corp.example.com"

For more information on available overrides, see the Registry overrides documentation.

1.5.2.7 How to perform the Kubeflow installation?

Install Kubeflow using the charts stored in your local registry mirror. Pass your airgapped-values.yaml configuration file to override the default registry values.

> helm upgrade --install kubeflow \
  oci://mirror.corp.example.com/ai/charts/kubeflow \
  --version 0.4.1 \
  -n kubeflow \
  --force-conflicts \ 1
  --server-side=true \
  --wait --timeout 15m \
  -f airgapped-values.yaml

1

The --force-conflicts flag is required because several sub-components update shared Kubernetes resources that Helm tracks.

1.6 How do I access Kubeflow?

Once Kubeflow is deployed, the next step is choosing how to access its Web interface. The right method depends on your environment — whether you are running a local cluster, working inside a private network, or exposing Kubeflow externally.

The following options cover common access patterns, from simple local port forwarding to fully configured external endpoints with custom host names and TLS.

1.6.1 Option A: Port-forward (local dev)

If NodePorts are not directly reachable, use port-forward instead:

# Run in a separate terminal and keep it open
> kubectl port-forward svc/istio -n istio-system 8080:80

Navigate to http://localhost:8080.

1.6.2 Option B: NodePort (standard Linux cluster)

The Istio gateway Service is of type LoadBalancer and always has NodePorts assigned, even without a load balancer controller:

NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
HTTP_PORT=$(kubectl get svc istio -n istio-system -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
echo "http://${NODE_IP}:${HTTP_PORT}"

1.6.3 Option C: LoadBalancer external IP

If your cluster has MetalLB or a cloud load-balancer controller, the Service receives an external IP:

> kubectl get svc istio -n istio-system   # wait for EXTERNAL-IP

Navigate to http://<EXTERNAL-IP>.

Tip
Tip

No Helm values changes are needed for options A, B, or C.

1.6.4 Option D: Named host name (HTTP)

Set a host name to restrict the gateway to a specific FQDN. TLS is not required.

See Section 1.7.2, “Non-production: Named host name over HTTP” for override values.

1.6.5 Option E: Named host name with TLS (HTTPS)

See Section 1.7.3, “Non-production: Self-signed TLS” or Section 1.7.4, “Production: Let’s Encrypt TLS + external-dns” for override values.

Note
Note: external-dns is required for the Let’s Encrypt issuer

When using Let’s Encrypt as the issuer for Gateway TLS certificates, external-dns is required for DNS-01 challenges. Furthermore, a load balancer (i.e. MetalLB) must be used with external-dns. This ensures that external-dns can properly obtain an external IP from the load balancer to create the DNS record.

1.7 How do I configure Kubeflow for different scenarios?

Following is a set of common configuration scenarios for deploying Kubeflow. They range from simple non-production setups for development and testing to production-ready configurations with TLS and automated DNS.

1.7.1 Non-production: NodePort access (zero-configuration)

No custom values file is required. Install with the default values and access via NodePort or port-forward as described in Section 1.6, “How do I access Kubeflow?”.

Default credentials:

Email:    user@example.com
Password: 12341234

1.7.2 Non-production: Named host name over HTTP

Use this if you want a stable URL for a shared development cluster. You can point /etc/hosts at the cluster IP or use external-dns to automate DNS.

# my-values.yaml
global:
  kserveDomain: "models.dev.example.com"   # inference domain
kubeflow-istio-resources:
  hostname: "kubeflow.dev.example.com"
  externalDNSEnabled: false

After the installation, obtain the cluster IP and add a local DNS entry:

> kubectl get svc istio -n istio-system

# /etc/hosts entry (on your local machine or in the cluster)
192.168.1.100  kubeflow.dev.example.com

Navigate to: http://kubeflow.dev.example.com.

1.7.3 Non-production: Self-signed TLS

Suitable for shared development clusters where you can distribute the self-signed CA manually. Requires cert-manager. The chart creates a self-signed ClusterIssuer and requests a Certificate automatically. No kubectl steps are required beyond the Helm install. Add the self-signed CA to your Web browser trust store to avoid certificate warnings.

# my-values.yaml
global:
  kserveDomain: "models.dev.example.com"   # inference domain — see note above
kubeflow-istio-resources:
  hostname: "kubeflow.dev.example.com"
  externalDNSEnabled: false
  tls:
    source: "selfSigned"
    credentialName: kubeflow-gateway-tls
    httpsRedirect: true
Note
Note

The self-signed path can also serve inference over external HTTPS. It issues the wildcard directly, therefore no DNS-01 solver is needed on that path. Refer to Section 1.7.7, “External HTTPS for KServe inference (opt-in)” for more details.

1.7.4 Production: Let’s Encrypt TLS + external-dns

Recommended for Internet-facing production deployments. Uses DNS-01 challenge via Cloudflare, so HTTP-01 port requirements are avoided. Requires a Cloudflare API token with DNS edit access.

# prod-values.yaml
# ── Access ──────────────────────────────────────────────────────────────────────
global:
  kserveDomain: "models.example.com"   # inference domain — see note above
kubeflow-istio-resources:
  hostname: "kubeflow.example.com"
  externalDNSEnabled: true
  tls:
    source: "letsEncrypt"
    credentialName: kubeflow-gateway-tls
    httpsRedirect: true
    letsEncrypt:
      email: "admin@example.com"   # your ACME account email
      server: prod                 # prod | staging (use staging first to test)
      solver: cloudflare           # dns01 via Cloudflare
    cloudflare:
      email: "admin@example.com"
      apiTokenSecretRef:
        name: cloudflare-api-key
        key: apiKey

# external-dns — watches the Istio Gateway and creates/updates DNS records
externaldns:
  enabled: true
  provider:
    name: cloudflare
  cloudflare:
    apiToken: "<YOUR-CLOUDFLARE-API-TOKEN>"  # chart creates the Secret automatically
  domainFilters:
    - "example.com"
  txtOwnerId: "kubeflow"   # unique per cluster — prevents conflicts
  sources:
    - istio-gateway
  env:
    - name: CF_API_TOKEN
      valueFrom:
        secretKeyRef:
          name: cloudflare-api-key
          key: apiKey

# ── Credentials — change ALL of these (see Production Hardening) ─────────────────
auth:
  oidc:
    clientSecret: "<STRONG-RANDOM-32-CHAR-SECRET>"
  initialUser:
    email: "admin@example.com"

dex:
  config:
    staticClients:
      - id: kubeflow-oidc-authservice
        redirectURIs:
          - /oauth2/callback
        name: kubeflow-oidc-authservice
        secret: "<STRONG-RANDOM-32-CHAR-SECRET>"   # must match auth.oidc.clientSecret
    staticPasswords:
      - email: "admin@example.com"
        # Generate: htpasswd -nbBC 12 "" 'YourPassword' | tr -d ':\n' | sed 's/$2y/$2a/'
        hash: "<BCRYPT-HASH-OF-YOUR-PASSWORD>"
        username: admin
        userID: "1"
    enablePasswordDB: true

# ── Storage credentials — change these (see Production Hardening) ────────────────
pipelines:
  seaweedfs:
    accessKey: "<STRONG-ACCESS-KEY>"
    secretKey: "<STRONG-SECRET-KEY>"
  mariadb:
    backup:
      enabled: true        # recommended for production
      schedule: "0 2 * * *"
      storageSize: 20Gi

# User namespace must use the SAME SeaweedFS credentials
user-namespace:
  pipelines:
    seaweedfs:
      accessKey: "<STRONG-ACCESS-KEY>"    # same as pipelines.seaweedfs.accessKey
      secretKey: "<STRONG-SECRET-KEY>"    # same as pipelines.seaweedfs.secretKey

# ── Optional hardening ───────────────────────────────────────────────────────────
networkPolicies:
  enabled: false   # set true when using a CNI that enforces NetworkPolicy (Calico, Cilium, Canal)
monitoring:
  enabled: false   # set true after Rancher Monitoring (kube-prometheus-stack) is installed

Installation:

> helm upgrade --install kubeflow \
  oci://registry.suse.com/ai/charts/kubeflow \
  --version 0.4.1 \
  -n kubeflow \
  --force-conflicts \
  --wait --timeout 15m \
  -f prod-values.yaml
Important
Important: Using staging first is strongly recommended

Let’s Encrypt rate-limits production certificate issuance. Test with server: staging until the certificate is issued, then switch to server: prod and run helm upgrade again.

1.7.5 Production: Bring-your-own certificate

Use this if your organization manages TLS certificates through an existing PKI or secret manager. Create your TLS Secret in the istio-system namespace before installing:

> kubectl create secret tls my-kubeflow-tls \
  --cert=path/to/tls.crt \
  --key=path/to/tls.key \
  -n istio-system

Then reference it in your custom values file:

# prod-values.yaml
global:
  kserveDomain: "models.example.com"   # inference domain — see note above
  kserveExternalHttps: false
kubeflow-istio-resources:
  hostname: "kubeflow.example.com"
  externalDNSEnabled: false   # manage DNS separately
  tls:
    source: "secret"
    existingSecret: "my-kubeflow-tls"
    httpsRedirect: true

# Change credentials as shown in the Let's Encrypt scenario / Production Hardening
auth:
  oidc:
    clientSecret: "<STRONG-RANDOM-32-CHAR-SECRET>"
# ... (rest of credentials)

1.7.6 Production: Use an existing external-dns and cert-manager

If the existing environment already has external-dns and cert-manager, Kubeflow can make use of them if the following conditions are satisfied:

  • external-dns must be configured to watch for the istio-gateway source. You can check the deployment with the kubectl command:

    > $ kubectl get deployment external-dns -n external-dns -o yaml | grep source=
      - --source=service
      - --source=ingress
      - --source=istio-gateway
  • A cluster issuer must exist in the environment and be configured to issue certificates from a production public CA such as Let’s Encrypt. You can check the deployment with the kubectl command:

    > $ kubectl get clusterissuer
    NAME                     READY   AGE
    letsencrypt-production   True    75m

    To configure Kubeflow to use existing external-dns and cert-manager, reference them in your values:

# prod-values.yaml
global:
  kserveDomain: "models.example.com"   # inference domain — see note above
kubeflow-istio-resources:
  hostname: "kubeflow.example.com"
  externalDNSEnabled: true
  tls:
    source: "issuerRef"
    httpsRedirect: true
    issuerRef:
      name: letsencrypt-production
externaldns:
  enabled: false

# Change credentials as shown in the Let's Encrypt scenario / Production Hardening
auth:
  oidc:
    clientSecret: "<STRONG-RANDOM-32-CHAR-SECRET>"
# ... (rest of credentials)
Note
Note: Verify Istio CRDs installation

When adding istio-gateway as a source to external-dns, make sure the Istio CRDs are installed. Otherwise, the external-dns pod may keep crashing with an error indicating a failure to list the Istio gateway resource. However, the error will eventually resolve after Istio is installed by Kubeflow.

1.7.7 External HTTPS for KServe inference (opt-in)

KServe inference always routes through its own dedicated kserve-ingress-gateway (global.kserveGateway.name, default kubeflow/kserve-ingress-gateway). This gateway is separate from the kubeflow-gateway of the dashboard. The net-istio component binds the Knative ingress VirtualServices to this gateway. Its readiness prober only probes the gateway ports. Therefore, inference readiness is independent of the dashboard TLS on port :443.

By default, the dedicated gateway serves inference on port :80. The status.url of each InferenceService uses the default KServe {name}-{namespace}.{domain} host over HTTP. This behavior is identical to upstream KServe. Serving inference over external HTTPS is optional. To enable this option, configure the two global parameters shown below and point the gateway at a TLS issuer. The Helm chart then adds a *.<kserveDomain>:443 server to the dedicated gateway. The chart also adds a redirect from port :80 to port :443.

# --- KServe external HTTPS opt-in ---
global:
  kserveDomain: "models.example.com"  # base domain for inference hosts + wildcard cert. Prefer a
                                      # scoped subdomain (models.example.com) over an org-wide domain:
                                      # the wildcard is "*.<kserveDomain>", so a bare domain widens the
                                      # cert's blast radius to your whole zone.
  kserveExternalHttps: true           # the only switch — everything else is auto-derived

kubeflow-istio-resources:
  hostname: "kubeflow.example.com"    # required: the dashboard host (chart fails fast if unset). Keep
                                      # it outside the *.models.example.com wildcard so the dashboard
                                      # gets its own cert.
  tls:
    source: "letsEncrypt"             # or selfSigned / secret / issuerRef
    letsEncrypt:
      email: "admin@example.com"
      solver: cloudflare              # DNS-01 required for wildcard certs
    cloudflare:                       # required for the cloudflare DNS-01 solver
      email: "admin@example.com"
      apiTokenSecretRef:
        name: cloudflare-api-token-secret   # must live in cert-manager's namespace
        key: api-token

You do not need to set kserve.urlScheme, kserve.ingressDomain, knativeServing.domain, or knativeServing.domainTemplate manually. The chart derives these parameters from the two global variables. The urlScheme becomes https, and the domains use the value of global.kserveDomain. The chart flattens the Knative route host to a single {name}-{namespace}.{domain} label so that the wildcard certificate covers it. Consequently, the dedicated kserve-ingress-gateway serves *.<kserveDomain> on port :443 with a separate certificate named kserve-wildcard-tls. Because this gateway is separate from the dashboard gateway, a DNS-01 failure on the wildcard does not affect the dashboard TLS. The Helm chart validation fails during rendering if the configuration is incomplete. This includes a missing hostname, an invalid tls.source, an HTTP-01 solver, an unpopulated Cloudflare block, or a disabled OAuth2 proxy.

1.7.7.1 Requirements, security model, and caveats

  • Requires the bundled oauth2-proxy (global.oauth2Proxy.enabled: true, the default). The chart only renders the gateway authentication (authn-filter ext_authz chain) when you enable the OAuth2 proxy. If you disable this proxy to use a custom identity provider (IdP), the dedicated gateway performs no authentication. Consequently, the wildcard server on *.<kserveDomain>:443 publishes the inference endpoints of all tenants without authentication. Therefore, the chart does not render the wildcard server and fails validation if global.kserveExternalHttps: true is combined with global.oauth2Proxy.enabled: false. If you use custom gateway authentication, do not enable this option. Expose the inference service through your own authenticated ingress instead.

  • Requires a DNS-01 solver (solver: cloudflare). Let’s Encrypt HTTP-01 cannot issue wildcard certificates. The apiTokenSecretRef secret must exist in the namespace of cert-manager. A ClusterIssuer resolves solver secrets in that namespace, not in the release namespace. Scope the Cloudflare API token to Zone > DNS > Edit and Zone > Zone > Read for zone discovery. Apply this scope to the specific zones that cover kserveDomain or the dashboard host, rather than selecting all zones. This restriction ensures that a leaked token can only access those specific zones.

  • The Knative route host is flattened to {name}-{namespace}.{domain} to allow a single *.\{domain\} wildcard to cover them. Let’s Encrypt cannot issue a two-level wildcard like *.*.\{domain\}. Knative and KServe render this host using the Go standard library text/template package instead of Helm or Sprig. Therefore, the namespace cannot be hashed in the template.

  • This option is designed for single-tenant environments. Do not enable this option on an untrusted multi-tenant cluster. Name and namespace are joined by a hyphen into one DNS label. Consequently, two different name and namespace pairs can resolve to the identical host. For example, the name victim-team in the alpha namespace and the name victim in the team-alpha namespace both resolve to victim-team-alpha.{domain}. Similarly, the name a in the b-c namespace collides with the name a-b in the c namespace. Istio resolves host collisions by prioritizing the oldest VirtualService. A user who can create an InferenceService in any namespace can hijack or deny the inference host of another user with a colliding name. Because the user chooses the name, admin-provisioned namespaces do not prevent this risk. Only enable the wildcard option when you trust every namespace that can host an InferenceService. This applies to single-tenant environments or mutually trusting teams. Avoid using namespace names that are hyphenated suffixes of other hostnames. For strict isolation between namespaces on an untrusted cluster, keep inference traffic cluster-local and disable this option. Alternatively, issue one wildcard certificate per namespace using the dotted {name}.{namespace}.{domain} host format.

  • The gateway authenticates but does not authorize per tenant. Any authenticated user can invoke any InferenceService. The external authorization chain (ext_authz) checks that the caller has a valid Dex session or Bearer token. It does not check if the caller owns the namespace of the model. No gateway-scoped AuthorizationPolicy binds *.<kserveDomain> hosts to their owning namespace. The AuthorizationPolicy of the Knative activator allows all traffic. Additionally, the OAuth2 proxy accepts any domain. Testing with two users confirmed that a token for any Dex user is accepted at every inference host, regardless of the namespace containing the model. This behavior is acceptable for single tenants or mutually trusting teams. However, on an untrusted multi-tenant cluster, one user can invoke and read outputs from the models of another user. To isolate invocations between tenants, disable this option and keep inference traffic cluster-local. In this local configuration, namespace-scoped policies still apply. Alternatively, add a custom gateway-scoped AuthorizationPolicy that binds each host to its namespace.

  • The flattened label must stay under 63 characters. The Domain Name System (DNS) standard limits any single label to 63 characters. The wildcard certificate must cover the component Knative service rather than the InferenceService. For example, a serverless predictor routes as {isvc}-predictor-{namespace}. You must budget for the ten-character -predictor suffix. Long InferenceService or namespace names can exceed the limit.

  • External access requires a Bearer token or session cookie. Serving over HTTPS does not bypass Kubeflow authentication. See the Tutorial: Programmatic Access for instructions on how to authenticate external curl or Python clients.

  • The configuration path for custom certificates differs. For more information, see Section 1.7.5, “Production: Bring-your-own certificate”.

  • To keep inference on plain HTTP, leave the kserveExternalHttps parameter set to false. In this case, the dedicated gateway serves port :80 only without a wildcard certificate. The status.url parameter does not change and uses the default {name}-{namespace}.{domain} host over HTTP.

1.8 How do I harden Kubeflow for production use?

The override values for the chart ship with defaults that are not suitable for production. Update these values before exposing the deployment to any network or storing sensitive data.

Warning
Warning: Demo credentials

By default (global.demoMode: false), the chart fails at render time with a security error if any well-known demo credential is still present. To suppress this during local development, set global.demoMode: true in your override values file but never set this in production.

  1. Change all default credentials. The credentials that trigger the render-time security check:

    # in your kubeflow-override-values.yaml
    auth:
      oidc:
        clientSecret: "<STRONG-RANDOM-SECRET>"
        cookieSecret: "<STRONG-RANDOM-32-BYTE-BASE64>"  # generate: openssl rand -base64 32
    
    dex:
      config:
        staticClients:
          - id: kubeflow-oidc-authservice
            redirectURIs:
              - /oauth2/callback
            name: kubeflow-oidc-authservice
            secret: "<STRONG-RANDOM-SECRET>"   # must match auth.oidc.clientSecret above
        staticPasswords:
          - email: "admin@yourcompany.com"
            # Generate: htpasswd -nbBC 12 "" 'YourPassword' | tr -d ':\n' | sed 's/$2y/$2a/'
            hash: "<BCRYPT-HASH>"
            username: admin
            userID: "1"
        enablePasswordDB: true
    
    pipelines:
      seaweedfs:
        accessKey: "<STRONG-ACCESS-KEY>"
        secretKey: "<STRONG-SECRET-KEY>"
    
    user-namespace:
      pipelines:
        seaweedfs:
          accessKey: "<STRONG-ACCESS-KEY>"    # must match pipelines.seaweedfs.accessKey
          secretKey: "<STRONG-SECRET-KEY>"    # must match pipelines.seaweedfs.secretKey
    Note
    Note: MariaDB root password

    Both the KFP and Katib MySQL secrets are auto-generated (24-char random password) on the first install. They are preserved across upgrades with no action required. To rotate them, delete the secret and run helm upgrade to regenerate:

    > kubectl delete secret mysql-secret -n kubeflow        # for KFP
    > kubectl delete secret katib-mysql-secrets -n kubeflow # for Katib
    > helm upgrade kubeflow . -f kubeflow-override-values.yaml -n kubeflow
  2. Use an external identity provider. Replace Dex static passwords with an LDAP, SAML, or upstream OIDC connector. Add a connectors block to dex.config and remove staticPasswords and enablePasswordDB: true.

  3. NetworkPolicies. NetworkPolicies are disabled by default. They use an 'ingress-only deny-by-default' model where egress is unrestricted so that components can reach external services such as Hugging Face and container registries. Such configurations are supported by Calico, Cilium, Canal, and any other CNI that enforces NetworkPolicy. If your CNI does not enforce NetworkPolicy, enable it:

    # in your kubeflow-override-values.yaml
    networkPolicies:
      enabled: true
  4. Enable TLS. Refer to Section 1.7.4, “Production: Let’s Encrypt TLS + external-dns” or Section 1.7.5, “Production: Bring-your-own certificate” for more details.

  5. Enable database backups.

    # in your kubeflow-override-values.yaml
    pipelines:
      mariadb:
        backup:
          enabled: true
          schedule: "0 2 * * *"   # daily at 02:00 UTC
          storageSize: 20Gi

    To restore the backup:

    # List available backups
    > kubectl exec -n kubeflow sts/mysql -- ls /backup/
    
    # Restore
    > kubectl exec -n kubeflow sts/mysql -- \
      sh -c "mariadb --ssl=false -u root < /backup/<filename>.sql"
  6. Enable pre-install validation.

    # in your kubeflow-override-values.yaml
    preflightChecks:
      enabled: true

    Runs a hook job before the installation that validates that the default StorageClass exists and that cert-manager CRDs are registered.

  7. Enable High Availability. Apply ha-overrides.yaml (provided in the repository) on top of your base values to scale the Katib controller, training-operator, and KServe controller to 2 replicas. KFP and Dex PodDisruptionBudgets are already enabled by default.

    > helm upgrade kubeflow . -f kubeflow-override-values.yaml -f ha-overrides.yaml -n kubeflow

    PDBs protect against voluntary disruptions (node drains) but only provide meaningful coverage with 2 or more replicas. With a single replica, the PDB allows full eviction. See Known limitations (Section 1.2.3, “What are known Kubeflow limitations?”) for supported HA controllers.

  8. Apply resource quotas per user namespace.

    # in your kubeflow-override-values.yaml
    additionalUsers:
      - email: alice@example.com
        namespace: alice
        resourceQuota:
          requests.cpu: "4"
          requests.memory: "8Gi"
          requests.nvidia.com/gpu: "1"

1.9 How do I manage user profiles and namespaces?

Kubeflow uses a profile-per-user model. The default user namespace that Kubeflow creates during the installation is kubeflow-user-example-com.

1.9.1 Adding users during installation

Insert or update the following snippet in your kubeflow-override-values.yaml file:

# in your kubeflow-override-values.yaml
user-namespace:
  additionalUsers:
    - email: tux@example.com
      namespace: tux
      resourceQuota:
        requests.cpu: "4"
        requests.memory: 8Gi
    - email: geeko@example.com
      namespace: geeko
Important
Important: Use an explicit namespace

The namespace field is optional but strongly recommended. Without it, the namespace is auto-generated from the e-mail by replacing @ with -- and . with -. e-mail that differ only by . vs - — such as tux.geeko@example.com and tux-geeko@example.com — produce the same auto-generated namespace. Use an explicit namespace to disambiguate.

1.9.2 Adding users after installation

Add the user to your values file and run helm upgrade:

> helm upgrade kubeflow oci://registry.suse.com/ai/charts/kubeflow \
  --version <VERSION> \
  -n kubeflow --reuse-values \
  --set "user-namespace.additionalUsers[0].email=tux@example.com" \
  --set "user-namespace.additionalUsers[0].namespace=tux"

Alternatively, add the user to your values file and rerun the upgrade:

# in your kubeflow-override-values.yaml
user-namespace:
  additionalUsers:
    - email: tux@example.com
      namespace: tux
> helm upgrade kubeflow oci://registry.suse.com/ai/charts/kubeflow \
  --version <VERSION> \
  -n kubeflow -f kubeflow-override-values.yaml

This creates the Profile CR and deploys all required KFP per-namespace resources (pipeline artifact server, visualization server, credentials, authorization policies) in a single step.

Note
Note

Do not add users by applying a Profile CR directly with kubectl. The profiles controller only creates namespace-level RBAC — it does not deploy the KFP per-namespace resources that pipelines depend on. Users added this way will have an incomplete environment and the pipeline runs will fail.

1.10 How do I upgrade Kubeflow?

You can upgrade Kubeflow either using Rancher Prime or on the command line with Helm.

1.10.1 Upgrading with Rancher Prime

Refer to AI application upgrade for a general upgrade procedure.

1.10.2 Upgrading with Helm

  1. Log in to SUSE Registry. To prevent the password from being stored in the shell history, save it to the password.txt file.

    > helm registry login registry.suse.com \
      --username=regcode \
      --password-stdin < password.txt
  2. Upgrade Kubeflow.

    > helm upgrade kubeflow \
      oci://registry.suse.com/ai/charts/kubeflow \
      -n kubeflow \
      --version <VERSION_NUMBER> \ 1
      --wait --timeout 15m \
      --force-conflicts

    1

    If you omit --version, Kubeflow gets upgraded to the latest available version.

1.10.3 Upgrade notes

Following are important considerations and specific instructions for managing and upgrading your Kubeflow deployment. They cover critical aspects like configuration conflicts, credential rotation, and potential service interruptions.

Always use --force-conflicts

The --force-conflicts flag is required for helm upgrade. Certain components, such as cert-manager-cainjector and istiod, modify fields that Helm also manages. This flag tells Helm to overwrite these external changes and reclaim ownership.

> helm upgrade kubeflow \
  oci://registry.suse.com/ai/charts/kubeflow \
  --version 0.3.1 \
  -n kubeflow --force-conflicts --wait --timeout 15m
Switching database engines or changing StorageClass

PVCs with the option helm.sh/resource-policy: keep are not deleted by Helm. When switching database images or StorageClasses, delete the PVC manually first:

# For KFP MariaDB (Rancher MariaDB StatefulSet)
> kubectl delete pvc -n kubeflow data-mysql-0

# For Katib MariaDB (standalone Deployment, not StatefulSet)
> kubectl delete pvc -n kubeflow katib-mysql
Training operator webhook secret migration (auto)

On upgrade from chart versions prior to 0.3.0, a pre-upgrade hook automatically migrates the training-operator-webhook-cert Secret from type kubernetes.io/tls to Opaque. No manual action is required. If the kubectl get jobs -n kubeflow command reports a hook failure, delete the secret manually and rerun helm upgrade:

> kubectl delete secret training-operator-webhook-cert -n kubeflow --ignore-not-found
> helm upgrade kubeflow charts/kubeflow -n kubeflow --force-conflicts --wait --timeout 15m
Credential rotation (SeaweedFS)

Changing pipelines.seaweedfs.accessKey and secretKey requires restarting all KFP Deployments that read those credentials. Pods do not automatically restart when a Secret changes:

> kubectl rollout restart deployment -n kubeflow \
  ml-pipeline ml-pipeline-ui ml-pipeline-persistenceagent ml-pipeline-scheduledworkflow
> kubectl rollout restart deployment -n kubeflow-user-example-com \
  ml-pipeline-ui-artifact

SeaweedFS IAM accumulates credentials across restarts (the postStart hook adds, never removes). To clean stale entries after a rotation:

> kubectl exec -n kubeflow deploy/seaweedfs -- \
  sh -c "printf 's3.configure -user <OLD_USER> -access_key <OLD_KEY> -delete -apply\n' \
    | weed shell -master 127.0.0.1:9333"
SeaweedFS upgrade downtime

SeaweedFS uses a Recreate deployment strategy (single-node S3 store backed by a PVC). During helm upgrade, the old pod is terminated before the new pod starts. Expect a brief window of 10s to 60s where S3 artifact uploads and downloads are unavailable. In-flight pipeline runs may stall.

We recommend temporarily disabling active pipeline runs in the GUI before upgrading.

Supply the full values file on upgrade

Running helm upgrade --reuse-values does not update the user-namespace Secret with new SeaweedFS credentials. Always supply the full values file on upgrade, or patch the Secret manually after upgrade.

CRDs are not automatically upgraded by helm upgrade

Kubeflow CRDs are placed in crds/ subdirectories. Helm intentionally skips them on upgrade. After a chart version bump, run:

> helm show crds oci://registry.suse.com/ai/charts/kubeflow \
  --version <VERSION> | kubectl apply -f -

1.11 How do I remove Kubeflow?

You can remove Kubeflow either using Rancher Prime or on the command line with Helm.

1.11.1 Removal using Rancher Prime

Refer to AI application removal for a general removal procedure.

1.11.2 Removal using Helm

To remove Kubeflow using Helm, use the following command:

> helm uninstall kubeflow -n kubeflow

1.12 Troubleshooting

Learn about solutions for common issues encountered when using Kubeflow.

Pods are not starting — too many open files

Run on each cluster node:

> sudo sysctl -w fs.inotify.max_user_watches=524288
> sudo sysctl -w fs.inotify.max_user_instances=512

# Persist across reboots
> echo "fs.inotify.max_user_watches=524288" | \
  sudo tee -a /etc/sysctl.d/99-kubeflow.conf
> echo "fs.inotify.max_user_instances=512"  | \
  sudo tee -a /etc/sysctl.d/99-kubeflow.conf
Pods are stuck pending — storage issues
> kubectl get pvc -n kubeflow
> kubectl describe pvc -n kubeflow <PVC_NAME>

Ensure the default StorageClass exists:

> kubectl get storageclass

If none is marked as default, patch one:

kubectl patch storageclass <NAME> \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
metadata-grpc CrashLoopBackOff (MLMD)

The metadb database is created by a post-install hook job. If the job failed, rerun it:

> kubectl delete job metadb-init -n kubeflow --ignore-not-found
> helm template kubeflow oci://registry.suse.com/ai/charts/kubeflow --version <VERSION> -n kubeflow \
  --show-only 'charts/pipelines/templates/Job/metadb-init-kubeflow-Job.yaml' \
  | kubectl apply -n kubeflow -f -
TensorBoard unavailable / controller CrashLoopBackOff

Check that the tensorboard-controller-config ConfigMap contains the following required keys:

  • ISTIO_HOST: "*"

  • ISTIO_GATEWAY: kubeflow/kubeflow-gateway (must be in the namespace/name format)

> kubectl get configmap -n kubeflow -l app=tensorboard-controller -o yaml | grep -A5 'data:'

If either the key is wrong or missing, upgrade the chart with the following overrides:

tensorboard-controller:
  configMapData:
    ISTIO_HOST: "*"
    ISTIO_GATEWAY: kubeflow/kubeflow-gateway
KServe InferenceService not progressing

Verify that the ClusterStorageContainer CRD and a default object exist:

> kubectl get crd clusterstoragecontainers.serving.kserve.io
> kubectl get clusterstoragecontainer default

If either is missing, rerun helm upgrade to install them.

"RBAC: access denied" for user namespace traffic

Rancher Prime Istio uses the istio ServiceAccount (not istio-ingressgateway-service-account). The rancher-ingressgateway-access AuthorizationPolicy in each user namespace handles this.

If you pre-created the AuthorizationPolicy with kubectl before the first Helm install, you need to add Helm ownership labels/annotations before upgrading:

> kubectl label authorizationpolicy rancher-ingressgateway-access \
  -n kubeflow-user-example-com app.kubernetes.io/managed-by=Helm --overwrite
> kubectl annotate authorizationpolicy rancher-ingressgateway-access \
  -n kubeflow-user-example-com \
  meta.helm.sh/release-name=kubeflow \
  meta.helm.sh/release-namespace=kubeflow --overwrite
SeaweedFS pod stuck on ContainerCreating

On K3s 1.34 with cri-dockerd (Rancher Prime Desktop), a race between the CNI and the container runtime can leave the SeaweedFS pod stuck. SeaweedFS acquires a LevelDB file lock on startup. If the pod is stuck, the lock is held and the next pod will also fail to start. To recover, run the following commands:

# Find the Docker container ID for the stuck pod
> docker ps | grep seaweedfs

# Release the lock
> docker kill <container-id>

# Delete the stuck pod — a new pod will start cleanly
> kubectl delete pod -n kubeflow -l app=seaweedfs
Dex login loop (infinite redirect)

This usually means that oauth2-proxy is receiving a 403 from an Istio AuthorizationPolicy rather than from oauth2-proxy itself. The Lua-redirect filter only converts 403 → 302 when the response includes a set-cookie header; AuthorizationPolicy 403 messages do not have one.

Check for sidecar-level AuthorizationPolicy denials:

> kubectl logs -n istio-system -l app=istiod | grep "RBAC"
> kubectl get authorizationpolicy -A