1 Kubeflow #
Kubeflow (https://www.kubeflow.org) is a cloud-native platform that orchestrates and simplifies the entire machine learning (ML) lifecycle on Kubernetes. Unlike traditional software with hard-coded rules, ML applications learn to make decisions and predictions directly from data. Kubeflow automates this data-driven process, managing everything from initial development to production deployment.
Kubeflow is currently available as a Technology Preview.
1.1 What components does Kubeflow include? #
The following table describes all the components of the Kubeflow package.
| Component | Chart source | Namespace | Purpose |
|---|---|---|---|
Dex | SUSE Application Collection | kubeflow | OIDC identity provider |
oauth2-proxy | SUSE Application Collection | kubeflow | OIDC login broker / session cookie |
Central Dashboard | local | kubeflow | Web UI shell |
Kubeflow Pipelines | local | kubeflow | ML workflow engine (KFP v2) |
Notebook Controller | local | kubeflow | Jupyter Notebook CRD controller |
Jupyter Web App | local | kubeflow | Notebook creation UI |
Katib | local | kubeflow | Hyperparameter tuning |
KServe | local | kubeflow | Model serving |
Training Operator V1 | local | kubeflow | TFJob / PyTorchJob / etc. ( |
Trainer V2 | local | kubeflow | TrainJob / TrainingRuntime / ClusterTrainingRuntime ( |
Profiles & KFAM | local | kubeflow | Multi-tenancy / namespace management |
Volumes Web App | local | kubeflow | PVC management UI (co-enabled with PVCViewer Controller via |
PVCViewer Controller | local | kubeflow | File browser for PVCs (co-enabled with Volumes Web App) |
Tensorboard Controller | local | kubeflow | TensorBoard CRD controller |
Tensorboards Web App | local | kubeflow | TensorBoard UI |
Models Web App | local | kubeflow | Model serving UI (co-enabled with KServe via |
Model Registry | local | kubeflow | Model versioning and metadata |
Admission Webhook | local | kubeflow | PodDefault mutating webhook (always installed — no condition) |
Knative Serving | local | knative-serving | Serverless layer required by KServe |
Knative Eventing | local | knative-eventing | Optional event-driven pipeline triggers (disabled by default) |
MariaDB (KFP) | SUSE Application Collection | kubeflow | Pipeline + MLMD metadata DB |
MariaDB (Katib) | inline | kubeflow | Katib experiment DB |
SeaweedFS | inline | kubeflow | Pipeline artifact store (S3-compatible, Docker Hub image — requires internet access) |
external-dns | SUSE Application Collection | kubeflow | Automatic DNS record management for Cloudflare (disabled by default) |
NetworkPolicies | local | kubeflow / knative-serving | Deny-all + explicit allow NetworkPolicies (disabled by default) |
Monitoring | local | kubeflow | ServiceMonitors + PrometheusRules for Rancher Monitoring (disabled by default) |
1.2 What are Kubeflow requirements? #
Deploying and running Kubeflow has the following requirements.
1.2.1 Hardware requirements #
| Profile | CPU | RAM | GPU | Storage |
|---|---|---|---|---|
Minimal evaluation | 8 cores | 16 GB | None | 100+ GB |
Recommended evaluation | 8-18 cores | 32-64 GB | 1 GPU | 250+ GB |
Production / large language model development | 16+ cores | 64-128+ GB | 2+ GPUs (workload-dependent) | 500+ GB |
1.2.2 Software requirements #
| Requirement | Version | Notes |
|---|---|---|
Kubernetes | >= 1.31 | Tested on RKE2 / K3s |
Helm | >= 4.0.1 | Required for OCI chart support |
cert-manager | 1.20.2 | — |
Istio | mesh ≥ 1.9 (chart 1.3.0) | — |
Load Balancer (i.e. metallb) | — | Required when used in conjunction with external-dns and cert-manager. Tested with MetalLB v0.15.3 |
Default StorageClass | — | Local Path Provisioner (development) or Longhorn (production) |
SUSE Application Collection credentials | — | User name + token for |
SUSE Registry credentials | — | User name + token for |
- Storage class
All PVCs use the cluster default StorageClass unless
global.storageClassis set. For single-node clusters the Local Path Provisioner is sufficient for development. For production, Longhorn is recommended.NoteMake sure a default StorageClass is set for the cluster before installing. Run
kubectl get storageclassand confirm that one has the (default) designation next to its name. Otherwise, designate one, for example:> kubectl patch storageclass local-path \ -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'- Cloudflare API token
Only required if you want automatic DNS record management via
external-dnsor Let’s Encrypt DNS-01 challenges.include::../common/generic-attributes.adoc[]
1.2.3 What are known Kubeflow limitations? #
The current Kubeflow distribution has the following known limitations. Understanding these limitations is crucial for successful deployment and operation.
Namespace names, such as
kubeflow,knative-servingoristio-system, are hardcoded in most templates. Deploying to a non-standard namespace requires template modifications.Argo workflow executor image pull secrets must be set via the
workflow-controller-configmapworkflowDefaultsfield. They cannot be set via Helm values at runtime because the configmap is rendered at pod-create time, not Helm time.knativeServing.enabledmust betruefor KServe to function. Disabling Knative Serving will cause KServe InferenceService resources to remain in a non-Ready state.Only the Cloudflare provider is tested for external-dns out of the box. Other providers (AWS Route53, Azure, GCP) require additional
externaldns.envconfiguration.SeaweedFS is single-node, non-replicated. Pipeline artifact storage has no HA. A SeaweedFS pod restart causes a brief (~10–60s) S3 outage. For production HA, replace SeaweedFS with an external S3-compatible store.
Not all Kubeflow controllers support multiple replicas. Leader election is confirmed for katib-controller (v0.17+), training-operator, kserve-controller-manager, pvcviewer-controller, and model-registry-controller. These are safe to scale via
ha-overrides.yaml. It is not confirmed for notebook-controller, profiles-controller, tensorboard-controller, and several KFP background workers. SettingreplicaCount > 1for these controllers causes undefined behavior, such as duplicate reconciliation or data corruption.CRDs are not automatically upgraded by
helm upgrade. Kubeflow CRDs are placed incrds/subdirectories. Helm intentionally skips them on upgrade. After a chart version bump, run:> helm show crds oci://registry.suse.com/ai/charts/kubeflow \ --version <VERSION> | kubectl apply -f -Dex must be v0.23.0 (dex 2.42.0). v0.24.0 (dex 2.44.0) uses Go 1.25, which introduced strict IPv6 URL parsing. Kubernetes API server addresses like
[10.43.0.1]:443are rejected, crashing Dex on startup. The chart pins v0.23.0.SeaweedFS image is pulled from Docker Hub (
chrislusf/seaweedfs:4.00). Air-gapped clusters or environments with Docker Hub pull-rate limits will fail to start Kubeflow Pipelines. Mirror the image to a private registry and setpipelines.seaweedfs.image.registryto override.
1.3 How do I get the values list for Kubeflow Helm chart? #
Before deploying Kubeflow, it is important to know more about the supported configurations.
(Optional) To get the values for a specific Kubeflow chart version, log in to SUSE Registry and identify the version. To prevent the password from being stored in the shell history, save it to the
password.txtfile.> skopeo login registry.suse.com \ --username regcode \ --password-stdin < password.txt > skopeo list-tags docker://registry.suse.com/ai/charts/kubeflowLog in to SUSE Registry where Kubeflow chart is stored. To prevent the password from being stored in the shell history, save it to the
password.txtfile.> helm registry login registry.suse.com \ --username regcode \ --password-stdin < password.txtDisplay all values of the Kubeflow Helm chart. Omit
--versionto display the latest chart.> helm show values oci://registry.suse.com/ai/charts/kubeflow \ --version <VERSION>
1.4 How do I install Kubeflow? #
You can install Kubeflow on the command line with Helm.
1.4.1 Install using Helm #
To install Kubeflow on the command line using Helm, follow these steps:
Log in to the SUSE Application Collection registry. To prevent the password from being stored in the shell history, save it to the
paassword.txtfile.> helm registry login dp.apps.rancher.io \ --username=<APPCO-REGISTRY-USERNAME> \ --password-stdin < paassword.txtLog in to SUSE Registry. To prevent the password from being stored in the shell history, save it to the
paassword.txtfile.> helm registry login registry.suse.com \ --username=regcode \ --password-stdin < paassword.txtCreate namespaces.
for ns in cert-manager istio-system kubeflow knative-serving; do kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f - doneNotekubeflow-user-example-comand all other user namespaces are created by the Kubeflow chart and receive registry pull secrets automatically via ESO — no manual creation needed.Create image pull secrets.
Create secrets only in the static system namespaces. ESO propagates them to user namespaces automatically by copying
suse-ai-registryandapplication-collectionfrom the Kubeflow namespace into every user namespace labelledapp.kubernetes.io/part-of: kubeflow-profile, including namespaces created dynamically by the profiles controller at runtime.Create SUSE Application Collection image pull secrets.
for ns in cert-manager istio-system kubeflow knative-serving; do kubectl create secret docker-registry application-collection \ --docker-server=dp.apps.rancher.io \ --docker-username=<appco-registry-username> \ --docker-password=<appco-registry-token> \ -n "$ns" \ --dry-run=client -o yaml | kubectl apply -f - doneCreate SUSE Registry image pull secrets.
for ns in cert-manager istio-system kubeflow knative-serving; do kubectl create secret docker-registry suse-ai-registry \ --docker-server=registry.suse.com \ --docker-username=regcode \ --docker-password=<suse-ai-registry-token> \ -n "$ns" \ --dry-run=client -o yaml | kubectl apply -f - done
Label namespaces for Helm.
for ns in kubeflow kubeflow-user-example-com knative-serving; do kubectl label namespace "$ns" app.kubernetes.io/managed-by=Helm --overwrite kubectl annotate namespace "$ns" \ meta.helm.sh/release-name=kubeflow \ meta.helm.sh/release-namespace=kubeflow \ --overwrite doneInstall cert-manager.
> helm upgrade --install cert-manager \ oci://dp.apps.rancher.io/charts/cert-manager \ --version 1.20.2 \ --namespace cert-manager \ --set crds.enabled=true \ --set crds.keep=true \ --set global.imagePullSecrets[0].name=application-collection \ --wait --timeout 5mInstall Istio.
> helm upgrade --install istio \ oci://dp.apps.rancher.io/charts/istio \ --version 1.3.0 \ --namespace istio-system \ --set global.imagePullSecrets[0].name=application-collection \ --set gateway.enabled=true \ --force-conflicts \ --server-side=true \ --wait --timeout 5mInstall External Secrets Operator (ESO). ESO must be installed before the Kubeflow chart. Its CRDs must be present when Helm applies the
ClusterSecretStoreandClusterExternalSecretresources.> helm upgrade --install external-secrets-operator \ oci://dp.apps.rancher.io/charts/external-secrets-operator \ --version 2.3.0 \ --namespace kubeflow \ --set installCRDs=true \ --set global.imagePullSecrets[0].name=application-collection \ --wait --timeout 5mTo override default installation values, list them in a
values.yamlfile. For a complete list of Kubeflow chart values, refer to Section 1.3, “How do I get the values list for Kubeflow Helm chart?”.Install Kubeflow. To keep the default installation values, omit
-f values.yaml.> helm upgrade --install kubeflow \ oci://registry.suse.com/ai/charts/kubeflow \ --version 0.4.0 \ -n kubeflow \ --force-conflicts \ 1 --server-side=true \ --wait --timeout 15m--force-conflictsis required because cert-manager-cainjector, istiod (pilot-discovery), and the clusterrole-aggregation-controller modify fields (caBundle, webhook failurePolicy, aggregated RBAC rules) that Helm tracks. This flag lets Helm reclaim ownership of those fields on each upgrade.
1.5 How do I install Kubeflow in air-gapped environments? #
Installing Kubeflow in an air-gapped environment allows you to deploy and manage machine learning workflows securely without active Internet connectivity. This deployment method uses local image registries and Helm repositories to satisfy all software dependencies, ensuring complete data privacy and network isolation.
1.5.1 Prerequisites #
A local private container registry mirror
Synced container images and Helm charts
A running Kubernetes cluster (v1.31 or higher)
1.5.2 How to install Kubeflow in an air-gapped environment? #
An air-gapped deployment requires you to host all container images and Helm charts in a local container registry. This process ensures the system can satisfy all software dependencies without contacting external networks.
Before starting the installation, you must mirror the required AI application charts and container images to your local private registry.
1.5.2.1 How to configure local registry credentials? #
To access your private registry, authenticate with your local mirror address.
To prevent passwords from appearing in the shell history, save your credentials to the mirror-password.txt file.
> helm registry login mirror.corp.example.com \
--username=<username> \
--password-stdin < mirror-password.txt1.5.2.2 How to prepare the Kubernetes namespaces? #
Create the required namespaces on the target cluster. Creating namespaces before the deployment allows you to pre-configure the necessary image pull secrets.
> for ns in cert-manager istio-system kubeflow knative-serving; do
kubectl create namespace "$ns" --dry-run=client -o yaml | kubectl apply -f -
done+
User namespaces, such as kubeflow-user-example-com, are automatically created by the Kubeflow chart.
The External Secrets Operator (ESO) handles registry pull secrets for these dynamic namespaces automatically.
1.5.2.3 How to create local image pull secrets? #
Create the pull secrets in all static system namespaces. The cluster uses these credentials to pull container images from your local mirror. ESO propagates these secrets to user namespaces at runtime.
Create SUSE Application Collection secrets pointing to your local registry.
> for ns in cert-manager istio-system kubeflow knative-serving; do kubectl create secret docker-registry application-collection \ --docker-server=mirror.corp.example.com \ --docker-username=<username> \ --docker-password=<token> \ -n "$ns" \ --dry-run=client -o yaml | kubectl apply -f - doneCreate SUSE Registry secrets pointing to your local registry.
> for ns in cert-manager istio-system kubeflow knative-serving; do kubectl create secret docker-registry suse-ai-registry \ --docker-server=mirror.corp.example.com \ --docker-username=<username> \ --docker-password=<token> \ -n "$ns" \ --dry-run=client -o yaml | kubectl apply -f - done
1.5.2.4 How to label and annotate namespaces? #
Configure the namespaces to allow Helm management. Applying these labels prevents resource ownership conflicts during the upgrade process.
> for ns in kubeflow kubeflow-user-example-com knative-serving; do
kubectl label namespace "$ns" app.kubernetes.io/managed-by=Helm --overwrite
kubectl annotate namespace "$ns" \
meta.helm.sh/release-name=kubeflow \
meta.helm.sh/release-namespace=kubeflow \
--overwrite
done1.5.2.5 How to install required infrastructure components? #
Install the supporting operators and controllers from your local registry mirror.
Install cert-manager from the local mirror.
> helm upgrade --install cert-manager \ oci://mirror.corp.example.com/charts/cert-manager \ --version 1.20.2 \ --namespace cert-manager \ --set crds.enabled=true \ --set crds.keep=true \ --set global.imagePullSecrets[0].name=application-collection \ --wait --timeout 5mInstall Istio from the local mirror.
> helm upgrade --install istio \ oci://mirror.corp.example.com/charts/istio \ --version 1.3.0 \ --namespace istio-system \ --set global.imagePullSecrets[0].name=application-collection \ --set gateway.enabled=true \ --force-conflicts \ --server-side=true \ --wait --timeout 5mInstall External Secrets Operator (ESO) from the local mirror.
> helm upgrade --install external-secrets-operator \ oci://mirror.corp.example.com/charts/external-secrets-operator \ --version 2.3.0 \ --namespace kubeflow \ --set installCRDs=true \ --set global.imagePullSecrets[0].name=application-collection \ --wait --timeout 5m
1.5.2.6 How to override default registry settings? #
You must instruct the Kubeflow chart and all its sub-charts to pull container images from your private registry instead of the default online repositories.
To configure registry overrides, create an airgapped-values.yaml file with the following settings:
# airgapped-values.yaml
global:
suseApplicationCollectionRegistry: "mirror.corp.example.com"
suseRegistry: "mirror.corp.example.com"For more information on available overrides, see the Registry overrides documentation.
1.5.2.7 How to perform the Kubeflow installation? #
Install Kubeflow using the charts stored in your local registry mirror.
Pass your airgapped-values.yaml configuration file to override the default registry values.
> helm upgrade --install kubeflow \
oci://mirror.corp.example.com/ai/charts/kubeflow \
--version 0.4.1 \
-n kubeflow \
--force-conflicts \ 1
--server-side=true \
--wait --timeout 15m \
-f airgapped-values.yamlThe |
1.6 How do I access Kubeflow? #
Once Kubeflow is deployed, the next step is choosing how to access its Web interface. The right method depends on your environment — whether you are running a local cluster, working inside a private network, or exposing Kubeflow externally.
The following options cover common access patterns, from simple local port forwarding to fully configured external endpoints with custom host names and TLS.
1.6.1 Option A: Port-forward (local dev) #
If NodePorts are not directly reachable, use port-forward instead:
# Run in a separate terminal and keep it open
> kubectl port-forward svc/istio -n istio-system 8080:80Navigate to http://localhost:8080.
1.6.2 Option B: NodePort (standard Linux cluster) #
The Istio gateway Service is of type LoadBalancer and always has NodePorts assigned, even without a load balancer controller:
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
HTTP_PORT=$(kubectl get svc istio -n istio-system -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')
echo "http://${NODE_IP}:${HTTP_PORT}"1.6.3 Option C: LoadBalancer external IP #
If your cluster has MetalLB or a cloud load-balancer controller, the Service receives an external IP:
> kubectl get svc istio -n istio-system # wait for EXTERNAL-IPNavigate to http://<EXTERNAL-IP>.
No Helm values changes are needed for options A, B, or C.
1.6.4 Option D: Named host name (HTTP) #
Set a host name to restrict the gateway to a specific FQDN. TLS is not required.
See Section 1.7.2, “Non-production: Named host name over HTTP” for override values.
1.6.5 Option E: Named host name with TLS (HTTPS) #
See Section 1.7.3, “Non-production: Self-signed TLS” or Section 1.7.4, “Production: Let’s Encrypt TLS + external-dns” for override values.
external-dns is required for the Let’s Encrypt issuerWhen using Let’s Encrypt as the issuer for Gateway TLS certificates, external-dns is required for DNS-01 challenges.
Furthermore, a load balancer (i.e. MetalLB) must be used with external-dns.
This ensures that external-dns can properly obtain an external IP from the load balancer to create the DNS record.
1.7 How do I configure Kubeflow for different scenarios? #
Following is a set of common configuration scenarios for deploying Kubeflow. They range from simple non-production setups for development and testing to production-ready configurations with TLS and automated DNS.
1.7.1 Non-production: NodePort access (zero-configuration) #
No custom values file is required. Install with the default values and access via NodePort or port-forward as described in Section 1.6, “How do I access Kubeflow?”.
Default credentials:
Email: user@example.com Password: 12341234
1.7.2 Non-production: Named host name over HTTP #
Use this if you want a stable URL for a shared development cluster.
You can point /etc/hosts at the cluster IP or use external-dns to automate DNS.
# my-values.yaml
global:
kserveDomain: "models.dev.example.com" # inference domain
kubeflow-istio-resources:
hostname: "kubeflow.dev.example.com"
externalDNSEnabled: falseAfter the installation, obtain the cluster IP and add a local DNS entry:
> kubectl get svc istio -n istio-system
# /etc/hosts entry (on your local machine or in the cluster)
192.168.1.100 kubeflow.dev.example.comNavigate to: http://kubeflow.dev.example.com.
1.7.3 Non-production: Self-signed TLS #
Suitable for shared development clusters where you can distribute the self-signed CA manually.
Requires cert-manager.
The chart creates a self-signed ClusterIssuer and requests a Certificate automatically.
No kubectl steps are required beyond the Helm install.
Add the self-signed CA to your Web browser trust store to avoid certificate warnings.
# my-values.yaml
global:
kserveDomain: "models.dev.example.com" # inference domain — see note above
kubeflow-istio-resources:
hostname: "kubeflow.dev.example.com"
externalDNSEnabled: false
tls:
source: "selfSigned"
credentialName: kubeflow-gateway-tls
httpsRedirect: trueThe self-signed path can also serve inference over external HTTPS. It issues the wildcard directly, therefore no DNS-01 solver is needed on that path. Refer to Section 1.7.7, “External HTTPS for KServe inference (opt-in)” for more details.
1.7.4 Production: Let’s Encrypt TLS + external-dns #
Recommended for Internet-facing production deployments. Uses DNS-01 challenge via Cloudflare, so HTTP-01 port requirements are avoided. Requires a Cloudflare API token with DNS edit access.
# prod-values.yaml
# ── Access ──────────────────────────────────────────────────────────────────────
global:
kserveDomain: "models.example.com" # inference domain — see note above
kubeflow-istio-resources:
hostname: "kubeflow.example.com"
externalDNSEnabled: true
tls:
source: "letsEncrypt"
credentialName: kubeflow-gateway-tls
httpsRedirect: true
letsEncrypt:
email: "admin@example.com" # your ACME account email
server: prod # prod | staging (use staging first to test)
solver: cloudflare # dns01 via Cloudflare
cloudflare:
email: "admin@example.com"
apiTokenSecretRef:
name: cloudflare-api-key
key: apiKey
# external-dns — watches the Istio Gateway and creates/updates DNS records
externaldns:
enabled: true
provider:
name: cloudflare
cloudflare:
apiToken: "<YOUR-CLOUDFLARE-API-TOKEN>" # chart creates the Secret automatically
domainFilters:
- "example.com"
txtOwnerId: "kubeflow" # unique per cluster — prevents conflicts
sources:
- istio-gateway
env:
- name: CF_API_TOKEN
valueFrom:
secretKeyRef:
name: cloudflare-api-key
key: apiKey
# ── Credentials — change ALL of these (see Production Hardening) ─────────────────
auth:
oidc:
clientSecret: "<STRONG-RANDOM-32-CHAR-SECRET>"
initialUser:
email: "admin@example.com"
dex:
config:
staticClients:
- id: kubeflow-oidc-authservice
redirectURIs:
- /oauth2/callback
name: kubeflow-oidc-authservice
secret: "<STRONG-RANDOM-32-CHAR-SECRET>" # must match auth.oidc.clientSecret
staticPasswords:
- email: "admin@example.com"
# Generate: htpasswd -nbBC 12 "" 'YourPassword' | tr -d ':\n' | sed 's/$2y/$2a/'
hash: "<BCRYPT-HASH-OF-YOUR-PASSWORD>"
username: admin
userID: "1"
enablePasswordDB: true
# ── Storage credentials — change these (see Production Hardening) ────────────────
pipelines:
seaweedfs:
accessKey: "<STRONG-ACCESS-KEY>"
secretKey: "<STRONG-SECRET-KEY>"
mariadb:
backup:
enabled: true # recommended for production
schedule: "0 2 * * *"
storageSize: 20Gi
# User namespace must use the SAME SeaweedFS credentials
user-namespace:
pipelines:
seaweedfs:
accessKey: "<STRONG-ACCESS-KEY>" # same as pipelines.seaweedfs.accessKey
secretKey: "<STRONG-SECRET-KEY>" # same as pipelines.seaweedfs.secretKey
# ── Optional hardening ───────────────────────────────────────────────────────────
networkPolicies:
enabled: false # set true when using a CNI that enforces NetworkPolicy (Calico, Cilium, Canal)
monitoring:
enabled: false # set true after Rancher Monitoring (kube-prometheus-stack) is installedInstallation:
> helm upgrade --install kubeflow \
oci://registry.suse.com/ai/charts/kubeflow \
--version 0.4.1 \
-n kubeflow \
--force-conflicts \
--wait --timeout 15m \
-f prod-values.yamlstaging first is strongly recommendedLet’s Encrypt rate-limits production certificate issuance.
Test with server: staging until the certificate is issued, then switch to server: prod and run helm upgrade again.
1.7.5 Production: Bring-your-own certificate #
Use this if your organization manages TLS certificates through an existing PKI or secret manager.
Create your TLS Secret in the istio-system namespace before installing:
> kubectl create secret tls my-kubeflow-tls \
--cert=path/to/tls.crt \
--key=path/to/tls.key \
-n istio-systemThen reference it in your custom values file:
# prod-values.yaml
global:
kserveDomain: "models.example.com" # inference domain — see note above
kserveExternalHttps: false
kubeflow-istio-resources:
hostname: "kubeflow.example.com"
externalDNSEnabled: false # manage DNS separately
tls:
source: "secret"
existingSecret: "my-kubeflow-tls"
httpsRedirect: true
# Change credentials as shown in the Let's Encrypt scenario / Production Hardening
auth:
oidc:
clientSecret: "<STRONG-RANDOM-32-CHAR-SECRET>"
# ... (rest of credentials)1.7.6 Production: Use an existing external-dns and cert-manager #
If the existing environment already has external-dns and cert-manager, Kubeflow can make use of them if the following conditions are satisfied:
external-dnsmust be configured to watch for theistio-gatewaysource. You can check the deployment with thekubectlcommand:> $ kubectl get deployment external-dns -n external-dns -o yaml | grep source= - --source=service - --source=ingress - --source=istio-gatewayA cluster issuer must exist in the environment and be configured to issue certificates from a production public CA such as Let’s Encrypt. You can check the deployment with the
kubectlcommand:> $ kubectl get clusterissuer NAME READY AGE letsencrypt-production True 75mTo configure Kubeflow to use existing
external-dnsandcert-manager, reference them in your values:
# prod-values.yaml
global:
kserveDomain: "models.example.com" # inference domain — see note above
kubeflow-istio-resources:
hostname: "kubeflow.example.com"
externalDNSEnabled: true
tls:
source: "issuerRef"
httpsRedirect: true
issuerRef:
name: letsencrypt-production
externaldns:
enabled: false
# Change credentials as shown in the Let's Encrypt scenario / Production Hardening
auth:
oidc:
clientSecret: "<STRONG-RANDOM-32-CHAR-SECRET>"
# ... (rest of credentials)When adding istio-gateway as a source to external-dns, make sure the Istio CRDs are installed.
Otherwise, the external-dns pod may keep crashing with an error indicating a failure to list the Istio gateway resource.
However, the error will eventually resolve after Istio is installed by Kubeflow.
1.7.7 External HTTPS for KServe inference (opt-in) #
KServe inference always routes through its own dedicated kserve-ingress-gateway
(global.kserveGateway.name, default kubeflow/kserve-ingress-gateway).
This gateway is separate from the kubeflow-gateway of the dashboard.
The net-istio component binds the Knative ingress VirtualServices to this gateway.
Its readiness prober only probes the gateway ports.
Therefore, inference readiness is independent of the dashboard TLS on port :443.
By default, the dedicated gateway serves inference on port :80.
The status.url of each InferenceService uses the default KServe {name}-{namespace}.{domain} host over HTTP.
This behavior is identical to upstream KServe.
Serving inference over external HTTPS is optional.
To enable this option, configure the two global parameters shown below and point the gateway at a TLS issuer.
The Helm chart then adds a *.<kserveDomain>:443 server to the dedicated gateway.
The chart also adds a redirect from port :80 to port :443.
# --- KServe external HTTPS opt-in ---
global:
kserveDomain: "models.example.com" # base domain for inference hosts + wildcard cert. Prefer a
# scoped subdomain (models.example.com) over an org-wide domain:
# the wildcard is "*.<kserveDomain>", so a bare domain widens the
# cert's blast radius to your whole zone.
kserveExternalHttps: true # the only switch — everything else is auto-derived
kubeflow-istio-resources:
hostname: "kubeflow.example.com" # required: the dashboard host (chart fails fast if unset). Keep
# it outside the *.models.example.com wildcard so the dashboard
# gets its own cert.
tls:
source: "letsEncrypt" # or selfSigned / secret / issuerRef
letsEncrypt:
email: "admin@example.com"
solver: cloudflare # DNS-01 required for wildcard certs
cloudflare: # required for the cloudflare DNS-01 solver
email: "admin@example.com"
apiTokenSecretRef:
name: cloudflare-api-token-secret # must live in cert-manager's namespace
key: api-tokenYou do not need to set kserve.urlScheme, kserve.ingressDomain, knativeServing.domain, or knativeServing.domainTemplate manually.
The chart derives these parameters from the two global variables.
The urlScheme becomes https, and the domains use the value of global.kserveDomain.
The chart flattens the Knative route host to a single {name}-{namespace}.{domain} label so that the wildcard certificate covers it.
Consequently, the dedicated kserve-ingress-gateway serves *.<kserveDomain> on port :443 with a separate certificate named kserve-wildcard-tls.
Because this gateway is separate from the dashboard gateway, a DNS-01 failure on the wildcard does not affect the dashboard TLS.
The Helm chart validation fails during rendering if the configuration is incomplete.
This includes a missing hostname, an invalid tls.source, an HTTP-01 solver, an unpopulated Cloudflare block, or a disabled OAuth2 proxy.
1.7.7.1 Requirements, security model, and caveats #
Requires the bundled oauth2-proxy (
global.oauth2Proxy.enabled: true, the default). The chart only renders the gateway authentication (authn-filterext_authz chain) when you enable the OAuth2 proxy. If you disable this proxy to use a custom identity provider (IdP), the dedicated gateway performs no authentication. Consequently, the wildcard server on*.<kserveDomain>:443publishes the inference endpoints of all tenants without authentication. Therefore, the chart does not render the wildcard server and fails validation ifglobal.kserveExternalHttps: trueis combined withglobal.oauth2Proxy.enabled: false. If you use custom gateway authentication, do not enable this option. Expose the inference service through your own authenticated ingress instead.Requires a DNS-01 solver (
solver: cloudflare). Let’s Encrypt HTTP-01 cannot issue wildcard certificates. TheapiTokenSecretRefsecret must exist in the namespace of cert-manager. AClusterIssuerresolves solver secrets in that namespace, not in the release namespace. Scope the Cloudflare API token to Zone > DNS > Edit and Zone > Zone > Read for zone discovery. Apply this scope to the specific zones that coverkserveDomainor the dashboard host, rather than selecting all zones. This restriction ensures that a leaked token can only access those specific zones.The Knative route host is flattened to
{name}-{namespace}.{domain}to allow a single*.\{domain\}wildcard to cover them. Let’s Encrypt cannot issue a two-level wildcard like*.*.\{domain\}. Knative and KServe render this host using the Go standard librarytext/templatepackage instead of Helm or Sprig. Therefore, the namespace cannot be hashed in the template.This option is designed for single-tenant environments. Do not enable this option on an untrusted multi-tenant cluster. Name and namespace are joined by a hyphen into one DNS label. Consequently, two different name and namespace pairs can resolve to the identical host. For example, the name
victim-teamin thealphanamespace and the namevictimin theteam-alphanamespace both resolve tovictim-team-alpha.{domain}. Similarly, the nameain theb-cnamespace collides with the namea-bin thecnamespace. Istio resolves host collisions by prioritizing the oldestVirtualService. A user who can create anInferenceServicein any namespace can hijack or deny the inference host of another user with a colliding name. Because the user chooses the name, admin-provisioned namespaces do not prevent this risk. Only enable the wildcard option when you trust every namespace that can host anInferenceService. This applies to single-tenant environments or mutually trusting teams. Avoid using namespace names that are hyphenated suffixes of other hostnames. For strict isolation between namespaces on an untrusted cluster, keep inference traffic cluster-local and disable this option. Alternatively, issue one wildcard certificate per namespace using the dotted{name}.{namespace}.{domain}host format.The gateway authenticates but does not authorize per tenant. Any authenticated user can invoke any
InferenceService. The external authorization chain (ext_authz) checks that the caller has a valid Dex session or Bearer token. It does not check if the caller owns the namespace of the model. No gateway-scopedAuthorizationPolicybinds*.<kserveDomain>hosts to their owning namespace. TheAuthorizationPolicyof the Knative activator allows all traffic. Additionally, the OAuth2 proxy accepts any domain. Testing with two users confirmed that a token for any Dex user is accepted at every inference host, regardless of the namespace containing the model. This behavior is acceptable for single tenants or mutually trusting teams. However, on an untrusted multi-tenant cluster, one user can invoke and read outputs from the models of another user. To isolate invocations between tenants, disable this option and keep inference traffic cluster-local. In this local configuration, namespace-scoped policies still apply. Alternatively, add a custom gateway-scopedAuthorizationPolicythat binds each host to its namespace.The flattened label must stay under 63 characters. The Domain Name System (DNS) standard limits any single label to 63 characters. The wildcard certificate must cover the component Knative service rather than the
InferenceService. For example, a serverless predictor routes as{isvc}-predictor-{namespace}. You must budget for the ten-character-predictorsuffix. Long InferenceService or namespace names can exceed the limit.External access requires a Bearer token or session cookie. Serving over HTTPS does not bypass Kubeflow authentication. See the Tutorial: Programmatic Access for instructions on how to authenticate external
curlor Python clients.The configuration path for custom certificates differs. For more information, see Section 1.7.5, “Production: Bring-your-own certificate”.
To keep inference on plain HTTP, leave the
kserveExternalHttpsparameter set tofalse. In this case, the dedicated gateway serves port:80only without a wildcard certificate. Thestatus.urlparameter does not change and uses the default{name}-{namespace}.{domain}host over HTTP.
1.8 How do I harden Kubeflow for production use? #
The override values for the chart ship with defaults that are not suitable for production. Update these values before exposing the deployment to any network or storing sensitive data.
By default (global.demoMode: false), the chart fails at render time with a security error if any well-known demo credential is still present.
To suppress this during local development, set global.demoMode: true in your override values file but never set this in production.
Change all default credentials. The credentials that trigger the render-time security check:
# in your kubeflow-override-values.yaml auth: oidc: clientSecret: "<STRONG-RANDOM-SECRET>" cookieSecret: "<STRONG-RANDOM-32-BYTE-BASE64>" # generate: openssl rand -base64 32 dex: config: staticClients: - id: kubeflow-oidc-authservice redirectURIs: - /oauth2/callback name: kubeflow-oidc-authservice secret: "<STRONG-RANDOM-SECRET>" # must match auth.oidc.clientSecret above staticPasswords: - email: "admin@yourcompany.com" # Generate: htpasswd -nbBC 12 "" 'YourPassword' | tr -d ':\n' | sed 's/$2y/$2a/' hash: "<BCRYPT-HASH>" username: admin userID: "1" enablePasswordDB: true pipelines: seaweedfs: accessKey: "<STRONG-ACCESS-KEY>" secretKey: "<STRONG-SECRET-KEY>" user-namespace: pipelines: seaweedfs: accessKey: "<STRONG-ACCESS-KEY>" # must match pipelines.seaweedfs.accessKey secretKey: "<STRONG-SECRET-KEY>" # must match pipelines.seaweedfs.secretKeyNote: MariaDB root passwordBoth the KFP and Katib MySQL secrets are auto-generated (24-char random password) on the first install. They are preserved across upgrades with no action required. To rotate them, delete the secret and run
helm upgradeto regenerate:> kubectl delete secret mysql-secret -n kubeflow # for KFP > kubectl delete secret katib-mysql-secrets -n kubeflow # for Katib > helm upgrade kubeflow . -f kubeflow-override-values.yaml -n kubeflowUse an external identity provider. Replace Dex static passwords with an LDAP, SAML, or upstream OIDC connector. Add a
connectorsblock todex.configand removestaticPasswordsandenablePasswordDB: true.NetworkPolicies. NetworkPolicies are disabled by default. They use an 'ingress-only deny-by-default' model where egress is unrestricted so that components can reach external services such as Hugging Face and container registries. Such configurations are supported by Calico, Cilium, Canal, and any other CNI that enforces NetworkPolicy. If your CNI does not enforce NetworkPolicy, enable it:
# in your kubeflow-override-values.yaml networkPolicies: enabled: trueEnable TLS. Refer to Section 1.7.4, “Production: Let’s Encrypt TLS + external-dns” or Section 1.7.5, “Production: Bring-your-own certificate” for more details.
Enable database backups.
# in your kubeflow-override-values.yaml pipelines: mariadb: backup: enabled: true schedule: "0 2 * * *" # daily at 02:00 UTC storageSize: 20GiTo restore the backup:
# List available backups > kubectl exec -n kubeflow sts/mysql -- ls /backup/ # Restore > kubectl exec -n kubeflow sts/mysql -- \ sh -c "mariadb --ssl=false -u root < /backup/<filename>.sql"Enable pre-install validation.
# in your kubeflow-override-values.yaml preflightChecks: enabled: trueRuns a hook job before the installation that validates that the default
StorageClassexists and that cert-manager CRDs are registered.Enable High Availability. Apply
ha-overrides.yaml(provided in the repository) on top of your base values to scale the Katib controller, training-operator, and KServe controller to 2 replicas. KFP and Dex PodDisruptionBudgets are already enabled by default.> helm upgrade kubeflow . -f kubeflow-override-values.yaml -f ha-overrides.yaml -n kubeflowPDBs protect against voluntary disruptions (node drains) but only provide meaningful coverage with 2 or more replicas. With a single replica, the PDB allows full eviction. See Known limitations (Section 1.2.3, “What are known Kubeflow limitations?”) for supported HA controllers.
Apply resource quotas per user namespace.
# in your kubeflow-override-values.yaml additionalUsers: - email: alice@example.com namespace: alice resourceQuota: requests.cpu: "4" requests.memory: "8Gi" requests.nvidia.com/gpu: "1"
1.9 How do I manage user profiles and namespaces? #
Kubeflow uses a profile-per-user model.
The default user namespace that Kubeflow creates during the installation is kubeflow-user-example-com.
1.9.1 Adding users during installation #
Insert or update the following snippet in your kubeflow-override-values.yaml file:
# in your kubeflow-override-values.yaml
user-namespace:
additionalUsers:
- email: tux@example.com
namespace: tux
resourceQuota:
requests.cpu: "4"
requests.memory: 8Gi
- email: geeko@example.com
namespace: geekoThe namespace field is optional but strongly recommended.
Without it, the namespace is auto-generated from the e-mail by replacing @ with -- and . with -.
e-mail that differ only by . vs - — such as tux.geeko@example.com and tux-geeko@example.com — produce the same auto-generated namespace.
Use an explicit namespace to disambiguate.
1.9.2 Adding users after installation #
Add the user to your values file and run helm upgrade:
> helm upgrade kubeflow oci://registry.suse.com/ai/charts/kubeflow \
--version <VERSION> \
-n kubeflow --reuse-values \
--set "user-namespace.additionalUsers[0].email=tux@example.com" \
--set "user-namespace.additionalUsers[0].namespace=tux"Alternatively, add the user to your values file and rerun the upgrade:
# in your kubeflow-override-values.yaml
user-namespace:
additionalUsers:
- email: tux@example.com
namespace: tux> helm upgrade kubeflow oci://registry.suse.com/ai/charts/kubeflow \
--version <VERSION> \
-n kubeflow -f kubeflow-override-values.yamlThis creates the Profile CR and deploys all required KFP per-namespace resources (pipeline artifact server, visualization server, credentials, authorization policies) in a single step.
Do not add users by applying a Profile CR directly with kubectl.
The profiles controller only creates namespace-level RBAC — it does not deploy the KFP per-namespace resources that pipelines depend on.
Users added this way will have an incomplete environment and the pipeline runs will fail.
1.10 How do I upgrade Kubeflow? #
You can upgrade Kubeflow either using Rancher Prime or on the command line with Helm.
1.10.1 Upgrading with Rancher Prime #
Refer to AI application upgrade for a general upgrade procedure.
1.10.2 Upgrading with Helm #
Log in to SUSE Registry. To prevent the password from being stored in the shell history, save it to the
password.txtfile.> helm registry login registry.suse.com \ --username=regcode \ --password-stdin < password.txtUpgrade Kubeflow.
> helm upgrade kubeflow \ oci://registry.suse.com/ai/charts/kubeflow \ -n kubeflow \ --version <VERSION_NUMBER> \ 1 --wait --timeout 15m \ --force-conflictsIf you omit
--version, Kubeflow gets upgraded to the latest available version.
1.10.3 Upgrade notes #
Following are important considerations and specific instructions for managing and upgrading your Kubeflow deployment. They cover critical aspects like configuration conflicts, credential rotation, and potential service interruptions.
- Always use
--force-conflicts The
--force-conflictsflag is required forhelm upgrade. Certain components, such ascert-manager-cainjectorandistiod, modify fields that Helm also manages. This flag tells Helm to overwrite these external changes and reclaim ownership.> helm upgrade kubeflow \ oci://registry.suse.com/ai/charts/kubeflow \ --version 0.3.1 \ -n kubeflow --force-conflicts --wait --timeout 15m- Switching database engines or changing StorageClass
PVCs with the option
helm.sh/resource-policy: keepare not deleted by Helm. When switching database images or StorageClasses, delete the PVC manually first:# For KFP MariaDB (Rancher MariaDB StatefulSet) > kubectl delete pvc -n kubeflow data-mysql-0 # For Katib MariaDB (standalone Deployment, not StatefulSet) > kubectl delete pvc -n kubeflow katib-mysql- Training operator webhook secret migration (auto)
On upgrade from chart versions prior to 0.3.0, a pre-upgrade hook automatically migrates the
training-operator-webhook-certSecret from typekubernetes.io/tlstoOpaque. No manual action is required. If thekubectl get jobs -n kubeflowcommand reports a hook failure, delete the secret manually and rerunhelm upgrade:> kubectl delete secret training-operator-webhook-cert -n kubeflow --ignore-not-found > helm upgrade kubeflow charts/kubeflow -n kubeflow --force-conflicts --wait --timeout 15m- Credential rotation (SeaweedFS)
Changing
pipelines.seaweedfs.accessKeyandsecretKeyrequires restarting all KFP Deployments that read those credentials. Pods do not automatically restart when a Secret changes:> kubectl rollout restart deployment -n kubeflow \ ml-pipeline ml-pipeline-ui ml-pipeline-persistenceagent ml-pipeline-scheduledworkflow > kubectl rollout restart deployment -n kubeflow-user-example-com \ ml-pipeline-ui-artifactSeaweedFS IAM accumulates credentials across restarts (the
postStarthook adds, never removes). To clean stale entries after a rotation:> kubectl exec -n kubeflow deploy/seaweedfs -- \ sh -c "printf 's3.configure -user <OLD_USER> -access_key <OLD_KEY> -delete -apply\n' \ | weed shell -master 127.0.0.1:9333"- SeaweedFS upgrade downtime
SeaweedFS uses a
Recreatedeployment strategy (single-node S3 store backed by a PVC). Duringhelm upgrade, the old pod is terminated before the new pod starts. Expect a brief window of 10s to 60s where S3 artifact uploads and downloads are unavailable. In-flight pipeline runs may stall.We recommend temporarily disabling active pipeline runs in the GUI before upgrading.
- Supply the full values file on upgrade
Running
helm upgrade --reuse-valuesdoes not update theuser-namespaceSecret with new SeaweedFS credentials. Always supply the full values file on upgrade, or patch the Secret manually after upgrade.- CRDs are not automatically upgraded by
helm upgrade Kubeflow CRDs are placed in
crds/subdirectories. Helm intentionally skips them on upgrade. After a chart version bump, run:> helm show crds oci://registry.suse.com/ai/charts/kubeflow \ --version <VERSION> | kubectl apply -f -
1.11 How do I remove Kubeflow? #
You can remove Kubeflow either using Rancher Prime or on the command line with Helm.
1.11.1 Removal using Rancher Prime #
Refer to AI application removal for a general removal procedure.
1.11.2 Removal using Helm #
To remove Kubeflow using Helm, use the following command:
> helm uninstall kubeflow -n kubeflow1.12 Troubleshooting #
Learn about solutions for common issues encountered when using Kubeflow.
- Pods are not starting — too many open files
Run on each cluster node:
> sudo sysctl -w fs.inotify.max_user_watches=524288 > sudo sysctl -w fs.inotify.max_user_instances=512 # Persist across reboots > echo "fs.inotify.max_user_watches=524288" | \ sudo tee -a /etc/sysctl.d/99-kubeflow.conf > echo "fs.inotify.max_user_instances=512" | \ sudo tee -a /etc/sysctl.d/99-kubeflow.conf- Pods are stuck pending — storage issues
> kubectl get pvc -n kubeflow > kubectl describe pvc -n kubeflow <PVC_NAME>Ensure the default StorageClass exists:
> kubectl get storageclassIf none is marked as default, patch one:
kubectl patch storageclass <NAME> \ -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'metadata-grpcCrashLoopBackOff (MLMD)The
metadbdatabase is created by a post-install hook job. If the job failed, rerun it:> kubectl delete job metadb-init -n kubeflow --ignore-not-found > helm template kubeflow oci://registry.suse.com/ai/charts/kubeflow --version <VERSION> -n kubeflow \ --show-only 'charts/pipelines/templates/Job/metadb-init-kubeflow-Job.yaml' \ | kubectl apply -n kubeflow -f -- TensorBoard unavailable / controller CrashLoopBackOff
Check that the
tensorboard-controller-configConfigMap contains the following required keys:ISTIO_HOST: "*"ISTIO_GATEWAY: kubeflow/kubeflow-gateway(must be in thenamespace/nameformat)
> kubectl get configmap -n kubeflow -l app=tensorboard-controller -o yaml | grep -A5 'data:'If either the key is wrong or missing, upgrade the chart with the following overrides:
tensorboard-controller: configMapData: ISTIO_HOST: "*" ISTIO_GATEWAY: kubeflow/kubeflow-gateway- KServe InferenceService not progressing
Verify that the
ClusterStorageContainerCRD and a default object exist:> kubectl get crd clusterstoragecontainers.serving.kserve.io > kubectl get clusterstoragecontainer defaultIf either is missing, rerun
helm upgradeto install them.- "RBAC: access denied" for user namespace traffic
Rancher Prime Istio uses the
istioServiceAccount (notistio-ingressgateway-service-account). Therancher-ingressgateway-accessAuthorizationPolicy in each user namespace handles this.If you pre-created the AuthorizationPolicy with
kubectlbefore the first Helm install, you need to add Helm ownership labels/annotations before upgrading:> kubectl label authorizationpolicy rancher-ingressgateway-access \ -n kubeflow-user-example-com app.kubernetes.io/managed-by=Helm --overwrite > kubectl annotate authorizationpolicy rancher-ingressgateway-access \ -n kubeflow-user-example-com \ meta.helm.sh/release-name=kubeflow \ meta.helm.sh/release-namespace=kubeflow --overwrite- SeaweedFS pod stuck on ContainerCreating
On K3s 1.34 with cri-dockerd (Rancher Prime Desktop), a race between the CNI and the container runtime can leave the SeaweedFS pod stuck. SeaweedFS acquires a LevelDB file lock on startup. If the pod is stuck, the lock is held and the next pod will also fail to start. To recover, run the following commands:
# Find the Docker container ID for the stuck pod > docker ps | grep seaweedfs # Release the lock > docker kill <container-id> # Delete the stuck pod — a new pod will start cleanly > kubectl delete pod -n kubeflow -l app=seaweedfs- Dex login loop (infinite redirect)
This usually means that oauth2-proxy is receiving a 403 from an Istio AuthorizationPolicy rather than from oauth2-proxy itself. The Lua-redirect filter only converts 403 → 302 when the response includes a
set-cookieheader; AuthorizationPolicy 403 messages do not have one.Check for sidecar-level AuthorizationPolicy denials:
> kubectl logs -n istio-system -l app=istiod | grep "RBAC" > kubectl get authorizationpolicy -A