This is unreleased documentation for SUSE® Virtual Clusters v1.0.3 (Dev).

Adding custom certificates to k3s from k3k cluster

Generate and add the custom certificates

  1. Use the custom certificate script to generate certificates.

  2. After generating the certificates, create a local directory as the default directory will not be accessible. For example, k3k/custom-certs.

  3. Copy the generated certificates to the directory - $ sudo cp -r /var/lib/rancher/k3s/server/tls k3k/custom-certs/

  4. Use the path /var/lib/rancher/k3s/server/tls k3k/custom-certs/ to create the virtual clusters in 2 modes:

    • Virtual mode

    • Shared mode

Virtual mode

  1. Create the k3k cluster:

    $ k3kcli cluster create --namespace virtual-ns --persistence-type ephemeral --version v1.33.3-k3s1 --mode virtual --custom-certs /home/ubuntu/k3k/custom-certs/tls mdvirtcl1
INFO[0000] Creating cluster [mdvirtcl1] in namespace [virtual-ns]
INFO[0000] Waiting for cluster to be available..
INFO[0070] Extracting Kubeconfig for [mdvirtcl1] cluster
INFO[0070] certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1756155858: notBefore=2025-08-25 21:04:18 +0000 UTC notAfter=2026-08-25 21:06:55 +0000 UTC
INFO[0070] You can start using the cluster with:

	export KUBECONFIG=/home/ubuntu/virtual-ns-mdvirtcl1-kubeconfig.yaml
	kubectl cluster-info

  2. k3k cluster:

    $ KUBECONFIG=/home/ubuntu/virtual-ns-mdvirtcl1-kubeconfig.yaml kubectl get nodes,pods -A
NAME                          STATUS   ROLES                       AGE   VERSION
node/k3k-mdvirtcl1-server-0   Ready    control-plane,etcd,master   88s   v1.33.3+k3s1

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-5688667fd4-mwt7d                  1/1     Running     0          81s
kube-system   pod/helm-install-traefik-crd-qkb9n            0/1     Completed   0          82s
kube-system   pod/helm-install-traefik-q5v59                0/1     Completed   2          82s
kube-system   pod/local-path-provisioner-774c6665dc-qsv9w   1/1     Running     0          81s
kube-system   pod/metrics-server-6f4c6675d5-sn86r           1/1     Running     0          81s
kube-system   pod/svclb-traefik-faf86720-fs5sv              2/2     Running     0          35s
kube-system   pod/traefik-c98fdf6fb-nqsqw                   1/1     Running     0          36s

  3. Generate custom certificates via script:

    $ sudo ls -la k3k/custom-certs/tls/
total 92
drwxr-xr-x 3 root root 4096 Aug 25 21:04 .
drwxr-xr-x 3 root root 4096 Aug 25 21:04 ..
-rwxr-xr-x 1 root root 4961 Aug 25 21:04 client-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 client-ca.key
-rwxr-xr-x 1 root root 1241 Aug 25 21:04 client-ca.pem
drwxr-xr-x 2 root root 4096 Aug 25 21:04 etcd
-rwxr-xr-x 1 root root 3720 Aug 25 21:04 intermediate-ca.crt
-rwxr-xr-x 1 root root 3243 Aug 25 21:04 intermediate-ca.key
-rwxr-xr-x 1 root root 1858 Aug 25 21:04 intermediate-ca.pem
-rwxr-xr-x 1 root root 4969 Aug 25 21:04 request-header-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 request-header-ca.key
-rwxr-xr-x 1 root root 1249 Aug 25 21:04 request-header-ca.pem
-rwxr-xr-x 1 root root 1862 Aug 25 21:04 root-ca.crt
-rwxr-xr-x 1 root root 3243 Aug 25 21:04 root-ca.key
-rwxr-xr-x 1 root root 1862 Aug 25 21:04 root-ca.pem
-rwxr-xr-x 1 root root 4961 Aug 25 21:04 server-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 server-ca.key
-rwxr-xr-x 1 root root 1241 Aug 25 21:04 server-ca.pem
-rwxr-xr-x 1 root root 5025 Aug 25 21:04 service.key

  4. Verify the certificates on the server pod:

    $ kubectl exec --stdin --tty -n virtual-ns k3k-mdvirtcl1-server-0 -- ls -la /var/lib/rancher/k3s/server/tls
total 204
drwxr-xr-x 6 root root  4096 Aug 25 21:05 .
drwxr-xr-x 8 root root  4096 Aug 25 21:06 ..
-rw-r--r-- 1 root root  5568 Aug 25 21:05 client-admin.crt
-rw------- 1 root root   227 Aug 25 21:05 client-admin.key
-rw-r--r-- 1 root root  5556 Aug 25 21:05 client-auth-proxy.crt
-rw------- 1 root root   227 Aug 25 21:05 client-auth-proxy.key
-rw-r--r-- 1 root root  4961 Aug 25 21:05 client-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:05 client-ca.key
-rw-r--r-- 1 root root  1241 Aug 25 21:05 client-ca.nochain.crt
-rw-r--r-- 1 root root  5556 Aug 25 21:05 client-controller.crt
-rw------- 1 root root   227 Aug 25 21:05 client-controller.key
-rw-r--r-- 1 root root  5552 Aug 25 21:05 client-k3s-cloud-controller.crt
-rw------- 1 root root   227 Aug 25 21:05 client-k3s-cloud-controller.key
-rw------- 1 root root   227 Aug 25 21:05 client-k3s-controller.key
-rw-r--r-- 1 root root  5572 Aug 25 21:05 client-kube-apiserver.crt
-rw------- 1 root root   227 Aug 25 21:05 client-kube-apiserver.key
-rw------- 1 root root   227 Aug 25 21:05 client-kube-proxy.key
-rw------- 1 root root   227 Aug 25 21:05 client-kubelet.key
-rw-r--r-- 1 root root  5544 Aug 25 21:05 client-scheduler.crt
-rw------- 1 root root   227 Aug 25 21:05 client-scheduler.key
-rw-r--r-- 1 root root  5576 Aug 25 21:05 client-supervisor.crt
-rw------- 1 root root   227 Aug 25 21:05 client-supervisor.key
-rw-r--r-- 1 root root 10224 Aug 25 21:06 dynamic-cert.json
drwxr-xr-x 2 root root  4096 Aug 25 21:05 etcd
drwxr-xr-x 2 root root  4096 Aug 25 21:05 kube-controller-manager
drwxr-xr-x 2 root root  4096 Aug 25 21:05 kube-scheduler
-rw-r--r-- 1 root root  4969 Aug 25 21:05 request-header-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:05 request-header-ca.key
-rw-r--r-- 1 root root  4961 Aug 25 21:05 server-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:05 server-ca.key
-rw-r--r-- 1 root root  1241 Aug 25 21:05 server-ca.nochain.crt
-rw------- 1 root root  1675 Aug 25 21:05 service.current.key
-rw-r--r-- 1 root root  5025 Aug 25 21:05 service.key
-rw-r--r-- 1 root root  5877 Aug 25 21:05 serving-kube-apiserver.crt
-rw------- 1 root root   227 Aug 25 21:05 serving-kube-apiserver.key
-rw------- 1 root root   227 Aug 25 21:05 serving-kubelet.key
drwx------ 2 root root  4096 Aug 25 21:05 temporary-certs

  5. Copy and verify the certificates on the local directory:

    $ kubectl cp -n virtual-ns k3k-mdvirtcl1-server-0:var/lib/rancher/k3s/server/tls /home/ubuntu/k3k-server-pod/

$ sudo diff -sr k3k/custom-certs/tls/ k3k-server-pod/ | grep -i identical | awk '{print $2}' | xargs basename -a | awk 'BEGIN{print "Identical Files:  "}; {print $1}'
Identical Files:
client-ca.crt
client-ca.key
peer-ca.crt
peer-ca.key
server-ca.crt
server-ca.key
request-header-ca.crt
request-header-ca.key
server-ca.crt
server-ca.key
service.key

Shared mode

  1. Create k3k cluster:

    $ k3kcli cluster create --namespace shared-ns --persistence-type ephemeral --version v1.33.3-k3s1 --mode shared --custom-certs /home/ubuntu/k3k/custom-certs/tls mdshcl1
INFO[0000] Creating namespace [shared-ns]
INFO[0000] Creating cluster [mdshcl1] in namespace [shared-ns]
INFO[0000] Waiting for cluster to be available..
INFO[0075] Extracting Kubeconfig for [mdshcl1] cluster
INFO[0075] certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1756155858: notBefore=2025-08-25 21:04:18 +0000 UTC notAfter=2026-08-25 21:49:42 +0000 UTC
INFO[0075] You can start using the cluster with:

	export KUBECONFIG=/home/ubuntu/shared-ns-mdshcl1-kubeconfig.yaml
	kubectl cluster-info

  2. Verify the newly created k3k cluster:

    $ KUBECONFIG=/home/ubuntu/shared-ns-mdshcl1-kubeconfig.yaml kubectl get nodes,pods -A
NAME                                             STATUS   ROLES   AGE   VERSION
node/ip-172-31-28-7.us-east-2.compute.internal   Ready    agent   84s   v1.33.3-k3s1

NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-5688667fd4-m4zgr   1/1     Running   0          2m32s

  3. Generate custom certificates via script:

    $ sudo ls -la k3k/custom-certs/tls/
total 92
drwxr-xr-x 3 root root 4096 Aug 25 21:04 .
drwxr-xr-x 3 root root 4096 Aug 25 21:04 ..
-rwxr-xr-x 1 root root 4961 Aug 25 21:04 client-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 client-ca.key
-rwxr-xr-x 1 root root 1241 Aug 25 21:04 client-ca.pem
drwxr-xr-x 2 root root 4096 Aug 25 21:04 etcd
-rwxr-xr-x 1 root root 3720 Aug 25 21:04 intermediate-ca.crt
-rwxr-xr-x 1 root root 3243 Aug 25 21:04 intermediate-ca.key
-rwxr-xr-x 1 root root 1858 Aug 25 21:04 intermediate-ca.pem
-rwxr-xr-x 1 root root 4969 Aug 25 21:04 request-header-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 request-header-ca.key
-rwxr-xr-x 1 root root 1249 Aug 25 21:04 request-header-ca.pem
-rwxr-xr-x 1 root root 1862 Aug 25 21:04 root-ca.crt
-rwxr-xr-x 1 root root 3243 Aug 25 21:04 root-ca.key
-rwxr-xr-x 1 root root 1862 Aug 25 21:04 root-ca.pem
-rwxr-xr-x 1 root root 4961 Aug 25 21:04 server-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 server-ca.key
-rwxr-xr-x 1 root root 1241 Aug 25 21:04 server-ca.pem
-rwxr-xr-x 1 root root 5025 Aug 25 21:04 service.key

  4. Verify the certificates on the server pod:

    $ kubectl exec --stdin --tty -n shared-ns k3k-mdshcl1-server-0 -- ls -la /var/lib/rancher/k3s/server/tls
total 204
drwxr-xr-x 6 root root  4096 Aug 25 21:48 .
drwxr-xr-x 8 root root  4096 Aug 25 21:48 ..
-rw-r--r-- 1 root root  5568 Aug 25 21:48 client-admin.crt
-rw------- 1 root root   227 Aug 25 21:48 client-admin.key
-rw-r--r-- 1 root root  5560 Aug 25 21:48 client-auth-proxy.crt
-rw------- 1 root root   227 Aug 25 21:48 client-auth-proxy.key
-rw-r--r-- 1 root root  4961 Aug 25 21:48 client-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:48 client-ca.key
-rw-r--r-- 1 root root  1241 Aug 25 21:48 client-ca.nochain.crt
-rw-r--r-- 1 root root  5556 Aug 25 21:48 client-controller.crt
-rw------- 1 root root   227 Aug 25 21:48 client-controller.key
-rw-r--r-- 1 root root  5556 Aug 25 21:48 client-k3s-cloud-controller.crt
-rw------- 1 root root   227 Aug 25 21:48 client-k3s-cloud-controller.key
-rw------- 1 root root   227 Aug 25 21:48 client-k3s-controller.key
-rw-r--r-- 1 root root  5572 Aug 25 21:48 client-kube-apiserver.crt
-rw------- 1 root root   227 Aug 25 21:48 client-kube-apiserver.key
-rw------- 1 root root   227 Aug 25 21:48 client-kube-proxy.key
-rw------- 1 root root   227 Aug 25 21:48 client-kubelet.key
-rw-r--r-- 1 root root  5544 Aug 25 21:48 client-scheduler.crt
-rw------- 1 root root   227 Aug 25 21:48 client-scheduler.key
-rw-r--r-- 1 root root  5580 Aug 25 21:48 client-supervisor.crt
-rw------- 1 root root   227 Aug 25 21:48 client-supervisor.key
-rw-r--r-- 1 root root 10185 Aug 25 21:48 dynamic-cert.json
drwxr-xr-x 2 root root  4096 Aug 25 21:48 etcd
drwxr-xr-x 2 root root  4096 Aug 25 21:48 kube-controller-manager
drwxr-xr-x 2 root root  4096 Aug 25 21:48 kube-scheduler
-rw-r--r-- 1 root root  4969 Aug 25 21:48 request-header-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:48 request-header-ca.key
-rw-r--r-- 1 root root  4961 Aug 25 21:48 server-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:48 server-ca.key
-rw-r--r-- 1 root root  1241 Aug 25 21:48 server-ca.nochain.crt
-rw------- 1 root root  1675 Aug 25 21:48 service.current.key
-rw-r--r-- 1 root root  5025 Aug 25 21:48 service.key
-rw-r--r-- 1 root root  5865 Aug 25 21:48 serving-kube-apiserver.crt
-rw------- 1 root root   227 Aug 25 21:48 serving-kube-apiserver.key
-rw------- 1 root root   227 Aug 25 21:48 serving-kubelet.key
drwx------ 2 root root  4096 Aug 25 21:48 temporary-certs

  5. Copy and verify the certificates on the local directory:

    ~$ kubectl cp -n shared-ns k3k-mdshcl1-server-0:var/lib/rancher/k3s/server/tls /home/ubuntu/k3k-server-pod/shared-tls

~$ sudo diff -sr k3k/custom-certs/tls/ k3k-server-pod/shared-tls/ | grep -i identical | awk '{print $2}' | xargs basename -a | awk 'BEGIN{print "Identical Files:  "}; {print $1}'
Identical Files:
client-ca.crt
client-ca.key
peer-ca.crt
peer-ca.key
server-ca.crt
server-ca.key
request-header-ca.crt
request-header-ca.key
server-ca.crt
server-ca.key
service.key