Adding custom certificates to k3s from k3k cluster

Generate and add the custom certificates

Use the custom certificate script to generate certificates.
After generating the certificates, create a local directory as the default directory will not be accessible. For example, k3k/custom-certs.
Copy the generated certificates to the directory - $ sudo cp -r /var/lib/rancher/k3s/server/tls k3k/custom-certs/
Use the path /var/lib/rancher/k3s/server/tls k3k/custom-certs/ to create the virtual clusters in 2 modes:
- Virtual mode
- Shared mode

Virtual mode

Create the k3k cluster:

$ k3kcli cluster create --namespace virtual-ns --persistence-type ephemeral --version v1.33.3-k3s1 --mode virtual --custom-certs /home/ubuntu/k3k/custom-certs/tls mdvirtcl1
INFO[0000] Creating cluster [mdvirtcl1] in namespace [virtual-ns]
INFO[0000] Waiting for cluster to be available..
INFO[0070] Extracting Kubeconfig for [mdvirtcl1] cluster
INFO[0070] certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1756155858: notBefore=2025-08-25 21:04:18 +0000 UTC notAfter=2026-08-25 21:06:55 +0000 UTC
INFO[0070] You can start using the cluster with:

	export KUBECONFIG=/home/ubuntu/virtual-ns-mdvirtcl1-kubeconfig.yaml
	kubectl cluster-info

k3k cluster:

$ KUBECONFIG=/home/ubuntu/virtual-ns-mdvirtcl1-kubeconfig.yaml kubectl get nodes,pods -A
NAME                          STATUS   ROLES                       AGE   VERSION
node/k3k-mdvirtcl1-server-0   Ready    control-plane,etcd,master   88s   v1.33.3+k3s1

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE
kube-system   pod/coredns-5688667fd4-mwt7d                  1/1     Running     0          81s
kube-system   pod/helm-install-traefik-crd-qkb9n            0/1     Completed   0          82s
kube-system   pod/helm-install-traefik-q5v59                0/1     Completed   2          82s
kube-system   pod/local-path-provisioner-774c6665dc-qsv9w   1/1     Running     0          81s
kube-system   pod/metrics-server-6f4c6675d5-sn86r           1/1     Running     0          81s
kube-system   pod/svclb-traefik-faf86720-fs5sv              2/2     Running     0          35s
kube-system   pod/traefik-c98fdf6fb-nqsqw                   1/1     Running     0          36s

Generate custom certificates via script:

$ sudo ls -la k3k/custom-certs/tls/
total 92
drwxr-xr-x 3 root root 4096 Aug 25 21:04 .
drwxr-xr-x 3 root root 4096 Aug 25 21:04 ..
-rwxr-xr-x 1 root root 4961 Aug 25 21:04 client-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 client-ca.key
-rwxr-xr-x 1 root root 1241 Aug 25 21:04 client-ca.pem
drwxr-xr-x 2 root root 4096 Aug 25 21:04 etcd
-rwxr-xr-x 1 root root 3720 Aug 25 21:04 intermediate-ca.crt
-rwxr-xr-x 1 root root 3243 Aug 25 21:04 intermediate-ca.key
-rwxr-xr-x 1 root root 1858 Aug 25 21:04 intermediate-ca.pem
-rwxr-xr-x 1 root root 4969 Aug 25 21:04 request-header-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 request-header-ca.key
-rwxr-xr-x 1 root root 1249 Aug 25 21:04 request-header-ca.pem
-rwxr-xr-x 1 root root 1862 Aug 25 21:04 root-ca.crt
-rwxr-xr-x 1 root root 3243 Aug 25 21:04 root-ca.key
-rwxr-xr-x 1 root root 1862 Aug 25 21:04 root-ca.pem
-rwxr-xr-x 1 root root 4961 Aug 25 21:04 server-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 server-ca.key
-rwxr-xr-x 1 root root 1241 Aug 25 21:04 server-ca.pem
-rwxr-xr-x 1 root root 5025 Aug 25 21:04 service.key

Verify the certificates on the server pod:

$ kubectl exec --stdin --tty -n virtual-ns k3k-mdvirtcl1-server-0 -- ls -la /var/lib/rancher/k3s/server/tls
total 204
drwxr-xr-x 6 root root  4096 Aug 25 21:05 .
drwxr-xr-x 8 root root  4096 Aug 25 21:06 ..
-rw-r--r-- 1 root root  5568 Aug 25 21:05 client-admin.crt
-rw------- 1 root root   227 Aug 25 21:05 client-admin.key
-rw-r--r-- 1 root root  5556 Aug 25 21:05 client-auth-proxy.crt
-rw------- 1 root root   227 Aug 25 21:05 client-auth-proxy.key
-rw-r--r-- 1 root root  4961 Aug 25 21:05 client-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:05 client-ca.key
-rw-r--r-- 1 root root  1241 Aug 25 21:05 client-ca.nochain.crt
-rw-r--r-- 1 root root  5556 Aug 25 21:05 client-controller.crt
-rw------- 1 root root   227 Aug 25 21:05 client-controller.key
-rw-r--r-- 1 root root  5552 Aug 25 21:05 client-k3s-cloud-controller.crt
-rw------- 1 root root   227 Aug 25 21:05 client-k3s-cloud-controller.key
-rw------- 1 root root   227 Aug 25 21:05 client-k3s-controller.key
-rw-r--r-- 1 root root  5572 Aug 25 21:05 client-kube-apiserver.crt
-rw------- 1 root root   227 Aug 25 21:05 client-kube-apiserver.key
-rw------- 1 root root   227 Aug 25 21:05 client-kube-proxy.key
-rw------- 1 root root   227 Aug 25 21:05 client-kubelet.key
-rw-r--r-- 1 root root  5544 Aug 25 21:05 client-scheduler.crt
-rw------- 1 root root   227 Aug 25 21:05 client-scheduler.key
-rw-r--r-- 1 root root  5576 Aug 25 21:05 client-supervisor.crt
-rw------- 1 root root   227 Aug 25 21:05 client-supervisor.key
-rw-r--r-- 1 root root 10224 Aug 25 21:06 dynamic-cert.json
drwxr-xr-x 2 root root  4096 Aug 25 21:05 etcd
drwxr-xr-x 2 root root  4096 Aug 25 21:05 kube-controller-manager
drwxr-xr-x 2 root root  4096 Aug 25 21:05 kube-scheduler
-rw-r--r-- 1 root root  4969 Aug 25 21:05 request-header-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:05 request-header-ca.key
-rw-r--r-- 1 root root  4961 Aug 25 21:05 server-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:05 server-ca.key
-rw-r--r-- 1 root root  1241 Aug 25 21:05 server-ca.nochain.crt
-rw------- 1 root root  1675 Aug 25 21:05 service.current.key
-rw-r--r-- 1 root root  5025 Aug 25 21:05 service.key
-rw-r--r-- 1 root root  5877 Aug 25 21:05 serving-kube-apiserver.crt
-rw------- 1 root root   227 Aug 25 21:05 serving-kube-apiserver.key
-rw------- 1 root root   227 Aug 25 21:05 serving-kubelet.key
drwx------ 2 root root  4096 Aug 25 21:05 temporary-certs

Copy and verify the certificates on the local directory:

$ kubectl cp -n virtual-ns k3k-mdvirtcl1-server-0:var/lib/rancher/k3s/server/tls /home/ubuntu/k3k-server-pod/

$ sudo diff -sr k3k/custom-certs/tls/ k3k-server-pod/ | grep -i identical | awk '{print $2}' | xargs basename -a | awk 'BEGIN{print "Identical Files:  "}; {print $1}'
Identical Files:
client-ca.crt
client-ca.key
peer-ca.crt
peer-ca.key
server-ca.crt
server-ca.key
request-header-ca.crt
request-header-ca.key
server-ca.crt
server-ca.key
service.key

Shared mode

Create k3k cluster:

$ k3kcli cluster create --namespace shared-ns --persistence-type ephemeral --version v1.33.3-k3s1 --mode shared --custom-certs /home/ubuntu/k3k/custom-certs/tls mdshcl1
INFO[0000] Creating namespace [shared-ns]
INFO[0000] Creating cluster [mdshcl1] in namespace [shared-ns]
INFO[0000] Waiting for cluster to be available..
INFO[0075] Extracting Kubeconfig for [mdshcl1] cluster
INFO[0075] certificate CN=system:admin,O=system:masters signed by CN=k3s-client-ca@1756155858: notBefore=2025-08-25 21:04:18 +0000 UTC notAfter=2026-08-25 21:49:42 +0000 UTC
INFO[0075] You can start using the cluster with:

	export KUBECONFIG=/home/ubuntu/shared-ns-mdshcl1-kubeconfig.yaml
	kubectl cluster-info

Verify the newly created k3k cluster:

$ KUBECONFIG=/home/ubuntu/shared-ns-mdshcl1-kubeconfig.yaml kubectl get nodes,pods -A
NAME                                             STATUS   ROLES   AGE   VERSION
node/ip-172-31-28-7.us-east-2.compute.internal   Ready    agent   84s   v1.33.3-k3s1

NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-5688667fd4-m4zgr   1/1     Running   0          2m32s

Generate custom certificates via script:

$ sudo ls -la k3k/custom-certs/tls/
total 92
drwxr-xr-x 3 root root 4096 Aug 25 21:04 .
drwxr-xr-x 3 root root 4096 Aug 25 21:04 ..
-rwxr-xr-x 1 root root 4961 Aug 25 21:04 client-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 client-ca.key
-rwxr-xr-x 1 root root 1241 Aug 25 21:04 client-ca.pem
drwxr-xr-x 2 root root 4096 Aug 25 21:04 etcd
-rwxr-xr-x 1 root root 3720 Aug 25 21:04 intermediate-ca.crt
-rwxr-xr-x 1 root root 3243 Aug 25 21:04 intermediate-ca.key
-rwxr-xr-x 1 root root 1858 Aug 25 21:04 intermediate-ca.pem
-rwxr-xr-x 1 root root 4969 Aug 25 21:04 request-header-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 request-header-ca.key
-rwxr-xr-x 1 root root 1249 Aug 25 21:04 request-header-ca.pem
-rwxr-xr-x 1 root root 1862 Aug 25 21:04 root-ca.crt
-rwxr-xr-x 1 root root 3243 Aug 25 21:04 root-ca.key
-rwxr-xr-x 1 root root 1862 Aug 25 21:04 root-ca.pem
-rwxr-xr-x 1 root root 4961 Aug 25 21:04 server-ca.crt
-rwxr-xr-x 1 root root  227 Aug 25 21:04 server-ca.key
-rwxr-xr-x 1 root root 1241 Aug 25 21:04 server-ca.pem
-rwxr-xr-x 1 root root 5025 Aug 25 21:04 service.key

Verify the certificates on the server pod:

$ kubectl exec --stdin --tty -n shared-ns k3k-mdshcl1-server-0 -- ls -la /var/lib/rancher/k3s/server/tls
total 204
drwxr-xr-x 6 root root  4096 Aug 25 21:48 .
drwxr-xr-x 8 root root  4096 Aug 25 21:48 ..
-rw-r--r-- 1 root root  5568 Aug 25 21:48 client-admin.crt
-rw------- 1 root root   227 Aug 25 21:48 client-admin.key
-rw-r--r-- 1 root root  5560 Aug 25 21:48 client-auth-proxy.crt
-rw------- 1 root root   227 Aug 25 21:48 client-auth-proxy.key
-rw-r--r-- 1 root root  4961 Aug 25 21:48 client-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:48 client-ca.key
-rw-r--r-- 1 root root  1241 Aug 25 21:48 client-ca.nochain.crt
-rw-r--r-- 1 root root  5556 Aug 25 21:48 client-controller.crt
-rw------- 1 root root   227 Aug 25 21:48 client-controller.key
-rw-r--r-- 1 root root  5556 Aug 25 21:48 client-k3s-cloud-controller.crt
-rw------- 1 root root   227 Aug 25 21:48 client-k3s-cloud-controller.key
-rw------- 1 root root   227 Aug 25 21:48 client-k3s-controller.key
-rw-r--r-- 1 root root  5572 Aug 25 21:48 client-kube-apiserver.crt
-rw------- 1 root root   227 Aug 25 21:48 client-kube-apiserver.key
-rw------- 1 root root   227 Aug 25 21:48 client-kube-proxy.key
-rw------- 1 root root   227 Aug 25 21:48 client-kubelet.key
-rw-r--r-- 1 root root  5544 Aug 25 21:48 client-scheduler.crt
-rw------- 1 root root   227 Aug 25 21:48 client-scheduler.key
-rw-r--r-- 1 root root  5580 Aug 25 21:48 client-supervisor.crt
-rw------- 1 root root   227 Aug 25 21:48 client-supervisor.key
-rw-r--r-- 1 root root 10185 Aug 25 21:48 dynamic-cert.json
drwxr-xr-x 2 root root  4096 Aug 25 21:48 etcd
drwxr-xr-x 2 root root  4096 Aug 25 21:48 kube-controller-manager
drwxr-xr-x 2 root root  4096 Aug 25 21:48 kube-scheduler
-rw-r--r-- 1 root root  4969 Aug 25 21:48 request-header-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:48 request-header-ca.key
-rw-r--r-- 1 root root  4961 Aug 25 21:48 server-ca.crt
-rw-r--r-- 1 root root   227 Aug 25 21:48 server-ca.key
-rw-r--r-- 1 root root  1241 Aug 25 21:48 server-ca.nochain.crt
-rw------- 1 root root  1675 Aug 25 21:48 service.current.key
-rw-r--r-- 1 root root  5025 Aug 25 21:48 service.key
-rw-r--r-- 1 root root  5865 Aug 25 21:48 serving-kube-apiserver.crt
-rw------- 1 root root   227 Aug 25 21:48 serving-kube-apiserver.key
-rw------- 1 root root   227 Aug 25 21:48 serving-kubelet.key
drwx------ 2 root root  4096 Aug 25 21:48 temporary-certs

Copy and verify the certificates on the local directory:

~$ kubectl cp -n shared-ns k3k-mdshcl1-server-0:var/lib/rancher/k3s/server/tls /home/ubuntu/k3k-server-pod/shared-tls

~$ sudo diff -sr k3k/custom-certs/tls/ k3k-server-pod/shared-tls/ | grep -i identical | awk '{print $2}' | xargs basename -a | awk 'BEGIN{print "Identical Files:  "}; {print $1}'
Identical Files:
client-ca.crt
client-ca.key
peer-ca.crt
peer-ca.key
server-ca.crt
server-ca.key
request-header-ca.crt
request-header-ca.key
server-ca.crt
server-ca.key
service.key