v1.27.X

Bump k3s-root to v0.14.0

Bump github.com/hashicorp/go-retryablehttp from 0.7.4 to 0.7.7

Bump Local Path Provisioner version

Ensure remotedialer kubelet connections use kubelet bind address

Chore: Bump Trivy version

Add etcd s3 config secret implementation

Add WithSkipMissing to not fail import on missing blobs

Use fixed stream server bind address for cri-dockerd

Switch stargz over to cri registry config_path

Bump to containerd v1.7.17, etcd v3.5.13

Bump spegel version

Fix issue with externalTrafficPolicy: Local for single-stack services on dual-stack nodes

ServiceLB now sets the priorityClassName on svclb pods to system-node-critical by default. This can be overridden on a per-service basis via the svccontroller.k3s.cattle.io/priorityclassname annotation.

Bump minio-go to v7.0.70

Bump kine to v0.11.9 to fix pagination

Update valid resolv conf

Add missing kernel config check

Symlinked sub-directories are now respected when scanning Auto-Deploying Manifests (AddOns)

Fix bug: allow helm controller set owner reference

Bump klipper-helm image for tls secret support

Fix issue with k3s-etcd informers not starting

--Enable-pprof can now be set on agents to enable the debug/pprof endpoints. When set, agents will listen on the supervisor port.

--Supervisor-metrics can now be set on servers to enable serving internal metrics on the supervisor endpoint; when set agents will listen on the supervisor port.

Fix netpol crash when node remains tainted uninitialized

The embedded load-balancer will now fall back to trying all servers with health-checks ignored, if all servers have been marked unavailable due to failed health checks.

Update Kubernetes to v1.27.15

Send error response if member list cannot be retrieved

The k3s stub cloud provider now respects the kubelet’s requested provider-id, instance type, and topology labels

Fix error when image has already been pulled

Add /etc/passwd and /etc/group to k3s docker image

Fix etcd snapshot reconcile for agentless servers

Add health-check support to loadbalancer

Add certificate expiry check, events, and metrics

Add workaround for containerd hosts.toml bug when passing config for default registry endpoint

Add supervisor cert/key to rotate list

The embedded containerd has been bumped to v1.7.15

The embedded cri-dockerd has been bumped to v0.3.12

The k3s etcd-snapshot command has been reworked for improved consistency. All snapshots operations are now performed by the server process, with the CLI acting as a client to initiate and report results. As a side effect, the CLI is now less noisy when managing snapshots.

Improve etcd load-balancer startup behavior

Actually fix agent certificate rotation

Traefik has been bumped to v2.10.7.

Traefik pod annotations are now set properly in the default chart values.

The system-default-registry value now supports RFC2732 IPv6 literals.

The local-path provisioner now defaults to creating local volumes, instead of hostPath.

Fix: use correct wasm shims names

The embedded flannel cni-plugin binary is now built and versioned separate from the rest of the cni plugins and the embedded flannel controller.

Bump spegel to v0.0.18-k3s3

Adds wildcard registry support

Fixes issue with excessive CPU utilization while waiting for containerd to start

Add env var to allow spegel mirroring of latest tag

Tweak netpol node wait logs

Fix coredns NodeHosts on dual-stack clusters

Bump helm-controller/klipper-helm versions

Fix snapshot prune

Fix issue with etcd node name missing hostname

Rootless mode should also bind service nodePort to host for LoadBalancer type, matching UX of rootful mode.

To enable raw output for the check-config subcommand, you may now set NO_COLOR=1

Fix additional corner cases in registries handling

Bump metrics-server to v0.7.0

K3s will now warn and suppress duplicate entries in the mirror endpoint list for a registry. Containerd does not support listing the same endpoint multiple times as a mirror for a single upstream registry.

Directories containing runtimes need to be included in the $PATH environment variable for effective runtime detection.

Bump runc to v1.1.12 and helm-controller to v0.15.7

Fix handling of bare hostname or IP as endpoint address in registries.yaml

Added runtime classes for wasm/nvidia/crun

Added default runtime flag for containerd

New timezone info in Docker image allows the use of spec.timeZone in CronJobs

Bumped kine to v0.11.0 to resolve issues with postgres and NATS, fix performance of watch channels under heavy load, and improve compatibility with the reference implementation.

Containerd may now be configured to use rdt or blockio configuration by defining rdt_config.yaml or blockio_config.yaml files.

Add agent flag disable-apiserver-lb, agent will not start load balance proxy.

Improved ingress IP ordering from ServiceLB

Disable helm CRD installation for disable-helm-controller

Omit snapshot list configmap entries for snapshots without extra metadata

Add jitter to client config retry to avoid hammering servers when they are starting up

Bumped dynamiclistener to address a race condition that could cause a server to fail to sync its certificates into the Kubernetes secret

Reduced etcd snapshot log spam during initial cluster startup

Don’t apply S3 retention if S3 client failed to initialize

Don’t request metadata when listing S3 snapshots

Print key instead of file path in snapshot metadata log message

Fixed an issue with identifying additional container runtimes

The user will receive a error when --cluster-reset with the --server flag

Users can now rotate the server token using k3s token rotate -t <OLD_TOKEN> --new-token <NEW_TOKEN>. After command succeeds, all server nodes must be restarted with the new token.

Fixed an issue that could cause k3s to attempt to remove members from the etcd cluster immediately following a cluster-reset/restore, if they were queued for removal at the time the snapshot was taken.

Add --image-service-endpoint flag to specify an external image service socket.

Re-enable etcd endpoint auto-sync

Manually requeue configmap reconcile when no nodes have reconciled snapshots

Bump embedded containerd to v1.7.6

Bump embedded stargz-snapshotter plugin to latest

Fixed intermittent drone CI failures due to race conditions in test environment setup scripts

Fixed CI failures due to changes to api discovery changes in Kubernetes 1.28

Upgraded cni-plugins to v1.3.0

Update flannel to v0.22.1

User-provided containerd config templates may now use {{ template "base" . }} to include the default K3s template content. This makes it easier to maintain user configuration if the only need is to add additional sections to the file.

K3s no longer enables the apiserver’s enable-aggregator-routing flag when the egress proxy is not being used to route connections to in-cluster endpoints.

K3s’s external apiserver listener now declines to add to its certificate any subject names not associated with the kubernetes apiserver service, server nodes, or values of the --tls-san option. This prevents the certificate’s SAN list from being filled with unwanted entries.

Bump docker/docker module version to fix issues with cri-dockerd caused by recent releases of golang rejecting invalid host headers sent by the docker client.

Updated the embedded containerd to v1.7.3+k3s1

Updated the embedded runc to v1.1.8

Updated the embedded etcd to v3.5.9+k3s1

Updated kine to v0.10.2

Bumped dynamiclistener to address an issue that could cause the apiserver/supervisor listener on 6443 to stop serving requests on etcd-only nodes.

The K3s external apiserver/supervisor listener on 6443 now sends a complete certificate chain in the TLS handshake.

The version of helm used by the bundled helm controller’s job image has been updated to v3.12.3

Move flannel to v0.22.2

Added a new --tls-san-security option. This flag defaults to false, but can be set to true to disable automatically adding SANs to the server’s TLS certificate to satisfy any hostname requested by a client.

The default helm-controller job image can now be overridden with the --helm-job-image CLI flag

Resolved an issue that caused agents joined with kubeadm-style bootstrap tokens to fail to rejoin the cluster when their node object is deleted.

Support connecting tailscale to a separate server (e.g. headscale)

Update flannel to v0.22.0

The coredns-custom ConfigMap now allows for *.override sections to be included in the .:53 default server block.

Bumped klipper-lb image to v0.4.4 to resolve an issue that prevented access to ServiceLB ports from localhost when the Service ExternalTrafficPolicy was set to Local.

The bundled metrics-server has been bumped to v0.6.3, and now uses only secure TLS ciphers by default.

The K3s core controllers (supervisor, deploy, and helm) no longer use the admin kubeconfig. This makes it easier to determine from access and audit logs which actions are performed by the system, and which are performed by an administrative user.

K3s now allows nodes to join the cluster even if the node password secret cannot be created at the time the node joins. The secret create will be retried in the background. This resolves a potential deadlock created by fail-closed validating webhooks that block secret creation, where the webhook is unavailable until new nodes join the cluster to run the webhook pod.

The bundled containerd’s aufs/devmapper/zfs snapshotter plugins have been restored. These were unintentionally omitted when moving containerd back into the k3s multicall binary in the previous release.

Non root users can now call k3s --help and k3s --version commands without running into permission errors over the default config file.

Integration of tailscale VPN into k3s

The embedded helm controller has been bumped to v0.15.0, and now supports creating the chart’s target namespace if it does not exist.

Fixed an issue where Addon, HelmChart, and HelmChartConfig CRDs were created without structural schema, allowing the creation of custom resources of these types with invalid content.

K3s once again supports aarch64 nodes with page size > 4k

The packaged Traefik version has been bumped to v2.9.10 / chart 21.2.0

K3s now prints a more meaningful error when attempting to run from a filesystem mounted noexec.

Servers started with the (experimental) --disable-agent flag no longer attempt to run the tunnel authorizer agent component.

Fixed an regression that prevented the pod and cluster egress-selector modes from working properly.

K3s now retries the cluster join operation when receiving a "too many learners" error from etcd. This most frequently occurred when attempting to add multiple servers at the same time.

K3s now correctly passes through etcd-args to the temporary etcd that is used to extract cluster bootstrap data when restarting managed etcd nodes.

K3s now properly handles errors obtaining the current etcd cluster member list when a new server is joining the managed etcd cluster.

The bundled CNI plugins have been upgraded to v1.2.0-k3s1. The bandwidth and firewall plugins are now included in the bundle.

K3s now exits with a proper error message when the server token uses a bootstrap token id.secret format.

The embedded kine version has been bumped to v0.10.1. This replaces the legacy lib/pq postgres driver with pgx.

The embedded containerd version has been bumped to v1.7.0-k3s1, and has been reintegrated into the main k3s binary for a significant savings in release artifact size.

The embedded Helm controller now supports authenticating to chart repositories via credentials stored in a Secret, as well as passing repo CAs via ConfigMap.

The bundled containerd and runc versions have been bumped to v1.7.1-k3s1/v1.1.7

--flannel-backed=wireguard has been completely replaced with --flannel-backend=wireguard-native

The k3s etcd-snapshot command will now print a help message, to save a snapshot use: k3s etcd-snapshot save

The following flags will now cause fatal errors (with full removal coming in v1.28.0):