v1.28.X
|
Upgrade Notice
Before upgrading from earlier releases, be sure to read the Kubernetes Urgent Upgrade Notes. |
| Version | Release date | Kubernetes | Kine | SQLite | Etcd | Containerd | Runc | Flannel | Metrics-server | Traefik | CoreDNS | Helm-controller | Local-path-provisioner |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Sep 19 2024 |
|||||||||||||
Aug 21 2024 |
|||||||||||||
Jul 31 2024 |
|||||||||||||
Jul 03 2024 |
|||||||||||||
Jun 25 2024 |
|||||||||||||
May 22 2024 |
|||||||||||||
Apr 25 2024 |
|||||||||||||
Mar 25 2024 |
|||||||||||||
Feb 29 2024 |
|||||||||||||
Feb 06 2024 |
|||||||||||||
Dec 27 2023 |
|||||||||||||
Dec 06 2023 |
|||||||||||||
Nov 08 2023 |
|||||||||||||
Oct 30 2023 |
|||||||||||||
Sep 20 2023 |
|||||||||||||
Sep 08 2023 |
Release v1.28.14+k3s1
This release updates Kubernetes to v1.28.14, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.13+k3s1:
-
Testing Backports for 2024-09 (#10804)
-
Update to newer OS images for install testing
-
Fix caching name for e2e vagrant box
-
Fix deploy latest commit on E2E tests
-
DRY E2E Upgrade test setup
-
Cover edge case when on new minor release for E2E upgrade test
-
-
Update CNI plugins version (#10820)
-
Backports for 2024-09 (#10845)
-
Fix hosts.toml header var (#10874)
-
Update to v1.28.14-k3s1 and Go 1.22.6 (#10884)
-
Update Kubernetes to v1.28.14-k3s2 (#10907)
Release v1.28.13+k3s1
This release updates Kubernetes to v1.28.13, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.12+k3s1:
-
Fixing setproctitle function (#10624)
-
Bump docker/docker to v24.0.10-0.20240723193628-852759a7df45 (#10651)
-
Backports for 2024-08 release cycle (#10666)
-
Use pagination when listing large numbers of resources
-
Fix multiple issues with servicelb
-
Remove deprecated use of wait. functions
-
Wire lasso metrics up to metrics endpoint
-
-
Backports for August 2024 (#10673)
-
Bump containerd to v1.7.20 (#10662)
-
Add tolerations support for DaemonSet pods (#10705)
-
New Feature: Users can now define Kubernetes tolerations for ServiceLB DaemonSet directly in the
svccontroller.k3s.cattle.io/tolerationsannotation on services.
-
-
Update to v1.28.13-k3s1 and Go 1.22.5 (#10719)
Release v1.28.12+k3s1
This release updates Kubernetes to v1.28.12, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.11+k3s2:
-
Backports for 2024-07 release cycle (#10499)
-
Bump k3s-root to v0.14.0
-
Bump github.com/hashicorp/go-retryablehttp from 0.7.4 to 0.7.7
-
Bump Local Path Provisioner version
-
Ensure remotedialer kubelet connections use kubelet bind address
-
Chore: Bump Trivy version
-
Add etcd s3 config secret implementation
-
-
July Test Backports (#10509)
-
Update to v1.28.12-k3s1 and Go 1.22.5 (#10541)
-
Fix issues loading data-dir value from env vars or dropping config files (#10598)
Release v1.28.11+k3s2
This release updates Kubernetes to v1.28.11, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Release v1.28.11+k3s1
This release updates Kubernetes to v1.28.11, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.10+k3s1:
-
Replace deprecated ruby function (#10090)
-
Fix bug when using tailscale config by file (#10144)
-
Bump flannel version to v0.25.2 (#10221)
-
Update kube-router version to v2.1.2 (#10182)
-
Improve tailscale test & add extra log in e2e tests (#10213)
-
Backports for 2024-06 release cycle (#10258)
-
Add WithSkipMissing to not fail import on missing blobs
-
Use fixed stream server bind address for cri-dockerd
-
Switch stargz over to cri registry config_path
-
Bump to containerd v1.7.17, etcd v3.5.13
-
Bump spegel version
-
Fix issue with externalTrafficPolicy: Local for single-stack services on dual-stack nodes
-
ServiceLB now sets the priorityClassName on svclb pods to
system-node-criticalby default. This can be overridden on a per-service basis via thesvccontroller.k3s.cattle.io/priorityclassnameannotation. -
Bump minio-go to v7.0.70
-
Bump kine to v0.11.9 to fix pagination
-
Update valid resolv conf
-
Add missing kernel config check
-
Symlinked sub-directories are now respected when scanning Auto-Deploying Manifests (AddOns)
-
Fix bug: allow helm controller set owner reference
-
Bump klipper-helm image for tls secret support
-
Fix issue with k3s-etcd informers not starting
-
--Enable-pprofcan now be set on agents to enable the debug/pprof endpoints. When set, agents will listen on the supervisor port. -
--Supervisor-metricscan now be set on servers to enable serving internal metrics on the supervisor endpoint; when set agents will listen on the supervisor port. -
Fix netpol crash when node remains tainted uninitialized
-
The embedded load-balancer will now fall back to trying all servers with health-checks ignored, if all servers have been marked unavailable due to failed health checks.
-
-
More backports for 2024-06 release cycle (#10289)
-
Add snapshot retention etcd-s3-folder fix (#10315)
-
Add test for
isValidResolvConf(#10302) (#10331) -
Fix race condition panic in loadbalancer.nextServer (#10323)
-
Fix typo, use
rancher/permissions(#10299) -
Update Kubernetes to v1.28.11 (#10347)
-
Fix agent supervisor port using apiserver port instead (#10355)
-
Fix issue that allowed multiple simultaneous snapshots to be allowed (#10377)
Release v1.28.10+k3s1
This release updates Kubernetes to v1.28.10, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Release v1.28.9+k3s1
This release updates Kubernetes to v1.28.9, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.8+k3s1:
-
Add a new error when kine is with disable apiserver or disable etcd (#9804)
-
Remove old pinned dependencies (#9827)
-
Transition from deprecated pointer library to ptr (#9824)
-
Golang caching and E2E ubuntu 23.10 (#9821)
-
Add tls for kine (#9849)
-
Bump spegel to v0.0.20-k3s1 (#9880)
-
Backports for 2024-04 release cycle (#9911)
-
Send error response if member list cannot be retrieved
-
The k3s stub cloud provider now respects the kubelet’s requested provider-id, instance type, and topology labels
-
Fix error when image has already been pulled
-
Add /etc/passwd and /etc/group to k3s docker image
-
Fix etcd snapshot reconcile for agentless servers
-
Add health-check support to loadbalancer
-
Add certificate expiry check, events, and metrics
-
Add workaround for containerd hosts.toml bug when passing config for default registry endpoint
-
Add supervisor cert/key to rotate list
-
The embedded containerd has been bumped to v1.7.15
-
The embedded cri-dockerd has been bumped to v0.3.12
-
The
k3s etcd-snapshotcommand has been reworked for improved consistency. All snapshots operations are now performed by the server process, with the CLI acting as a client to initiate and report results. As a side effect, the CLI is now less noisy when managing snapshots. -
Improve etcd load-balancer startup behavior
-
Actually fix agent certificate rotation
-
Traefik has been bumped to v2.10.7.
-
Traefik pod annotations are now set properly in the default chart values.
-
The system-default-registry value now supports RFC2732 IPv6 literals.
-
The local-path provisioner now defaults to creating
localvolumes, instead ofhostPath.
-
-
Allow LPP to read helper logs (#9938)
-
Update kube-router to v2.1.0 (#9942)
-
Update to v1.28.9-k3s1 and Go 1.21.9 (#9959)
-
Fix on-demand snapshots timing out; not honoring folder (#9994)
-
Make /db/info available anonymously from localhost (#10002)
Release v1.28.8+k3s1
This release updates Kubernetes to v1.28.8, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.7+k3s1:
-
Add an integration test for flannel-backend=none (#9608)
-
Install and Unit test backports (#9641)
-
Update klipper-lb image version (#9605)
-
Chore(deps): Remediating CVE-2023-45142 CVE-2023-48795 (#9647)
-
Adjust first node-ip based on configured clusterCIDR (#9631)
-
Improve tailscale e2e test (#9653)
-
Backports for 2024-03 release cycle (#9669)
-
Fix: use correct wasm shims names
-
The embedded flannel cni-plugin binary is now built and versioned separate from the rest of the cni plugins and the embedded flannel controller.
-
Bump spegel to v0.0.18-k3s3
-
Adds wildcard registry support
-
Fixes issue with excessive CPU utilization while waiting for containerd to start
-
Add env var to allow spegel mirroring of latest tag
-
Tweak netpol node wait logs
-
Fix coredns NodeHosts on dual-stack clusters
-
Bump helm-controller/klipper-helm versions
-
Fix snapshot prune
-
Fix issue with etcd node name missing hostname
-
Rootless mode should also bind service nodePort to host for LoadBalancer type, matching UX of rootful mode.
-
To enable raw output for the
check-configsubcommand, you may now set NO_COLOR=1 -
Fix additional corner cases in registries handling
-
Bump metrics-server to v0.7.0
-
K3s will now warn and suppress duplicate entries in the mirror endpoint list for a registry. Containerd does not support listing the same endpoint multiple times as a mirror for a single upstream registry.
-
-
Docker and E2E Test Backports (#9707)
-
Fix wildcard entry upstream fallback (#9733)
-
Update to v1.28.8-k3s1 and Go 1.21.8 (#9746)
Release v1.28.7+k3s1
This release updates Kubernetes to v1.28.7, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.6+k3s2:
-
Chore: bump Local Path Provisioner version (#9426)
-
Bump cri-dockerd to fix compat with Docker Engine 25 (#9293)
-
Auto Dependency Bump (#9419)
-
Runtimes refactor using exec.LookPath (#9431)
-
Directories containing runtimes need to be included in the $PATH environment variable for effective runtime detection.
-
-
Changed how lastHeartBeatTime works in the etcd condition (#9424)
-
Bump Flannel v0.24.2 + remove multiclustercidr (#9401)
-
Allow executors to define containerd and docker behavior (#9254)
-
Update Kube-router to v2.0.1 (#9404)
-
Backports for 2024-02 release cycle (#9462)
-
Enable longer http timeout requests (#9444)
-
Test_UnitApplyContainerdQoSClassConfigFileIfPresent (#9440)
-
Support PR testing installs (#9469)
-
Update Kubernetes to v1.28.7 (#9492)
-
Fix drone publish for arm (#9508)
-
Remove failing Drone step (#9516)
-
Restore original order of agent startup functions (#9545)
-
Fix netpol startup when flannel is disabled (#9578)
Release v1.28.6+k3s2
This release updates Kubernetes to v1.28.6, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Important Notes
Addresses the runc CVE: CVE-2024-21626 by updating runc to v1.1.12.
Changes since v1.28.5+k3s1:
-
Add a retry around updating a secrets-encrypt node annotations (#9125)
-
Wait for taint to be gone in the node before starting the netpol controller (#9175)
-
Etcd condition (#9181)
-
Backports for 2024-01 (#9203)
-
Pin opa version for missing dependency chain (#9216)
-
Added support for env *_PROXY variables for agent loadbalancer (#9206)
-
Etcd node is nil (#9228)
-
Update to v1.28.6 and Go 1.20.13 (#9260)
-
Use
ipFamilyPolicy: RequireDualStackfor dual-stack kube-dns (#9269) -
Backports for 2024-01 k3s2 (#9336)
-
Bump runc to v1.1.12 and helm-controller to v0.15.7
-
Fix handling of bare hostname or IP as endpoint address in registries.yaml
-
-
Bump helm-controller to fix issue with ChartContent (#9346)
Release v1.28.5+k3s1
This release updates Kubernetes to v1.28.5, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.4+k3s1:
-
Remove s390x steps temporarily since runners are disabled (#8983)
-
Remove s390x from manifest (#8998)
-
Fix overlapping address range (#8913)
-
Modify CONTRIBUTING.md guide (#8954)
-
Nov 2023 stable channel update (#9022)
-
Default runtime and runtime classes for wasm/nvidia/crun (#8936)
-
Added runtime classes for wasm/nvidia/crun
-
Added default runtime flag for containerd
-
-
Bump containerd/runc to v1.7.10-k3s1/v1.1.10 (#8962)
-
Allow setting default-runtime on servers (#9027)
-
Bump containerd to v1.7.11 (#9040)
-
Update to v1.28.5-k3s1 (#9081)
Release v1.28.4+k3s2
This release updates Kubernetes to v1.28.4, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.3+k3s2:
-
Update channels latest to v1.27.7+k3s2 (#8799)
-
Add etcd status condition (#8724)
-
Now the user can see the etcd status from each node in a simple way
-
-
ADR for etcd status (#8355)
-
Wasm shims detection (#8751)
-
Automatic discovery of WebAssembly runtimes
-
-
Add warning for removal of multiclustercidr flag (#8758)
-
Improve dualStack log (#8798)
-
Optimize: Simplify and clean up Dockerfile (#8244)
-
Add: timezone info in image (#8764)
-
-
New timezone info in Docker image allows the use of
spec.timeZonein CronJobs
-
-
-
Bump kine to fix nats, postgres, and watch issues (#8778)
-
Bumped kine to v0.11.0 to resolve issues with postgres and NATS, fix performance of watch channels under heavy load, and improve compatibility with the reference implementation.
-
-
QoS-class resource configuration (#8726)
-
Containerd may now be configured to use rdt or blockio configuration by defining
rdt_config.yamlorblockio_config.yamlfiles.
-
-
Add agent flag disable-apiserver-lb (#8717)
-
Add agent flag disable-apiserver-lb, agent will not start load balance proxy.
-
-
Force umount for NFS mount (like with longhorn) (#8521)
-
General updates to README (#8786)
-
Fix wrong warning from restorecon in install script (#8871)
-
Fix issue with snapshot metadata configmap (#8835)
-
Omit snapshot list configmap entries for snapshots without extra metadata
-
-
Skip initial datastore reconcile during cluster-reset (#8861)
-
Tweaked order of ingress IPs in ServiceLB (#8711)
-
Improved ingress IP ordering from ServiceLB
-
-
Disable helm CRD installation for disable-helm-controller (#8702)
-
More improves for K3s patch release docs (#8800)
-
Update install.sh sha256sum (#8885)
-
Add jitter to client config retry to avoid hammering servers when they are starting up (#8863)
-
Handle nil pointer when runtime core is not ready in etcd (#8886)
-
Bump dynamiclistener; reduce snapshot controller log spew (#8894)
-
Bumped dynamiclistener to address a race condition that could cause a server to fail to sync its certificates into the Kubernetes secret
-
Reduced etcd snapshot log spam during initial cluster startup
-
-
Remove depends_on for e2e step; fix cert rotate e2e (#8906)
-
Fix etcd snapshot S3 issues (#8926)
-
Don’t apply S3 retention if S3 client failed to initialize
-
Don’t request metadata when listing S3 snapshots
-
Print key instead of file path in snapshot metadata log message
-
-
Update to v1.28.4 and Go to v1.20.11 (#8920)
-
Remove s390x steps temporarily since runners are disabled (#8983)
-
Remove s390x from manifest (#8998)
Release v1.28.3+k3s2
This release updates Kubernetes to v1.28.3, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.3+k3s1:
-
Restore selinux context systemd unit file (#8593)
-
Update channel to v1.27.7+k3s1 (#8753)
-
Bump Sonobuoy version (#8710)
-
Bump Trivy version (#8739)
-
Fix: Access outer scope .SystemdCgroup (#8761)
-
Fixed failing to start with nvidia-container-runtime
-
-
Upgrade traefik chart to v25.0.0 (#8771)
-
Update traefik to fix registry value (#8792)
-
Don’t use iptables-save/iptables-restore if it will corrupt rules (#8795)
Release v1.28.3+k3s1
This release updates Kubernetes to v1.28.3, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.2+k3s1:
-
Fix error reporting (#8250)
-
Add context to flannel errors (#8284)
-
Update channel, September patch release (#8397)
-
Add missing link to drone in documentation (#8295)
-
Include the interface name in the error message (#8346)
-
Add extraArgs to vpn provider (#8354)
-
Allow to pass extra args to the vpn provider
-
-
Disable HTTP on main etcd client port (#8402)
-
Embedded etcd no longer serves http requests on the client port, only grpc. This addresses a performance issue that could cause watch stream starvation under load. For more information, see https://github.com/etcd-io/etcd/issues/15402
-
-
Server token rotation (#8215)
-
Fix issues with etcd member removal after reset (#8392)
-
Fixed an issue that could cause k3s to attempt to remove members from the etcd cluster immediately following a cluster-reset/restore, if they were queued for removal at the time the snapshot was taken.
-
-
Fix gofmt error (#8439)
-
Added advertise address integration test (#8344)
-
Added cluster reset from non bootstrap nodes on snapshot restore e2e test (#8292)
-
Fix .github regex to skip drone runs on gh action bumps (#8433)
-
Added error when cluster reset while using server flag (#8385)
-
The user will receive a error when --cluster-reset with the --server flag
-
-
Update kube-router (#8423)
-
Update kube-router to v2.0.0-rc7 to fix performance issues
-
-
Add SHA256 signatures of the install script (#8312)
-
-
Add SHA256 signatures of the install script.
-
-
-
Add --image-service-endpoint flag (#8279)
-
Add
--image-service-endpointflag to specify an external image service socket.
-
-
Don’t ignore assets in home dir if system assets exist (#8458)
-
Pass SystemdCgroup setting through to nvidia runtime options (#8470)
-
Fixed issue that would cause pods using nvidia container runtime to be killed after a few seconds, when using newer versions of nvidia-container-toolkit.
-
-
Improve release docs - updated (#8414)
-
Take IPFamily precedence based on order (#8460)
-
Fix spellcheck problem (#8507)
-
Network defaults are duplicated, remove one (#8523)
-
Fix slemicro check for selinux (#8526)
-
Update install.sh.sha256sum (#8566)
-
System agent push tags fix (#8568)
-
Fixed tailscale node IP dualstack mode in case of IPv4 only node (#8524)
-
Server Token Rotation (#8265)
-
Users can now rotate the server token using
k3s token rotate -t <OLD_TOKEN> --new-token <NEW_TOKEN>. After command succeeds, all server nodes must be restarted with the new token.
-
-
E2E Domain Drone Cleanup (#8579)
-
Bump containerd to v1.7.7-k3s1 (#8604)
-
Bump busybox to v1.36.1 (#8602)
-
Migrate to using custom resource to store etcd snapshot metadata (#8064)
-
Switch build target from main.go to a package. (#8342)
-
Use IPv6 in case is the first configured IP with dualstack (#8581)
-
Bump traefik, golang.org/x/net, google.golang.org/grpc (#8624)
-
Update kube-router package in build script (#8630)
-
Add etcd-only/control-plane-only server test and fix control-plane-only server crash (#8638)
-
Use
version.Programnot K3s in token rotate logs (#8653) -
[Windows Port (#7259)
-
Fix CloudDualStackNodeIPs feature-gate inconsistency (#8667)
-
Re-enable etcd endpoint auto-sync (#8675)
-
Manually requeue configmap reconcile when no nodes have reconciled snapshots (#8683)
-
Update to v1.28.3 and Go to v1.20.10 (#8682)
-
Fix s3 snapshot restore (#8729)
Release v1.28.2+k3s1
This release updates Kubernetes to v1.28.2, and fixes a number of issues.
For more details on what’s new, see the Kubernetes release notes.
Changes since v1.28.1+k3s1:
-
Update channel for version v1.28 (#8305)
-
Bump kine to v0.10.3 (#8323)
-
Update to v1.28.2 and go v1.20.8 (#8364)
-
Bump embedded containerd to v1.7.6
-
Bump embedded stargz-snapshotter plugin to latest
-
Fixed intermittent drone CI failures due to race conditions in test environment setup scripts
-
Fixed CI failures due to changes to api discovery changes in Kubernetes 1.28
-
Release v1.28.1+k3s1
This release is K3S’s first in the v1.28 line. This release updates Kubernetes to v1.28.1.
|
Important
This release includes remediation for CVE-2023-32187, a potential Denial of Service attack vector on K3s servers. See https://github.com/k3s-io/k3s/security/advisories/GHSA-m4hf-6vgr-75r2 for more information, including documentation on changes in behavior that harden clusters against this vulnerability. |
|
Critical Regression
Kubernetes v1.28 contains a critical regression (kubernetes/kubernetes#120247) that causes init containers to run at the same time as app containers following a restart of the node. This issue will be fixed in v1.28.2. We do not recommend using K3s v1.28 at this time if your application depends on init containers. |
For more details on what’s new, see the Kubernetes release notes.