Architecture

Because of this shared infrastructure, the CNI is the same as the one configured in the host cluster. The same goes for the available storage where the Storage Classes and Volumes are those of the host cluster.

To provide the needed isolation, K3k leverages Network Policies.

In shared mode, K3k leverages Kubernetes 'ResourceQuotas' and 'LimitRanges' to manage resource sharing and enforce limits. Since all virtual cluster workloads run within the same namespace on the host cluster, ResourceQuotas are applied to this namespace to limit the total resources consumed by a virtual cluster. LimitRanges are used to set default resource requests and limits for pods, ensuring that workloads have reasonable resource allocations even if they do not explicitly specify them.

Each pod in a virtual cluster is assigned a unique name that incorporates the pod name, namespace and cluster name. This prevents naming collisions in the shared host cluster namespace.

It is important to understand that ResourceQuotas are applied at the namespace level. This means that all pods within a virtual cluster share the same quota. While this provides overall limits for the virtual cluster, it also means that resource allocation is dynamic. If one workload is not using its full resource allocation, other workloads within the same virtual cluster can utilize those resources, even if they belong to different deployments or services.

This dynamic sharing can be both a benefit and a challenge. It allows for efficient resource utilization, but it can also lead to unpredictable performance if workloads have varying resource demands. Furthermore, this approach makes it difficult to guarantee strict resource isolation between workloads within the same virtual cluster.

Isolation between virtual clusters in shared mode relies heavily on Kubernetes Network Policies. Network Policies define rules that control the network traffic allowed to and from pods. K3k configures Network Policies to ensure that pods in one virtual cluster cannot communicate with pods in other virtual clusters or with pods in the host cluster itself, providing a strong foundation for network isolation.

While Network Policies offer robust isolation capabilities, it is important to understand their characteristics:

K3k also utilizes Kubernetes Pod Security Admission (PSA) to enforce security policies within virtual clusters based on Pod Security Standards (PSS). PSS define different levels of security for pods, restricting what actions pods can perform. By configuring PSA to enforce a specific PSS level (for example, baseline or restricted) for a virtual cluster, K3k ensures that pods adhere to established security best practices and prevents them from using privileged features or performing potentially dangerous operations.

Key aspects of PSA integration include:

The shared mode architecture is designed with security in mind. K3k employs multiple layers of security controls, including Network Policies and PSA, to protect virtual clusters and the host cluster. While the shared namespace model requires careful configuration and management, these controls provide a robust security foundation for running workloads in a multi-tenant environment. K3k continuously evaluates and enhances its security mechanisms to address evolving threats and ensure the highest level of protection for its users.

Virtual clusters in virtual mode each have their own independent networking configuration managed by their respective K3s servers. Each virtual cluster runs its own CNI plugin, configured within its K3s server, providing complete network isolation from other virtual clusters and the host cluster. While the virtual cluster networks ultimately operate on top of the host cluster’s network infrastructure, the networking configuration and traffic management are entirely separate.

Resource sharing in virtual mode is managed by applying resource limits to the pods that make up the virtual cluster (both the K3s server pod and the K3s agent pods). Each pod is assigned a specific amount of CPU, memory, and other resources. The workloads running within the virtual cluster then utilize these allocated resources. This means that the virtual cluster as a whole has a defined resource pool determined by the limits on its constituent pods.

This approach provides a clear and direct way to control the resources available to each virtual cluster. However, it requires careful resource planning to ensure that each virtual cluster has sufficient capacity for its workloads.

The virtual mode offers strong isolation due to the dedicated K3s clusters deployed for each virtual cluster. Because each virtual cluster runs its own separate control plane and worker nodes, workloads are effectively isolated from each other and from the host cluster. This architecture minimizes the risk of one virtual cluster impacting others or the host cluster.

Security in virtual mode benefits from the inherent isolation provided by the separate K3s clusters. However, standard Kubernetes security best practices still apply, and K3k emphasizes a layered security approach. While the K3s server pods often run with elevated privileges (due to the nature of their function, requiring access to system resources), K3k recommends minimizing these privileges whenever possible and adhering to the principle of least privilege. This can be achieved by carefully configuring the necessary capabilities instead of relying on full privileged mode. Further information on K3s security best practices can be found in the official K3s documentation: https://docs.k3s.io/security (This link provides general security guidance, including discussions of capabilities and other relevant topics).

Currently, security in virtual mode has a risk of privilege escalation as the server pods run with elevated privileges due to the nature of their function, requiring access to system resources.