Cluster Load Balancer

This section describes how to install an external load balancer in front of a High Availability (HA) K3s cluster’s server nodes. Two examples are provided: Nginx and HAProxy.

External load-balancers should not be confused with the embedded ServiceLB, which is an embedded controller that allows for use of Kubernetes LoadBalancer Services without deploying a third-party load-balancer controller. For more details, see Service Load Balancer.

External load-balancers can be used to provide a fixed registration address for registering nodes, or for external access to the Kubernetes API Server. For exposing LoadBalancer Services, external load-balancers can be used alongside or instead of ServiceLB, but in most cases, replacement load-balancer controllers such as MetalLB or Kube-VIP are a better choice.

Prerequisites

All nodes in this example are running Ubuntu 20.04.

For both examples, assume that a HA K3s cluster with embedded etcd has been installed on 3 nodes.

Each k3s server is configured with:

# /etc/rancher/k3s/config.yaml
token: lb-cluster-gd
tls-san: 10.10.10.100
yaml

The nodes have hostnames and IPs of:

  • server-1: 10.10.10.50

  • server-2: 10.10.10.51

  • server-3: 10.10.10.52

Two additional nodes for load balancing are configured with hostnames and IPs of:

  • lb-1: 10.10.10.98

  • lb-2: 10.10.10.99

Three additional nodes exist with hostnames and IPs of:

  • agent-1: 10.10.10.101

  • agent-2: 10.10.10.102

  • agent-3: 10.10.10.103

Setup Load Balancer

HAProxy is an open source option that provides a TCP load balancer. It also supports HA for the load balancer itself, ensuring redundancy at all levels. See HAProxy Documentation for more info.

Additionally, we will use KeepAlived to generate a virtual IP (VIP) that will be used to access the cluster. See KeepAlived Documentation for more info.

  1. Install HAProxy and KeepAlived:

    sudo apt-get install haproxy keepalived
    bash
  2. Add the following to /etc/haproxy/haproxy.cfg on lb-1 and lb-2:

    frontend k3s-frontend
        bind *:6443
        mode tcp
        option tcplog
        default_backend k3s-backend
    
    backend k3s-backend
        mode tcp
        option tcp-check
        balance roundrobin
        default-server inter 10s downinter 5s
        server server-1 10.10.10.50:6443 check
        server server-2 10.10.10.51:6443 check
        server server-3 10.10.10.52:6443 check
  3. Add the following to /etc/keepalived/keepalived.conf on lb-1 and lb-2:

    global_defs {
      enable_script_security
      script_user root
    }
    
    vrrp_script chk_haproxy {
        script 'killall -0 haproxy' # faster than pidof
        interval 2
    }
    
    vrrp_instance haproxy-vip {
        interface eth1
        state <STATE> # MASTER on lb-1, BACKUP on lb-2
        priority <PRIORITY> # 200 on lb-1, 100 on lb-2
    
        virtual_router_id 51
    
        virtual_ipaddress {
            10.10.10.100/24
        }
    
        track_script {
            chk_haproxy
        }
    }
  4. Restart HAProxy and KeepAlived on lb-1 and lb-2:

    systemctl restart haproxy
    systemctl restart keepalived
    bash
  5. On agent-1, agent-2, and agent-3, run the following command to install k3s and join the cluster:

    curl -sfL https://get.k3s.io | K3S_TOKEN=lb-cluster-gd sh -s - agent --server https://10.10.10.100:6443
    bash

    You can now use kubectl from server node to interact with the cluster.

    root@server-1 $ k3s kubectl get nodes -A
    NAME       STATUS   ROLES                       AGE     VERSION
    agent-1    Ready    <none>                      32s     v1.27.3+k3s1
    agent-2    Ready    <none>                      20s     v1.27.3+k3s1
    agent-3    Ready    <none>                      9s      v1.27.3+k3s1
    server-1   Ready    control-plane,etcd,master   4m22s   v1.27.3+k3s1
    server-2   Ready    control-plane,etcd,master   3m58s   v1.27.3+k3s1
    server-3   Ready    control-plane,etcd,master   3m12s   v1.27.3+k3s1
    bash