29 使用 clusterclass 部署下游群集 #
29.1 简介 #
置备 Kubernetes 群集是一项复杂的任务，需要具备深入的群集组件配置专业知识。随着配置变得愈发复杂，或者不同提供商的需求引入了大量提供商专用的资源定义，群集创建可能会让人望而生畏。值得庆幸的是，Kubernetes Cluster API (CAPI) 提供了一种更简洁的声明式方法，而 ClusterClass 进一步增强了这种方法。此功能引入了一种基于模板的模型，允许您定义可重用的群集类，该类封装了复杂性并提升了一致性。
29.2 什么是 ClusterClass？ #
CAPI 项目引入了 ClusterClass 功能，通过采用基于模板的群集实例化方法，实现了 Kubernetes 群集生命周期管理的范式转变。用户无需为每个群集独立定义资源，而是定义一个 ClusterClass，将它作为一个全面且可重用的蓝图。这种抽象表示封装了 Kubernetes 群集的期望状态和配置，让您能快速、一致地创建多个符合已定义规范的群集。这种抽象化减轻了配置负担，使部署清单更易于管理。这意味着工作负载群集的核心组件在类级别定义，这样用户便可将这些模板用作 Kubernetes 群集版本，可多次重用以置备群集。ClusterClass 的实现提供了多项关键优势，解决了传统 CAPI 大规模管理所固有的挑战：
大幅降低复杂性和 YAML 冗余度
优化维护和更新过程
增强部署间的一致性和标准化
提高可扩展性和自动化能力
声明式管理和强大的版本控制
29.3 当前 CAPI 置备文件示例 #
利用 Cluster API (CAPI) 和 RKE2 提供程序部署 Kubernetes 群集需要定义多个自定义资源。这些资源定义了群集及其底层基础架构的期望状态，使 CAPI 能够编排置备和管理生命周期。下面的代码段说明了必须配置的资源类型：
群集：此资源封装了总体配置，包括将管理节点间通讯和服务发现的网络拓扑。此外，它还建立了与控制平面规范和指定的基础架构提供程序资源的必要关联，从而告知 CAPI 期望的群集体系结构及将用于置备群集的底层基础架构。
Metal3Cluster：此资源定义 Metal3 特有的基础架构级属性，例如 Kubernetes API 服务器可通过其访问的外部端点。
RKE2ControlPlane：此资源定义群集控制平面节点的特性和行为。此规范中配置了期望的控制平面复本数量（对确保高可用性和容错能力至关重要）、特定的 Kubernetes 发行版版本（与所选的 RKE2 版本一致），以及控制平面组件的滚动更新策略等参数。此外，此资源规定了将在群集中采用的容器网络接口 (CNI)，便于注入特定代理的配置，通常利用 Ignition 来无缝自动完成控制平面节点的 RKE2 代理置备。
Metal3MachineTemplate：此资源充当将构成 Kubernetes 群集工作节点的各计算实例的创建蓝图，定义了要使用的映像。
Metal3DataTemplate：此资源作为 Metal3MachineTemplate 的补充，可用于为新置备的计算机实例指定其他元数据。
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: emea-spa-cluster-3
namespace: emea-spa
spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/18
services:
cidrBlocks:
- 10.96.0.0/12
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlane
name: emea-spa-cluster-3
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3Cluster
name: emea-spa-cluster-3
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3Cluster
metadata:
name: emea-spa-cluster-3
namespace: emea-spa
spec:
controlPlaneEndpoint:
host: 192.168.122.203
port: 6443
noCloudProvider: true
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlane
metadata:
name: emea-spa-cluster-3
namespace: emea-spa
spec:
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
name: emea-spa-cluster-3
replicas: 1
version: v1.32.4+rke2r1
rolloutStrategy:
type: "RollingUpdate"
rollingUpdate:
maxSurge: 1
registrationMethod: "control-plane-endpoint"
registrationAddress: 192.168.122.203
serverConfig:
cni: cilium
cniMultusEnable: true
tlsSan:
- 192.168.122.203
- https://192.168.122.203.sslip.io
agentConfig:
format: ignition
additionalUserData:
config: |
variant: fcos
version: 1.4.0
storage:
files:
- path: /var/lib/rancher/rke2/server/manifests/endpoint-copier-operator.yaml
overwrite: true
contents:
inline: |
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: endpoint-copier-operator
namespace: kube-system
spec:
chart: oci://registry.suse.com/edge/charts/endpoint-copier-operator
targetNamespace: endpoint-copier-operator
version: 303.0.0+up0.2.1
createNamespace: true
- path: /var/lib/rancher/rke2/server/manifests/metallb.yaml
overwrite: true
contents:
inline: |
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: metallb
namespace: kube-system
spec:
chart: oci://registry.suse.com/edge/charts/metallb
targetNamespace: metallb-system
version: 303.0.0+up0.14.9
createNamespace: true
- path: /var/lib/rancher/rke2/server/manifests/metallb-cr.yaml
overwrite: true
contents:
inline: |
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: kubernetes-vip-ip-pool
namespace: metallb-system
spec:
addresses:
- 192.168.122.203/32
serviceAllocation:
priority: 100
namespaces:
- default
serviceSelectors:
- matchExpressions:
- {key: "serviceType", operator: In, values: [kubernetes-vip]}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: ip-pool-l2-adv
namespace: metallb-system
spec:
ipAddressPools:
- kubernetes-vip-ip-pool
- path: /var/lib/rancher/rke2/server/manifests/endpoint-svc.yaml
overwrite: true
contents:
inline: |
apiVersion: v1
kind: Service
metadata:
name: kubernetes-vip
namespace: default
labels:
serviceType: kubernetes-vip
spec:
ports:
- name: rke2-api
port: 9345
protocol: TCP
targetPort: 9345
- name: k8s-api
port: 6443
protocol: TCP
targetPort: 6443
type: LoadBalancer
systemd:
units:
- name: rke2-preinstall.service
enabled: true
contents: |
[Unit]
Description=rke2-preinstall
Wants=network-online.target
Before=rke2-install.service
ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
[Service]
Type=oneshot
User=root
ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
ExecStartPost=/bin/sh -c "umount /mnt"
[Install]
WantedBy=multi-user.target
kubelet:
extraArgs:
- provider-id=metal3://BAREMETALHOST_UUID
nodeName: "localhost.localdomain"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
name: emea-spa-cluster-3
namespace: emea-spa
spec:
nodeReuse: True
template:
spec:
automatedCleaningMode: metadata
dataTemplate:
name: emea-spa-cluster-3
hostSelector:
matchLabels:
cluster-role: control-plane
deploy-region: emea-spa
node: group-3
image:
checksum: http://fileserver.local:8080/eibimage-downstream-cluster.raw.sha256
checksumType: sha256
format: raw
url: http://fileserver.local:8080/eibimage-downstream-cluster.raw
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
name: emea-spa-cluster-3
namespace: emea-spa
spec:
clusterName: emea-spa-cluster-3
metaData:
objectNames:
- key: name
object: machine
- key: local-hostname
object: machine
- key: local_hostname
object: machine29.4 将 CAPI 置备文件转换为 ClusterClass #
29.4.1 ClusterClass 定义 #
以下代码定义了一个 ClusterClass 资源，这是一个用于采用一致方式部署特定 Kubernetes 群集类型的声明式模板。该规范包含通用的基础架构和控制平面配置，可实现群集 Fleet 的高效置备和统一生命周期管理。下面的 clusterclass 示例中包含一些变量，这些变量将在群集实例化过程中被实际值替换。示例中使用的变量如下：
controlPlaneMachineTemplate：这是用于定义要使用的控制平面计算机模板引用的名称controlPlaneEndpointHost：这是控制平面端点的主机名或 IP 地址tlsSan：这是控制平面端点的 TLS 主题备用名称
clusterclass 定义文件基于以下 3 种资源定义：
ClusterClass：此资源封装了整个群集类定义，包括控制平面和基础架构模板。此外，它还包含在实例化过程中将被替换的变量列表。
RKE2ControlPlaneTemplate：此资源定义控制平面模板，指定期望的控制平面节点配置。它包含复本数量、Kubernetes 版本、要使用的 CNI 等参数。此外，一些参数在实例化过程中将被替换为适当的值。
Metal3ClusterTemplate：此资源定义基础架构模板，指定期望的底层基础架构配置。它包含控制平面端点、noCloudProvider 标志等参数。此外，一些参数在实例化过程中将被替换为适当的值。
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlaneTemplate
metadata:
name: example-controlplane-type2
namespace: emea-spa
spec:
template:
spec:
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
name: example-controlplane # This will be replaced by the patch applied in each cluster instances
namespace: emea-spa
replicas: 1
version: v1.32.4+rke2r1
rolloutStrategy:
type: "RollingUpdate"
rollingUpdate:
maxSurge: 1
registrationMethod: "control-plane-endpoint"
registrationAddress: "default" # This will be replaced by the patch applied in each cluster instances
serverConfig:
cni: cilium
cniMultusEnable: true
tlsSan:
- "default" # This will be replaced by the patch applied in each cluster instances
agentConfig:
format: ignition
additionalUserData:
config: |
default
kubelet:
extraArgs:
- provider-id=metal3://BAREMETALHOST_UUID
nodeName: "localhost.localdomain"
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3ClusterTemplate
metadata:
name: example-cluster-template-type2
namespace: emea-spa
spec:
template:
spec:
controlPlaneEndpoint:
host: "default" # This will be replaced by the patch applied in each cluster instances
port: 6443
noCloudProvider: true
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: example-clusterclass-type2
namespace: emea-spa
spec:
variables:
- name: controlPlaneMachineTemplate
required: true
schema:
openAPIV3Schema:
type: string
- name: controlPlaneEndpointHost
required: true
schema:
openAPIV3Schema:
type: string
- name: tlsSan
required: true
schema:
openAPIV3Schema:
type: array
items:
type: string
infrastructure:
ref:
kind: Metal3ClusterTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
name: example-cluster-template-type2
controlPlane:
ref:
kind: RKE2ControlPlaneTemplate
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
name: example-controlplane-type2
patches:
- name: setControlPlaneMachineTemplate
definitions:
- selector:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlaneTemplate
matchResources:
controlPlane: true
jsonPatches:
- op: replace
path: "/spec/template/spec/infrastructureRef/name"
valueFrom:
variable: controlPlaneMachineTemplate
- name: setControlPlaneEndpoint
definitions:
- selector:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3ClusterTemplate
matchResources:
infrastructureCluster: true # Added to select InfraCluster
jsonPatches:
- op: replace
path: "/spec/template/spec/controlPlaneEndpoint/host"
valueFrom:
variable: controlPlaneEndpointHost
- name: setRegistrationAddress
definitions:
- selector:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlaneTemplate
matchResources:
controlPlane: true # Added to select ControlPlane
jsonPatches:
- op: replace
path: "/spec/template/spec/registrationAddress"
valueFrom:
variable: controlPlaneEndpointHost
- name: setTlsSan
definitions:
- selector:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlaneTemplate
matchResources:
controlPlane: true # Added to select ControlPlane
jsonPatches:
- op: replace
path: "/spec/template/spec/serverConfig/tlsSan"
valueFrom:
variable: tlsSan
- name: updateAdditionalUserData
definitions:
- selector:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlaneTemplate
matchResources:
controlPlane: true
jsonPatches:
- op: replace
path: "/spec/template/spec/agentConfig/additionalUserData"
valueFrom:
template: |
config: |
variant: fcos
version: 1.4.0
storage:
files:
- path: /var/lib/rancher/rke2/server/manifests/endpoint-copier-operator.yaml
overwrite: true
contents:
inline: |
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: endpoint-copier-operator
namespace: kube-system
spec:
chart: oci://registry.suse.com/edge/charts/endpoint-copier-operator
targetNamespace: endpoint-copier-operator
version: 303.0.0+up0.2.1
createNamespace: true
- path: /var/lib/rancher/rke2/server/manifests/metallb.yaml
overwrite: true
contents:
inline: |
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: metallb
namespace: kube-system
spec:
chart: oci://registry.suse.com/edge/charts/metallb
targetNamespace: metallb-system
version: 303.0.0+up0.14.9
createNamespace: true
- path: /var/lib/rancher/rke2/server/manifests/metallb-cr.yaml
overwrite: true
contents:
inline: |
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: kubernetes-vip-ip-pool
namespace: metallb-system
spec:
addresses:
- {{ .controlPlaneEndpointHost }}/32
serviceAllocation:
priority: 100
namespaces:
- default
serviceSelectors:
- matchExpressions:
- {key: "serviceType", operator: In, values: [kubernetes-vip]}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: ip-pool-l2-adv
namespace: metallb-system
spec:
ipAddressPools:
- kubernetes-vip-ip-pool
- path: /var/lib/rancher/rke2/server/manifests/endpoint-svc.yaml
overwrite: true
contents:
inline: |
apiVersion: v1
kind: Service
metadata:
name: kubernetes-vip
namespace: default
labels:
serviceType: kubernetes-vip
spec:
ports:
- name: rke2-api
port: 9345
protocol: TCP
targetPort: 9345
- name: k8s-api
port: 6443
protocol: TCP
targetPort: 6443
type: LoadBalancer
systemd:
units:
- name: rke2-preinstall.service
enabled: true
contents: |
[Unit]
Description=rke2-preinstall
Wants=network-online.target
Before=rke2-install.service
ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
[Service]
Type=oneshot
User=root
ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
ExecStartPost=/bin/sh -c "umount /mnt"
[Install]
WantedBy=multi-user.target29.4.2 群集实例定义 #
在 ClusterClass 的概念中，群集实例指的是基于定义的 ClusterClass 创建且已实例化的特定运行中群集。它代表一个具体的部署，基于 ClusterClass 中指定的蓝图直接衍生而成，具有独特的配置、资源和运行状态。这包括正在运行的一组特定计算机、网络配置以及相关的 Kubernetes 组件。要管理使用 ClusterClass 框架置备的特定已部署群集的生命周期、执行升级、进行扩容操作和实施监控，理解群集实例至关重要。
要定义一个群集实例，需要定义以下资源：
群集
Metal3MachineTemplate
Metal3DataTemplate
之前在模板（clusterclass 定义文件）中定义的变量将被替换为该群集实例化后的最终值：
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: emea-spa-cluster-3
namespace: emea-spa
spec:
topology:
class: example-clusterclass-type2 # Correct way to reference ClusterClass
version: v1.32.4+rke2r1
controlPlane:
replicas: 1
variables: # Variables to be replaced for this cluster instance
- name: controlPlaneMachineTemplate
value: emea-spa-cluster-3-machinetemplate
- name: controlPlaneEndpointHost
value: 192.168.122.203
- name: tlsSan
value:
- 192.168.122.203
- https://192.168.122.203.sslip.io
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
name: emea-spa-cluster-3-machinetemplate
namespace: emea-spa
spec:
nodeReuse: True
template:
spec:
automatedCleaningMode: metadata
dataTemplate:
name: emea-spa-cluster-3
hostSelector:
matchLabels:
cluster-role: control-plane
deploy-region: emea-spa
cluster-type: type2
image:
checksum: http://fileserver.local:8080/eibimage-downstream-cluster.raw.sha256
checksumType: sha256
format: raw
url: http://fileserver.local:8080/eibimage-downstream-cluster.raw
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
name: emea-spa-cluster-3
namespace: emea-spa
spec:
clusterName: emea-spa-cluster-3
metaData:
objectNames:
- key: name
object: machine
- key: local-hostname
object: machine这种方法使整个过程更加精简，一旦定义了 clusterclass，只需定义 3 种资源即可部署一个群集。