This is unreleased documentation for Cluster API v0.28-dev.

ClusterClass

In this section we cover using ClusterClass with the SUSE® Rancher Prime Cluster API.

Prerequisites

  • Rancher Manager cluster with SUSE® Rancher Prime Cluster API installed.

  • The CAAPF provider, if required by the ClusterClass.

In previous versions of SUSE® Rancher Prime Cluster API, CAAPF was installed by default. From Rancher 2.14.1 onwards, it is no longer part of SUSE® Rancher Prime Cluster API and if required it needs to be installed separately. The majority of the examples shown in this page are relying on CAAPF being installed and enabled. For more information on how to do this, see the CAAPF section.

CAAPF depends on the WatchList Kubernetes feature gate. This feature needs to be explicitly enabled on Kubernetes 1.33 versions. See the Kubernetes upstream documentation for further information.

  • Depending on your engine, either the CAPRKE2 or Kubeadm Bootstrap/Control Plane installation is necessary before further CAPIProvider setup:

    • CAPRKE2

    • Kubeadm

    Bootstrap and Control Plane provider installation:

    Click to Expand 
    apiVersion: v1
kind: Namespace
metadata:
  name: rke2-bootstrap-system
---
apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: rke2-bootstrap
  namespace: rke2-bootstrap-system
spec:
  name: rke2
  type: bootstrap
---
apiVersion: v1
kind: Namespace
metadata:
  name: rke2-control-plane-system
---
apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: rke2-control-plane
  namespace: rke2-control-plane-system
spec:
  name: rke2
  type: controlPlane

    Bootstrap and Control Plane provider installation:

    Click to Expand 
    apiVersion: v1
kind: Namespace
metadata:
  name: capi-kubeadm-bootstrap-system
---
apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: kubeadm-bootstrap
  namespace: capi-kubeadm-bootstrap-system
spec:
  name: kubeadm
  type: bootstrap
---
apiVersion: v1
kind: Namespace
metadata:
  name: capi-kubeadm-control-plane-system
---
apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: kubeadm-control-plane
  namespace: capi-kubeadm-control-plane-system
spec:
  name: kubeadm
  type: controlPlane

Setup

  • Azure

  • AWS

  • GCP

  • Docker

  • vSphere

To prepare the management Cluster, we are going to install the Cluster API Provider Azure, and create a ServicePrincipal identity to provision a new Cluster on Azure.

Before we start, a ServicePrincipal needs to be created, with at least Contributor access to an Azure subscription. Refer to the CAPZ documentation for more details.

  • Provider installation

    apiVersion: v1
kind: Namespace
metadata:
  name: capz-system
---
apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: azure
  namespace: capz-system
spec:
  type: infrastructure
  name: azure

  • Identity Setup

    Authentication for AKS cluster provisioning

    If you are deploying a managed cluster on AKS, you can skip the Identity Setup step and jump to Create a Cluster from a ClusterClass. Instead of using AzureClusterIdentity, you will create a Secret with the Service Principal credentials for the Azure Service Operator to use when applying the cluster manifest.

    In this example we are going to use an AzureClusterIdentity to provision Azure Clusters. A Secret containing the Service Principal credentials needs to be created first, to be referenced by the AzureClusterIdentity resource. Note that the AzureClusterIdentity is a namespaced resource and it needs to be created in the same namespace as the Cluster. For more information on best practices when using Azure identities, please refer to the official documentation.

    Note that some variables are left to the user to substitute.

    apiVersion: v1
kind: Secret
metadata:
  name: <AZURE_CLUSTER_IDENTITY_SECRET_NAME>
  namespace: <AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE>
type: Opaque
stringData:
  clientSecret: <AZURE_CLIENT_SECRET>
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureClusterIdentity
metadata:
  labels:
    clusterctl.cluster.x-k8s.io/move-hierarchy: "true"
  name: cluster-identity
spec:
  allowedNamespaces: {}
  clientID: <AZURE_APP_ID>
  clientSecret:
    name: <AZURE_CLUSTER_IDENTITY_SECRET_NAME>
    namespace: <AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE>
  tenantID: <AZURE_TENANT_ID>
  type: ServicePrincipal

To prepare the management Cluster, we are going to install the Cluster API Provider AWS, and create a secret with the required credentials to provision a new Cluster on AWS. The global credentials are set to blanks, as we are going to use AWSClusterStaticIdentity instead.

  • Provider installation

    apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: aws
  namespace: capa-system
spec:
  type: infrastructure
  variables:
    AWS_B64ENCODED_CREDENTIALS: ""

  • Identity Setup

    In this example we are going to use a AWSClusterStaticIdentity to provision AWS Clusters.
    A Secret containing the credentials needs to be created in the namespace where the AWS provider is installed.
    For more information on how to setup the credentials, refer to the clusterawsadm documentation.
    The AWSClusterStaticIdentity can reference this Secret to allow Cluster provisioning. For this example we are allowing usage of the identity across all namespaces, so that it can be easily reused.
    You can refer to the official documentation to learn more about identity management.

    Note that some variables are left to the user to substitute.

    apiVersion: v1
kind: Secret
metadata:
  name: <AWS_IDENTITY_SECRET_NAME>
  namespace: capa-system
type: Opaque
stringData:
  AccessKeyID: <AWS_ACCESS_KEY_ID>
  SecretAccessKey: <AWS_SECRET_ACCESS_KEY>
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSClusterStaticIdentity
metadata:
  name: cluster-identity
spec:
  secretRef: <AWS_IDENTITY_SECRET_NAME>
  allowedNamespaces: {}

To prepare the management Cluster, we are going to install the Cluster API Provider GCP, and create a secret with the credentials required to provision a new Cluster on GCP. A Service Account is required to create and manage clusters in GCP and this will require Editor permissions. You can follow the offical guide from the CAPG Book. The base64-encoded Service Account key needs to be set in the GCP_B64ENCODED_CREDENTIALS variable of the provider.

  • Provider installation

    apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: gcp
  namespace: capg-system
spec:
  type: infrastructure
  variables:
    GCP_B64ENCODED_CREDENTIALS: xxx

  • Network Setup

    Provisioning a self-managed GCP cluster requires that a GCP network is configured to allow Kubernetes nodes to communicate with the control plane and pull images from the container registry for which machines need to have NAT access or a public IP. The default provider behavior is to create virtual machines with no public IP attached, so a Cloud NAT is required to allow the nodes to establish a connection with the load balancer and the outside world. Please, refer to the official CAPG Book guide on how to prepare your GCP network to provision a self-managed GCP cluster.

    The following steps are required to prepare the GCP network for Cluster provisioning:

    • Create a router.

    • Create a NAT associated with the router.

To prepare the management Cluster, we are going to install the Docker Cluster API Provider.

  • Infrastructure Docker provider installation

    apiVersion: v1
kind: Namespace
metadata:
  name: capd-system
---
apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: docker
  namespace: capd-system
spec:
  type: infrastructure

To prepare the management Cluster, we are going to install the Cluster API Provider vSphere. The global credentials are set to blanks, as we are going to use VSphereClusterIdentity instead.

  • Provider installation

    apiVersion: v1
kind: Namespace
metadata:
  name: capv-system
---
apiVersion: turtles-capi.cattle.io/v1alpha1
kind: CAPIProvider
metadata:
  name: vsphere
  namespace: capv-system
spec:
  type: infrastructure
  variables:
    VSPHERE_USERNAME: ""
    VSPHERE_PASSWORD: ""

  • Identity Setup

    In this example we are going to use a VSphereClusterIdentity to provision vSphere Clusters.
    A Secret containing the credentials needs to be created in the namespace where the vSphere provider is installed. The VSphereClusterIdentity can reference this Secret to allow Cluster provisioning. For this example we are allowing usage of the identity across all namespaces, so that it can be easily reused. You can refer to the official documentation to learn more about identity management.

    apiVersion: v1
kind: Secret
metadata:
  name: cluster-identity
  namespace: capv-system
type: Opaque
stringData:
  username: xxx
  password: xxx
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereClusterIdentity
metadata:
  name: cluster-identity
spec:
  secretName: cluster-identity
  allowedNamespaces:
    selector:
      matchLabels: {}

Create a Cluster from a ClusterClass

  • Azure RKE2

  • Azure AKS

  • Azure Kubeadm

  • AWS EKS

  • AWS Kubeadm

  • AWS RKE2

  • GCP GKE

  • GCP Kubeadm

  • Docker Kubeadm

  • Docker RKE2

  • vSphere Kubeadm

  • vSphere RKE2

  • An Azure ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    An Azure RKE2 ClusterClass and associated applications can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r azure-rke2 | kubectl apply -f -

    • Alternatively, you can apply the Azure RKE2 ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/azure/rke2/clusterclass-rke2-example.yaml

  • Additionally, the Azure Cloud Provider will need to be installed on each downstream Cluster, for the nodes to be initialized correctly.

    We can do this automatically using a ClusterResourceSet to apply the HelmChart downstream.

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-provider-azure
data:
  cloud-provider-azure.yaml: |-
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: azure-ccm
      namespace: kube-system
    spec:
      repo: https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo
      chart: cloud-provider-azure
      bootstrap: true
      version: "1.36.0"
      targetNamespace: kube-system
      valuesContent: |-
        infra:
          clusterName: azure-quickstart
        cloudControllerManager:
          clusterCIDR: 192.168.0.0/16
---
apiVersion: addons.cluster.x-k8s.io/v1beta2
kind: ClusterResourceSet
metadata:
  name: cloud-provider-azure
spec:
  strategy: Reconcile
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: 'azure-quickstart'
  resources:
    - name: cloud-provider-azure
      kind: ConfigMap

  • Create the Azure Cluster from the example ClusterClass

    Note that some variables are left to the user to substitute.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: azure-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: azure-rke2-example
    controlPlane:
      replicas: 1
    variables:
    - name: subscriptionID
      value: <AZURE_SUBSCRIPTION_ID>
    - name: location
      value: <AZURE_LOCATION>
    - name: resourceGroup
      value: <AZURE_RESOURCE_GROUP>
    - name: azureClusterIdentityName
      value: cluster-identity
    version: v1.36.0+rke2r1
    workers:
      machineDeployments:
      - class: rke2-default-worker
        name: md-0
        replicas: 1

  • An Azure AKS ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    An Azure AKS ClusterClass can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r azure-aks | kubectl apply -f -

    • Alternatively, you can apply the Azure AKS ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/azure/aks/clusterclass-aks-example.yaml

  • Create the Azure AKS Cluster from the example ClusterClass.

    The Azure AKS ClusterClass that the Cluster is created from uses the AzureASOManagedCluster, AzureASOManagedControlPlane and AzureASOManagedMachinePool resources that are managed by Azure Service Operator. For this reason you won’t use a AzureClusterIdentity as you would for a self-managed Azure cluster (e.g. Azure with RKE2). Instead a Secret is created with the Service Principal credentials in the same namespace as the Cluster. Note that the Secret definition is embedded in the Cluster manifest for simplicity, but it can be created beforehand as well.

    You can refer to the Azure Service Operator documentation for more information on this method of authentication.

    Note that some variables are left to the user to substitute.

    apiVersion: v1
kind: Secret
metadata:
  name: aks-credentials
stringData:
  AZURE_SUBSCRIPTION_ID: <AZURE_SUBSCRIPTION_ID>
  AZURE_TENANT_ID: <AZURE_TENANT_ID>
  AZURE_CLIENT_ID: <AZURE_CLIENT_ID>
  AZURE_CLIENT_SECRET: <AZURE_CLIENT_SECRET>
type: Opaque
---
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: azure-aks-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    class: azure-aks-example
    variables:
    - name: location
      value: <AZURE_LOCATION>
    - name: resourceGroup
      value: <AZURE_RESOURCE_GROUP>
    - name: asoCredentialSecretName
      value: aks-credentials
    version: v1.36.0
    workers:
      machinePools:
      - class: default-system
        name: system-1
        replicas: 1
      - class: default-worker
        name: worker-1
        replicas: 1

  • An Azure ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    An Azure Kubeadm ClusterClass can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r azure-rke2 | kubectl apply -f -

    • Alternatively, you can apply the Azure RKE2 ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/azure/kubeadm/clusterclass-kubeadm-example.yaml

  • Create the Azure Cluster from the example ClusterClass.

    Note that some variables are left to the user to substitute.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: azure-kubeadm-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: azure-kubeadm-example
    controlPlane:
      replicas: 1
    variables:
    - name: subscriptionID
      value: <AZURE_SUBSCRIPTION_ID>
    - name: location
      value: <AZURE_LOCATION>
    - name: resourceGroup
      value: <AZURE_RESOURCE_GROUP>
    - name: azureClusterIdentityName
      value: cluster-identity
    version: v1.36.1
    workers:
      machineDeployments:
      - class: kubeadm-default-worker
        name: md-0
        replicas: 1

  • Once the Cluster is initialized, you can install a CNI of your choice and the Azure Cloud Provider. Refer to the upstream CAPZ documentation.

  • An AWS EKS ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    An AWS EKS ClusterClass and associated applications can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@main -r aws-eks | kubectl apply -f -

    • Alternatively, you can apply the AWS EKS ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/{turtles_version}/examples/clusterclasses/aws/eks/clusterclass-eks-example.yaml

  • Create the AWS Cluster from the example ClusterClass

    Note that some variables are left to the user to substitute.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: aws-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: aws-eks-example
    variables:
    - name: region
      value: eu-west-2
    - name: instanceType
      value: <AWS_NODE_MACHINE_TYPE>
    - name: awsClusterIdentityName
      value: cluster-identity
    version: v1.35.0
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        replicas: 1

  • An AWS Kubeadm ClusterClass can be found among the Turtles examples.

    This ClusterClass relies on the CAAPF provider being installed and enabled. For information on how to do this, see the CAAPF section.

    Applications like Calico CNI, AWS Cloud Controller Manager, and the AWS EBS CSI Driver will be installed on downstream Clusters. This is done automatically at Cluster creation by targeted Clusters with specific labels, such as cni: calico, cloud-provider: aws, and csi: aws-ebs-csi-driver.

    • CLI

    • kubectl

    An AWS Kubeadm ClusterClass and associated applications can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r aws-kubeadm | kubectl apply -f -

    • Alternatively, you can apply the AWS Kubeadm ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/aws/kubeadm/clusterclass-kubeadm-example.yaml

    • For this example we are also going to install Calico as the default CNI.

    • The AWS Cloud Controller Manager will need to be installed on each downstream Cluster for the nodes to be functional.

    • Additionally, we will also enable AWS EBS CSI Driver.

      We can do this automatically at Cluster creation using the Cluster API Add-on Provider Fleet. The HelmOps need to be created first, to be applied on the new Cluster via label selectors. This will take care of deploying Calico, the EBS CSI Driver, and the AWS Cloud Controller Manager in the workload cluster.

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/applications/csi/aws/helm-chart.yaml
kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/applications/cni/aws/calico/helm-chart.yaml
kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/applications/ccm/aws/helm-chart.yaml

  • Create the AWS Cluster from the example ClusterClass.

    Note that some variables are left to the user to substitute.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cluster-api.cattle.io/rancher-auto-import: "true"
    cni: calico
    cloud-provider: aws
    csi: aws-ebs-csi-driver
  name: aws-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: aws-kubeadm-example
    controlPlane:
      replicas: 3
    variables:
    - name: region
      value: eu-west-2
    - name: sshKeyName
      value: <AWS_SSH_KEY_NAME>
    - name: controlPlaneMachineType
      value: <AWS_CONTROL_PLANE_MACHINE_TYPE>
    - name: workerMachineType
      value: <AWS_NODE_MACHINE_TYPE>
    - name: awsClusterIdentityName
      value: cluster-identity
    version: v1.36.1
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        replicas: 1

Before creating an AWS+RKE2 workload cluster, it is required to either build an AMI for the RKE2 version that is going to be installed on the cluster or find one that will work for non-airgapped installations. You can follow the steps in the RKE2 image-builder README to build the AMI.

  • An AWS RKE2 ClusterClass can be found among the Turtles examples.

    This ClusterClass relies on the CAAPF provider being installed and enabled. For information on how to do this, see the CAAPF section.

    Applications like Calico CNI, AWS Cloud Controller Manager, and the AWS EBS CSI Driver will be installed on downstream Clusters. This is done automatically at Cluster creation by targeted Clusters with specific labels, such as cni: calico, cloud-provider: aws, and csi: aws-ebs-csi-driver.

    • CLI

    • kubectl

    An AWS RKE2 ClusterClass and associated applications can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r aws-rke2 | kubectl apply -f -

    • Alternatively, you can apply the AWS RKE2 ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/aws/rke2/clusterclass-ec2-rke2-example.yaml

  • For this example we are also going to install Calico as the default CNI.

  • The AWS Cloud Controller Manager will need to be installed on each downstream Cluster for the nodes to be functional.

  • Additionally, we will also enable AWS EBS CSI Driver.

    We can do this automatically at Cluster creation using the Cluster API Add-on Provider Fleet. The HelmOps need to be created first, to be applied on the new Cluster via label selectors. This will take care of deploying Calico, the EBS CSI Driver, and the AWS Cloud Controller Manager in the workload cluster.

    kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/applications/csi/aws/helm-chart.yaml
kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/applications/cni/aws/calico/helm-chart.yaml
kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/applications/ccm/aws/helm-chart.yaml

  • Create the AWS Cluster from the example ClusterClass

    Note that some variables are left to the user to substitute.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cloud-provider: aws
    cni: calico
    csi: aws-ebs-csi-driver
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: aws-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: aws-rke2-example
    controlPlane:
      replicas: 3
    variables:
    - name: cni
      value: none
    - name: region
      value: <AWS_REGION>
    - name: sshKeyName
      value: <AWS_SSH_KEY_NAME>
    - name: controlPlaneMachineType
      value: <AWS_RKE2_CONTROL_PLANE_MACHINE_TYPE>
    - name: workerMachineType
      value: <AWS_RKE2_NODE_MACHINE_TYPE>
    - name: amiID
      value: <AWS_AMI_ID>
    - name: awsClusterIdentityName
      value: cluster-identity
    version: v1.36.0+rke2r1
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        replicas: 1

  • A GCP GKE ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    A GCP GKE ClusterClass and associated applications can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@main -r gcp-gke | kubectl apply -f -

    • Alternatively, you can apply the GCP GKE ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/{turtles_version}/examples/clusterclasses/gcp/gke/clusterclass-gke-example.yaml

  • Create the GCP Cluster from the example ClusterClass

    Note that some variables are left to the user to substitute.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: gcp-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: gcp-gke-example
    variables:
      - name: project
        value: <GCP_PROJECT>
      - name: region
        value: <GCP_REGION>
      - name: networkName
        value: <GCP_NETWORK_NAME>
    version: v1.35.3
    workers:
      machinePools:
        - class: default-system
          name: system-pool
          replicas: 1
        - class: default-worker
          name: worker-pool
          replicas: 1

Before creating a GCP+Kubeadm workload cluster, it is required to either build an Image for the Kubernetes version that is going to be installed on the cluster or find one that will work for your use case. You can follow the steps in the Kubernetes GCP Image Builder book.

  • A GCP Kubeadm ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    A GCP Kubeadm ClusterClass can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r gcp-kubeadm | kubectl apply -f -

    • Alternatively, you can apply the GCP Kubeadm ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/gcp/kubeadm/clusterclass-kubeadm-example.yaml

  • Create the GCP Cluster from the example ClusterClass

    Note that some variables are left to the user to substitute. The default configuration of GCP Cloud Controller Manager is configured to use a single zone cluster, so the clusterFailureDomains variable is set to a single zone. If you need to provision a multi-zone cluster, we recommend you inspect the parameters provided by GCP Cloud Controller Manager and how CAPG leverages these variables to create cluster-specific configurations.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: gcp-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: gcp-kubeadm-example
    controlPlane:
      replicas: 1
    workers:
      machineDeployments:
        - class: "default-worker"
          name: "md-0"
          replicas: 1
    variables:
      - name: gcpProject
        value: <GCP_PROJECT>
      - name: region
        value: <GCP_REGION>
      - name: gcpNetworkName
        value: <GCP_NETWORK_NAME>
      - name: clusterFailureDomains
        value:
          - "<GCP_REGION>-a"
      - name: imageId
        value: <GCP_IMAGE_ID>
      - name: machineType
        value: <GCP_MACHINE_TYPE>
    version: v1.36.1

  • Once the Cluster is initialized, you can install a CNI of your choice and the GCP Cloud Provider. Refer to the upstream CAPG documentation.

  • A Docker Kubeadm ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    A Docker Kubeadm ClusterClass can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r docker-kubeadm | kubectl apply -f -

    • Alternatively, you can apply the Docker Kubeadm ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/docker/kubeadm/clusterclass-docker-kubeadm.yaml

  • Create the Docker Kubeadm Cluster from the example ClusterClass.

    Note that some variables are left to the user to substitute.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: docker-kubeadm-quickstart
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 192.168.0.0/16
    serviceDomain: cluster.local
    services:
      cidrBlocks:
        - 10.96.0.0/24
  topology:
    classRef:
      name: docker-kubeadm-example
    controlPlane:
      replicas: 1
    version: v1.36.1
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 1

  • Once the Cluster is initialized, you can install a CNI of your choice. Refer to the upstream CAPI documentation.

  • A Docker RKE2 ClusterClass can be found among the Turtles examples.

    In this example, no CNI is being defined so RKE2 installs Canal by default. This ClusterClass does not rely on CAAPF being installed and enabled.

    • CLI

    • kubectl

    A Docker RKE2 ClusterClass and associated applications can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r docker-rke2 | kubectl apply -f -

    • Alternatively, you can apply the Docker RKE2 ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/docker/rke2/clusterclass-docker-rke2.yaml

    • Create the LoadBalancer ConfigMap for Docker RKEv2 Cluster.

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/applications/lb/docker/configmap.yaml

  • Create the Docker RKE2 Cluster from the example ClusterClass.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: docker-rke2-example
  labels:
    cni: calico
  annotations:
    cluster-api.cattle.io/upstream-system-agent: "true"
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/24
    serviceDomain: cluster.local
  topology:
    classRef:
      name: docker-rke2-example
    controlPlane:
      replicas: 1
    variables:
    - name: rke2CNI
      value: none
    - name: dockerImage
      value: kindest/node:v1.35.0
    version: v1.36.0+rke2r1
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        replicas: 1

  • A vSphere ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    A vSphere Kubeadm ClusterClass and associated applications can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r vsphere-kubeadm | kubectl apply -f -

    • Alternatively, you can apply the vSphere Kubeadm ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/vsphere/kubeadm/clusterclass-kubeadm-example.yaml

  • Create the vSphere Cluster from the example ClusterClass

    Note that for this example we are using kube-vip as a Control Plane load balancer. The KUBE_VIP_INTERFACE will be used to bind the CONTROL_PLANE_IP in ARP mode. Depending on your operating system and network device configuration, you need to configure this value accordingly - for example, to eth0. The kube-vip static manifest is embedded in the ClusterClass definition. For more information on how to generate a static kube-vip manifest for your own ClusterClasses, please consult the official documentation.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cni: calico
    cloud-provider: vsphere
    csi: vsphere
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: 'vsphere-kubeadm-quickstart'
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: vsphere-kubeadm-example
    version: v1.36.1
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
      - class: vsphere-kubeadm-example-worker
        name: md-0
        replicas: 1
    variables:
    - name: vSphereClusterIdentityName
      value: cluster-identity
    - name: vSphereTLSThumbprint
      value: <VSPHERE_THUMBPRINT>
    - name: vSphereDataCenter
      value: <VSPHERE_DATACENTER>
    - name: vSphereDataStore
      value: <VSPHERE_DATASTORE>
    - name: vSphereFolder
      value: <VSPHERE_FOLDER>
    - name: vSphereNetwork
      value: <VSPHERE_NETWORK>
    - name: vSphereResourcePool
      value: <VSPHERE_RESOURCE_POOL>
    - name: vSphereServer
      value: <VSPHERE_SERVER>
    - name: vSphereTemplate
      value: <VSPHERE_TEMPLATE>
    - name: controlPlaneIpAddr
      value: <CONTROL_PLANE_IP>
    - name: controlPlanePort
      value: 6443
    - name: sshKey
      value: <SSH_KEY>
    - name: kubeVIPInterface
      value: <KUBE_VIP_INTERFACE>

  • Once the Cluster is initialized, you can install a CNI of your choice, the vSphere Cloud Provider, and the Container Storage Interface (CSI) driver for vSphere.

  • A vSphere ClusterClass can be found among the Turtles examples.

    • CLI

    • kubectl

    A vSphere RKE2 ClusterClass can be applied using the examples tool:

    go run github.com/rancher/turtles/examples@v0.28.0 -r vsphere-rke2 | kubectl apply -f -

    • Alternatively, you can apply the vSphere RKE2 ClusterClass directly using kubectl:

      kubectl apply -f https://raw.githubusercontent.com/rancher/turtles/refs/tags/v0.28.0/examples/clusterclasses/vsphere/rke2/clusterclass-rke2-example.yaml

  • Cluster configuration

    The vSphere Cloud Provider and the vSphere CSI controller need additional configuration to be applied on the downstream Cluster. We can do this automatically using a ClusterResourceSet. The CPI and CSI charts can also be installed using the same workflow by applying the HelmChart downstream.

    • Configure the vSphere Cloud Provider controller:

      apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-provider-vsphere
data:
  cloud-provider-vsphere.yaml: |-
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: vsphere-ccm
      namespace: kube-system
    spec:
      repo: https://rke2-charts.rancher.io
      chart: rancher-vsphere-cpi
      bootstrap: true
      targetNamespace: kube-system
      version: "1.14.000"
      valuesContent: |-
        vCenter:
          host: "<VSPHERE_SERVER>"
          datacenters: "<VSPHERE_DATACENTER>"
          credentialsSecret:
            generate: false
            name: cloud-provider-vsphere-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: cloud-provider-vsphere-credentials
type: addons.cluster.x-k8s.io/resource-set
stringData:
  cloud-provider-vsphere-credentials.yaml: |-
    apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: cloud-provider-vsphere-credentials
      namespace: kube-system
    stringData:
      <VSPHERE_SERVER>.username: "<VSPHERE_USERNAME>"
      <VSPHERE_SERVER>.password: "<VSPHERE_PASSWORD>"
---
apiVersion: addons.cluster.x-k8s.io/v1beta2
kind: ClusterResourceSet
metadata:
  name: cloud-provider-vsphere
spec:
  strategy: Reconcile
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: 'vsphere-rke2-quickstart'
  resources:
    - name: cloud-provider-vsphere
      kind: ConfigMap
    - name: cloud-provider-vsphere-credentials
      kind: Secret

    • Configure the vSphere CSI controller:

      apiVersion: v1
kind: Secret
metadata:
  name: csi-vsphere-config
type: addons.cluster.x-k8s.io/resource-set
stringData:
  csi-vsphere-config.yaml: |-
    apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: vsphere-config-secret
      namespace: vmware-system-csi
    stringData:
      csi-vsphere.conf: |+
        [Global]
        thumbprint = "<VSPHERE_TLS_THUMBPRINT>"

        [VirtualCenter "<VSPHERE_SERVER>"]
        user = "<VSPHERE_USERNAME>"
        password = "<VSPHERE_PASSWORD>"
        datacenters = "<VSPHERE_DATACENTER>"

        [Network]
        public-network = "<VSPHERE_NETWORK>"

        [Labels]
        region = ""
        zone = ""
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: csi-vsphere
data:
  csi-vsphere.yaml: |-
    apiVersion: helm.cattle.io/v1
    kind: HelmChart
    metadata:
      name: vsphere-csi
      namespace: kube-system
    spec:
      repo: https://rke2-charts.rancher.io
      chart: rancher-vsphere-csi
      bootstrap: true
      targetNamespace: vmware-system-csi
      createNamespace: true
      version: "3.7.0-rancher100"
      valuesContent: |-
        vCenter:
          configSecret:
            generate: false
            name: vsphere-config-secret
---
apiVersion: addons.cluster.x-k8s.io/v1beta2
kind: ClusterResourceSet
metadata:
  name: csi-vsphere
spec:
  strategy: Reconcile
  clusterSelector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: 'vsphere-rke2-quickstart'
  resources:
    - name: csi-vsphere
      kind: ConfigMap
    - name: csi-vsphere-config
      kind: Secret

  • Create the vSphere Cluster from the example ClusterClass

    Note that for this example we are using kube-vip as a Control Plane load balancer. The KUBE_VIP_INTERFACE will be used to bind the CONTROL_PLANE_IP in ARP mode. Depending on your operating system and network device configuration, you need to configure this value accordingly - for example, to eth0. The kube-vip static manifest is embedded in the ClusterClass definition. For more information on how to generate a static kube-vip manifest for your own ClusterClasses, please consult the official documentation. In case you are using a VM template based on SUSE Linux Micro, you may optionally provide a productKey variable to enable automatic SL Micro registration against SUSE Customer Center.

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  labels:
    cluster-api.cattle.io/rancher-auto-import: "true"
  name: 'vsphere-rke2-quickstart'
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  topology:
    classRef:
      name: vsphere-rke2-example
    version: v1.36.0+rke2r1
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
      - class: vsphere-rke2-example-worker
        name: md-0
        replicas: 1
    variables:
    - name: vSphereClusterIdentityName
      value: cluster-identity
    - name: vSphereTLSThumbprint
      value: <VSPHERE_THUMBPRINT>
    - name: vSphereDataCenter
      value: <VSPHERE_DATACENTER>
    - name: vSphereDataStore
      value: <VSPHERE_DATASTORE>
    - name: vSphereFolder
      value: <VSPHERE_FOLDER>
    - name: vSphereNetwork
      value: <VSPHERE_NETWORK>
    - name: vSphereResourcePool
      value: <VSPHERE_RESOURCE_POOL>
    - name: vSphereServer
      value: <VSPHERE_SERVER>
    - name: vSphereTemplate
      value: <VSPHERE_TEMPLATE>
    - name: controlPlaneIpAddr
      value: <CONTROL_PLANE_IP>
    - name: controlPlanePort
      value: 6443
    - name: sshKey
      value: <SSH_KEY>
    - name: kubeVIPInterface
      value: <KUBE_VIP_INTERFACE>
    - name: productKey
      value: <SL_MICRO_PRODUCT_KEY>