Aplica-se a SUSE Telco Cloud 3.6

56 Provisionamento de cluster downstream em cenários air-gapped #

O fluxo de trabalho de provisionamento de rede direcionado permite automatizar o provisionamento de clusters downstream em cenários air-gapped.

56.1 Requisitos para cenários air-gapped #

A imagem raw gerada usando EIB deve incluir as imagens de contêiner específicas (OCI de helm-chart e imagens de contêiner) necessárias para executar o cluster downstream em um cenário air-gapped. Para obter mais informações, consulte esta seção (Capítulo 50, Preparar imagem de cluster downstream para cenários de air gap).
No caso de usar SR-IOV ou qualquer outra carga de trabalho personalizada, as imagens necessárias para executar as cargas de trabalho devem ser pré-carregadas em seu registro privado seguindo a seção de pré-carregamento de registro privado (Seção 50.2.7, “Preparando os artefatos de air-gap”).

56.2 Registrar os hosts bare metal em cenários air-gapped #

O processo para registrar os hosts bare metal no cluster de gerenciamento é o mesmo descrito na seção anterior (Capítulo 51, Provisionamento de cluster downstream com provisionamento de rede direcionado (nó único)).

56.3 Provisionar o cluster downstream em cenários air-gapped #

Existem algumas alterações importantes necessárias para provisionar o cluster downstream em cenários air-gapped:

O bloco RKE2ControlPlane no arquivo capi-provisioning-example.yaml deve incluir a diretiva spec.agentConfig.airGapped: true.
A configuração do registro privado deve ser incluída no bloco RKE2ControlPlane no arquivo capi-provisioning-airgap-example.yaml seguindo a seção de registro privado (Capítulo 55, Registro privado).
Se você estiver usando SR-IOV ou qualquer outra configuração AdditionalUserData (script de combustão) que exija a instalação do helm-chart, você deve modificar o conteúdo para referenciar o registro privado em vez de usar o registro público.

O exemplo a seguir mostra a configuração de SR-IOV no bloco AdditionalUserData no arquivo capi-provisioning-airgap-example.yaml com as modificações necessárias para referenciar o registro privado

Referências de segredos do Registro Privado
Definição de Helm-Chart usando o registro privado em vez das imagens OCI públicas.

# secret to include the private registry certificates
apiVersion: v1
kind: Secret
metadata:
  name: private-registry-cert
  namespace: default
data:
  tls.crt: ${TLS_BASE64_CERT}
  tls.key: ${TLS_BASE64_KEY}
  ca.crt: ${CA_BASE64_CERT}
type: kubernetes.io/tls
---
# secret to include the private registry auth credentials
apiVersion: v1
kind: Secret
metadata:
  name: private-registry-auth
  namespace: default
data:
  username: ${REGISTRY_USERNAME}
  password: ${REGISTRY_PASSWORD}
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: RKE2ControlPlane
metadata:
  name: single-node-cluster
  namespace: default
spec:
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: Metal3MachineTemplate
    name: single-node-cluster-controlplane
  replicas: 1
  version: ${RKE2_VERSION}
  rolloutStrategy:
    type: "RollingUpdate"
    rollingUpdate:
      maxSurge: 0
  privateRegistriesConfig:       # Private registry configuration to add your own mirror and credentials
    mirrors:
      docker.io:
        endpoint:
          - "${PRIVATE_REGISTRY_URL}"
        rewrite:
          "^(.*)$": "mirror/$1"
      registry.suse.com:
        endpoint:
          - "${PRIVATE_REGISTRY_URL}"
        rewrite:
          "^(.*)$": "mirror/$1"
      registry.suse.de:
        endpoint:
          - "${PRIVATE_REGISTRY_URL}"
        rewrite:
          "^(.*)$": "mirror/$1"
      registry.opensuse.org:
        endpoint:
          - "${PRIVATE_REGISTRY_URL}"
        rewrite:
          "^(.*)$": "mirror/$1"
      registry.rancher.com:
        endpoint:
          - "${PRIVATE_REGISTRY_URL}"
        rewrite:
          "^(.*)$": "mirror/$1"
    configs:
      "192.168.100.22:5000":
        authSecret:
          apiVersion: v1
          kind: Secret
          namespace: default
          name: private-registry-auth
        tls:
          tlsConfigSecret:
            apiVersion: v1
            kind: Secret
            namespace: default
            name: private-registry-cert
          insecureSkipVerify: false
  serverConfig:
    cni: calico
    cniMultusEnable: true
  agentConfig:
    airGapped: true       # Airgap true to enable airgap mode
    format: ignition
    additionalUserData:
      config: |
        variant: fcos
        version: 1.4.0
        storage:
          files:
          - path: /var/lib/rancher/rke2/server/manifests/configmap-sriov-custom-auto.yaml
            overwrite: true
            contents:
              inline: |
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  name: sriov-custom-auto-config
                  namespace: sriov-network-operator
                data:
                  config.json: |
                    [
                        {
                          "resourceName": "${RESOURCE_NAME1}",
                          "interface": "${SRIOV-NIC-NAME1}",
                          "pfname": "${PF_NAME1}",
                          "driver": "${DRIVER_NAME1}",
                          "numVFsToCreate": ${NUM_VFS1}
                        },
                        {
                          "resourceName": "${RESOURCE_NAME2}",
                          "interface": "${SRIOV-NIC-NAME2}",
                          "pfname": "${PF_NAME2}",
                          "driver": "${DRIVER_NAME2}",
                          "numVFsToCreate": ${NUM_VFS2}
                        }
                    ]
            mode: 0644
            user:
              name: root
            group:
              name: root
          - path: /var/lib/rancher/rke2/server/manifests/sriov.yaml
            overwrite: true
            contents:
              inline: |
                apiVersion: v1
                data:
                  .dockerconfigjson: ${REGISTRY_AUTH_DOCKERCONFIGJSON}
                kind: Secret
                metadata:
                  name: privregauth
                  namespace: kube-system
                type: kubernetes.io/dockerconfigjson
                ---
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  namespace: kube-system
                  name: example-repo-ca
                data:
                  ca.crt: |-
                    -----BEGIN CERTIFICATE-----
                    ${CA_BASE64_CERT}
                    -----END CERTIFICATE-----
                ---
                apiVersion: helm.cattle.io/v1
                kind: HelmChart
                metadata:
                  name: sriov-crd
                  namespace: kube-system
                spec:
                  chart: oci://${PRIVATE_REGISTRY_URL}/mirror/sriov-crd
                  dockerRegistrySecret:
                    name: privregauth
                  repoCAConfigMap:
                    name: example-repo-ca
                  createNamespace: true
                  set:
                    global.clusterCIDR: 192.168.0.0/18
                    global.clusterCIDRv4: 192.168.0.0/18
                    global.clusterDNS: 10.96.0.10
                    global.clusterDomain: cluster.local
                    global.rke2DataDir: /var/lib/rancher/rke2
                    global.serviceCIDR: 10.96.0.0/12
                  targetNamespace: sriov-network-operator
                  version: 306.0.4+up1.6.0
                ---
                apiVersion: helm.cattle.io/v1
                kind: HelmChart
                metadata:
                  name: sriov-network-operator
                  namespace: kube-system
                spec:
                  chart: oci://${PRIVATE_REGISTRY_URL}/mirror/sriov-network-operator
                  dockerRegistrySecret:
                    name: privregauth
                  repoCAConfigMap:
                    name: example-repo-ca
                  createNamespace: true
                  set:
                    global.clusterCIDR: 192.168.0.0/18
                    global.clusterCIDRv4: 192.168.0.0/18
                    global.clusterDNS: 10.96.0.10
                    global.clusterDomain: cluster.local
                    global.rke2DataDir: /var/lib/rancher/rke2
                    global.serviceCIDR: 10.96.0.0/12
                  targetNamespace: sriov-network-operator
                  version: 306.0.4+up1.6.0
            mode: 0644
            user:
              name: root
            group:
              name: root
        kernel_arguments:
          should_exist:
            - intel_iommu=on
            - iommu=pt
            - idle=poll
            - mce=off
            - hugepagesz=1G hugepages=40
            - hugepagesz=2M hugepages=0
            - default_hugepagesz=1G
            - irqaffinity=${NON-ISOLATED_CPU_CORES}
            - isolcpus=domain,nohz,managed_irq,${ISOLATED_CPU_CORES}
            - nohz_full=${ISOLATED_CPU_CORES}
            - rcu_nocbs=${ISOLATED_CPU_CORES}
            - rcu_nocb_poll
            - nosoftlockup
            - nowatchdog
            - nohz=on
            - nmi_watchdog=0
            - skew_tick=1
            - quiet
        systemd:
          units:
          - name: rke2-preinstall.service
            enabled: true
            contents: |
              [Unit]
              Description=rke2-preinstall
              Wants=network-online.target
              Before=rke2-install.service
              ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
              [Service]
              Type=oneshot
              User=root
              ExecStartPre=/bin/sh -c "mount -L config-2 /mnt"
              ExecStart=/bin/sh -c "sed -i \"s/BAREMETALHOST_UUID/$(jq -r .uuid /mnt/openstack/latest/meta_data.json)/\" /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"node-name: $(jq -r .name /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"node-label:\" >> /etc/rancher/rke2/config.yaml"
              ExecStart=/bin/sh -c "echo \"  - metal3.io/uuid=$(jq -r .uuid /mnt/openstack/latest/meta_data.json)\" >> /etc/rancher/rke2/config.yaml"
              ExecStartPost=/bin/sh -c "umount /mnt"
              [Install]
              WantedBy=multi-user.target
          # rke2-traefik-deployment.service unit to be removed once "traefik" being the default ingress controller (starting with RKE2 v1.36)
          - name: rke2-traefik-deployment.service
            enabled: true
            contents: |
              [Unit]
              Description=rke2-traefik-deployment
              Wants=rke2-preinstall.service
              Before=rke2-install.service
              ConditionPathExists=!/run/cluster-api/bootstrap-success.complete
              [Service]
              Type=oneshot
              User=root
              ExecStart=/bin/sh -c "echo \"ingress-controller: traefik\" >> /etc/rancher/rke2/config.yaml"
              [Install]
              WantedBy=multi-user.target
          - name: cpu-partitioning.service
            enabled: true
            contents: |
              [Unit]
              Description=cpu-partitioning
              Wants=network-online.target
              After=network.target network-online.target
              [Service]
              Type=oneshot
              User=root
              ExecStart=/bin/sh -c "echo isolated_cores=${ISOLATED_CPU_CORES} > /etc/tuned/cpu-partitioning-variables.conf"
              ExecStartPost=/bin/sh -c "tuned-adm profile cpu-partitioning"
              ExecStartPost=/bin/sh -c "systemctl enable tuned.service"
              [Install]
              WantedBy=multi-user.target
          - name: performance-settings.service
            enabled: true
            contents: |
              [Unit]
              Description=performance-settings
              Wants=network-online.target
              After=network.target network-online.target cpu-partitioning.service
              [Service]
              Type=oneshot
              User=root
              ExecStart=/bin/sh -c "/opt/performance-settings/performance-settings.sh"
              [Install]
              WantedBy=multi-user.target
          - name: sriov-custom-auto-vfs.service
            enabled: true
            contents: |
              [Unit]
              Description=SRIOV Custom Auto VF Creation
              Wants=network-online.target  rke2-server.target
              After=network.target network-online.target rke2-server.target
              [Service]
              User=root
              Type=forking
              TimeoutStartSec=1800
              ExecStart=/bin/sh -c "while ! /var/lib/rancher/rke2/bin/kubectl --kubeconfig=/etc/rancher/rke2/rke2.yaml wait --for condition=ready nodes --timeout=30m --all ; do sleep 10 ; done"
              ExecStartPost=/bin/sh -c "/opt/sriov/sriov-auto-filler.sh"
              RemainAfterExit=yes
              KillMode=process
              [Install]
              WantedBy=multi-user.target
    kubelet:
      extraArgs:
      - provider-id=metal3://BAREMETALHOST_UUID
    nodeName: "localhost.localdomain"