Register Downstream Clusters
Overview
There are two specific styles to registering clusters. These styles will be referred to as agent-initiated and manager-initiated registration. Typically one would go with the agent-initiated registration but there are specific use cases in which manager-initiated is a better workflow.
Agent-Initiated Registration
Agent-initiated refers to a pattern in which the downstream cluster installs an agent with a cluster registration token and optionally a client ID. The cluster agent will then make a API request to the SUSE® Rancher Prime Continuous Delivery manager and initiate the registration process. Using this process the Manager will never make an outbound API request to the downstream clusters and will thus never need to have direct network access. The downstream cluster only needs to make outbound HTTPS calls to the manager.
Manager-Initiated Registration
Manager-initiated registration is a process in which you register an existing Kubernetes cluster with the SUSE® Rancher Prime Continuous Delivery manager and the SUSE® Rancher Prime Continuous Delivery manager will make an API call to the downstream cluster to deploy the agent. This style can place additional network access requirements because the Fleet manager must be able to communicate with the downstream cluster API server for the registration process. After the cluster is registered there is no further need for the manager to contact the downstream cluster API. This style is more compatible if you wish to manage the creation of all your Kubernetes clusters through GitOps using something like cluster-api or Rancher.
Agent Initiated
A downstream cluster is registered by installing an agent via helm and using the cluster registration token and optionally a client ID or cluster labels.
It’s not necessary to configure the fleet manager for multi cluster, as the downstream agent we install via Helm will connect to the Kubernetes API of the upstream cluster directly. Agent-initiated registration is normally not used with Rancher. |
Cluster Registration Token and Client ID
The cluster registration token is a credential that will authorize the downstream cluster agent to be
able to initiate the registration process. This is required.
The cluster registration token is manifested as a values.yaml
file that will be passed to the helm install
process.
Alternatively one can pass the token directly to the helm install command via --set token="$token"
.
There are two styles of registering an agent. You can have the cluster for this agent dynamically created, in which case you will probably want to specify cluster labels upon registration. Or you can have the agent register to a predefined cluster in the SUSE® Rancher Prime Continuous Delivery manager, in which case you will need a client ID. The former approach is typically the easiest.
Install Agent For a New Cluster
The SUSE® Rancher Prime Continuous Delivery agent is installed as a Helm chart. Following are explanations how to determine and set its parameters.
First, follow the cluster registration token instructions to obtain the values.yaml
which contains
the registration token to authenticate against the SUSE® Rancher Prime Continuous Delivery cluster.
Second, optionally you can define labels that will assigned to the newly created cluster upon registration. After
registration is completed an agent cannot change the labels of the cluster. To add cluster labels add
--set-string labels.KEY=VALUE
to the below Helm command. To add the labels foo=bar
and bar=baz
then you would
add --set-string labels.foo=bar --set-string labels.bar=baz
to the command line.
# Leave blank if you do not want any labels
CLUSTER_LABELS="--set-string labels.example=true --set-string labels.env=dev"
Third, set variables with the SUSE® Rancher Prime Continuous Delivery cluster’s API Server URL and CA, for the downstream cluster to use for connecting.
API_SERVER_URL=https://...
API_SERVER_CA_DATA=...
Value in API_SERVER_CA_DATA
can be obtained from a .kube/config
file with valid data to connect to the upstream cluster
(under the certificate-authority-data
key). Alternatively it can be obtained from within the upstream cluster itself,
by looking up the default ServiceAccount secret name (typically prefixed with default-token-
, in the default namespace),
under the ca.crt
key.
Use proper namespace and release name:
For the agent chart the namespace must be |
Kubectl Context
Ensure you are installing to the right cluster:
Helm will use the default context in |
SUSE® Rancher Prime Continuous Delivery in Rancher
Rancher has separate helm charts for SUSE® Rancher Prime Continuous Delivery and uses a different repository. |
Add Fleet’s Helm repo.
helm repo add fleet https://rancher.github.io/fleet-helm-charts/
Finally, install the agent using Helm.
-
Install
-
Validate
helm -n cattle-fleet-system install --create-namespace --wait \
--set clientID="$CLUSTER_CLIENT_ID" \
--values values.yaml \
fleet-agent fleet/fleet-agent
You can check that status of the fleet pods by running the below commands:
# Ensure kubectl is pointing to the right cluster kubectl -n cattle-fleet-system logs -l app=fleet-agent kubectl -n cattle-fleet-system get pods -l app=fleet-agent
The agent should now be deployed.
Additionally you should see a new cluster registered in the SUSE® Rancher Prime Continuous Delivery manager. Below is an example of checking that a new cluster
was registered in the clusters
namespace. Please ensure your ${HOME}/.kube/config
is pointed to the Fleet
manager to run this command.
kubectl -n clusters get clusters.fleet.cattle.io
NAME BUNDLES-READY NODES-READY SAMPLE-NODE LAST-SEEN STATUS cluster-ab13e54400f1 1/1 1/1 k3d-cluster2-server-0 2020-08-31T19:23:10Z
Install Agent For a Predefined Cluster
Client IDs are for the purpose of predefining clusters in the SUSE® Rancher Prime Continuous Delivery manager with existing labels and repos targeted to them.
A client ID is not required and is just one approach to managing clusters.
The client ID is a unique string that will identify the cluster.
This string is user generated and opaque to the SUSE® Rancher Prime Continuous Delivery manager and agent. It is assumed to be sufficiently unique. For security reasons one should not be able to easily guess this value
as then one cluster could impersonate another. The client ID is optional and if not specified the UID field of the kube-system
namespace
resource will be used as the client ID. Upon registration if the client ID is found on a Cluster
resource in the SUSE® Rancher Prime Continuous Delivery manager it will associate
the agent with that Cluster
. If no Cluster
resource is found with that client ID a new Cluster
resource will be created with the specific
client ID.
The SUSE® Rancher Prime Continuous Delivery agent is installed as a Helm chart. The only parameters to the helm chart installation should be the cluster registration token, which
is represented by the values.yaml
file and the client ID. The client ID is optional.
First, create a Cluster
in the SUSE® Rancher Prime Continuous Delivery Manager with the random client ID you have chosen.
kind: Cluster
apiVersion: fleet.cattle.io/v1alpha1
metadata:
name: my-cluster
namespace: clusters
spec:
clientID: "really-random"
Second, follow the [cluster registration token instructions]((#create-cluster-registration-tokens) to obtain the values.yaml
file to be used.
Third, setup your environment to use the client ID.
CLUSTER_CLIENT_ID="really-random"
Use proper namespace and release name:
For the agent chart the namespace must be |
Ensure you are installing to the right cluster:
Helm will use the default context in |
Add Fleet’s Helm repo.
helm repo add fleet https://rancher.github.io/fleet-helm-charts/
Finally, install the agent using Helm.
-
Install
-
Validate
helm -n cattle-fleet-system install --create-namespace --wait \
--set clientID="$CLUSTER_CLIENT_ID" \
--values values.yaml \
fleet-agent fleet/fleet-agent
You can check that status of the fleet pods by running the below commands:
# Ensure kubectl is pointing to the right cluster kubectl -n cattle-fleet-system logs -l app=fleet-agent kubectl -n cattle-fleet-system get pods -l app=fleet-agent
The agent should now be deployed.
Additionally you should see a new cluster registered in the SUSE® Rancher Prime Continuous Delivery manager. Below is an example of checking that a new cluster
was registered in the clusters
namespace. Please ensure your ${HOME}/.kube/config
is pointed to the Fleet
manager to run this command.
kubectl -n clusters get clusters.fleet.cattle.io
NAME BUNDLES-READY NODES-READY SAMPLE-NODE LAST-SEEN STATUS my-cluster 1/1 1/1 k3d-cluster2-server-0 2020-08-31T19:23:10Z
Create Cluster Registration Tokens
Not needed for Manager-initiated registration: For manager-initiated registrations the token is managed by the SUSE® Rancher Prime Continuous Delivery manager and does not need to be manually created and obtained. |
For an agent-initiated registration the downstream cluster must have a cluster registration token.
Cluster registration tokens are used to establish a new identity for a cluster. Internally
cluster registration tokens are managed by creating Kubernetes service accounts that have the
permissions to create ClusterRegistrationRequests
within a specific namespace. Once the
cluster is registered a new ServiceAccount
is created for that cluster that is used as
the unique identity of the cluster. The agent is designed to forget the cluster registration
token after registration. While the agent will not maintain a reference to the cluster registration
token after a successful registration please note that usually other system bootstrap scripts do.
Since the cluster registration token is forgotten, if you need to re-register a cluster you must give the cluster a new registration token.
Token TTL
Cluster registration tokens can be reused by any cluster in a namespace. The tokens can be given a TTL such that it will expire after a specific time.
Create a new Token
The ClusterRegistationToken
is a namespaced type and should be created in the same namespace
in which you will create GitRepo
and ClusterGroup
resources. For in depth details on how namespaces
are used in SUSE® Rancher Prime Continuous Delivery refer to the documentation on namespaces. Create a new
token with the below YAML.
kind: ClusterRegistrationToken
apiVersion: "fleet.cattle.io/v1alpha1"
metadata:
name: new-token
namespace: clusters
spec:
# A duration string for how long this token is valid for. A value <= 0 or null means infinite time.
ttl: 240h
After the ClusterRegistrationToken
is created, SUSE® Rancher Prime Continuous Delivery will create a corresponding Secret
with the same name.
As the Secret
creation is performed asynchronously, you will need to wait until it’s available before using it.
One way to do so is via the following one-liner:
while ! kubectl --namespace=clusters get secret new-token; do sleep 5; done
Obtaining Token Value (Agent values.yaml)
The token value contains YAML content for a values.yaml
file that is expected to be passed to helm install
to install the SUSE® Rancher Prime Continuous Delivery agent on a downstream cluster.
Such value is contained in the values
field of the Secret
mentioned above. To obtain the YAML content for the
above example one can run the following one-liner:
kubectl --namespace clusters get secret new-token -o 'jsonpath={.data.values}' | base64 --decode > values.yaml
Once the values.yaml
is ready it can be used repeatedly by clusters to register until the TTL expires.
Manager Initiated
The manager-initiated registration flow is accomplished by creating a
Cluster
resource in the SUSE® Rancher Prime Continuous Delivery Manager that refers to a Kubernetes
Secret
containing a valid kubeconfig file in the data field called value
.
If you are using SUSE® Rancher Prime Continuous Delivery standalone without Rancher, it must be installed as described in installation details. The manager-initiated registration is used when you add a cluster from the Rancher dashboard. |
Create Kubeconfig Secret
The format of this secret is intended to match the format of the kubeconfig
secret used in cluster-api.
This means you can use cluster-api
to create a cluster that is dynamically registered withSUSE® Rancher Prime Continuous Delivery.
title="Kubeconfig Secret Example"
kind: Secret
apiVersion: v1
metadata:
name: my-cluster-kubeconfig
namespace: clusters
data:
value: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIHNlcnZlcjogaHR0cHM6Ly9leGFtcGxlLmNvbTo2NDQzCiAgbmFtZTogY2x1c3Rlcgpjb250ZXh0czoKLSBjb250ZXh0OgogICAgY2x1c3RlcjogY2x1c3RlcgogICAgdXNlcjogdXNlcgogIG5hbWU6IGRlZmF1bHQKY3VycmVudC1jb250ZXh0OiBkZWZhdWx0CmtpbmQ6IENvbmZpZwpwcmVmZXJlbmNlczoge30KdXNlcnM6Ci0gbmFtZTogdXNlcgogIHVzZXI6CiAgICB0b2tlbjogc29tZXRoaW5nCg==