If you wish to remove a node temporarily, the recommended approach is to first drain the node.

When you want to bring the node back, you only have to uncordon it.

Tip

For instructions on how to perform these operations refer to Section 2.6, “Node Operations”.

Important

Nodes removed with this method cannot be added back to the cluster or any other skuba-initiated cluster. You must reinstall the entire node and then join it again to the cluster.

The skuba node remove command serves to permanently remove nodes. Running this command will work even if the target virtual machine is down, so it is the safest way to remove the node.

skuba node remove <NODE_NAME> [flags]

Note

Per default, node removal has an unlimited timeout on waiting for the node to drain. If the node is unreachable it can not be drained and thus the removal will fail or get stuck indefinitely. You can specify a time after which removal will be performed without waiting for the node to drain with the flag --drain-timeout <DURATION>.

For example, waiting for the node to drain for 1 minute and 5 seconds:

skuba node remove caasp-worker1 --drain-timeout 1m5s

For a list of supported time formats run skuba node remove -h.

Important

After the removal of a master node, you have to manually delete its entries from your load balancer’s configuration.

These to commands respectively define if a node is marked as schedulable or unschedulable. This means that a node is allowed to or not allowed to receive any new workloads. This can be useful when troubleshooting a node.

To mark a node as unschedulable run:

kubectl cordon <NODE_NAME>

To mark a node as schedulable run:

kubectl uncordon <NODE_NAME>

Draining a node consists of evicting all the running pods from the current node in order to perform maintenance. This is a mandatory step in order to ensure a proper functioning of the workloads. This is achieved using kubectl.

To drain a node run:

kubectl drain <NODE_NAME>

This action will also implicitly cordon the node. Therefore once the maintenance is done, uncordon the node to set it back to schedulable.

Refer to the official Kubernetes documentation for more information: https://v1-18.docs.kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/#use-kubectl-drain-to-remove-a-node-from-service

Applications that will be deployed to Kubernetes will typically contain all the required software to be executed. In some cases, especially when it comes to the hardware layer abstraction (storage backends, GPU), additional packages must be installed on the underlying operating system outside of Kubernetes.

Note

The following examples show installation of required packages for Ceph, please adjust the list of packages and repositories to whichever software you need to install.

While you can install any software package from the SLES ecosystem this falls outside of the support scope for SUSE CaaS Platform.

sudo zypper in ceph-common xfsprogs

Run sudo zypper up on your management workstation to get the latest version of skuba and its dependencies. Reboot the machine to make sure that all system changes are correctly applied.

In order to get an overview of the updates available, you can run:

skuba cluster upgrade plan

This will show you a list of updates (if available) for different components installed on the cluster. If the cluster is already running the latest available versions, the output should look like this:

Current Kubernetes cluster version: 1.16.2
Latest Kubernetes version: 1.16.2

Congratulations! You are already at the latest version available

If the cluster has a new patch-level or minor Kubernetes version available, the output should look like this:

Current Kubernetes cluster version: 1.15.2
Latest Kubernetes version: 1.16.2

Upgrade path to update from 1.15.2 to 1.16.2:
 - 1.15.2 -> 1.16.2

Similarly, you can also fetch this information on a per-node basis with the following command:

skuba node upgrade plan <NODE>

For example, if the cluster has a node named worker0 which is running the latest available versions, the output should look like this:

Current Kubernetes cluster version: 1.16.2
Latest Kubernetes version: 1.16.2

Node worker0 is up to date

On the other hand, if this same node has a new patch-level or minor Kubernetes version available, the output should look like this:

Current Kubernetes cluster version: 1.15.2
Latest Kubernetes version: 1.16.2

Current Node version: 1.15.2

Component versions in worker0
  - kubelet: 1.15.2 -> 1.16.2
  - cri-o: 1.15.0 -> 1.16.0

You will get a similar output if there is a version available on a master node (named master0 in this example):

Current Kubernetes cluster version: 1.15.2
Latest Kubernetes version: 1.16.2

Current Node version: 1.15.2

Component versions in master0
  - apiserver: 1.15.2 -> 1.16.2
  - controller-manager: 1.15.2 -> 1.16.2
  - scheduler: 1.15.2 -> 1.16.2
  - etcd: 3.3.11 -> 3.3.15
  - kubelet: 1.15.2 -> 1.16.2
  - cri-o: 1.15.0 -> 1.16.0

It may happen that the Kubernetes version on the control plane is too outdated for the update to progress. In this case, you would get output similar to the following:

Current Kubernetes cluster version: 1.15.0
Latest Kubernetes version: 1.15.0

Unable to plan node upgrade: at least one control plane does not tolerate the current cluster version

Tip

The control plane consists of these components:

• apiserver

• controller-manager

• scheduler

• etcd

• kubelet

• cri-o

Note

Due to changes to the way skuba handles addons some existing components might be shown as new addon in the status output. This is expected and no cause for concern. For any upgrade afterwards the addon will be considered known and only show available upgrades.

Important

SUSE CaaS Platform 4.2.1 provides the update of Cilium from 1.5.3 to 1.6.6. The important change in Cilium 1.6 is usage of Kubernetes CRDs instead of etcd. skuba performs and automated migration of data from etcd to CRDs. If that migration is not successful, skuba shows the following warning:

"Could not migrate data from etcd to CRD. Addons upgrade will be continued without it, which will result in temporary connection loss for currently existing pods and services."

That warning means that Cilium is going to regenerate all internal data on the first run after upgrade. It can result in temporary connection loss for pods and services which might take few minutes.

In order to get an overview of the addon updates available, you can run:

skuba addon upgrade plan

This will show you a list of updates (if available) for different addons installed on the cluster:

Current Kubernetes cluster version: 1.17.4
Latest Kubernetes version: 1.17.4

Addon upgrades for 1.17.4:
  - cilium: 1.5.3 -> 1.6.6
  - dex: 2.16.0 (manifest version from 5 to 6)
  - gangway: 3.1.0-rev4 (manifest version from 4 to 5)
  - metrics-server: 0.3.6 (new addon)

If the cluster is already running the latest available versions, the output should look like this:

Current Kubernetes cluster version: 1.17.4
Latest Kubernetes version: 1.17.4

Congratulations! Addons are already at the latest version available

Before updating the nodes you must apply the addon upgrades to your management workstation. Please run:

skuba addon upgrade apply

Once you have upgraded all nodes, please run skuba cluster upgrade plan again. This will show if any upgrades are available that required the versions you just installed. If there are upgrades available please repeat the procedure until no more new upgrades are shown.

Nodes added to a cluster have the service skuba-update.timer, which is responsible for running automatic updates, activated by default.

This service calls the skuba-update utility and it can be configured with the /etc/sysconfig/skuba-update file.

Note: How skuba-update non-interactive mode works

skuba-update uses the flags --non-interactive and --non-interactive-include-reboot-patches. The --non-interactive flag causes zypper to use default answers to questions rather than prompting a user for answers. In non-interactive mode, the --non-interactive-include-reboot-patches flag causes patches with the rebootSuggested-flag to not be skipped. Zypper does not perform the reboot directly. Instead, kured will be used to safely schedule reboots as needed.

To disable the automatic updates on a node, simply ssh to it and then configure the skuba-update service by editing the /etc/sysconfig/skuba-update file with the following runtime options:

## Path           : System/Management
## Description    : Extra switches for skuba-update
## Type           : string
## Default        : ""
## ServiceRestart : skuba-update
#
SKUBA_UPDATE_OPTIONS="--annotate-only"

Tip

It is not required to reload or restart skuba-update.timer.

The --annotate-only flag makes the skuba-update utility only check if updates are available and annotate the node accordingly. When this flag is activated no updates are installed at all.

When OS updates are disabled, then you will have to manage OS updates manually. In order to do so, you will have to call skuba-update manually on each node.

Warning

Do not use zypper up/zypper patch commands as these do not manage the Kubernetes annotations used by kured. If you perform a manual update using these commands you might render your cluster unusable.

After that, rebooting the node will depend on whether you have also disabled reboots or not. If you have disabled reboots for this node, then you will have to follow the instructions as given in Section 4.4.2, “Completely Disabling Reboots”. Otherwise, you will have to wait until kured performs the reboot of the node

If you would like to take care of reboots manually, either as a temporary measure or permanently, you can disable them by creating a lock:

kubectl -n kube-system annotate ds kured weave.works/kured-node-lock='{"nodeID":"manual"}'

This command modifies an annotation (annotate) on the daemonset (ds) named kured.

When automatic reboots are disabled, you will have to manage reboots yourself. In order to do this, you will have to follow some steps whenever you want to issue a reboot marker for a node. First of all, you will have to cordon and drain the node:

kubectl cordon <NODE_ID>
kubectl drain --force=true \
  --ignore-daemonsets=true \ 1
  --delete-local-data=false \ 2
  --grace-period 600 \ 3
  --timeout=900s \ 4
  <NODE_ID>

Important

Depending on your deployed applications, you must adjust the values for --grace-period and --timeout to grant the applications enough time to safely shut down without losing data. The values here are meant to represent a conservative default for an application like SUSE Cloud Application Platform.

If you do not set these values, applications might never finish and draining of the pod will hang indefinitely.

Only then you will be able to manually reboot the node safely.

Once the node is back, remember to uncordon it so it is scheduleable again:

kubectl uncordon <NODE_ID>

Perform the above steps first on control plane nodes, and afterwards on worker nodes.

Tip

If the node that should be rebooted does not contain any workload you can skip the above steps and simply reboot the node.

In exceptional circumstances, such as a node experiencing a permanent failure whilst rebooting, manual intervention may be required to remove the cluster lock:

kubectl -n kube-system annotate ds kured weave.works/kured-node-lock-

This command modifies an annotation (annotate) on the daemonset (ds) named kured. It explicitly performs an "unset" (-) for the value for the annotation named weave.works/kured-node-lock.

After following all these instructions you should be running SUSE CaaS Platform 4.5. Refer to the release notes for further information on the new features that this release brings. Enjoy!

Authentication is composed of:

For authorization (Role-Based Access Control, RBAC), administrators can use kubectl to create corresponding RoleBinding or ClusterRoleBinding for a user or group to limit resource access.

6.2.1.1 Web Flow #

1. The user requests access through Gangway.

2. Gangway redirects to Dex.

3. Dex redirects to a connected identity provider (connector). User login and a request to approve access are generated.

4. Dex continues with OIDC authentication flow on behalf of the user and creates/updates data to Kubernetes CRDs.

5. Dex redirects the user to Gangway. This redirect includes (ID/refresh) tokens.

6. Gangway returns a link to download kubeconfig or self-configures kubectl instructions to the user.

   

7. User downloads kubeconf or self-configures kubectl.

8. User uses kubectl to connect to the Kubernetes API server.

9. Kubernetes CRDs validate the Kubernetes API server request and return a response.

10. The kubectl connects to the authorized Kubernetes resources through the Kubernetes API server.

6.2.1.2 CLI Flow #

1. User requests access through skuba auth login with the Dex server URL, username and password.

2. Dex uses received username and password to log in and approve the access request to the connected identity providers (connectors).

3. Dex continues with the OIDC authentication flow on behalf of the user and creates/updates data to the Kubernetes CRDs.

4. Dex returns the ID token and refreshes token to skuba auth login.

5. skuba auth login generates the kubeconfig file kubeconf.txt.

6. User uses kubectl to connect the Kubernetes API server.

7. Kubernetes CRDs validate the Kubernetes API server request and return a response.

8. The kubectl connects to the authorized Kubernetes resources through Kubernetes API server.

If you already have an existed LDAP server, you could skip this part.

Note: Known Issues

The error message is a warning for 389-ds version 1.4.3 when replacing external certificates.

ERR - attrcrypt_cipher_init - No symmetric key found for cipher AES in backend exampleDB, attempting to create one...
INFO - attrcrypt_cipher_init - Key for cipher AES successfully generated and stored
ERR - attrcrypt_cipher_init - No symmetric key found for cipher 3DES in backend exampleDB, attempting to create one...
INFO - attrcrypt_cipher_init - Key for cipher 3DES successfully generated and stored

It is due to the encrypted key being stored in the dse.ldif. When replacing the key and certificate in /data/config, 389ds searches dse.ldif for symmetric key and create one if it does not exist. 389-ds developers are planning a fix that switches 389-ds to use the nssdb exclusively.

In both directories, user-regular1 and user-regular2 are members of the k8s-users group, and user-admin is a member of the k8s-admins group.

In Active Directory, user-bind is a simple user that is a member of the default Domain Users group. Hence, we can use it to authenticate, because it has read-only access to Active Directory. The mail attribute is used to create the RBAC rules.

Administrators can update the authentication connector settings before or after SUSE CaaS Platform deployment as follows:

Roles define, which subjects (users or groups) can use which verbs (operations) on which resources. The following sections provide an overview of the verbs, resources, and how to create roles. Roles can then be assigned to users and groups.

6.5.1.4 Create Role Bindings #

To bind a group or user to a role or cluster role, create a YAML file that contains the role or cluster role binding description. Then apply the binding with kubectl apply -f YAML_FILE. The following examples provide an overview of different use cases of role bindings.

• Binding a User to a Role:

  This example shows how to bind a user to a defined role.

  kind: RoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: <ROLE_BINDING_NAME> 1
    namespace: <NAMESPACE> 2
  subjects:
  - kind: User
    name: <LDAP_USER_NAME> 3
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: Role
    name: <ROLE_NAME> 4
    apiGroup: rbac.authorization.k8s.io

  1	Defines a name for this new role binding.
  2	Name of the namespace to which the binding applies.
  3	Name of the LDAP user to which this binding applies.
  4	Name of the role used. For defining rules, refer to Section 6.5.1.3, “Creating Roles”.

• Binding a User to a Cluster Role:

  This example shows how to bind a user to a defined cluster role.

  kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: <CLUSTER_ROLE_BINDING_NAME> 1
  subjects:
  - kind: User
    name: <LDAP_USER_NAME> 2
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: ClusterRole
    name: <CLUSER_ROLE_NAME> 3
    apiGroup: rbac.authorization.k8s.io

  1	Defines a name for this new cluster role binding.
  2	Name of the LDAP user to which this binding applies.
  3	Name of the cluster role used. For defining rules, refer to Section 6.5.1.3, “Creating Roles”.

• Binding a Group to a Role:

  This example shows how to bind a group to a defined role.

  kind: RoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: <ROLE_BINDING_NAME> 1
    namespace: <NAMESPACE> 2
  subjects:
  - kind: Group
    name: <LDAP_GROUP_NAME> 3
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: Role
    name: <ROLE_NAME> 4
    apiGroup: rbac.authorization.k8s.io

  1	Defines a name for this new role binding.
  2	Name of the namespace to which the binding applies.
  3	Name of the LDAP group to which this binding applies.
  4	Name of the role used. For defining rules, refer to Section 6.5.1.3, “Creating Roles”.

• Binding a Group to a Cluster Role:

  This example shows how to bind a group to a defined cluster role.

  kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: <CLUSTER_ROLE_BINDING_NAME> 1
  subjects:
  - kind: Group
    name: <LDAP_GROUP_NAME> 2
    apiGroup: rbac.authorization.k8s.io
  roleRef:
    kind: ClusterRole
    name: <CLUSER_ROLE_NAME> 3
    apiGroup: rbac.authorization.k8s.io

  1	Defines a name for this new cluster role binding.
  2	Name of the LDAP group to which this binding applies.
  3	Name of the cluster role used. For defining rules, refer to Section 6.5.1.3, “Creating Roles”.

Important

When creating new Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings, it is important to keep in mind the Principle of Least Privilege:

"define rules such that the account bound to the Role or ClusterRole has the minimum amount of permissions needed to fulfill its purpose and no more."

For instance, granting the admin ClusterRole to most accounts is most likely unnecessary, when a reduced-scope role would be enough to fulfill the account’s purpose. This helps reduce the attack surface if an account is compromised.

It is also recommended to periodically review your Roles and ClusterRoles to ensure they are still required and are not overly-permissive.

The user can now access resources in the authorized <NAMESPACE>.

If the user has the proper permissions to access the resources, the output should look like this:

# kubectl -n <NAMESPACE> get pod

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   dex-844dc9b8bb-w2zkm                 1/1     Running   0          19d
kube-system   gangway-944dc9b8cb-w2zkm             1/1     Running   0          19d
kube-system   cilium-76glw                         1/1     Running   0          27d
kube-system   cilium-fvgcv                         1/1     Running   0          27d
kube-system   cilium-j5lpx                         1/1     Running   0          27d
kube-system   cilium-operator-5d9cc4fbb7-g5plc     1/1     Running   0          34d
kube-system   cilium-vjf6p                         1/1     Running   8          27d
kube-system   coredns-559fbd6bb4-2r982             1/1     Running   9          46d
kube-system   coredns-559fbd6bb4-89k2j             1/1     Running   9          46d
kube-system   etcd-my-master                       1/1     Running   5          46d
kube-system   kube-apiserver-<CLUSTER_NAME>            1/1     Running   0          19d
kube-system   kube-controller-manager-my-master    1/1     Running   14         46d
kube-system   kube-proxy-62hls                     1/1     Running   4          46d
kube-system   kube-proxy-fhswj                     1/1     Running   0          46d
kube-system   kube-proxy-r4h42                     1/1     Running   1          39d
kube-system   kube-proxy-xsdf4                     1/1     Running   0          39d
kube-system   kube-scheduler-my-master             1/1     Running   13         46d

If the user does not have the right permissions to access a resource, they will receive a Forbidden message.

Error from server (Forbidden): pods is forbidden

The kubeconfig file (kubeconf.txt) contains the OIDC tokens necessary to perform authentication and authorization in the cluster. OIDC tokens have an expiration date which means that they need to be refreshed after some time.

Important

If you use the same user in multiple kubeconfig files distributed among multiple machines, this can lead to issues. Due to the nature of access and refresh tokens (https://tools.ietf.org/html/rfc6749#page-10) only one of the machines will be fully able to refresh the token set at any given time.

The user will be able to download multiple 'kubeconfig' files. However, the file with the same user is likely to be valid only for single access until expiration.

Dex regards one session per user. The id-token and refresh-token are refreshed together. If a second user is trying to login to get a new id-token, Dex will invalidate the previous id-token and refresh-token for the first user. The first user is still able to continue using the old id-token until expiration. After expiration, the first user is not allowed to refresh the id-token due to the invalid refresh-token. Only the second user will have a valid refresh-token now. The first user will encounter an error like: "msg="failed to rotate keys: keys already rotated by another server instance".

If sharing the same id-token in many places, all of them can be used until expiration. The first user to refresh the id-token and refresh token will be able to continue accessing the cluster. All other users will encounter an error Refresh token is invalid or has already been claimed by another client because the refresh-token got updated by the first user.

Please use separate users for each kubeconfig file to avoid this situation. Find out how to add more users in Section 6.4.4.1, “Adding a New User”. You can also check information about the user and the respective OIDC tokens in the kubeconfig file under the users section:

users:
- name: myuser
  user:
    auth-provider:
      config:
        client-id: oidc
        client-secret: <SECRET>
        id-token:  <ID_TOKEN>
        idp-issuer-url: https://<IP>:<PORT>
        refresh-token: <REFRESH_TOKEN>
      name: oidc

After user authentication and authorization, admission takes place to complete the access control for the Kubernetes API. As the final step in the access control process, admission enhances the security layer by mandating a reasonable security baseline across a specific namespace or the entire cluster. The built-in PodSecurityPolicy admission controller is perhaps the most prominent example of it.

Apart from the security aspect, admission controllers can enforce custom policies to adhere to certain best-practices such as having good labels, annotation, resource limits, or other settings. It is worth noting that instead of only validating the request, admission controllers are also capable of "fixing" a request by mutating it, such as automatically adding resource limits if the user forgets to.

The admission is controlled by admission controllers which may only be configured by the cluster administrator. The admission control process happens in two phases:

Important

If any of the controllers in either phase reject the request, the entire request is rejected immediately and an error is returned to the end-user.

Important

Any modification of this list prior to the creation of the cluster will be overwritten by these default settings.

The ability to add or remove individual admission controllers will be provided with one of the upcoming releases of SUSE CaaS Platform.

The complete list of admission controllers can be found at https://v1-18.docs.kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#what-does-each-admission-controller-do

The default admission controllers enabled in SUSE CaaS Platform are:

SUSE CaaS Platform 4 currently ships with two default policies:

All pods running the containers for the basic SUSE CaaS Platform software are deployed into the kube-system namespace and run with the "privileged" policy.

All authenticated system users (group system:authenticated) and service accounts in kube-system (system:serviceaccounts:kube-system) have a RoleBinding (suse:caasp:psp:privileged) to run pods using the privileged policy in the kube-system namespace.

Any other pods launched in any other namespace are, by default, deployed in unprivileged mode.

Important

You must configure RBAC rules and PodSecurityPolicy to provide proper functionality and security.

The policy definitions are embedded in the cluster bootstrap manifest (GitHub).

During the bootstrap with skuba, the policy files will be stored on your workstation in the cluster definition folder under addons/psp/base. These policy files will be installed automatically for all cluster nodes.

The file names of the files created are:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: suse.caasp.psp.unprivileged
  annotations:
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
spec:
  # Privileged
  privileged: false
  # Volumes and File Systems
  volumes:
    # Kubernetes Pseudo Volume Types
    - configMap
    - secret
    - emptyDir
    - downwardAPI
    - projected
    - persistentVolumeClaim
    # Networked Storage
    - nfs
    - rbd
    - cephFS
    - glusterfs
    - fc
    - iscsi
    # Cloud Volumes
    - cinder
    - gcePersistentDisk
    - awsElasticBlockStore
    - azureDisk
    - azureFile
    - vsphereVolume
  allowedHostPaths:
    # Note: We don't allow hostPath volumes above, but set this to a path we
    # control anyway as a belt+braces protection. /dev/null may be a better
    # option, but the implications of pointing this towards a device are
    # unclear.
    - pathPrefix: /opt/kubernetes-hostpath-volumes
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: []
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: false
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: suse:caasp:psp:unprivileged
rules:
  - apiGroups: ['extensions']
    resources: ['podsecuritypolicies']
    verbs: ['use']
    resourceNames: ['suse.caasp.psp.unprivileged']
---
# Allow all users and serviceaccounts to use the unprivileged
# PodSecurityPolicy
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: suse:caasp:psp:default
roleRef:
  kind: ClusterRole
  name: suse:caasp:psp:unprivileged
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:serviceaccounts
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:authenticated

In order to properly secure and run your Kubernetes workloads you must configure RBAC rules for your desired users create a PodSecurityPolicy adequate for your respective workloads and then link the user accounts to the PodSecurityPolicy using (Cluster)RoleBinding.

https://v1-18.docs.kubernetes.io/docs/concepts/policy/pod-security-policy/

Communication is secured with TLS v1.2 using the AES 128 CBC cipher. All certificates are 2048 bit RSA encrypted.

The CA certificate is valid for 3650 days (10 years) by default. Client and server certificates are valid for 365 days (1 year) by default.

Required CAs for SUSE CaaS Platform are stored on all control plane nodes:

The control plane certificates stored in the Kubernetes cluster on control plane nodes:

The addon certificates stored in the Kubernetes cluster Secret resource:

We use cert-exporter to monitor nodes' on-host certificates and addons' secret certificates. The cert-exporter collects the metrics of certificates expiration periodically (1 hour by default) and exposes them through the /metrics endpoint. Then, the Prometheus server can scrape these metrics from the endpoint periodically.

helm repo add suse https://kubernetes-charts.suse.com
helm install suse/cert-exporter --name ${RELEASE_NAME}

Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

helm repo add suse https://kubernetes-charts.suse.com
helm install ${RELEASE_NAME} suse/cert-exporter

SUSE CaaS Platform uses oidc-dex and oidc-gangway servers to do authentication and authorization. Administrators might choose to replace these server’s certificates by issuing a trusted CA certificate after cluster deployment. This way, the user does not have to add specific certificates to their trusted keychain.

SUSE CaaS Platform renews the control plane certificates and kubeconfigs automatically in two ways:

Note: kubelet client certificate

During the kubelet client certificate signing flow, kubelet sends the kubelet client CSR into the Kubernetes cluster. The kube-controller-manager signs the kubelet client CSR with the Kubernetes CA cert/key pair. The kucero signs the kubelet server CSR with the kubelet CA cert/key pair.

The kubelet client certificate renews automatically at approximately 70%-90% of the total lifetime of the certificate, the kubelet would use new client certificates without downtime.

The kubelet client certificate uses the /var/lib/kubelet/pki/kubelet-client-current.pem, this file is a symlink to the latest signed client certificate.

6.9.7.3 Addon Certificate Rotation #

The addon certificates can be automatically rotated by leveraging the functions of the open-source solutions cert-manager and reloader. cert-manager is for automatically rotating certificates stored in Secrets, and reloader is for watching and reconciling the updated Secrets to execute a rolling upgrade of the affected Deployments or DaemonSet.

• Prerequisites

  1. To let reloader do an automatic rolling upgrade of the addon Deployments or DaemonSet, we need to label the addons:

     kubectl annotate --overwrite deployment/oidc-dex -n kube-system secret.reloader.stakater.com/reload=oidc-dex-cert
     
     kubectl annotate --overwrite deployment/oidc-gangway -n kube-system secret.reloader.stakater.com/reload=oidc-gangway-cert
     
     kubectl annotate --overwrite deployment/metrics-server -n kube-system secret.reloader.stakater.com/reload=metrics-server-cert

  2. Upload the Kubernetes CA cert/key pair to Secret in the kube-system namespace:

     kubectl create secret tls kubernetes-ca --cert=pki/ca.crt --key=pki/ca.key -n kube-system

     

     Note

     If you want to use a custom trusted CA certificate/key to sign the certificate, upload to the secret resource.

     kubectl create secret tls custom-trusted-ca --cert=<CUSTOM_TRUSTED_CA_CERT> --key=<CUSTOM_TRUSTED_CA_KEY> -n kube-system

  3. Install reloader via helm chart:

     helm install suse/reloader \
         --name ${RELEASE_NAME} \
         --namespace cert-manager

  4. Install cert-manager via helm chart:

     helm install suse/cert-manager \
         --name ${RELEASE_NAME} \
         --namespace cert-manager \
         --set global.leaderElection.namespace=cert-manager \
         --set installCRDs=true

• Create a Cert-Manager CA Issuer Resource

  The cert-manager CA issuer is a Kubernetes resource that represents a certificate authority (CA), which is able to generate signed certificates by honoring certificate signing requests (CSR). Each cert-manager certificate resource requires one referenced issuer in the ready state to be able to honor CSR requests.

  

  Note

  An Issuer is a namespaced resource, and it can not issue certificates to the certificate resources in other namespaces.

  If you want to create a single Issuer that can be consumed in multiple namespaces, you should consider creating a ClusterIssuer resource. This is almost identical to the Issuer resource, however, it is cluster-wide so it can be used to issue certificates in all namespaces.

  Create a CA issuer called kubernetes-ca that will sign incoming certificate requests based on the CA certificate and private key stored in the secret kubernetes-ca used to trust newly signed certificates.

  cat << EOF > issuer.yaml
  apiVersion: cert-manager.io/v1alpha3
  kind: Issuer
  metadata:
    name: kubernetes-ca 1
    namespace: kube-system
  spec:
    ca:
      secretName: kubernetes-ca 2
  EOF
  
  kubectl apply -f issuer.yaml

  1	The issuer name.
  2	The secret reference name.

  

  Note

  If you want to use custom trusted CA certificate/key to sign the certificate, create a custom trusted CA issuer.

  cat << EOF > custom-trusted-ca-issuer.yaml
  apiVersion: cert-manager.io/v1alpha3
  kind: Issuer 1
  metadata:
    name: custom-trusted-ca
    namespace: kube-system
  spec:
    ca:
      secretName: custom-trusted-ca
  EOF
  
  kubectl apply -f custom-trusted-ca-issuer.yaml

  1	Issuer or ClusterIssuer.

• Create a Cert-Manager Certificate Resource

  The cert-manager has a custom resource, Certificate, which can be used to define a requested x509 certificate which will be renewed and kept up to date by an Issuer or ClusterIssuer resource.

  • oidc-dex certificate

    cat << EOF > oidc-dex-certificate.yaml
    apiVersion: cert-manager.io/v1alpha3
    kind: Certificate
    metadata:
      name: oidc-dex-cert
      namespace: kube-system
    spec:
      subject:
        organizations:
        - system:masters
      commonName: oidc-dex
      duration: 8760h # 1 year 1
      renewBefore: 720h # 1 month 2
      # At least one of a DNS Name or IP address is required.
      dnsNames:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 3
      ipAddresses:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 4
      secretName: oidc-dex-cert
      issuerRef:
        name: kubernetes-ca 5
        kind: Issuer 6
        group: cert-manager.io
      isCA: false
      usages:
        - digital signature
        - key encipherment
        - server auth
      keySize: 2048
      keyAlgorithm: rsa
      keyEncoding: pkcs1
    EOF
    
    kubectl apply -f oidc-dex-certificate.yaml

    1	Default length of certificate validity, in the format (XhYmZs).
    2	Certificate renewal time before validity expires, in the format (XhYmZs).
    3	DNSNames is a list of subject alt names to be used on the Certificate.
    4	IPAddresses is a list of IP addresses to be used on the Certificate.
    5	The cert-manager issuer name.
    6	Issuer or ClusterIssuer.

    This certificate will tell cert-manager to attempt to use the Issuer named kubernetes-ca to obtain a certificate key pair for the domain list in dnsNames and ipAddresses. If successful, the resulting key and certificate will be stored in a secret named oidc-dex-cert with keys of tls.key and tls.crt respectively.

    The dnsNames and ipAddresses fields specify a list of Subject Alternative Names to be associated with the certificate.

    The referenced Issuer must exist in the same namespace as the Certificate. A Certificate can alternatively reference a ClusterIssuer which is cluster-wide so it can be referenced from any namespace.

    

    Note

    If you want to use a custom trusted CA Issuer/ClusterIssuer, change the value of name under issuerRef to custom-trusted-ca and the value of kind under issuerRef to Issuer/ClusterIssuer.

  • oidc-gangway certificate

    cat << EOF > oidc-gangway-certificate.yaml
    apiVersion: cert-manager.io/v1alpha3
    kind: Certificate
    metadata:
      name: oidc-gangway-cert
      namespace: kube-system
    spec:
      subject:
        organizations:
        - system:masters
      commonName: oidc-gangway
      duration: 8760h # 1 year 1
      renewBefore: 720h # 1 month 2
      # At least one of a DNS Name or IP address is required.
      dnsNames:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 3
      ipAddresses:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 4
      secretName: oidc-gangway-cert
      issuerRef:
        name: kubernetes-ca 5
        kind: Issuer 6
        group: cert-manager.io
      isCA: false
      usages:
        - digital signature
        - key encipherment
        - server auth
      keySize: 2048
      keyAlgorithm: rsa
      keyEncoding: pkcs1
    EOF
    
    kubectl apply -f oidc-gangway-certificate.yaml

    1	Default length of certificate validity, in the format (XhYmZs).
    2	Certificate renewal time before validity expires, in the format (XhYmZs).
    3	DNSNames is a list of subject alt names to be used on the Certificate.
    4	IPAddresses is a list of IP addresses to be used on the Certificate.
    5	The cert-manager issuer name.
    6	Issuer or ClusterIssuer.

    

    Note

    If you want to use a custom trusted CA Issuer/ClusterIssuer, change the value of name under issuerRef to custom-trusted-ca and the value of kind under issuerRef to Issuer/ClusterIssuer.

  • metrics-server certificate

    cat << EOF > metrics-server-certificate.yaml
    apiVersion: cert-manager.io/v1alpha3
    kind: Certificate
    metadata:
      name: metrics-server-cert
      namespace: kube-system
    spec:
      subject:
        organizations:
        - system:masters
      commonName: metrics-server.kube-system.svc
      duration: 8760h # 1 year 1
      renewBefore: 720h # 1 month 2
      # At least one of a DNS Name or IP address is required.
      dnsNames:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 3
      ipAddresses:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 4
      secretName: metrics-server-cert
      issuerRef:
        name: kubernetes-ca 5
        kind: Issuer 6
        group: cert-manager.io
      isCA: false
      usages:
        - digital signature
        - key encipherment
        - server auth
      keySize: 2048
      keyAlgorithm: rsa
      keyEncoding: pkcs1
    EOF
    
    kubectl apply -f metrics-server-certificate.yaml

    1	Default length of certificate validity, in the format (XhYmZs).
    2	Certificate renewal time before validity expires, in the format (XhYmZs).
    3	DNSNames is a list of subject alt names to be used on the Certificate.
    4	IPAddresses is a list of IP addresses to be used on the Certificate.
    5	The cert-manager issuer name.
    6	Issuer or ClusterIssuer.

Warning: Cert-Manager Known Issue

Once the cert-manager has issued a certificate to the secret, if you change the certificate inside the secret manually, or you manually change the current certificate duration to a value lower than the value renewBefore, the certificate won’t be renewed immediately but will be scheduled to renew near the certificate expiry date.

This is because the cert-manager is not designed to pick up changes you make to the certificate in the secret.

The Kubernetes audit log only collects and stores actions performed on the level of the cluster. This does not include any resulting actions of application services.

In order to successfully use Centralized Logging, you first need to install Helm, and Tiller if using Helm 2. Helm is used to install the log agents and provide custom logging settings.

Refer to Section 3.1.2.1, “Installing Helm”.

You can log the following groups of services. See Section 7.4.4, “Deployment” for more information on how to select and customize the logs.

Centralized Logging is also restricted to the following protocols: UDP, TCP, TCP + TLS, TCP + mTLS.

The two supported syslog message formats are RFC 5424 and RFC 3164.

Note

The Kubernetes cluster metadata is included in the RFC 5424 message.

Example RFC 3164

2019-05-30T09:11:21.968458+00:00 worker1 k8s.system/crio[12080] level=debug msg="Endpoint successfully created" containerID=caa46f14a68e766b877af01442e58731845bb45d8ce1f856553440a02c958b2f eventUUID=e2405f2a-82ba-11e9-9a06-fa163eebdfd6 subsys=cilium-cni

Example RFC 5424

<133>1 2019-05-30T08:28:38.784214+00:00 master0 k8s.pod/kube-system/kube-apiserver-master0/kube-apiserver - - [kube_meta namespace_id="1e030def-81db-11e9-a62b-fa163e1876c9" container_name="kube-apiserver" creation_timestamp="2019-05-29T06:29:31Z" host="master0" namespace_name="kube-system" master_url="https://kubernetes.default.svc.cluster.local:443" pod_id="4aaf10f9-81db-11e9-a62b-fa163e1876c9" pod_name="kube-apiserver-master0"] 2019-05-30T08:28:38.783780355+00:00 stderr F I0530 08:28:38.783710       1 log.go:172] http: TLS handshake error from 172.28.0.19:45888: tls: client offered only unsupported versions: [300]

After you have successfully installed it, use Helm CLI to install log agents on each node, and provide customized settings via specific command options.

The only three mandatory parameters for a successful deployment of Centralized Logging are:

See Section 7.4.6, “Optional settings” for the optional parameters and their default values.

helm repo add suse https://kubernetes-charts.suse.com
helm install suse/log-agent-rsyslog --name ${RELEASE_NAME} --namespace kube-system --set server.host=${SERVER_HOST} --set server.port=${SERVER_PORT}

Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

helm repo add suse https://kubernetes-charts.suse.com
helm install ${RELEASE_NAME} suse/log-agent-rsyslog --namespace kube-system --set server.host=${SERVER_HOST} --set server.port=${SERVER_PORT}

Note

If not specified otherwise, Helm will install log agents with TCP as the default value for server.protocol.

Warning

Running Rsyslog in the host machine as well as in the Kubernetes cluster with imjournal and imfile input modules enabled may causes issues. Rsyslog can crash. To avoid that, it is possible to disable the imjournal module in the helm chart installation adding the following command line argument:

--set logs.osSystem.enabled=false

After this step, all of the log agents will initialize then start to forward logs from each node to the configured remote Rsyslog server.

helm status ${RELEASE_NAME}

helm delete --purge ${RELEASE_NAME}

Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

helm uninstall ${RELEASE_NAME}

Centralized Logging supports a configurable buffered queue. This can be used to improve log processing throughput and eliminate possible data loss, for instance after log agents shutdown, restart or in case of an unresponsive remote server. The queue files are located under /var/log/containers/{RELEASE_NAME}-log-agent-rsyslog on every node in the cluster. Queue files remain even after the log agents are deleted.

The buffered queue can be enabled/disabled with the following parameter:

queue.enabled, default value = false

Setting queue.enabled to false means that data will be stored in-memory only. Setting the parameter to true will set the data store to a mixture of in-memory and in-disk. Data will then be stored in memory until the queue is filled up, after which storing is switched to disk. Enabling the queue also automatically saves the queue to disk at service shutdown.

Additional parameters to define queue size and its disk usage are:

queue.size, default value = 50000

This option sets the number of messages allowed for the in-memory queue. This setting affects the Kubernetes cluster logs (kubernetes-control-plane and kubernetes-USER_NAME-space).

queue.maxDiskSpace, default value = 2147483648

This option sets the maximum size allowed for disk storage (in bytes). The storage is divided so that 20 percent of it is for journal logs and 80 percent for the remaining logs.

This document aims to describe monitoring in a Kubernetes cluster.

The monitoring stack consists of a monitoring/trending system and a visualization platform. Additionally you can use the in-memory metrics-server to perform automatic scaling (Refer to: Section 8.3, “Horizontal Pod Autoscaler”).

There will be two different ways of using ingress for accessing the monitoring system.

8.1.3.1 Installation For Subdomains #

This installation example shows how to install and configure Prometheus and Grafana using subdomains such as prometheus.example.com, prometheus-alertmanager.example.com, and grafana.example.com.

Important

In order to provide additional security by using TLS certificates, please make sure you have the Section 6.8, “NGINX Ingress Controller” installed and configured.

If you don’t need TLS, you may use other methods for exposing these web services as native LBaaS in OpenStack, haproxy service or k8s native methods as port-forwarding or NodePort but this is out of scope of this document.

8.1.3.2 Create DNS entries #

In this example, we will use a master node with IP 10.86.4.158 in the case of NodePort service of the Ingress Controller.

Note

You should configure proper DNS names in any production environment. These values are only for example purposes.

1. Configure the DNS server

   monitoring.example.com                      IN  A       10.86.4.158
   prometheus.example.com                      IN  CNAME   monitoring.example.com
   prometheus-alertmanager.example.com         IN  CNAME   monitoring.example.com
   grafana.example.com                         IN  CNAME   monitoring.example.com

2. Configure the management workstation /etc/hosts (optional)

   10.86.4.158 prometheus.example.com prometheus-alertmanager.example.com grafana.example.com

8.1.3.2.1 TLS Certificate #

You must configure your certificates for the components as secrets in the Kubernetes cluster. Get certificates from your certificate authority.

1. Individual certificate

   Single-name TLS certificate protects a single sub-domain, and it means each sub-domain owns its private key. From the security perspective, it is recommended to use individual certificates. However, you have to manage the private key and the certificate rotation separately.

   

   Important: Note Down Secret Names For Configuration

   When you choose to secure each service with an individual certificate, you must repeat the step below for each component and adjust the name for the individual secret each time. Please note down the names of the secrets you have created.

   In this example, the secret name is monitoring-tls.

2. Wildcard certificate

   Wildcard TLS allows you to secure multiple sub-domains with one certificate and it means multiple sub-domains share the same private key. You can then add more sub-domains without having to redeploy the certificate and moreover, save the additional certificate costs.

Refer to Section 6.9.9.1.1, “Trusted Server Certificate” on how to sign the trusted certificate or refer to Section 6.9.9.2.2, “Self-signed Server Certificate” on how to sign the self-signed certificate. The server.conf for DNS.1 is prometheus.example.com and prometheus-alertmanager.example.com grafana.example.com for individual certificates separately. The server.conf for DNS.1 is *.example.com for a wildcard certificate.

Then, import your certificate and key pair into the Kubernetes cluster secret name monitoring-tls. In this example, the certificate and key are monitoring.crt and monitoring.key.

kubectl create -n monitoring secret tls monitoring-tls  \
--key  ./monitoring.key \
--cert ./monitoring.crt

8.1.3.2.2 Prometheus #

1. Create a configuration file prometheus-config-values.yaml

   We need to configure the storage for our deployment. Choose among the options and uncomment the line in the config file. In production environments you must configure persistent storage.

   • Use an existing PersistentVolumeClaim

   • Use a StorageClass (preferred)

   # Alertmanager configuration
   alertmanager:
     enabled: true
     ingress:
       enabled: true
       hosts:
       -  prometheus-alertmanager.example.com
       annotations:
         kubernetes.io/ingress.class: nginx
         nginx.ingress.kubernetes.io/auth-type: basic
         nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
         nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
       tls:
         - hosts:
           - prometheus-alertmanager.example.com
           secretName: monitoring-tls
     persistentVolume:
       enabled: true
       ## Use a StorageClass
       storageClass: my-storage-class
       ## Create a PersistentVolumeClaim of 2Gi
       size: 2Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
   
   ## Alertmanager is configured through alertmanager.yml. This file and any others
   ## listed in alertmanagerFiles will be mounted into the alertmanager pod.
   ## See configuration options https://prometheus.io/docs/alerting/configuration/
   #alertmanagerFiles:
   #  alertmanager.yml:
   
   # Create a specific service account
   serviceAccounts:
     nodeExporter:
       name: prometheus-node-exporter
   
   # Node tolerations for node-exporter scheduling to nodes with taints
   # Allow scheduling of node-exporter on master nodes
   nodeExporter:
     hostNetwork: false
     hostPID: false
     podSecurityPolicy:
       enabled: true
       annotations:
         apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
         apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
         seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
         seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
     tolerations:
       - key: node-role.kubernetes.io/master
         operator: Exists
         effect: NoSchedule
   
   # Disable Pushgateway
   pushgateway:
     enabled: false
   
   # Prometheus configuration
   server:
     ingress:
       enabled: true
       hosts:
       - prometheus.example.com
       annotations:
         kubernetes.io/ingress.class: nginx
         nginx.ingress.kubernetes.io/auth-type: basic
         nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
         nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
       tls:
         - hosts:
           - prometheus.example.com
           secretName: monitoring-tls
     persistentVolume:
       enabled: true
       ## Use a StorageClass
       storageClass: my-storage-class
       ## Create a PersistentVolumeClaim of 8Gi
       size: 8Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
   
   ## Prometheus is configured through prometheus.yml. This file and any others
   ## listed in serverFiles will be mounted into the server pod.
   ## See configuration options
   ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/
   #serverFiles:
   #  prometheus.yml:

2. Add SUSE helm charts repository

   helm repo add suse https://kubernetes-charts.suse.com

3. Deploy SUSE prometheus helm chart and pass our configuration values file.

   helm install --name prometheus suse/prometheus \
   --namespace monitoring \
   --values prometheus-config-values.yaml

   Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

   helm install prometheus suse/prometheus \
   --namespace monitoring \
   --values prometheus-config-values.yaml

   There need to be 3 pods running (3 node-exporter pods because we have 3 nodes).

   kubectl -n monitoring get pod | grep prometheus
   NAME                                             READY     STATUS    RESTARTS   AGE
   prometheus-alertmanager-5487596d54-kcdd6         2/2       Running   0          2m
   prometheus-kube-state-metrics-566669df8c-krblx   1/1       Running   0          2m
   prometheus-node-exporter-jnc5w                   1/1       Running   0          2m
   prometheus-node-exporter-qfwp9                   1/1       Running   0          2m
   prometheus-node-exporter-sc4ls                   1/1       Running   0          2m
   prometheus-server-6488f6c4cd-5n9w8               2/2       Running   0          2m

   There need to be be 2 ingresses configured

   kubectl get ingress -n monitoring
   NAME                      HOSTS                                 ADDRESS   PORTS     AGE
   prometheus-alertmanager   prometheus-alertmanager.example.com             80, 443   87s
   prometheus-server         prometheus.example.com                          80, 443   87s

4. At this stage, the Prometheus Expression browser/API should be accessible, depending on your network configuration

   • NodePort: https://prometheus.example.com:32443

   • External IPs: https://prometheus.example.com

   • LoadBalancer: https://prometheus.example.com

8.1.3.2.3 Alertmanager Configuration Example #

The configuration example sets one "receiver" to get notified by email when one of below conditions is met:

• Node is unschedulable: severity is critical because the node cannot accept new pods

• Node runs out of disk space: severity is critical because the node cannot accept new pods

• Node has memory pressure: severity is warning

• Node has disk pressure: severity is warning

• Certificates is going to expire in 7 days: severity is critical

• Certificates is going to expire in 30 days: severity is warning

• Certificates is going to expire in 3 months: severity is info

  1. Configure alerting receiver in Alertmanager

     The Alertmanager handles alerts sent by Prometheus server, it takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email. It also takes care of silencing and inhibition of alerts.

     Add the alertmanagerFiles section to your Prometheus configuration file prometheus-config-values.yaml.

     For more information on how to configure Alertmanager, refer to Prometheus: Alerting - Configuration.

     alertmanagerFiles:
       alertmanager.yml:
         global:
           # The smarthost and SMTP sender used for mail notifications.
           smtp_from: alertmanager@example.com
           smtp_smarthost: smtp.example.com:587
           smtp_auth_username: admin@example.com
           smtp_auth_password: <PASSWORD>
           smtp_require_tls: true
     
         route:
           # The labels by which incoming alerts are grouped together.
           group_by: ['node']
     
           # When a new group of alerts is created by an incoming alert, wait at
           # least 'group_wait' to send the initial notification.
           # This way ensures that you get multiple alerts for the same group that start
           # firing shortly after another are batched together on the first
           # notification.
           group_wait: 30s
     
           # When the first notification was sent, wait 'group_interval' to send a batch
           # of new alerts that started firing for that group.
           group_interval: 5m
     
           # If an alert has successfully been sent, wait 'repeat_interval' to
           # resend them.
           repeat_interval: 3h
     
           # A default receiver
           receiver: admin-example
     
         receivers:
         - name: 'admin-example'
           email_configs:
           - to: 'admin@example.com'

  2. Configures alerting rules in Prometheus server

     Replace the serverFiles section of the Prometheus configuration file prometheus-config-values.yaml.

     For more information on how to configure alerts, refer to: Prometheus: Alerting - Notification Template Examples

     serverFiles:
       alerts: {}
       rules:
         groups:
         - name: caasp.node.rules
           rules:
           - alert: NodeIsNotReady
             expr: kube_node_status_condition{condition="Ready",status="false"} == 1 or kube_node_status_condition{condition="Ready",status="unknown"} == 1
             for: 1m
             labels:
               severity: critical
             annotations:
               description: '{{ $labels.node }} is not ready'
           - alert: NodeIsOutOfDisk
             expr: kube_node_status_condition{condition="OutOfDisk",status="true"} == 1
             labels:
               severity: critical
             annotations:
               description: '{{ $labels.node }} has insufficient free disk space'
           - alert: NodeHasDiskPressure
             expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
             labels:
               severity: warning
             annotations:
               description: '{{ $labels.node }} has insufficient available disk space'
           - alert: NodeHasInsufficientMemory
             expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
             labels:
               severity: warning
             annotations:
               description: '{{ $labels.node }} has insufficient available memory'
         - name: caasp.certs.rules
           rules:
           - alert: KubernetesCertificateExpiry3Months
             expr: (cert_exporter_cert_expires_in_seconds / 86400) < 90
             labels:
               severity: info
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 3 months'
           - alert: KubernetesCertificateExpiry30Days
             expr: (cert_exporter_cert_expires_in_seconds / 86400) < 30
             labels:
               severity: warning
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 30 days'
           - alert: KubernetesCertificateExpiry7Days
             expr: (cert_exporter_cert_expires_in_seconds / 86400) < 7
             labels:
               severity: critical
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 7 days'
           - alert: KubeconfigCertificateExpiry3Months
             expr: (cert_exporter_kubeconfig_expires_in_seconds / 86400) < 90
             labels:
               severity: info
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 3 months'
           - alert: KubeconfigCertificateExpiry30Days
             expr: (cert_exporter_kubeconfig_expires_in_seconds / 86400) < 30
             labels:
               severity: warning
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 30 days'
           - alert: KubeconfigCertificateExpiry7Days
             expr: (cert_exporter_kubeconfig_expires_in_seconds / 86400) < 7
             labels:
               severity: critical
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 7 days'
           - alert: AddonCertificateExpiry3Months
             expr: (cert_exporter_secret_expires_in_seconds / 86400) < 90
             labels:
               severity: info
             annotations:
               description: 'The cert for {{ $labels.secret_name }} is going to expire in 3 months'
           - alert: AddonCertificateExpiry30Days
             expr: (cert_exporter_secret_expires_in_seconds / 86400) < 30
             labels:
               severity: warning
             annotations:
               description: 'The cert for {{ $labels.secret_name }} is going to expire in 30 days'
           - alert: AddonCertificateExpiry7Days
             expr: (cert_exporter_secret_expires_in_seconds / 86400) < 7
             labels:
               severity: critical
             annotations:
               description: 'The cert for {{ $labels.secret_name }} is going to expire in 7 days'

  3. To apply the changed configuration, run:

     helm upgrade prometheus suse/prometheus --namespace monitoring --values prometheus-config-values.yaml

  4. You should now be able to see your Alertmanager, depending on your network configuration

• NodePort: https://prometheus-alertmanager.example.com:32443

• External IPs: https://prometheus-alertmanager.example.com

• LoadBalancer: https://prometheus-alertmanager.example.com

8.1.3.2.4 Recording Rules Configuration Example #

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. Querying the precomputed result will then often be much faster than executing the original expression every time it is needed. This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh. Another common use case is federation where precomputed metrics are scraped from one Prometheus instance by another.

For more information on how to configure recording rules, refer to Prometheus:Recording Rules - Configuration.

1. Configuring recording rules

   Add the following group of rules in the serverFiles section of the prometheus-config-values.yaml configuration file.

   serverFiles:
     alerts: {}
     rules:
       groups:
       - name: node-exporter.rules
         rules:
         - expr: count by (instance) (count without (mode) (node_cpu_seconds_total{component="node-exporter"}))
           record: instance:node_num_cpu:sum
         - expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{component="node-exporter",mode="idle"}[5m]))
           record: instance:node_cpu_utilisation:rate5m
         - expr: node_load1{component="node-exporter"} / on (instance) instance:node_num_cpu:sum
           record: instance:node_load1_per_cpu:ratio
         - expr: node_memory_MemAvailable_bytes / on (instance) node_memory_MemTotal_bytes
           record: instance:node_memory_utilisation:ratio
         - expr: rate(node_vmstat_pgmajfault{component="node-exporter"}[5m])
           record: instance:node_vmstat_pgmajfault:rate5m
         - expr: rate(node_disk_io_time_seconds_total{component="node-exporter", device=~"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+"}[5m])
           record: instance_device:node_disk_io_time_seconds:rate5m
         - expr: rate(node_disk_io_time_weighted_seconds_total{component="node-exporter", device=~"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+"}[5m])
           record: instance_device:node_disk_io_time_weighted_seconds:rate5m
         - expr: sum by (instance) (rate(node_network_receive_bytes_total{component="node-exporter", device!="lo"}[5m]))
           record: instance:node_network_receive_bytes_excluding_lo:rate5m
         - expr: sum by (instance) (rate(node_network_transmit_bytes_total{component="node-exporter", device!="lo"}[5m]))
           record: instance:node_network_transmit_bytes_excluding_lo:rate5m
         - expr: sum by (instance) (rate(node_network_receive_drop_total{component="node-exporter", device!="lo"}[5m]))
           record: instance:node_network_receive_drop_excluding_lo:rate5m
         - expr: sum by (instance) (rate(node_network_transmit_drop_total{component="node-exporter", device!="lo"}[5m]))
           record: instance:node_network_transmit_drop_excluding_lo:rate5m

2. To apply the changed configuration, run:

   helm upgrade prometheus suse/prometheus --namespace monitoring --values prometheus-config-values.yaml

3. You should now be able to see your configured rules, depending on your network configuration

   • NodePort: https://prometheus.example.com:32443/rules

   • External IPs: https://prometheus.example.com/rules

   • LoadBalancer: https://prometheus.example.com/rules

8.1.3.2.5 Grafana #

Starting from Grafana 5.0, it is possible to dynamically provision the data sources and dashboards via files. In a Kubernetes cluster, these files are provided via the utilization of ConfigMap, editing a ConfigMap will result by the modification of the configuration without having to delete/recreate the pod.

1. Configure Grafana provisioning

   Create the default datasource configuration file grafana-datasources.yaml which point to our Prometheus server

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: grafana-datasources
     namespace: monitoring
     labels:
        grafana_datasource: "1"
   data:
     datasource.yaml: |-
       apiVersion: 1
       deleteDatasources:
         - name: Prometheus
           orgId: 1
       datasources:
       - name: Prometheus
         type: prometheus
         url: http://prometheus-server.monitoring.svc.cluster.local:80
         access: proxy
         orgId: 1
         isDefault: true

2. Create the ConfigMap in Kubernetes cluster

   kubectl create -f grafana-datasources.yaml

3. Configure storage for the deployment

   Choose among the options and uncomment the line in the config file. In production environments you must configure persistent storage.

   • Use an existing PersistentVolumeClaim

   • Use a StorageClass (preferred)

     Create a file grafana-config-values.yaml with the appropriate values

     # Configure admin password
     adminPassword: <PASSWORD>
     
     # Ingress configuration
     ingress:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: nginx
       hosts:
         - grafana.example.com
       tls:
         - hosts:
           - grafana.example.com
           secretName: monitoring-tls
     
     # Configure persistent storage
     persistence:
       enabled: true
       accessModes:
         - ReadWriteOnce
       ## Use a StorageClass
       storageClassName: my-storage-class
       ## Create a PersistentVolumeClaim of 10Gi
       size: 10Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
     
     # Enable sidecar for provisioning
     sidecar:
       datasources:
         enabled: true
         label: grafana_datasource
       dashboards:
         enabled: true
         label: grafana_dashboard

4. Add SUSE helm charts repository

   helm repo add suse https://kubernetes-charts.suse.com

5. Deploy SUSE grafana helm chart and pass our configuration values file

   helm install --name grafana suse/grafana \
   --namespace monitoring \
   --values grafana-config-values.yaml

   Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

   helm install grafana suse/grafana \
   --namespace monitoring \
   --values grafana-config-values.yaml

6. The result should be a running Grafana pod

   kubectl -n monitoring get pod | grep grafana
   NAME                                             READY     STATUS    RESTARTS   AGE
   grafana-dbf7ddb7d-fxg6d                          3/3       Running   0          2m

7. At this stage, Grafana should be accessible, depending on your network configuration

   • NodePort: https://grafana.example.com:32443

   • External IPs: https://grafana.example.com

   • LoadBalancer: https://grafana.example.com

8. Now you can add Grafana dashboards.

8.1.3.2.6 Adding Grafana Dashboards #

There are three ways to add dashboards to Grafana:

• Deploy an existing dashboard from Grafana dashboards

  1. Open the deployed Grafana in your browser and log in.

  2. On the home page of Grafana, hover your mousecursor over the + button on the left sidebar and click on the import menuitem.

  3. Select an existing dashboard for your purpose from Grafana dashboards. Copy the URL to the clipboard.

  4. Paste the URL (for example) https://grafana.com/dashboards/3131 into the first input field to import the "Kubernetes All Nodes" Grafana Dashboard. After pasting in the url, the view will change to another form.

  5. Now select the "Prometheus" datasource in the prometheus field and click on the import button.

  6. The browser will redirect you to your newly created dashboard.

• Use our pre-built dashboards to monitor the SUSE CaaS Platform system

  # monitor SUSE CaaS Platform cluster
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-cluster.yaml
  # monitor SUSE CaaS Platform etcd cluster
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-etcd-cluster.yaml
  # monitor SUSE CaaS Platform nodes
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-nodes.yaml
  # monitor SUSE CaaS Platform namespaces
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-namespaces.yaml
  # monitor SUSE CaaS Platform pods
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-pods.yaml
  # monitor SUSE CaaS Platform certificates
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-certificates.yaml

• Build your own dashboard Deploy your own dashboard by configuration file containing the dashboard definition.

  1. Create your dashboard definition file as a ConfigMap, for example grafana-dashboards-caasp-cluster.yaml.

     ---
     apiVersion: v1
     kind: ConfigMap
     metadata:
       name: grafana-dashboards-caasp-cluster
       namespace: monitoring
       labels:
          grafana_dashboard: "1"
     data:
       caasp-cluster.json: |-
         {
           "__inputs": [
             {
               "name": "DS_PROMETHEUS",
               "label": "Prometheus",
               "description": "",
               "type": "datasource",
               "pluginId": "prometheus",
               "pluginName": "Prometheus"
             }
           ],
           "__requires": [
             {
               "type": "grafana",
     [...]
     continues with definition of dashboard JSON
     [...]

  2. Apply the ConfigMap to the cluster.

     kubectl apply -f grafana-dashboards-caasp-cluster.yaml

8.1.3.3 Installation For Subpaths #

This installation example shows how to install and configure Prometheus and Grafana using subpaths such as example.com/prometheus, example.com/alertmanager, and example.com/grafana.

Important

Overlapped instructions from subdomains will be omitted. Refer to the instruction from subdomains.

8.1.3.4 Create DNS entries #

In this example, we will use a master node with IP 10.86.4.158 in the case of NodePort service of the Ingress Controller.

Note

You should configure proper DNS names in any production environment. These values are only for example purposes.

1. Configure the DNS server

   example.com                      IN  A       10.86.4.158

2. Configure the management workstation /etc/hosts (optional)

   10.86.4.158 example.com

8.1.3.4.1 TLS Certificate #

You must configure your certificates for the components as secrets in the Kubernetes cluster. Get certificates from your certificate authority.

Refer to Section 6.9.9.1.1, “Trusted Server Certificate” on how to sign the trusted certificate or refer to Section 6.9.9.2.2, “Self-signed Server Certificate” on how to sign the self-signed certificate. The server.conf for DNS.1 is example.com.

Then, import your certificate and key pair into the Kubernetes cluster secret name monitoring-tls. In this example, the certificate and key are monitoring.crt and monitoring.key.

kubectl create -n monitoring secret tls monitoring-tls  \
--key  ./monitoring.key \
--cert ./monitoring.crt

8.1.3.4.2 Prometheus #

1. Create a configuration file prometheus-config-values.yaml

   We need to configure the storage for our deployment. Choose among the options and uncomment the line in the config file. In production environments you must configure persistent storage.

   • Use an existing PersistentVolumeClaim

   • Use a StorageClass (preferred)

   • Add the external URL to baseURL at which the server can be accessed. The baseURL depends on your network configuration.

     • NodePort: https://example.com:32443/prometheus and https://example.com:32443/alertmanager

     • External IPs: https://example.com/prometheus and https://example.com/alertmanager

     • LoadBalancer: https://example.com/prometheus and https://example.com/alertmanager

   # Alertmanager configuration
   alertmanager:
     enabled: true
     baseURL: https://example.com:32443/alertmanager
     prefixURL: /alertmanager
     ingress:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: nginx
         nginx.ingress.kubernetes.io/auth-type: basic
         nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
         nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
       hosts:
         - example.com/alertmanager
       tls:
         - secretName: monitoring-tls
           hosts:
           - example.com
     persistentVolume:
       enabled: true
       ## Use a StorageClass
       storageClass: my-storage-class
       ## Create a PersistentVolumeClaim of 2Gi
       size: 2Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
   
   ## Alertmanager is configured through alertmanager.yml. This file and any others
   ## listed in alertmanagerFiles will be mounted into the alertmanager pod.
   ## See configuration options https://prometheus.io/docs/alerting/configuration/
   #alertmanagerFiles:
   #  alertmanager.yml:
   
   # Create a specific service account
   serviceAccounts:
     nodeExporter:
       name: prometheus-node-exporter
   
   # Node tolerations for node-exporter scheduling to nodes with taints
   # Allow scheduling of node-exporter on master nodes
   nodeExporter:
     hostNetwork: false
     hostPID: false
     podSecurityPolicy:
       enabled: true
       annotations:
         apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
         apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
         seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
         seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
     tolerations:
       - key: node-role.kubernetes.io/master
         operator: Exists
         effect: NoSchedule
   
   # Disable Pushgateway
   pushgateway:
     enabled: false
   
   # Prometheus configuration
   server:
     baseURL: https://example.com:32443/prometheus
     prefixURL: /prometheus
     ingress:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: nginx
         nginx.ingress.kubernetes.io/auth-type: basic
         nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
         nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
       hosts:
         - example.com/prometheus
       tls:
         - secretName: monitoring-tls
           hosts:
           - example.com
     persistentVolume:
       enabled: true
       ## Use a StorageClass
       storageClass: my-storage-class
       ## Create a PersistentVolumeClaim of 8Gi
       size: 8Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
   
   ## Prometheus is configured through prometheus.yml. This file and any others
   ## listed in serverFiles will be mounted into the server pod.
   ## See configuration options
   ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/
   #serverFiles:
   #  prometheus.yml:

2. Add SUSE helm charts repository

   helm repo add suse https://kubernetes-charts.suse.com

3. Deploy SUSE prometheus helm chart and pass our configuration values file.

   helm install --name prometheus suse/prometheus \
   --namespace monitoring \
   --values prometheus-config-values.yaml

   Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

   helm install prometheus suse/prometheus \
   --namespace monitoring \
   --values prometheus-config-values.yaml

   There need to be 3 pods running (3 node-exporter pods because we have 3 nodes).

   kubectl -n monitoring get pod | grep prometheus
   NAME                                             READY     STATUS    RESTARTS   AGE
   prometheus-alertmanager-5487596d54-kcdd6         2/2       Running   0          2m
   prometheus-kube-state-metrics-566669df8c-krblx   1/1       Running   0          2m
   prometheus-node-exporter-jnc5w                   1/1       Running   0          2m
   prometheus-node-exporter-qfwp9                   1/1       Running   0          2m
   prometheus-node-exporter-sc4ls                   1/1       Running   0          2m
   prometheus-server-6488f6c4cd-5n9w8               2/2       Running   0          2m

8.1.3.4.3 Alertmanager Configuration Example #

Refer to Section 8.1.3.2.3, “Alertmanager Configuration Example”

8.1.3.4.4 Recording Rules Configuration Example #

Refer to Section 8.1.3.2.4, “Recording Rules Configuration Example”

8.1.3.4.5 Grafana #

Starting from Grafana 5.0, it is possible to dynamically provision the data sources and dashboards via files. In Kubernetes cluster, these files are provided via the utilization of ConfigMap, editing a ConfigMap will result by the modification of the configuration without having to delete/recreate the pod.

1. Configure Grafana provisioning

   Create the default datasource configuration file grafana-datasources.yaml which point to our Prometheus server

   ---
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: grafana-datasources
     namespace: monitoring
     labels:
        grafana_datasource: "1"
   data:
     datasource.yaml: |-
       apiVersion: 1
       deleteDatasources:
         - name: Prometheus
           orgId: 1
       datasources:
       - name: Prometheus
         type: prometheus
         url: http://prometheus-server.monitoring.svc.cluster.local:80
         access: proxy
         orgId: 1
         isDefault: true

2. Create the ConfigMap in Kubernetes cluster

   kubectl create -f grafana-datasources.yaml

3. Configure storage for the deployment

   Choose among the options and uncomment the line in the config file. In production environments you must configure persistent storage.

   • Use an existing PersistentVolumeClaim

   • Use a StorageClass (preferred)

   • Add the external URL to root_url at which the server can be accessed. The root_url depends on your network configuration.

     • NodePort: https://example.com:32443/grafana

     • External IPs: https://example.com/grafana

     • LoadBalancer: https://example.com/grafana

   Create a file grafana-config-values.yaml with the appropriate values

   +

   # Configure admin password
   adminPassword: <PASSWORD>
   
   # Ingress configuration
   ingress:
     enabled: true
     annotations:
       kubernetes.io/ingress.class: nginx
       nginx.ingress.kubernetes.io/rewrite-target: /
     hosts:
       - example.com
     path: /grafana
     tls:
       - secretName: monitoring-tls
         hosts:
         - example.com
   
   # subpath for grafana
   grafana.ini:
     server:
       root_url: https://example.com:32443/grafana
   
   # Configure persistent storage
   persistence:
     enabled: true
     accessModes:
       - ReadWriteOnce
     ## Use a StorageClass
     storageClassName: my-storage-class
     ## Create a PersistentVolumeClaim of 10Gi
     size: 10Gi
     ## Use an existing PersistentVolumeClaim (my-pvc)
     #existingClaim: my-pvc
   
   # Enable sidecar for provisioning
   sidecar:
     datasources:
       enabled: true
       label: grafana_datasource
     dashboards:
       enabled: true
       label: grafana_dashboard

4. Add SUSE helm charts repository

   helm repo add suse https://kubernetes-charts.suse.com

5. Deploy SUSE grafana helm chart and pass our configuration values file

   helm install --name grafana suse/grafana \
   --namespace monitoring \
   --values grafana-config-values.yaml

   Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

   helm install grafana suse/grafana \
   --namespace monitoring \
   --values grafana-config-values.yaml

6. The result should be a running Grafana pod

   kubectl -n monitoring get pod | grep grafana
   NAME                                             READY     STATUS    RESTARTS   AGE
   grafana-dbf7ddb7d-fxg6d                          3/3       Running   0          2m

7. Access Prometheus, Alertmanager, and Grafana

   At this stage, the Prometheus Expression browser/API, Alertmanager, and Grafana should be accessible, depending on your network configuration

   • Prometheus Expression browser/API

     • NodePort: https://example.com:32443/prometheus

     • External IPs: https://example.com/prometheus

     • LoadBalancer: https://example.com/prometheus

   • Alertmanager

     • NodePort: https://example.com:32443/alertmanager

     • External IPs: https://example.com/alertmanager

     • LoadBalancer: https://example.com/alertmanager

   • Grafana

     • NodePort: https://example.com:32443/grafana

     • External IPs: https://example.com/grafana

     • LoadBalancer: https://example.com/grafana

8. Now you can add the Grafana dashboards.

8.1.3.4.6 Adding Grafana Dashboards #

Refer to Section 8.1.3.2.6, “Adding Grafana Dashboards”

The basic check if a cluster is working correctly is based on a few criteria:

Note

For further understanding cluster health information, consider reading https://v1-18.docs.kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/

8.2.1.1 Kubernetes master #

All components in Kubernetes cluster expose a /healthz endpoint. The expected (healthy) HTTP response status code is 200.

The minimal services for the master to work properly are:

• kube-apiserver:

  The component that receives your requests from kubectl and from the rest of the Kubernetes components. The URL is https://<CONTROL_PLANE_IP/FQDN>:6443/healthz

  • Local Check

    curl -k -i https://localhost:6443/healthz

  • Remote Check

    curl -k -i https://<CONTROL_PLANE_IP/FQDN>:6443/healthz

• kube-controller-manager:

  The component that contains the control loop, driving current state to the desired state. The URL is http://<CONTROL_PLANE_IP/FQDN>:10252/healthz

  • Local Check

    curl -i http://localhost:10252/healthz

  • Remote Check

    Make sure firewall allows port 10252.

    curl -i http://<CONTROL_PLANE_IP/FQDN>:10252/healthz

• kube-scheduler:

  The component that schedules workloads to nodes. The URL is http://<CONTROL_PLANE_IP/FQDN>:10251/healthz

  • Local Check

    curl -i http://localhost:10251/healthz

  • Remote Check

    Make sure firewall allows port 10251.

    curl -i http://<CONTROL_PLANE_IP/FQDN>:10251/healthz

Note: High-Availability Environments

In a HA environment you can monitor kube-apiserver on https://<LOAD_BALANCER_IP/FQDN>:6443/healthz.

If any one of the master nodes is running correctly, you will receive a valid response.

This does, however, not mean that all master nodes necessarily work correctly. To ensure that all master nodes work properly, the health checks must be repeated individually for each deployed master node.

This endpoint will return a successful HTTP response if the cluster is operational; otherwise it will fail. It will for example check that it can access etcd. This should not be used to infer that the overall cluster health is ideal. It will return a successful response even when only minimal operational cluster health exists.

To probe for full cluster health, you must perform individual health checking for all machines.

This basic node health check consists of two parts. It checks:

curl -i http://localhost:10248/healthz

kubectl get deployments -n kube-system
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
cilium-operator   1/1     1            1           8d
coredns           2/2     2            2           8d
oidc-dex          1/1     1            1           8d
oidc-gangway      1/1     1            1           8d

If the deployed services contain a health endpoint, or if they contain an endpoint that can be used to determine if the service is up, you can use livenessProbes and/or readinessProbes.

Note: Health check endpoints vs. functional endpoints

A proper health check is always preferred if designed correctly.

Despite the fact that any endpoint could potentially be used to infer if your application is up, it is better to have an endpoint specifically for health in your application. Such an endpoint will only respond affirmatively when all your setup code on the server has finished and the application is running in a desired state.

The livenessProbes and readinessProbes share configuration options and probe types.

There are different options for the livenessProbes to check:

We recommend to apply other best practices from system administration to your monitoring and health checking approach. These steps are not specific to SUSE CaaS Platform and are beyond the scope of this document.

It is useful to first find out about the available resources of your cluster.

8.3.1.1 Using Horizontal Pod Autoscaler (HPA) #

You can set the HPA to scale according to various metrics. These include average CPU utilization, average CPU value, average memory utilization and average memory value. The following sections show the recommended configuration for each of the aforementioned options.

8.3.1.1.1 Creating an HPA Using Average CPU Utilization #

The following code is an example of what this type of HPA can look like. You will have to run the code on your admin node or user local machine. Note that you need a kubeconfig file with RBAC permission that allow setting up autoscale rules into your Kubernetes cluster.

# deployment
kubectl autoscale deployment <DEPLOYMENT_NAME> \
    --min=<MIN_REPLICAS_NUMBER> \
    --max=<MAX_REPLICAS_NUMBER> \
    --cpu-percent=<PERCENT>

# replication controller
kubectl autoscale replicationcontrollers <REPLICATIONCONTROLLERS_NAME> \
    --min=<MIN_REPLICAS_NUMBER> \
    --max=<MAX_REPLICAS_NUMBER> \
    --cpu-percent=<PERCENT>

You could for example use the following values:

kubectl autoscale deployment oidc-dex \
    --name=avg-cpu-util \
    --min=1 \
    --max=10 \
    --cpu-percent=50

The example output below shows autoscaling works in case of the oidc-dex deployment. The HPA increases the minimum number of pods to 1 and will increase the pods up to 10, if the average CPU utilization of the pods reaches 50%. For more details about the inner workings of the scaling, refer to The Kubernetes documentation on the horizontal pod autoscale algorithm.

To check the current status of the HPA run:

kubectl get hpa

Example output:

NAME       REFERENCE             TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
oidc-dex   Deployment/oidc-dex   0%/50%          1         10        3          115s

Note

To calculate pod CPU utilization HPA divides the total CPU usage of all containers by the total number of CPU requests:

POD CPU UTILIZATION = TOTAL CPU USAGE OF ALL CONTAINERS / NUMBER OF CPU REQUESTS

For example:

• Container1 requests 0.5 CPU and uses 0 CPU.

• Container2 requests 1 CPU and uses 2 CPU.

The CPU utilization will be (0+2)/(0.5+1)*100 (%)=133 (%)

If a replication controller, deployment, replica set or stateful set does not specify the CPU request, the output of kubectl get hpa TARGETS will be unknown.

8.3.1.1.2 Creating an HPA Using the Average CPU Value #

1. Create a yaml manifest file hpa-avg-cpu-value.yaml with the following content:

   apiVersion: autoscaling/v2beta2
   kind: HorizontalPodAutoscaler
   metadata:
     name: avg-cpu-value 1
     namespace: kube-system 2
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment 3
       name: example 4
     minReplicas: 1 5
     maxReplicas: 10 6
     metrics:
     - type: Resource
       resource:
         name: cpu
         target:
           type: AverageValue
           averageValue: 500Mi 7

   1	Name of the HPA.
   2	Namespace of the HPA.
   3	Specifies the kind of object to scale (a replication controller, deployment, replica set or stateful set).
   4	Specifies the name of the object to scale.
   5	Specifies the minimum number of replicas.
   6	Specifies the maximum number of replicas.
   7	The average value of the requested CPU that each pod uses.

2. Apply the yaml manifest by running:

   kubectl apply -f hpa-avg-cpu-value.yaml

3. Check the current status of the HPA:

   kubectl get hpa
   
   NAME            REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
   avg-cpu-value   Deployment/php-apache   1m/500Mi   1         10        1          39s

8.3.1.1.3 Creating an HPA Using Average Memory Utilization #

1. Create a yaml manifest file hpa-avg-memory-util.yaml with the following content:

   apiVersion: autoscaling/v2beta2
   kind: HorizontalPodAutoscaler
   metadata:
     name: avg-memory-util 1
     namespace: kube-system 2
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment 3
       name: example 4
     minReplicas: 1 5
     maxReplicas: 10 6
     metrics:
     - type: Resource
       resource:
         name: memory
         target:
           type: Utilization
           averageUtilization: 50 7

   1	Name of the HPA.
   2	Namespace of the HPA.
   3	Specifies the kind of object to scale (a replication controller, deployment, replica set or stateful set).
   4	Specifies the name of the object to scale.
   5	Specifies the minimum number of replicas.
   6	Specifies the maximum number of replicas.
   7	The average utilization of the requested memory that each pod uses.

2. Apply the yaml manifest by running:

   kubectl apply -f hpa-avg-memory-util.yaml

3. Check the current status of the HPA:

   kubectl get hpa
   
   NAME              REFERENCE            TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
   avg-memory-util   Deployment/example   5%/50%           1         10        1          4m54s

   

   Note

   HPA calculates pod memory utilization as: total memory usage of all containers / total memory requests. If a deployment or replication controller does not specify the memory request, the ouput of kubectl get hpa TARGETS is <unknown>.

8.3.1.1.4 Creating an HPA Using Average Memory Value #

1. Create a yaml manifest file hpa-avg-memory-value.yaml with the following content:

   apiVersion: autoscaling/v2beta2
   kind: HorizontalPodAutoscaler
   metadata:
     name: avg-memory-value 1
     namespace: kube-system 2
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment 3
       name: example 4
     minReplicas: 1 5
     maxReplicas: 10 6
     metrics:
     - type: Resource
       resource:
         name: memory
         target:
           type: AverageValue
           averageValue: 500Mi 7

   1	Name of the HPA.
   2	Namespace of the HPA.
   3	Specifies the kind of object to scale (a replication controller, deployment, replica set or stateful set).
   4	Specifies the name of the object to scale.
   5	Specifies the minimum number of replicas.
   6	Specifies the maximum number of replicas.
   7	The average value of the requested memory that each pod uses.

2. Apply the yaml manifest by running:

   kubectl apply -f hpa-avg-memory-value.yaml

3. Check the current status of the HPA:

   kubectl get hpa
   
   NAME                     REFERENCE            TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
   avg-memory-value         Deployment/example   11603968/500Mi   1         10        1          6m24s