If you are using a virtual machine template for creating new cluster nodes, you must make sure that before joining the cloned machine to the cluster it is updated to the same software versions than the other nodes in the cluster.

Refer to Section 4.1, “Update Requirements”.

Nodes with mismatching package or container software versions might not be fully functional.

If you wish to remove a node temporarily, the recommended approach is to first drain the node.

When you want to bring the node back, you only have to uncordon it.

Tip

For instructions on how to perform these operations refer to Section 2.6, “Node Operations”.

Important

Nodes removed with this method cannot be added back to the cluster or any other skuba-initiated cluster. You must reinstall the entire node and then join it again to the cluster.

The skuba node remove command serves to permanently remove nodes. Running this command will work even if the target virtual machine is down, so it is the safest way to remove the node.

skuba node remove <NODE_NAME> [flags]

Note

Per default, node removal has an unlimited timeout on waiting for the node to drain. If the node is unreachable it can not be drained and thus the removal will fail or get stuck indefinitely. You can specify a time after which removal will be performed without waiting for the node to drain with the flag --drain-timeout <DURATION>.

For example, waiting for the node to drain for 1 minute and 5 seconds:

skuba node remove caasp-worker1 --drain-timeout 1m5s

For a list of supported time formats run skuba node remove -h.

Important

After the removal of a master node, you have to manually delete its entries from your load balancer’s configuration.

These to commands respectively define if a node is marked as schedulable or unschedulable. This means that a node is allowed to or not allowed to receive any new workloads. This can be useful when troubleshooting a node.

To mark a node as unschedulable run:

kubectl cordon <NODE_NAME>

To mark a node as schedulable run:

kubectl uncordon <NODE_NAME>

Draining a node consists of evicting all the running pods from the current node in order to perform maintenance. This is a mandatory step in order to ensure a proper functioning of the workloads. This is achieved using kubectl.

To drain a node run:

kubectl drain <NODE_NAME>

This action will also implicitly cordon the node. Therefore once the maintenance is done, uncordon the node to set it back to schedulable.

Refer to the official Kubernetes documentation for more information: https://v1-17.docs.kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/#use-kubectl-drain-to-remove-a-node-from-service

Applications that will be deployed to Kubernetes will typically contain all the required software to be executed. In some cases, especially when it comes to the hardware layer abstraction (storage backends, GPU), additional packages must be installed on the underlying operating system outside of Kubernetes.

Note

The following examples show installation of required packages for Ceph, please adjust the list of packages and repositories to whichever software you need to install.

While you can install any software package from the SLES ecosystem this falls outside of the support scope for SUSE CaaS Platform.

sudo zypper in ceph-common xfsprogs

Run sudo zypper up on your management workstation to get the latest version of skuba and its dependencies. Reboot the machine to make sure that all system changes are correctly applied.

In order to get an overview of the updates available, you can run:

skuba cluster upgrade plan

This will show you a list of updates (if available) for different components installed on the cluster. If the cluster is already running the latest available versions, the output should look like this:

Current Kubernetes cluster version: 1.16.2
Latest Kubernetes version: 1.16.2

Congratulations! You are already at the latest version available

If the cluster has a new patch-level or minor Kubernetes version available, the output should look like this:

Current Kubernetes cluster version: 1.15.2
Latest Kubernetes version: 1.16.2

Upgrade path to update from 1.15.2 to 1.16.2:
 - 1.15.2 -> 1.16.2

Similarly, you can also fetch this information on a per-node basis with the following command:

skuba node upgrade plan <NODE>

For example, if the cluster has a node named worker0 which is running the latest available versions, the output should look like this:

Current Kubernetes cluster version: 1.16.2
Latest Kubernetes version: 1.16.2

Node worker0 is up to date

On the other hand, if this same node has a new patch-level or minor Kubernetes version available, the output should look like this:

Current Kubernetes cluster version: 1.15.2
Latest Kubernetes version: 1.16.2

Current Node version: 1.15.2

Component versions in worker0
  - kubelet: 1.15.2 -> 1.16.2
  - cri-o: 1.15.0 -> 1.16.0

You will get a similar output if there is a version available on a master node (named master0 in this example):

Current Kubernetes cluster version: 1.15.2
Latest Kubernetes version: 1.16.2

Current Node version: 1.15.2

Component versions in master0
  - apiserver: 1.15.2 -> 1.16.2
  - controller-manager: 1.15.2 -> 1.16.2
  - scheduler: 1.15.2 -> 1.16.2
  - etcd: 3.3.11 -> 3.3.15
  - kubelet: 1.15.2 -> 1.16.2
  - cri-o: 1.15.0 -> 1.16.0

It may happen that the Kubernetes version on the control plane is too outdated for the update to progress. In this case, you would get output similar to the following:

Current Kubernetes cluster version: 1.15.0
Latest Kubernetes version: 1.15.0

Unable to plan node upgrade: at least one control plane does not tolerate the current cluster version

Tip

The control plane consists of these components:

• apiserver

• controller-manager

• scheduler

• etcd

• kubelet

• cri-o

Note

Due to changes to the way skuba handles addons some existing components might be shown as new addon in the status output. This is expected and no cause for concern. For any upgrade afterwards the addon will be considered known and only show available upgrades.

Important

SUSE CaaS Platform 4.2.1 provides the update of Cilium from 1.5.3 to 1.6.6. The important change in Cilium 1.6 is usage of Kubernetes CRDs instead of etcd. skuba performs and automated migration of data from etcd to CRDs. If that migration is not successful, skuba shows the following warning:

"Could not migrate data from etcd to CRD. Addons upgrade will be continued without it, which will result in temporary connection loss for currently existing pods and services."

That warning means that Cilium is going to regenerate all internal data on the first run after upgrade. It can result in temporary connection loss for pods and services which might take few minutes.

Each Kubernetes cluster version comes with different addons base manifests. To update your local addons cluster folder definition in-sync with current Kubernetes cluster version, please run:

skuba addon refresh localconfig

In order to get an overview of the addon updates available, you can run:

skuba addon upgrade plan

This will show you a list of updates (if available) for different addons installed on the cluster:

Current Kubernetes cluster version: 1.17.4
Latest Kubernetes version: 1.17.4

Addon upgrades for 1.17.4:
  - cilium: 1.5.3 -> 1.6.6
  - dex: 2.16.0 (manifest version from 5 to 6)
  - gangway: 3.1.0-rev4 (manifest version from 4 to 5)
  - metrics-server: 0.3.6 (new addon)

If the cluster is already running the latest available versions, the output should look like this:

Current Kubernetes cluster version: 1.17.4
Latest Kubernetes version: 1.17.4

Congratulations! Addons are already at the latest version available

Before updating the nodes you must apply the addon upgrades to your management workstation. Please run:

skuba addon upgrade apply

Once you have upgraded all nodes, please run skuba cluster upgrade plan again. This will show if any upgrades are available that required the versions you just installed. If there are upgrades available please repeat the procedure until no more new upgrades are shown.

Nodes added to a cluster have the service skuba-update.timer, which is responsible for running automatic updates, activated by default.

This service calls the skuba-update utility and it can be configured with the /etc/sysconfig/skuba-update file.

Note: How skuba-update non-interactive mode works

skuba-update uses the flags --non-interactive and --non-interactive-include-reboot-patches. The --non-interactive flag causes zypper to use default answers to questions rather than prompting a user for answers. In non-interactive mode, the --non-interactive-include-reboot-patches flag causes patches with the rebootSuggested-flag to not be skipped. Zypper does not perform the reboot directly. Instead, kured will be used to safely schedule reboots as needed.

To disable the automatic updates on a node, simply ssh to it and then configure the skuba-update service by editing the /etc/sysconfig/skuba-update file with the following runtime options:

## Path           : System/Management
## Description    : Extra switches for skuba-update
## Type           : string
## Default        : ""
## ServiceRestart : skuba-update
#
SKUBA_UPDATE_OPTIONS="--annotate-only"

Tip

It is not required to reload or restart skuba-update.timer.

The --annotate-only flag makes the skuba-update utility only check if updates are available and annotate the node accordingly. When this flag is activated no updates are installed at all.

When OS updates are disabled, then you will have to manage OS updates manually. In order to do so, you will have to call skuba-update manually on each node.

Warning

Do not use zypper up/zypper patch commands as these do not manage the Kubernetes annotations used by kured. If you perform a manual update using these commands you might render your cluster unusable.

After that, rebooting the node will depend on whether you have also disabled reboots or not. If you have disabled reboots for this node, then you will have to follow the instructions as given in Section 4.4.2, “Completely Disabling Reboots”. Otherwise, you will have to wait until kured performs the reboot of the node

If you would like to take care of reboots manually, either as a temporary measure or permanently, you can disable them by creating a lock:

kubectl -n kube-system annotate ds kured weave.works/kured-node-lock='{"nodeID":"manual"}'

This command modifies an annotation (annotate) on the daemonset (ds) named kured.

When automatic reboots are disabled, you will have to manage reboots yourself. In order to do this, you will have to follow some steps whenever you want to issue a reboot marker for a node. First of all, you will have to cordon and drain the node:

kubectl cordon <NODE_ID>
kubectl drain --force=true \
  --ignore-daemonsets=true \ 1
  --delete-local-data=false \ 2
  --grace-period 600 \ 3
  --timeout=900s \ 4
  <NODE_ID>

Important

Depending on your deployed applications, you must adjust the values for --grace-period and --timeout to grant the applications enough time to safely shut down without losing data. The values here are meant to represent a conservative default for an application like SUSE Cloud Application Platform.

If you do not set these values, applications might never finish and draining of the pod will hang indefinitely.

Only then you will be able to manually reboot the node safely.

Once the node is back, remember to uncordon it so it is scheduleable again:

kubectl uncordon <NODE_ID>

Perform the above steps first on control plane nodes, and afterwards on worker nodes.

Tip

If the node that should be rebooted does not contain any workload you can skip the above steps and simply reboot the node.

In exceptional circumstances, such as a node experiencing a permanent failure whilst rebooting, manual intervention may be required to remove the cluster lock:

kubectl -n kube-system annotate ds kured weave.works/kured-node-lock-

This command modifies an annotation (annotate) on the daemonset (ds) named kured. It explicitly performs an "unset" (-) for the value for the annotation named weave.works/kured-node-lock.

This section provides an overview of the most common verbs (operations) used for defining roles. Verbs correspond to sub-commands of kubectl.

This section provides an overview of the most common resources used for defining roles.

Roles are defined in YAML files. To apply role definitions to Kubernetes, use kubectl apply -f YAML_FILE. The following examples provide an overview about different use cases of roles.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: view-pods 1
  namespace: default 2
rules:
- apiGroups: [""] 3
  resources: ["pods"] 4
  verbs: ["get", "watch", "list"] 5

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: admin-create-pods 1
rules:
- apiGroups: [""] 2
  resources: ["pods"] 3
  verbs: ["create"] 4

To bind a group or user to a role, create a YAML file that contains the role binding description. Then apply the binding with kubectl apply -f YAML_FILE. The following examples provide an overview about different use cases of role bindings.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: <ROLE_BINDING_NAME> 1
  namespace: <NAMESPACE> 2
subjects:
- kind: Group
  name: <LDAP_GROUP_NAME> 3
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: <ROLE_NAME> 4
  apiGroup: rbac.authorization.k8s.io

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: <CLUSTER_ROLE_BINDING_NAME> 1
subjects:
- kind: Group
  name: <CLUSTER_GROUP_NAME> 2
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: <CLUSER_ROLE_NAME> 3
  apiGroup: rbac.authorization.k8s.io

Important

When creating new Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings, it is important to keep in mind the Principle of Least Privilege:

"define rules such that the account bound to the Role or ClusterRole has the minimum amount of permissions needed to fulfill its purpose and no more."

For instance, granting the admin ClusterRole to most accounts is most likely unnecessary, when a reduced-scope role would be enough fulfill the account’s purpose. This helps reduce the attack surface if an account is compromised.

It is also recommended to periodically review your Roles and ClusterRoles to ensure they are still required and are not overly-permissive.

RBAC uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing administrators to dynamically configure policies through the Kubernetes API.

The authentication components are deployed as part of the SUSE CaaS Platform installation. Administrators can update LDAP identity providers before or after platform deployment. After deploying SUSE CaaS Platform, administrators can use Kubernetes RBAC to design user or group authorizations. Users can access with a Web browser or command line to do the authentication and self-configure kubectl to access authorized resources.

Authentication is composed of:

For RBAC, administrators can use kubectl to create corresponding RoleBinding or ClusterRoleBinding for a user or group to limit resource access.

5.5.2.1 Web Flow #

1. User requests access through Gangway.

2. Gangway redirects to Dex.

3. Dex redirects to connected identity provider (connector). User login and a request to approve access are generated.

4. Dex continues with OIDC authentication flow on behalf of the user and creates/updates data to Kubernetes CRDs.

5. Dex redirects the user to Gangway. This redirect includes (ID/refresh) tokens.

6. Gangway returns a link to download kubeconfig or self-configures kubectl instructions to the user.

   

7. User downloads kubeconf or self-configures kubectl.

8. User uses kubectl to connect to the Kubernetes API server.

9. Kubernetes CRDs validate the Kubernetes API server request and return a response.

10. The kubectl connects to the authorized Kubernetes resources through the Kubernetes API server.

5.5.2.2 CLI Flow #

1. User requests access through skuba auth login with the Dex server URL, username and password.

2. Dex uses received username and password to log in and approve the access request to the connected identity providers (connectors).

3. Dex continues with the OIDC authentication flow on behalf of the user and creates/updates data to the Kubernetes CRDs.

4. Dex returns the ID token and refresh token to skuba auth login.

5. skuba auth login generates the kubeconfig file kubeconf.txt.

6. User uses kubectl to connect the Kubernetes API server.

7. Kubernetes CRDs validate the Kubernetes API server request and return a response.

8. The kubectl connects to the authorized Kubernetes resources through Kubernetes API server.

The 389 Directory Server image registry.suse.com/caasp/v4/389-ds:1.4.0 will automatically generate a self-signed certificate and key. The following instructions show how to deploy the "389 Directory Server" with a customized configuration using container commands.

To replace the automatically generated certificate with your own, follow these steps:

In both directories, user-regular1 and user-regular2 are members of the k8s-users group, and user-admin is a member of the k8s-admins group.

In Active Directory, user-bind is a simple user that is a member of the default Domain Users group. Hence, we can use it to authenticate, because it has read-only access to Active Directory. The mail attribute is used to create the RBAC rules.

Tip

The following examples might use PEM files encoded to a base64 string. These can be generated using:

cat <ROOT_CA_PEM_FILE> | base64 | awk '{print}' ORS='' && echo

dn: dc=example,dc=org
objectClass: top
objectClass: domain
dc: example

dn: cn=Directory Administrators,dc=example,dc=org
objectClass: top
objectClass: groupofuniquenames
cn: Directory Administrators
uniqueMember: cn=Directory Manager

dn: ou=Groups,dc=example,dc=org
objectClass: top
objectClass: organizationalunit
ou: Groups

dn: ou=People,dc=example,dc=org
objectClass: top
objectClass: organizationalunit
ou: People

dn: ou=Users,dc=example,dc=org
objectclass: top
objectclass: organizationalUnit
ou: Users

dn: dc=example,dc=org
changetype: modify
add: aci
aci: (targetattr!="userPassword || aci")(version 3.0; acl "Enable anonymous access"; allow (read, search, compare) userdn="ldap:///anyone";)
aci: (targetattr="carLicense || description || displayName || facsimileTelephoneNumber || homePhone || homePostalAddress || initials || jpegPhoto || labeledURI || mail || mobile || pager || photo || postOfficeBox || postalAddress || postalCode || preferredDeliveryMethod || preferredLanguage || registeredAddress || roomNumber || secretary || seeAlso || st || street || telephoneNumber || telexNumber || title || userCertificate || userPassword || userSMIMECertificate || x500UniqueIdentifier")(version 3.0; acl "Enable self write for common attributes"; allow (write) userdn="ldap:///self";)
aci: (targetattr ="*")(version 3.0;acl "Directory Administrators Group";allow (all) (groupdn = "ldap:///cn=Directory Administrators, dc=example,dc=org");)

dn: uid=user-regular1,ou=Users,dc=example,dc=org
changetype: add
uid: user-regular1
userPassword: SSHA_PASSWORD
objectClass: posixaccount
objectClass: inetOrgPerson
objectClass: person
objectClass: inetUser
objectClass: organizationalPerson
uidNumber: 1200
gidNumber: 500
givenName: User
mail: user-regular1@example.org
sn: Regular1
homeDirectory: /home/regular1
cn: User Regular1

/usr/sbin/slappasswd -h {SSHA} -s <USER_PASSWORD>

/usr/bin/pwdhash -s SSHA <USER_PASSWORD>

dn: uid=user-regular2,ou=Users,dc=example,dc=org
changetype: add
uid: user-regular2
userPassword: SSHA_PASSWORD
objectClass: posixaccount
objectClass: inetOrgPerson
objectClass: person
objectClass: inetUser
objectClass: organizationalPerson
uidNumber: 1300
gidNumber: 500
givenName: User
mail: user-regular2@example.org
sn: Regular1
homeDirectory: /home/regular2
cn: User Regular2

/usr/sbin/slappasswd -h {SSHA} -s <USER_PASSWORD>

/usr/bin/pwdhash -s SSHA <USER_PASSWORD>

dn: uid=user-admin,ou=Users,dc=example,dc=org
changetype: add
uid: user-admin
userPassword: SSHA_PASSWORD
objectClass: posixaccount
objectClass: inetOrgPerson
objectClass: person
objectClass: inetUser
objectClass: organizationalPerson
uidNumber: 1000
gidNumber: 100
givenName: User
mail: user-admin@example.org
sn: Admin
homeDirectory: /home/admin
cn: User Admin

/usr/sbin/slappasswd -h {SSHA} -s <USER_PASSWORD>

/usr/bin/pwdhash -s SSHA <USER_PASSWORD>

dn: cn=k8s-users,ou=Groups,dc=example,dc=org
changetype: add
gidNumber: 500
objectClass: groupOfNames
objectClass: posixGroup
cn: k8s-users
ou: Groups
memberUid: user-regular1
memberUid: user-regular2

dn: cn=k8s-admins,ou=Groups,dc=example,dc=org
changetype: add
gidNumber: 100
objectClass: groupOfNames
objectClass: posixGroup
cn: k8s-admins
ou: Groups
memberUid: user-admin

apiVersion: v1
kind: ConfigMap
metadata:
  name: oidc-dex-config
  namespace: kube-system
data:
  config.yaml: |
    connectors:
    - type: ldap
      # Required field for connector id.
      id: 389ds
      # Required field for connector name.
      name: 389ds
      config:
        # Host and optional port of the LDAP server in the form "host:port".
        # If the port is not supplied, it will be guessed based on "insecureNoSSL",
        # and "startTLS" flags. 389 for insecure or StartTLS connections, 636
        # otherwise.
        host: ldap.example.org:636

        # The following field is required if the LDAP host is not using TLS (port 389).
        # Because this option inherently leaks passwords to anyone on the same network
        # as dex, THIS OPTION MAY BE REMOVED WITHOUT WARNING IN A FUTURE RELEASE.
        #
        # insecureNoSSL: true

        # If a custom certificate isn't provide, this option can be used to turn on
        # TLS certificate checks. As noted, it is insecure and shouldn't be used outside
        # of explorative phases.
        #
        insecureSkipVerify: true

        # When connecting to the server, connect using the ldap:// protocol then issue
        # a StartTLS command. If unspecified, connections will use the ldaps:// protocol
        #
        # startTLS: true

        # Path to a trusted root certificate file. Default: use the host's root CA.
        # rootCA: /etc/dex/pki/ca.crt

        # A raw certificate file can also be provided inline.
        rootCAData: <BASE64_ENCODED_PEM_FILE>

        # The DN and password for an application service account. The connector uses
        # these credentials to search for users and groups. Not required if the LDAP
        # server provides access for anonymous auth.
        # Please note that if the bind password contains a `$`, it has to be saved in an
        # environment variable which should be given as the value to `bindPW`.
        bindDN: cn=Directory Manager
        bindPW: <BIND_DN_PASSWORD>

        # The attribute to display in the provided password prompt. If unset, will
        # display "Username"
        usernamePrompt: Email Address

        # User search maps a username and password entered by a user to a LDAP entry.
        userSearch:
          # BaseDN to start the search from. It will translate to the query
          # "(&(objectClass=person)(mail=<USERNAME>))".
          baseDN: ou=Users,dc=example,dc=org
          # Optional filter to apply when searching the directory.
          filter: "(objectClass=person)"

          # username attribute used for comparing user entries. This will be translated
          # and combined with the other filter as "(<attr>=<USERNAME>)".
          username: mail
          # The following three fields are direct mappings of attributes on the user entry.
          # String representation of the user.
          idAttr: DN
          # Required. Attribute to map to Email.
          emailAttr: mail
          # Maps to display name of users. No default value.
          nameAttr: cn

        # Group search queries for groups given a user entry.
        groupSearch:
          # BaseDN to start the search from. It will translate to the query
          # "(&(objectClass=group)(member=<USER_UID>))".
          baseDN: ou=Groups,dc=example,dc=org
          # Optional filter to apply when searching the directory.
          filter: "(objectClass=groupOfNames)"

          # Following two fields are used to match a user to a group. It adds an additional
          # requirement to the filter that an attribute in the group must match the user's
          # attribute value.
          userAttr: uid
          groupAttr: memberUid

          # Represents group name.
          nameAttr: cn

dn: cn=user-regular1,ou=Users,dc=example,dc=org
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: user-regular1
sn: Regular1
givenName: User
distinguishedName: cn=user-regular1,ou=Users,dc=example,dc=org
displayName: User Regular1
memberOf: cn=Domain Users,ou=Users,dc=example,dc=org
memberOf: cn=k8s-users,ou=Groups,dc=example,dc=org
name: user-regular1
sAMAccountName: user-regular1
objectCategory: cn=Person,cn=Schema,cn=Configuration,dc=example,dc=org
mail: user-regular1@example.org

dn: cn=user-regular2,ou=Users,dc=example,dc=org
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: user-regular2
sn: Regular2
givenName: User
distinguishedName: cn=user-regular2,ou=Users,dc=example,dc=org
displayName: User Regular2
memberOf: cn=Domain Users,ou=Users,dc=example,dc=org
memberOf: cn=k8s-users,ou=Groups,dc=example,dc=org
name: user-regular2
sAMAccountName: user-regular2
objectCategory: cn=Person,cn=Schema,cn=Configuration,dc=example,dc=org
mail: user-regular2@example.org

dn: cn=user-bind,ou=Users,dc=example,dc=org
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: user-bind
sn: Bind
givenName: User
distinguishedName: cn=user-bind,ou=Users,dc=example,dc=org
displayName: User Bind
memberOf: cn=Domain Users,ou=Users,dc=example,dc=org
name: user-bind
sAMAccountName: user-bind
objectCategory: cn=Person,cn=Schema,cn=Configuration,dc=example,dc=org
mail: user-bind@example.org

dn: cn=user-admin,ou=Users,dc=example,dc=org
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: user-admin
sn: Admin
givenName: User
distinguishedName: cn=user-admin,ou=Users,dc=example,dc=org
displayName: User Admin
memberOf: cn=Domain Users,ou=Users,dc=example,dc=org
memberOf: cn=k8s-admins,ou=Groups,dc=example,dc=org
name: user-admin
sAMAccountName: user-admin
objectCategory: cn=Person,cn=Schema,cn=Configuration,dc=example,dc=org
mail: user-admin@example.org

dn: cn=k8s-users,ou=Groups,dc=example,dc=org
objectClass: top
objectClass: group
cn: k8s-users
member: cn=user-regular1,ou=Users,dc=example,dc=org
member: cn=user-regular2,ou=Users,dc=example,dc=org
distinguishedName: cn=k8s-users,ou=Groups,dc=example,dc=org
name: k8s-users
sAMAccountName: k8s-users
objectCategory: cn=Group,cn=Schema,cn=Configuration,dc=example,dc=org

dn: cn=k8s-admins,ou=Groups,dc=example,dc=org
objectClass: top
objectClass: group
cn: k8s-admins
member: cn=user-admin,ou=Users,dc=example,dc=org
distinguishedName: cn=k8s-admins,ou=Groups,dc=example,dc=org
name: k8s-admins
sAMAccountName: k8s-admins
objectCategory: cn=Group,cn=Schema,cn=Configuration,dc=example,dc=org

connectors:
- type: ldap
  # Required field for connector id.
  id: AD
  # Required field for connector name.
  name: AD
  config:
    # Host and optional port of the LDAP server in the form "host:port".
    # If the port is not supplied, it will be guessed based on "insecureNoSSL",
    # and "startTLS" flags. 389 for insecure or StartTLS connections, 636
    # otherwise.
    host: ad.example.org:636

    # Following field is required if the LDAP host is not using TLS (port 389).
    # Because this option inherently leaks passwords to anyone on the same network
    # as dex, THIS OPTION MAY BE REMOVED WITHOUT WARNING IN A FUTURE RELEASE.
    #
    # insecureNoSSL: true

    # If a custom certificate isn't provide, this option can be used to turn on
    # TLS certificate checks. As noted, it is insecure and shouldn't be used outside
    # of explorative phases.
    #
    # insecureSkipVerify: true

    # When connecting to the server, connect using the ldap:// protocol then issue
    # a StartTLS command. If unspecified, connections will use the ldaps:// protocol
    #
    # startTLS: true

    # Path to a trusted root certificate file. Default: use the host's root CA.
    # rootCA: /etc/dex/ldap.ca

    # A raw certificate file can also be provided inline.
    rootCAData: <BASE_64_ENCODED_PEM_FILE>

    # The DN and password for an application service account. The connector uses
    # these credentials to search for users and groups. Not required if the LDAP
    # server provides access for anonymous auth.
    # Please note that if the bind password contains a `$`, it has to be saved in an
    # environment variable which should be given as the value to `bindPW`.
    bindDN: cn=user-admin,ou=Users,dc=example,dc=org
    bindPW: <BIND_DN_PASSWORD>

    # The attribute to display in the provided password prompt. If unset, will
    # display "Username"
    usernamePrompt: Email Address

    # User search maps a username and password entered by a user to a LDAP entry.
    userSearch:
      # BaseDN to start the search from. It will translate to the query
      # "(&(objectClass=person)(mail=<USERNAME>))".
      baseDN: ou=Users,dc=example,dc=org
      # Optional filter to apply when searching the directory.
      filter: "(objectClass=person)"

      # username attribute used for comparing user entries. This will be translated
      # and combined with the other filter as "(<attr>=<USERNAME>)".
      username: mail
      # The following three fields are direct mappings of attributes on the user entry.
      # String representation of the user.
      idAttr: distinguishedName
      # Required. Attribute to map to Email.
      emailAttr: mail
      # Maps to display name of users. No default value.
      nameAttr: sAMAccountName

    # Group search queries for groups given a user entry.
    groupSearch:
      # BaseDN to start the search from. It will translate to the query
      # "(&(objectClass=group)(member=<USER_UID>))".
      baseDN: ou=Groups,dc=example,dc=org
      # Optional filter to apply when searching the directory.
      filter: "(objectClass=group)"

      # Following two fields are used to match a user to a group. It adds an additional
      # requirement to the filter that an attribute in the group must match the user's
      # attribute value.
      userAttr: distinguishedName
      groupAttr: member

      # Represents group name.
      nameAttr: sAMAccountName

cat <ROOT_CA_PEM_FILE> | base64 | awk '{print}' ORS='' && echo

SUSE CaaS Platform 4 currently ships with two default policies:

All pods running the containers for the basic SUSE CaaS Platform software are deployed into the kube-system namespace and run with the "privileged" policy.

All authenticated system users (group system:authenticated) and service accounts in kube-system (system:serviceaccounts:kube-system) have a RoleBinding (suse:caasp:psp:privileged) to run pods using the privileged policy in the kube-system namespace.

Any other pods launched in any other namespace are, by default, deployed in unprivileged mode.

Important

You must configure RBAC rules and PodSecurityPolicy to provide proper functionality and security.

The policy definitions are embedded in the cluster bootstrap manifest (GitHub).

During the bootstrap with skuba, the policy files will be stored on your workstation in the cluster definition folder under addons/psp/base. These policy files will be installed automatically for all cluster nodes.

The file names of the files created are:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: suse.caasp.psp.unprivileged
  annotations:
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
spec:
  # Privileged
  privileged: false
  # Volumes and File Systems
  volumes:
    # Kubernetes Pseudo Volume Types
    - configMap
    - secret
    - emptyDir
    - downwardAPI
    - projected
    - persistentVolumeClaim
    # Networked Storage
    - nfs
    - rbd
    - cephFS
    - glusterfs
    - fc
    - iscsi
    # Cloud Volumes
    - cinder
    - gcePersistentDisk
    - awsElasticBlockStore
    - azureDisk
    - azureFile
    - vsphereVolume
  allowedHostPaths:
    # Note: We don't allow hostPath volumes above, but set this to a path we
    # control anyway as a belt+braces protection. /dev/null may be a better
    # option, but the implications of pointing this towards a device are
    # unclear.
    - pathPrefix: /opt/kubernetes-hostpath-volumes
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: []
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: false
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: suse:caasp:psp:unprivileged
rules:
  - apiGroups: ['extensions']
    resources: ['podsecuritypolicies']
    verbs: ['use']
    resourceNames: ['suse.caasp.psp.unprivileged']
---
# Allow all users and serviceaccounts to use the unprivileged
# PodSecurityPolicy
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: suse:caasp:psp:default
roleRef:
  kind: ClusterRole
  name: suse:caasp:psp:unprivileged
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:serviceaccounts
- kind: Group
  apiGroup: rbac.authorization.k8s.io
  name: system:authenticated

In order to properly secure and run your Kubernetes workloads you must configure RBAC rules for your desired users create a PodSecurityPolicy adequate for your respective workloads and then link the user accounts to the PodSecurityPolicy using (Cluster)RoleBinding.

https://v1-17.docs.kubernetes.io/docs/concepts/policy/pod-security-policy/

After user authentication and authorization, admission takes place to complete the access control for the Kubernetes API. As the final step in the access control process, admission enhances the security layer by mandating a reasonable security baseline across a specific namespace or the entire cluster. The built-in PodSecurityPolicy admission controller is perhaps the most prominent example of it.

Apart from the security aspect, admission controllers can enforce custom policies to adhere to certain best-practices such as having good labels, annotation, resource limits, or other settings. It is worth noting that instead of only validating the request, admission controllers are also capable of "fixing" a request by mutating it, such as automatically adding resource limits if the user forgets to.

The admission is controlled by admission controllers which may only be configured by the cluster administrator. The admission control process happens in two phases:

Important

If any of the controllers in either phase reject the request, the entire request is rejected immediately and an error is returned to the end-user.

Important

Any modification of this list prior to the creation of the cluster will be overwritten by these default settings.

The ability to add or remove individual admission controllers will be provided with one of the upcoming releases of SUSE CaaS Platform.

The complete list of admission controllers can be found at https://v1-17.docs.kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#what-does-each-admission-controller-do

The default admission controllers enabled in SUSE CaaS Platform are:

Communication is secured with TLS v1.2 using the AES 128 CBC cipher. All certificates are 2048 bit RSA encrypted.

The CA certificate is valid for 3650 days (10 years) by default. Client and server certificates are valid for 365 days (1 year) by default.

Required CAs for SUSE CaaS Platform are stored on all control plane nodes:

The control plane certificates stored in the Kubernetes cluster on control plane nodes:

Warning

If a node was bootstrapped/joined before Kubernetes version 1.17, you have to manually modify the contents of kubelet.conf to point to the automatically rotated kubelet client certificates by replacing client-certificate-data and client-key-data with:

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

The addon certificates stored in the Kubernetes cluster Secret resource:

We use cert-exporter to monitor nodes' on-host certificates and addons' secret certificates. The cert-exporter collects the metrics of certificates expiration periodically (1 hour by default) and exposes them through the /metrics endpoint. Then, the Prometheus server can scrape these metrics from the endpoint periodically.

helm repo add suse https://kubernetes-charts.suse.com
helm install suse/cert-exporter --name ${RELEASE_NAME}

Warning

Please plan carefully when deploying with a custom CA certificate. This certificate can not be reconfigured once deployed and requires a full re-installation of the cluster to replace.

Administrators can provide custom CA certificates (root CAs or intermediate CAs) during cluster deployment and decide which CA components to replace (multiple CA certificates) or if to replace all with a single CA certificate.

After you have run skuba cluster init, go to the my-cluster folder that has been generated, Create a pki folder and put your custom CA certificate into the pki folder.

Note: Extracting Certificate And Key From Combined PEM File

Some PKIs will issue certificates and keys in a combined .pem file. In order to use the contained certificate, you must extract them into separate files using openssl.

1. Extract the certificate:

   openssl x509 -in /path/to/file.pem -out /path/to/file.crt

2. Extract the key:

   openssl rsa -in /path/to/file.pem -out /path/to/file.key

After this process, bootstrap the cluster with skuba node bootstrap.

SUSE CaaS Platform uses oidc-dex and oidc-gangway servers to do authentication and authorization. Administrators might choose to replace these server’s certificates by issuing a trusted CA certificate after cluster deployment. This way, the user does not have to add specific certificates to their trusted keychain.

Warning

The custom trusted CA certificate key is not handled by skuba. Administrators must handle server certificate rotation manually before the certificate expires.

Warning

The oidc-dex and oidc-gangway server certificate and key would be replaced when skuba addon upgrade apply contains dex or gangway addon upgrade. Make sure to reapply your changes after running skuba addon upgrade apply, had you modified the default settings of oidc-dex and oidc-gangway addons.

SUSE CaaS Platform renews the control plane certificates and kubeconfigs automatically in two ways:

Note: Time to rotate the kubelet client and server certificate

The kubelet client and server certificate renews automatically at approximately 70%-90% of the total lifetime of the certificate, the kubelet daemon would use new client and server certificates without downtime.

Note: Kubelet client and server certificate signing flow

The configuration which controls the kubelet daemon to send out the CSR within the Kubernetes cluster or not is controlled by the configuration /var/lib/kubelet/config.yaml. The key rotateCertificates controls the kubelet client certificate; the key serverTLSBootstrap controls the kubelet server certificate.

When the client or server certificate is going to expire, the kubelet daemon sends the kubelet client or server CSR within the Kubernetes cluster. The kube-controller-manager signs the kubelet client CSR with the Kubernetes CA cert/key pair, the kucero signs the kubelet server CSR with the kubelet CA cert/key pair. Then, the kubelet daemon saves the signed certificate under the folder /var/lib/kubelet/pki and updates the client or server certificate symlink points to the latest signed certificate.

The path of kubelet client certificate is /var/lib/kubelet/pki/kubelet-client-current.pem. The path of kubelet server certificate is /var/lib/kubelet/pki/kubelet-server-current.pem.

5.10.8.3 Addon Certificate Rotation #

The addon certificates can be automatically rotated by leveraging the functions of the open-source solutions cert-manager and reloader. cert-manager is for automatically rotating certificates stored in Secrets, and reloader is for watching and reconciling the updated Secrets to execute a rolling upgrade of the affected Deployments or DaemonSet.

1. Install reloader via helm chart:

   helm install \
       --name <RELEASE_NAME> \
       --namespace cert-manager \
        suse/reloader

   Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

   helm install <RELEASE_NAME> \
       --namespace cert-manager \
       --create-namespace \
       suse/reloader

2. Install cert-manager via helm chart:

   helm install \
       --name <RELEASE_NAME> \
       --namespace cert-manager \
       --set global.leaderElection.namespace=cert-manager \
       --set installCRDs=true \
       suse/cert-manager

   Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

   helm install <RELEASE_NAME> \
       --namespace cert-manager \
       --create-namespace \
       --set global.leaderElection.namespace=cert-manager \
       --set installCRDs=true \
       suse/cert-manager

   • Cert-Manager CA Issuer Resource

     The cert-manager CA issuer is a Kubernetes resource that represents a certificate authority (CA), which can generate signed certificates by honoring certificate signing requests (CSR). Each cert-manager certificate resource requires one referenced issuer in the ready state to be able to honor CSR requests.

     

     Note

     An Issuer is a namespaced resource, and it can not issue certificates to the certificate resources in other namespaces.

     If you want to create a single Issuer that can be consumed in multiple namespaces, you should consider creating a ClusterIssuer resource. This is almost identical to the Issuer resource, however, it is cluster-wide so it can be used to issue certificates in all namespaces.

   • Cert-Manager Certificate Resource

     The cert-manager has a custom resource, Certificate, which can be used to define a requested x509 certificate which will be renewed and kept up to date by an Issuer or ClusterIssuer resource.

5.10.8.3.1 Client Certificate Rotation #

Warning

If you are running a cluster using cilium version before 1.6, the cilium data is stored in the ETCD cluster, not the custom resources (CR). `skuba` generates a client certificate to read/write the cilium date to the ETCD cluster and the client certificate will expire after 1 year. Please follow the below steps to use cert-manager to automatically renew the cilium client certificate.

1. Check the SUSE CaaS Platform cilium version before 1.6

   CILIUM_OPERATOR=`kubectl get pod -l name=cilium-operator --namespace kube-system -o jsonpath='{.items[0].metadata.name}'`
   kubectl exec -it ${CILIUM_OPERATOR} --namespace kube-system -- cilium-operator --version

2. To let reloader do an automatic rolling upgrade of the cilium addon DaemonSet, we need to label the addons:

   kubectl annotate --overwrite daemonset/cilium -n kube-system secret.reloader.stakater.com/reload=cilium-secret

3. Upload the ETCD CA cert/key pair to Secret in the kube-system namespace

   kubectl create secret tls etcd-ca --cert=pki/etcd/ca.crt --key=pki/etcd/ca.key -n kube-system

4. Create a Cert-Manager CA Issuer Resource

   Create a CA issuer called etcd-ca that will sign incoming certificate requests based on the CA certificate and private key stored in the secret etcd-ca used to trust newly signed certificates.

   cat << EOF > issuer-etcd-ca.yaml
   apiVersion: cert-manager.io/v1alpha3
   kind: Issuer
   metadata:
     name: etcd-ca
     namespace: kube-system
   spec:
     ca:
       secretName: etcd-ca
   EOF
   
   kubectl create -f issuer-etcd-ca.yaml

5. Create a Cert-Manager Certificate Resource

   Create a certificate resource cilium-etcd-client that will watch and auto-renews the secret cilium-secret if the certificate residual time is less than the renewBefore value.

   cat << EOF > cilium-etcd-client-certificate.yaml
   apiVersion: cert-manager.io/v1alpha3
   kind: Certificate
   metadata:
     name: cilium-etcd-client-cert
     namespace: kube-system
   spec:
     subject:
       organizations:
       - system:masters
     commonName: cilium-etcd-client
     duration: 8760h # 1 year
     renewBefore: 720h # 1 month
     secretName: cilium-secret
     issuerRef:
       name: etcd-ca
       kind: Issuer
       group: cert-manager.io
     isCA: false
     usages:
       - digital signature
       - key encipherment
       - client auth
     keySize: 2048
     keyAlgorithm: rsa
     keyEncoding: pkcs1
   EOF
   
   kubectl create -f cilium-etcd-client-certificate.yaml

5.10.8.3.2 Server Certificates Rotation #

• Prerequisites

  1. To let reloader do an automatic rolling upgrade of the addon Deployments or DaemonSet, we need to label the addons:

     kubectl annotate --overwrite deployment/oidc-dex -n kube-system secret.reloader.stakater.com/reload=oidc-dex-cert
     
     kubectl annotate --overwrite deployment/oidc-gangway -n kube-system secret.reloader.stakater.com/reload=oidc-gangway-cert
     
     kubectl annotate --overwrite deployment/metrics-server -n kube-system secret.reloader.stakater.com/reload=metrics-server-cert

  2. Upload the Kubernetes CA cert/key pair to Secret in the kube-system namespace:

     kubectl create secret tls kubernetes-ca --cert=pki/ca.crt --key=pki/ca.key -n kube-system

     

     Note

     If you want to use a custom trusted CA certificate/key to sign the certificate, upload to the secret resource.

     kubectl create secret tls custom-trusted-ca --cert=<CUSTOM_TRUSTED_CA_CERT> --key=<CUSTOM_TRUSTED_CA_KEY> -n kube-system

• Create a Cert-Manager CA Issuer Resource

  Create a CA issuer called kubernetes-ca that will sign incoming certificate requests based on the CA certificate and private key stored in the secret kubernetes-ca used to trust newly signed certificates.

  cat << EOF > issuer-kubernetes-ca.yaml
  apiVersion: cert-manager.io/v1alpha3
  kind: Issuer
  metadata:
    name: kubernetes-ca 1
    namespace: kube-system
  spec:
    ca:
      secretName: kubernetes-ca 2
  EOF
  
  kubectl create -f issuer-kubernetes-ca.yaml

  1	The issuer name.
  2	The secret reference name.

  

  Note

  If you want to use custom trusted CA certificate/key to sign the certificate, create a custom trusted CA issuer.

  cat << EOF > custom-trusted-kubernetes-ca-issuer.yaml
  apiVersion: cert-manager.io/v1alpha3
  kind: Issuer 1
  metadata:
    name: custom-trusted-kubernetes-ca
    namespace: kube-system
  spec:
    ca:
      secretName: custom-trusted-kubernetes-ca
  EOF
  
  kubectl create -f custom-trusted-kubernetes-ca-issuer.yaml

  1	Issuer or ClusterIssuer.

• Create a Cert-Manager Certificate Resource

  Create a certificate resource that will watch and auto-renews the secret if the certificate residual time is less than the renewBefore value.

  • oidc-dex certificate

    cat << EOF > oidc-dex-certificate.yaml
    apiVersion: cert-manager.io/v1alpha3
    kind: Certificate
    metadata:
      name: oidc-dex-cert
      namespace: kube-system
    spec:
      subject:
        organizations:
        - system:masters
      commonName: oidc-dex
      duration: 8760h # 1 year 1
      renewBefore: 720h # 1 month 2
      # At least one of a DNS Name or IP address is required.
      dnsNames:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 3
      ipAddresses:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 4
      secretName: oidc-dex-cert
      issuerRef:
        name: kubernetes-ca 5
        kind: Issuer 6
        group: cert-manager.io
      isCA: false
      usages:
        - digital signature
        - key encipherment
        - server auth
      keySize: 2048
      keyAlgorithm: rsa
      keyEncoding: pkcs1
    EOF
    
    kubectl create -f oidc-dex-certificate.yaml

    1	Default length of certificate validity, in the format (XhYmZs).
    2	Certificate renewal time before validity expires, in the format (XhYmZs).
    3	DNSNames is a list of subject alt names to be used on the Certificate.
    4	IPAddresses is a list of IP addresses to be used on the Certificate.
    5	The cert-manager issuer name.
    6	Issuer or ClusterIssuer.

    This certificate will tell cert-manager to attempt to use the Issuer named kubernetes-ca to obtain a certificate key pair for the domain list in dnsNames and ipAddresses. If successful, the resulting key and certificate will be stored in a secret named oidc-dex-cert with keys of tls.key and tls.crt respectively.

    The dnsNames and ipAddresses fields specify a list of Subject Alternative Names to be associated with the certificate.

    The referenced Issuer must exist in the same namespace as the Certificate. A Certificate can alternatively reference a ClusterIssuer which is cluster-wide so it can be referenced from any namespace.

    

    Note

    If you want to use a custom trusted CA Issuer/ClusterIssuer, change the value of name under issuerRef to custom-trusted-ca and the value of kind under issuerRef to Issuer/ClusterIssuer.

  • oidc-gangway certificate

    cat << EOF > oidc-gangway-certificate.yaml
    apiVersion: cert-manager.io/v1alpha3
    kind: Certificate
    metadata:
      name: oidc-gangway-cert
      namespace: kube-system
    spec:
      subject:
        organizations:
        - system:masters
      commonName: oidc-gangway
      duration: 8760h # 1 year 1
      renewBefore: 720h # 1 month 2
      # At least one of a DNS Name or IP address is required.
      dnsNames:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 3
      ipAddresses:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 4
      secretName: oidc-gangway-cert
      issuerRef:
        name: kubernetes-ca 5
        kind: Issuer 6
        group: cert-manager.io
      isCA: false
      usages:
        - digital signature
        - key encipherment
        - server auth
      keySize: 2048
      keyAlgorithm: rsa
      keyEncoding: pkcs1
    EOF
    
    kubectl create -f oidc-gangway-certificate.yaml

    1	Default length of certificate validity, in the format (XhYmZs).
    2	Certificate renewal time before validity expires, in the format (XhYmZs).
    3	DNSNames is a list of subject alt names to be used on the Certificate.
    4	IPAddresses is a list of IP addresses to be used on the Certificate.
    5	The cert-manager issuer name.
    6	Issuer or ClusterIssuer.

    

    Note

    If you want to use a custom trusted CA Issuer/ClusterIssuer, change the value of name under issuerRef to custom-trusted-ca and the value of kind under issuerRef to Issuer/ClusterIssuer.

  • metrics-server certificate

    cat << EOF > metrics-server-certificate.yaml
    apiVersion: cert-manager.io/v1alpha3
    kind: Certificate
    metadata:
      name: metrics-server-cert
      namespace: kube-system
    spec:
      subject:
        organizations:
        - system:masters
      commonName: metrics-server.kube-system.svc
      duration: 8760h # 1 year 1
      renewBefore: 720h # 1 month 2
      # At least one of a DNS Name or IP address is required.
      dnsNames:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 3
      ipAddresses:
      - $(cat admin.conf | grep server | awk '{print $2}' | sed 's/https:\/\///g' | sed 's/:6443//g') 4
      secretName: metrics-server-cert
      issuerRef:
        name: kubernetes-ca 5
        kind: Issuer 6
        group: cert-manager.io
      isCA: false
      usages:
        - digital signature
        - key encipherment
        - server auth
      keySize: 2048
      keyAlgorithm: rsa
      keyEncoding: pkcs1
    EOF
    
    kubectl create -f metrics-server-certificate.yaml

    1	Default length of certificate validity, in the format (XhYmZs).
    2	Certificate renewal time before validity expires, in the format (XhYmZs).
    3	DNSNames is a list of subject alt names to be used on the Certificate.
    4	IPAddresses is a list of IP addresses to be used on the Certificate.
    5	The cert-manager issuer name.
    6	Issuer or ClusterIssuer.

Warning: Cert-Manager Known Issue

Once the cert-manager has issued a certificate to the secret, if you change the certificate inside the secret manually, or you manually change the current certificate duration to a value lower than the value renewBefore, the certificate won’t be renewed immediately but will be scheduled to renew near the certificate expiry date.

This is because the cert-manager is not designed to pick up changes you make to the certificate in the secret.

The Kubernetes audit log only collects and stores actions performed on the level of the cluster. This does not include any resulting actions of application services.

In order to successfully use Centralized Logging, you first need to install Helm, and Tiller if using Helm 2. Helm is used to install the log agents and provide custom logging settings.

Refer to Section 3.1.2.1, “Installing Helm”.

You can log the following groups of services. See Section 6.4.4, “Deployment” for more information on how to select and customize the logs.

Centralized Logging is also restricted to the following protocols: UDP, TCP, TCP + TLS, TCP + mTLS.

The two supported syslog message formats are RFC 5424 and RFC 3164.

Note

The Kubernetes cluster metadata is included in the RFC 5424 message.

Example RFC 3164

2019-05-30T09:11:21.968458+00:00 worker1 k8s.system/crio[12080] level=debug msg="Endpoint successfully created" containerID=caa46f14a68e766b877af01442e58731845bb45d8ce1f856553440a02c958b2f eventUUID=e2405f2a-82ba-11e9-9a06-fa163eebdfd6 subsys=cilium-cni

Example RFC 5424

<133>1 2019-05-30T08:28:38.784214+00:00 master0 k8s.pod/kube-system/kube-apiserver-master0/kube-apiserver - - [kube_meta namespace_id="1e030def-81db-11e9-a62b-fa163e1876c9" container_name="kube-apiserver" creation_timestamp="2019-05-29T06:29:31Z" host="master0" namespace_name="kube-system" master_url="https://kubernetes.default.svc.cluster.local:443" pod_id="4aaf10f9-81db-11e9-a62b-fa163e1876c9" pod_name="kube-apiserver-master0"] 2019-05-30T08:28:38.783780355+00:00 stderr F I0530 08:28:38.783710       1 log.go:172] http: TLS handshake error from 172.28.0.19:45888: tls: client offered only unsupported versions: [300]

After you have successfully installed it, use Helm CLI to install log agents on each node, and provide customized settings via specific command options.

The only three mandatory parameters for a successful deployment of Centralized Logging are:

See Section 6.4.6, “Optional settings” for the optional parameters and their default values.

helm repo add suse https://kubernetes-charts.suse.com
helm install suse/log-agent-rsyslog --name <RELEASE_NAME> --namespace kube-system --set server.host=${SERVER_HOST} --set server.port=${SERVER_PORT}

Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

helm repo add suse https://kubernetes-charts.suse.com
helm install <RELEASE_NAME> suse/log-agent-rsyslog --namespace kube-system --set server.host=${SERVER_HOST} --set server.port=${SERVER_PORT}

Note

If not specified otherwise, Helm will install log agents with TCP as the default value for server.protocol.

Warning

Running Rsyslog in the host machine as well as in the Kubernetes cluster with imjournal and imfile input modules enabled may causes issues. Rsyslog can crash. To avoid that, it is possible to disable the imjournal module in the helm chart installation adding the following command line argument:

--set logs.osSystem.enabled=false

After this step, all of the log agents will initialize then start to forward logs from each node to the configured remote Rsyslog server.

helm status <RELEASE_NAME>

helm delete --purge <RELEASE_NAME>

Or if you have selected the Helm 3 alternative also see Section 3.1.2.1, “Installing Helm”:

helm uninstall <RELEASE_NAME>

Centralized Logging supports a configurable buffered queue. This can be used to improve log processing throughput and eliminate possible data loss, for instance after log agents shutdown, restart or in case of an unresponsive remote server. The queue files are located under /var/log/containers/{RELEASE_NAME}-log-agent-rsyslog on every node in the cluster. Queue files remain even after the log agents are deleted.

The buffered queue can be enabled/disabled with the following parameter:

queue.enabled, default value = false

Setting queue.enabled to false means that data will be stored in-memory only. Setting the parameter to true will set the data store to a mixture of in-memory and in-disk. Data will then be stored in memory until the queue is filled up, after which storing is switched to disk. Enabling the queue also automatically saves the queue to disk at service shutdown.

Additional parameters to define queue size and its disk usage are:

queue.size, default value = 50000

This option sets the number of messages allowed for the in-memory queue. This setting affects the Kubernetes cluster logs (kubernetes-control-plane and kubernetes-USER_NAME-space).

queue.maxDiskSpace, default value = 2147483648

This option sets the maximum size allowed for disk storage (in bytes). The storage is divided so that 20 percent of it is for journal logs and 80 percent for the remaining logs.

This document aims to describe monitoring in a Kubernetes cluster.

The monitoring stack consists of a metrics server and a visualization platform.

There will be two different ways of using ingress for accessing the monitoring system.

7.1.3.1 Installation For Subdomains #

This installation example shows how to install and configure Prometheus and Grafana using subdomains such as prometheus.example.com, prometheus-alertmanager.example.com, and grafana.example.com.

Important

In order to provide additional security by using TLS certificates, please make sure you have the Section 5.8, “NGINX Ingress Controller” installed and configured.

If you don’t need TLS, you may use other methods for exposing these web services as native LBaaS in OpenStack, haproxy service or k8s native methods as port-forwarding or NodePort but this is out of scope of this document.

7.1.3.2 Create DNS entries #

In this example, we will use a master node with IP 10.86.4.158 in the case of NodePort service of the Ingress Controller.

Note

You should configure proper DNS names in any production environment. These values are only for example purposes.

1. Configure the DNS server

   monitoring.example.com                      IN  A       10.86.4.158
   prometheus.example.com                      IN  CNAME   monitoring.example.com
   prometheus-alertmanager.example.com         IN  CNAME   monitoring.example.com
   grafana.example.com                         IN  CNAME   monitoring.example.com

2. Configure the management workstation /etc/hosts (optional)

   10.86.4.158 prometheus.example.com prometheus-alertmanager.example.com grafana.example.com

7.1.3.2.1 TLS Certificate #

You must configure your certificates for the components as secrets in the Kubernetes cluster. Get certificates from your certificate authority.

1. Individual certificate

   Single-name TLS certificate protects a single sub-domain, and it means each sub-domain owns its private key. From the security perspective, it is recommended to use individual certificates. However, you have to manage the private key and the certificate rotation separately.

   

   Important: Note Down Secret Names For Configuration

   When you choose to secure each service with an individual certificate, you must repeat the step below for each component and adjust the name for the individual secret each time. Please note down the names of the secrets you have created.

   In this example, the secret name is monitoring-tls.

2. Wildcard certificate

   Wildcard TLS allows you to secure multiple sub-domains with one certificate and it means multiple sub-domains share the same private key. You can then add more sub-domains without having to redeploy the certificate and moreover, save the additional certificate costs.

Refer to Section 5.10.10.1.1, “Trusted Server Certificate” on how to sign the trusted certificate or refer to Section 5.10.10.2.2, “Self-signed Server Certificate” on how to sign the self-signed certificate. The server.conf for DNS.1 is prometheus.example.com and prometheus-alertmanager.example.com grafana.example.com for individual certificates separately. The server.conf for DNS.1 is *.example.com for a wildcard certificate.

Then, import your certificate and key pair into the Kubernetes cluster secret name monitoring-tls. In this example, the certificate and key are monitoring.crt and monitoring.key.

kubectl create -n monitoring secret tls monitoring-tls  \
--key  ./monitoring.key \
--cert ./monitoring.crt

7.1.3.2.2 Prometheus #

1. Create a configuration file prometheus-config-values.yaml

   We need to configure the storage for our deployment. Choose among the options and uncomment the line in the config file. In production environments you must configure persistent storage.

   • Use an existing PersistentVolumeClaim

   • Use a StorageClass (preferred)

   # Alertmanager configuration
   alertmanager:
     enabled: true
     ingress:
       enabled: true
       hosts:
       -  prometheus-alertmanager.example.com
       annotations:
         kubernetes.io/ingress.class: nginx
         nginx.ingress.kubernetes.io/auth-type: basic
         nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
         nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
       tls:
         - hosts:
           - prometheus-alertmanager.example.com
           secretName: monitoring-tls
     persistentVolume:
       enabled: true
       ## Use a StorageClass
       storageClass: my-storage-class
       ## Create a PersistentVolumeClaim of 2Gi
       size: 2Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
   
   ## Alertmanager is configured through alertmanager.yml. This file and any others
   ## listed in alertmanagerFiles will be mounted into the alertmanager pod.
   ## See configuration options https://prometheus.io/docs/alerting/configuration/
   #alertmanagerFiles:
   #  alertmanager.yml:
   
   # Create a specific service account
   serviceAccounts:
     nodeExporter:
       name: prometheus-node-exporter
   
   # Node tolerations for node-exporter scheduling to nodes with taints
   # Allow scheduling of node-exporter on master nodes
   nodeExporter:
     hostNetwork: false
     hostPID: false
     podSecurityPolicy:
       enabled: true
       annotations:
         apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
         apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
         seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
         seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
     tolerations:
       - key: node-role.kubernetes.io/master
         operator: Exists
         effect: NoSchedule
   
   # Disable Pushgateway
   pushgateway:
     enabled: false
   
   # Prometheus configuration
   server:
     ingress:
       enabled: true
       hosts:
       - prometheus.example.com
       annotations:
         kubernetes.io/ingress.class: nginx
         nginx.ingress.kubernetes.io/auth-type: basic
         nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
         nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
       tls:
         - hosts:
           - prometheus.example.com
           secretName: monitoring-tls
     persistentVolume:
       enabled: true
       ## Use a StorageClass
       storageClass: my-storage-class
       ## Create a PersistentVolumeClaim of 8Gi
       size: 8Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
   
   ## Prometheus is configured through prometheus.yml. This file and any others
   ## listed in serverFiles will be mounted into the server pod.
   ## See configuration options
   ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/
   #serverFiles:
   #  prometheus.yml:

2. Add SUSE helm charts repository

   helm repo add suse https://kubernetes-charts.suse.com

3. Deploy SUSE prometheus helm chart and pass our configuration values file.

   helm install --name prometheus suse/prometheus \
   --namespace monitoring \
   --values prometheus-config-values.yaml

   There need to be 3 pods running (3 node-exporter pods because we have 3 nodes).

   kubectl -n monitoring get pod | grep prometheus
   NAME                                             READY     STATUS    RESTARTS   AGE
   prometheus-alertmanager-5487596d54-kcdd6         2/2       Running   0          2m
   prometheus-kube-state-metrics-566669df8c-krblx   1/1       Running   0          2m
   prometheus-node-exporter-jnc5w                   1/1       Running   0          2m
   prometheus-node-exporter-qfwp9                   1/1       Running   0          2m
   prometheus-node-exporter-sc4ls                   1/1       Running   0          2m
   prometheus-server-6488f6c4cd-5n9w8               2/2       Running   0          2m

   There need to be be 2 ingresses configured

   kubectl get ingress -n monitoring
   NAME                      HOSTS                                 ADDRESS   PORTS     AGE
   prometheus-alertmanager   prometheus-alertmanager.example.com             80, 443   87s
   prometheus-server         prometheus.example.com                          80, 443   87s

4. At this stage, the Prometheus Expression browser/API should be accessible, depending on your network configuration

   • NodePort: https://prometheus.example.com:32443

   • External IPs: https://prometheus.example.com

   • LoadBalancer: https://prometheus.example.com

7.1.3.2.3 Alertmanager Configuration Example #

The configuration example sets one "receiver" to get notified by email when one of below conditions is met:

• Node is unschedulable: severity is critical because the node cannot accept new pods

• Node runs out of disk space: severity is critical because the node cannot accept new pods

• Node has memory pressure: severity is warning

• Node has disk pressure: severity is warning

• Certificates is going to expire in 7 days: severity is critical

• Certificates is going to expire in 30 days: severity is warning

• Certificates is going to expire in 3 months: severity is info

  1. Configure alerting receiver in Alertmanager

     The Alertmanager handles alerts sent by Prometheus server, it takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email. It also takes care of silencing and inhibition of alerts.

     Add the alertmanagerFiles section to your Prometheus configuration file prometheus-config-values.yaml.

     For more information on how to configure Alertmanager, refer to Prometheus: Alerting - Configuration.

     alertmanagerFiles:
       alertmanager.yml:
         global:
           # The smarthost and SMTP sender used for mail notifications.
           smtp_from: alertmanager@example.com
           smtp_smarthost: smtp.example.com:587
           smtp_auth_username: admin@example.com
           smtp_auth_password: <PASSWORD>
           smtp_require_tls: true
     
         route:
           # The labels by which incoming alerts are grouped together.
           group_by: ['node']
     
           # When a new group of alerts is created by an incoming alert, wait at
           # least 'group_wait' to send the initial notification.
           # This way ensures that you get multiple alerts for the same group that start
           # firing shortly after another are batched together on the first
           # notification.
           group_wait: 30s
     
           # When the first notification was sent, wait 'group_interval' to send a batch
           # of new alerts that started firing for that group.
           group_interval: 5m
     
           # If an alert has successfully been sent, wait 'repeat_interval' to
           # resend them.
           repeat_interval: 3h
     
           # A default receiver
           receiver: admin-example
     
         receivers:
         - name: 'admin-example'
           email_configs:
           - to: 'admin@example.com'

  2. Configures alerting rules in Prometheus server

     Replace the serverFiles section of the Prometheus configuration file prometheus-config-values.yaml.

     For more information on how to configure alerts, refer to: Prometheus: Alerting - Notification Template Examples

     serverFiles:
       alerts: {}
       rules:
         groups:
         - name: caasp.node.rules
           rules:
           - alert: NodeIsNotReady
             expr: kube_node_status_condition{condition="Ready",status="false"} == 1 or kube_node_status_condition{condition="Ready",status="unknown"} == 1
             for: 1m
             labels:
               severity: critical
             annotations:
               description: '{{ $labels.node }} is not ready'
           - alert: NodeIsOutOfDisk
             expr: kube_node_status_condition{condition="OutOfDisk",status="true"} == 1
             labels:
               severity: critical
             annotations:
               description: '{{ $labels.node }} has insufficient free disk space'
           - alert: NodeHasDiskPressure
             expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
             labels:
               severity: warning
             annotations:
               description: '{{ $labels.node }} has insufficient available disk space'
           - alert: NodeHasInsufficientMemory
             expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
             labels:
               severity: warning
             annotations:
               description: '{{ $labels.node }} has insufficient available memory'
         - name: caasp.certs.rules
           rules:
           - alert: KubernetesCertificateExpiry3Months
             expr: (cert_exporter_cert_expires_in_seconds / 86400) < 90
             labels:
               severity: info
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 3 months'
           - alert: KubernetesCertificateExpiry30Days
             expr: (cert_exporter_cert_expires_in_seconds / 86400) < 30
             labels:
               severity: warning
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 30 days'
           - alert: KubernetesCertificateExpiry7Days
             expr: (cert_exporter_cert_expires_in_seconds / 86400) < 7
             labels:
               severity: critical
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 7 days'
           - alert: KubeconfigCertificateExpiry3Months
             expr: (cert_exporter_kubeconfig_expires_in_seconds / 86400) < 90
             labels:
               severity: info
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 3 months'
           - alert: KubeconfigCertificateExpiry30Days
             expr: (cert_exporter_kubeconfig_expires_in_seconds / 86400) < 30
             labels:
               severity: warning
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 30 days'
           - alert: KubeconfigCertificateExpiry7Days
             expr: (cert_exporter_kubeconfig_expires_in_seconds / 86400) < 7
             labels:
               severity: critical
             annotations:
               description: 'The cert for {{ $labels.filename }} on {{ $labels.nodename }} node is going to expire in 7 days'
           - alert: AddonCertificateExpiry3Months
             expr: (cert_exporter_secret_expires_in_seconds / 86400) < 90
             labels:
               severity: info
             annotations:
               description: 'The cert for {{ $labels.secret_name }} is going to expire in 3 months'
           - alert: AddonCertificateExpiry30Days
             expr: (cert_exporter_secret_expires_in_seconds / 86400) < 30
             labels:
               severity: warning
             annotations:
               description: 'The cert for {{ $labels.secret_name }} is going to expire in 30 days'
           - alert: AddonCertificateExpiry7Days
             expr: (cert_exporter_secret_expires_in_seconds / 86400) < 7
             labels:
               severity: critical
             annotations:
               description: 'The cert for {{ $labels.secret_name }} is going to expire in 7 days'

  3. To apply the changed configuration, run:

     helm upgrade prometheus suse/prometheus --namespace monitoring --values prometheus-config-values.yaml

  4. You should now be able to see your Alertmanager, depending on your network configuration

• NodePort: https://prometheus-alertmanager.example.com:32443

• External IPs: https://prometheus-alertmanager.example.com

• LoadBalancer: https://prometheus-alertmanager.example.com

7.1.3.2.4 Recording Rules Configuration Example #

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. Querying the precomputed result will then often be much faster than executing the original expression every time it is needed. This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh. Another common use case is federation where precomputed metrics are scraped from one Prometheus instance by another.

For more information on how to configure recording rules, refer to Prometheus:Recording Rules - Configuration.

1. Configuring recording rules

   Add the following group of rules in the serverFiles section of the prometheus-config-values.yaml configuration file.

   serverFiles:
     alerts: {}
     rules:
       groups:
       - name: node-exporter.rules
         rules:
         - expr: count by (instance) (count without (mode) (node_cpu_seconds_total{component="node-exporter"}))
           record: instance:node_num_cpu:sum
         - expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{component="node-exporter",mode="idle"}[5m]))
           record: instance:node_cpu_utilisation:rate5m
         - expr: node_load1{component="node-exporter"} / on (instance) instance:node_num_cpu:sum
           record: instance:node_load1_per_cpu:ratio
         - expr: node_memory_MemAvailable_bytes / on (instance) node_memory_MemTotal_bytes
           record: instance:node_memory_utilisation:ratio
         - expr: rate(node_vmstat_pgmajfault{component="node-exporter"}[5m])
           record: instance:node_vmstat_pgmajfault:rate5m
         - expr: rate(node_disk_io_time_seconds_total{component="node-exporter", device=~"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+"}[5m])
           record: instance_device:node_disk_io_time_seconds:rate5m
         - expr: rate(node_disk_io_time_weighted_seconds_total{component="node-exporter", device=~"nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|dasd.+"}[5m])
           record: instance_device:node_disk_io_time_weighted_seconds:rate5m
         - expr: sum by (instance) (rate(node_network_receive_bytes_total{component="node-exporter", device!="lo"}[5m]))
           record: instance:node_network_receive_bytes_excluding_lo:rate5m
         - expr: sum by (instance) (rate(node_network_transmit_bytes_total{component="node-exporter", device!="lo"}[5m]))
           record: instance:node_network_transmit_bytes_excluding_lo:rate5m
         - expr: sum by (instance) (rate(node_network_receive_drop_total{component="node-exporter", device!="lo"}[5m]))
           record: instance:node_network_receive_drop_excluding_lo:rate5m
         - expr: sum by (instance) (rate(node_network_transmit_drop_total{component="node-exporter", device!="lo"}[5m]))
           record: instance:node_network_transmit_drop_excluding_lo:rate5m

2. To apply the changed configuration, run:

   helm upgrade prometheus suse/prometheus --namespace monitoring --values prometheus-config-values.yaml

3. You should now be able to see your configured rules, depending on your network configuration

   • NodePort: https://prometheus.example.com:32443/rules

   • External IPs: https://prometheus.example.com/rules

   • LoadBalancer: https://prometheus.example.com/rules

7.1.3.2.5 Grafana #

Starting from Grafana 5.0, it is possible to dynamically provision the data sources and dashboards via files. In a Kubernetes cluster, these files are provided via the utilization of ConfigMap, editing a ConfigMap will result by the modification of the configuration without having to delete/recreate the pod.

1. Configure Grafana provisioning

   Create the default datasource configuration file grafana-datasources.yaml which point to our Prometheus server

   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: grafana-datasources
     namespace: monitoring
     labels:
        grafana_datasource: "1"
   data:
     datasource.yaml: |-
       apiVersion: 1
       deleteDatasources:
         - name: Prometheus
           orgId: 1
       datasources:
       - name: Prometheus
         type: prometheus
         url: http://prometheus-server.monitoring.svc.cluster.local:80
         access: proxy
         orgId: 1
         isDefault: true

2. Create the ConfigMap in Kubernetes cluster

   kubectl create -f grafana-datasources.yaml

3. Configure storage for the deployment

   Choose among the options and uncomment the line in the config file. In production environments you must configure persistent storage.

   • Use an existing PersistentVolumeClaim

   • Use a StorageClass (preferred)

     Create a file grafana-config-values.yaml with the appropriate values

     # Configure admin password
     adminPassword: <PASSWORD>
     
     # Ingress configuration
     ingress:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: nginx
       hosts:
         - grafana.example.com
       tls:
         - hosts:
           - grafana.example.com
           secretName: monitoring-tls
     
     # Configure persistent storage
     persistence:
       enabled: true
       accessModes:
         - ReadWriteOnce
       ## Use a StorageClass
       storageClassName: my-storage-class
       ## Create a PersistentVolumeClaim of 10Gi
       size: 10Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
     
     # Enable sidecar for provisioning
     sidecar:
       datasources:
         enabled: true
         label: grafana_datasource
       dashboards:
         enabled: true
         label: grafana_dashboard

4. Add SUSE helm charts repository

   helm repo add suse https://kubernetes-charts.suse.com

5. Deploy SUSE grafana helm chart and pass our configuration values file

   helm install --name grafana suse/grafana \
   --namespace monitoring \
   --values grafana-config-values.yaml

6. The result should be a running Grafana pod

   kubectl -n monitoring get pod | grep grafana
   NAME                                             READY     STATUS    RESTARTS   AGE
   grafana-dbf7ddb7d-fxg6d                          3/3       Running   0          2m

7. At this stage, Grafana should be accessible, depending on your network configuration

   • NodePort: https://grafana.example.com:32443

   • External IPs: https://grafana.example.com

   • LoadBalancer: https://grafana.example.com

8. Now you can add Grafana dashboards.

7.1.3.2.6 Adding Grafana Dashboards #

There are three ways to add dashboards to Grafana:

• Deploy an existing dashboard from Grafana dashboards

  1. Open the deployed Grafana in your browser and log in.

  2. On the home page of Grafana, hover your mousecursor over the + button on the left sidebar and click on the import menuitem.

  3. Select an existing dashboard for your purpose from Grafana dashboards. Copy the URL to the clipboard.

  4. Paste the URL (for example) https://grafana.com/dashboards/3131 into the first input field to import the "Kubernetes All Nodes" Grafana Dashboard. After pasting in the url, the view will change to another form.

  5. Now select the "Prometheus" datasource in the prometheus field and click on the import button.

  6. The browser will redirect you to your newly created dashboard.

• Use our pre-built dashboards to monitor the SUSE CaaS Platform system

  # monitor SUSE CaaS Platform cluster
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-cluster.yaml
  # monitor SUSE CaaS Platform etcd cluster
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-etcd-cluster.yaml
  # monitor SUSE CaaS Platform nodes
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-nodes.yaml
  # monitor SUSE CaaS Platform namespaces
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-namespaces.yaml
  # monitor SUSE CaaS Platform pods
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-pods.yaml
  # monitor SUSE CaaS Platform certificates
  kubectl apply -f https://raw.githubusercontent.com/SUSE/caasp-monitoring/master/grafana-dashboards-caasp-certificates.yaml

• Build your own dashboard Deploy your own dashboard by configuration file containing the dashboard definition.

  1. Create your dashboard definition file as a ConfigMap, for example grafana-dashboards-caasp-cluster.yaml.

     ---
     apiVersion: v1
     kind: ConfigMap
     metadata:
       name: grafana-dashboards-caasp-cluster
       namespace: monitoring
       labels:
          grafana_dashboard: "1"
     data:
       caasp-cluster.json: |-
         {
           "__inputs": [
             {
               "name": "DS_PROMETHEUS",
               "label": "Prometheus",
               "description": "",
               "type": "datasource",
               "pluginId": "prometheus",
               "pluginName": "Prometheus"
             }
           ],
           "__requires": [
             {
               "type": "grafana",
     [...]
     continues with definition of dashboard JSON
     [...]

  2. Apply the ConfigMap to the cluster.

     kubectl apply -f grafana-dashboards-caasp-cluster.yaml

7.1.3.3 Installation For Subpaths #

This installation example shows how to install and configure Prometheus and Grafana using subpaths such as example.com/prometheus, example.com/alertmanager, and example.com/grafana.

Important

Overlapped instructions from subdomains will be omitted. Refer to the instruction from subdomains.

7.1.3.4 Create DNS entries #

In this example, we will use a master node with IP 10.86.4.158 in the case of NodePort service of the Ingress Controller.

Note

You should configure proper DNS names in any production environment. These values are only for example purposes.

1. Configure the DNS server

   example.com                      IN  A       10.86.4.158

2. Configure the management workstation /etc/hosts (optional)

   10.86.4.158 example.com

7.1.3.4.1 TLS Certificate #

You must configure your certificates for the components as secrets in the Kubernetes cluster. Get certificates from your certificate authority.

Refer to Section 5.10.10.1.1, “Trusted Server Certificate” on how to sign the trusted certificate or refer to Section 5.10.10.2.2, “Self-signed Server Certificate” on how to sign the self-signed certificate. The server.conf for DNS.1 is example.com.

Then, import your certificate and key pair into the Kubernetes cluster secret name monitoring-tls. In this example, the certificate and key are monitoring.crt and monitoring.key.

kubectl create -n monitoring secret tls monitoring-tls  \
--key  ./monitoring.key \
--cert ./monitoring.crt

7.1.3.4.2 Prometheus #

1. Create a configuration file prometheus-config-values.yaml

   We need to configure the storage for our deployment. Choose among the options and uncomment the line in the config file. In production environments you must configure persistent storage.

   • Use an existing PersistentVolumeClaim

   • Use a StorageClass (preferred)

   • Add the external URL to baseURL at which the server can be accessed. The baseURL depends on your network configuration.

     • NodePort: https://example.com:32443/prometheus and https://example.com:32443/alertmanager

     • External IPs: https://example.com/prometheus and https://example.com/alertmanager

     • LoadBalancer: https://example.com/prometheus and https://example.com/alertmanager

   # Alertmanager configuration
   alertmanager:
     enabled: true
     baseURL: https://example.com:32443/alertmanager
     prefixURL: /alertmanager
     ingress:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: nginx
         nginx.ingress.kubernetes.io/auth-type: basic
         nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
         nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
       hosts:
         - example.com/alertmanager
       tls:
         - secretName: monitoring-tls
           hosts:
           - example.com
     persistentVolume:
       enabled: true
       ## Use a StorageClass
       storageClass: my-storage-class
       ## Create a PersistentVolumeClaim of 2Gi
       size: 2Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
   
   ## Alertmanager is configured through alertmanager.yml. This file and any others
   ## listed in alertmanagerFiles will be mounted into the alertmanager pod.
   ## See configuration options https://prometheus.io/docs/alerting/configuration/
   #alertmanagerFiles:
   #  alertmanager.yml:
   
   # Create a specific service account
   serviceAccounts:
     nodeExporter:
       name: prometheus-node-exporter
   
   # Node tolerations for node-exporter scheduling to nodes with taints
   # Allow scheduling of node-exporter on master nodes
   nodeExporter:
     hostNetwork: false
     hostPID: false
     podSecurityPolicy:
       enabled: true
       annotations:
         apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
         apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
         seccomp.security.alpha.kubernetes.io/allowedProfileNames: runtime/default
         seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
     tolerations:
       - key: node-role.kubernetes.io/master
         operator: Exists
         effect: NoSchedule
   
   # Disable Pushgateway
   pushgateway:
     enabled: false
   
   # Prometheus configuration
   server:
     baseURL: https://example.com:32443/prometheus
     prefixURL: /prometheus
     ingress:
       enabled: true
       annotations:
         kubernetes.io/ingress.class: nginx
         nginx.ingress.kubernetes.io/auth-type: basic
         nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
         nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
       hosts:
         - example.com/prometheus
       tls:
         - secretName: monitoring-tls
           hosts:
           - example.com
     persistentVolume:
       enabled: true
       ## Use a StorageClass
       storageClass: my-storage-class
       ## Create a PersistentVolumeClaim of 8Gi
       size: 8Gi
       ## Use an existing PersistentVolumeClaim (my-pvc)
       #existingClaim: my-pvc
   
   ## Prometheus is configured through prometheus.yml. This file and any others
   ## listed in serverFiles will be mounted into the server pod.
   ## See configuration options
   ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/
   #serverFiles:
   #  prometheus.yml:

2. Add SUSE helm charts repository

   helm repo add suse https://kubernetes-charts.suse.com

3. Deploy SUSE prometheus helm chart and pass our configuration values file.

   helm install --name prometheus suse/prometheus \
   --namespace monitoring \
   --values prometheus-config-values.yaml

   There need to be 3 pods running (3 node-exporter pods because we have 3 nodes).

   kubectl -n monitoring get pod | grep prometheus
   NAME                                             READY     STATUS    RESTARTS   AGE
   prometheus-alertmanager-5487596d54-kcdd6         2/2       Running   0          2m
   prometheus-kube-state-metrics-566669df8c-krblx   1/1       Running   0          2m
   prometheus-node-exporter-jnc5w                   1/1       Running   0          2m
   prometheus-node-exporter-qfwp9                   1/1       Running   0          2m
   prometheus-node-exporter-sc4ls                   1/1       Running   0          2m
   prometheus-server-6488f6c4cd-5n9w8               2/2       Running   0          2m

7.1.3.4.3 Alertmanager Configuration Example #

Refer to Section 7.1.3.2.3, “Alertmanager Configuration Example”

7.1.3.4.4 Recording Rules Configuration Example #

Refer to Section 7.1.3.2.4, “Recording Rules Configuration Example”

7.1.3.4.5 Grafana #

Starting from Grafana 5.0, it is possible to dynamically provision the data sources and dashboards via files. In Kubernetes cluster, these files are provided via the utilization of ConfigMap, editing a ConfigMap will result by the modification of the configuration without having to delete/recreate the pod.

1. Configure Grafana provisioning

   Create the default datasource configuration file grafana-datasources.yaml which point to our Prometheus server

   ---
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: grafana-datasources
     namespace: monitoring
     labels:
        grafana_datasource: "1"
   data:
     datasource.yaml: |-
       apiVersion: 1
       deleteDatasources:
         - name: Prometheus
           orgId: 1
       datasources:
       - name: Prometheus
         type: prometheus
         url: http://prometheus-server.monitoring.svc.cluster.local:80
         access: proxy
         orgId: 1
         isDefault: true

2. Create the ConfigMap in Kubernetes cluster

   kubectl create -f grafana-datasources.yaml

3. Configure storage for the deployment

   Choose among the options and uncomment the line in the config file. In production environments you must configure persistent storage.

   • Use an existing PersistentVolumeClaim

   • Use a StorageClass (preferred)

   • Add the external URL to root_url at which the server can be accessed. The root_url depends on your network configuration.

     • NodePort: https://example.com:32443/grafana

     • External IPs: https://example.com/grafana

     • LoadBalancer: https://example.com/grafana

   Create a file grafana-config-values.yaml with the appropriate values

   +

   # Configure admin password
   adminPassword: <PASSWORD>
   
   # Ingress configuration
   ingress:
     enabled: true
     annotations:
       kubernetes.io/ingress.class: nginx
       nginx.ingress.kubernetes.io/rewrite-target: /
     hosts:
       - example.com
     path: /grafana
     tls:
       - secretName: monitoring-tls
         hosts:
         - example.com
   
   # subpath for grafana
   grafana.ini:
     server:
       root_url: https://example.com:32443/grafana
   
   # Configure persistent storage
   persistence:
     enabled: true
     accessModes:
       - ReadWriteOnce
     ## Use a StorageClass
     storageClassName: my-storage-class
     ## Create a PersistentVolumeClaim of 10Gi
     size: 10Gi
     ## Use an existing PersistentVolumeClaim (my-pvc)
     #existingClaim: my-pvc
   
   # Enable sidecar for provisioning
   sidecar:
     datasources:
       enabled: true
       label: grafana_datasource
     dashboards:
       enabled: true
       label: grafana_dashboard

4. Add SUSE helm charts repository

   helm repo add suse https://kubernetes-charts.suse.com

5. Deploy SUSE grafana helm chart and pass our configuration values file

   helm install --name grafana suse/grafana \
   --namespace monitoring \
   --values grafana-config-values.yaml

6. The result should be a running Grafana pod

   kubectl -n monitoring get pod | grep grafana
   NAME                                             READY     STATUS    RESTARTS   AGE
   grafana-dbf7ddb7d-fxg6d                          3/3       Running   0          2m

7. Access Prometheus, Alertmanager, and Grafana

   At this stage, the Prometheus Expression browser/API, Alertmanager, and Grafana should be accessible, depending on your network configuration

   • Prometheus Expression browser/API

     • NodePort: https://example.com:32443/prometheus

     • External IPs: https://example.com/prometheus

     • LoadBalancer: https://example.com/prometheus

   • Alertmanager

     • NodePort: https://example.com:32443/alertmanager

     • External IPs: https://example.com/alertmanager

     • LoadBalancer: https://example.com/alertmanager

   • Grafana

     • NodePort: https://example.com:32443/grafana

     • External IPs: https://example.com/grafana

     • LoadBalancer: https://example.com/grafana

8. Now you can add the Grafana dashboards.

7.1.3.4.6 Adding Grafana Dashboards #

Refer to Section 7.1.3.2.6, “Adding Grafana Dashboards”

The basic check if a cluster is working correctly is based on a few criteria:

Note

For further understanding cluster health information, consider reading https://v1-17.docs.kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/

7.2.1.1 Kubernetes master #

All components in Kubernetes cluster expose a /healthz endpoint. The expected (healthy) HTTP response status code is 200.

The minimal services for the master to work properly are:

• kube-apiserver:

  The component that receives your requests from kubectl and from the rest of the Kubernetes components. The URL is https://<CONTROL_PLANE_IP/FQDN>:6443/healthz

  • Local Check

    curl -k -i https://localhost:6443/healthz

  • Remote Check

    curl -k -i https://<CONTROL_PLANE_IP/FQDN>:6443/healthz

• kube-controller-manager:

  The component that contains the control loop, driving current state to the desired state. The URL is http://<CONTROL_PLANE_IP/FQDN>:10252/healthz

  • Local Check

    curl -i http://localhost:10252/healthz

  • Remote Check

    Make sure firewall allows port 10252.

    curl -i http://<CONTROL_PLANE_IP/FQDN>:10252/healthz

• kube-scheduler:

  The component that schedules workloads to nodes. The URL is http://<CONTROL_PLANE_IP/FQDN>:10251/healthz

  • Local Check

    curl -i http://localhost:10251/healthz

  • Remote Check

    Make sure firewall allows port 10251.

    curl -i http://<CONTROL_PLANE_IP/FQDN>:10251/healthz

Note: High-Availability Environments

In a HA environment you can monitor kube-apiserver on https://<LOAD_BALANCER_IP/FQDN>:6443/healthz.

If any one of the master nodes is running correctly, you will receive a valid response.

This does, however, not mean that all master nodes necessarily work correctly. To ensure that all master nodes work properly, the health checks must be repeated individually for each deployed master node.

This endpoint will return a successful HTTP response if the cluster is operational; otherwise it will fail. It will for example check that it can access etcd. This should not be used to infer that the overall cluster health is ideal. It will return a successful response even when only minimal operational cluster health exists.

To probe for full cluster health, you must perform individual health checking for all machines.

This basic node health check consists of two parts. It checks:

curl -i http://localhost:10248/healthz

kubectl get deployments -n kube-system
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
cilium-operator   1/1     1            1           8d
coredns           2/2     2            2           8d
oidc-dex          1/1     1            1           8d
oidc-gangway      1/1     1            1           8d

If the deployed services contain a health endpoint, or if they contain an endpoint that can be used to determine if the service is up, you can use livenessProbes and/or readinessProbes.

Note: Health check endpoints vs. functional endpoints

A proper health check is always preferred if designed correctly.

Despite the fact that any endpoint could potentially be used to infer if your application is up, it is better to have an endpoint specifically for health in your application. Such an endpoint will only respond affirmatively when all your setup code on the server has finished and the application is running in a desired state.

The livenessProbes and readinessProbes share configuration options and probe types.

There are different options for the livenessProbes to check:

We recommend to apply other best practices from system administration to your monitoring and health checking approach. These steps are not specific to SUSE CaaS Platform and are beyond the scope of this document.

It is useful to first find out about the available resources of your cluster.

7.3.1.1 Using Horizontal Pod Autoscaler (HPA) #

You can set the HPA to scale according to various metrics. These include average CPU utilization, average CPU value, average memory utilization and average memory value. The following sections show the recommended configuration for each of the aforementioned options.

7.3.1.1.1 Creating an HPA Using Average CPU Utilization #

The following code is an example of what this type of HPA can look like. You will have to run the code on your admin node or user local machine. Note that you need a kubeconfig file with RBAC permission that allow setting up autoscale rules into your Kubernetes cluster.

# deployment
kubectl autoscale deployment <DEPLOYMENT_NAME> \
    --min=<MIN_REPLICAS_NUMBER> \
    --max=<MAX_REPLICAS_NUMBER> \
    --cpu-percent=<PERCENT>

# replication controller
kubectl autoscale replicationcontrollers <REPLICATIONCONTROLLERS_NAME> \
    --min=<MIN_REPLICAS_NUMBER> \
    --max=<MAX_REPLICAS_NUMBER> \
    --cpu-percent=<PERCENT>

You could for example use the following values:

kubectl autoscale deployment oidc-dex \
    --name=avg-cpu-util \
    --min=1 \
    --max=10 \
    --cpu-percent=50

The example output below shows autoscaling works in case of the oidc-dex deployment. The HPA increases the minimum number of pods to 1 and will increase the pods up to 10, if the average CPU utilization of the pods reaches 50%. For more details about the inner workings of the scaling, refer to The Kubernetes documentation on the horizontal pod autoscale algorithm.

To check the current status of the HPA run:

kubectl get hpa

Example output:

NAME       REFERENCE             TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
oidc-dex   Deployment/oidc-dex   0%/50%          1         10        3          115s

Note

To calculate pod CPU utilization HPA divides the total CPU usage of all containers by the total number of CPU requests:

POD CPU UTILIZATION = TOTAL CPU USAGE OF ALL CONTAINERS / NUMBER OF CPU REQUESTS

For example:

• Container1 requests 0.5 CPU and uses 0 CPU.

• Container2 requests 1 CPU and uses 2 CPU.

The CPU utilization will be (0+2)/(0.5+1)*100 (%)=133 (%)

If a replication controller, deployment, replica set or stateful set does not specify the CPU request, the output of kubectl get hpa TARGETS will be unknown.

7.3.1.1.2 Creating an HPA Using the Average CPU Value #

1. Create a yaml manifest file hpa-avg-cpu-value.yaml with the following content:

   apiVersion: autoscaling/v2beta2
   kind: HorizontalPodAutoscaler
   metadata:
     name: avg-cpu-value 1
     namespace: kube-system 2
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment 3
       name: example 4
     minReplicas: 1 5
     maxReplicas: 10 6
     metrics:
     - type: Resource
       resource:
         name: cpu
         target:
           type: AverageValue
           averageValue: 500Mi 7

   1	Name of the HPA.
   2	Namespace of the HPA.
   3	Specifies the kind of object to scale (a replication controller, deployment, replica set or stateful set).
   4	Specifies the name of the object to scale.
   5	Specifies the minimum number of replicas.
   6	Specifies the maximum number of replicas.
   7	The average value of the requested CPU that each pod uses.

2. Apply the yaml manifest by running:

   kubectl apply -f hpa-avg-cpu-value.yaml

3. Check the current status of the HPA:

   kubectl get hpa
   
   NAME            REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
   avg-cpu-value   Deployment/php-apache   1m/500Mi   1         10        1          39s

7.3.1.1.3 Creating an HPA Using Average Memory Utilization #

1. Create a yaml manifest file hpa-avg-memory-util.yaml with the following content:

   apiVersion: autoscaling/v2beta2
   kind: HorizontalPodAutoscaler
   metadata:
     name: avg-memory-util 1
     namespace: kube-system 2
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment 3
       name: example 4
     minReplicas: 1 5
     maxReplicas: 10 6
     metrics:
     - type: Resource
       resource:
         name: memory
         target:
           type: Utilization
           averageUtilization: 50 7

   1	Name of the HPA.
   2	Namespace of the HPA.
   3	Specifies the kind of object to scale (a replication controller, deployment, replica set or stateful set).
   4	Specifies the name of the object to scale.
   5	Specifies the minimum number of replicas.
   6	Specifies the maximum number of replicas.
   7	The average utilization of the requested memory that each pod uses.

2. Apply the yaml manifest by running:

   kubectl apply -f hpa-avg-memory-util.yaml

3. Check the current status of the HPA:

   kubectl get hpa
   
   NAME              REFERENCE            TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
   avg-memory-util   Deployment/example   5%/50%           1         10        1          4m54s

   

   Note

   HPA calculates pod memory utilization as: total memory usage of all containers / total memory requests. If a deployment or replication controller does not specify the memory request, the ouput of kubectl get hpa TARGETS is <unknown>.

7.3.1.1.4 Creating an HPA Using Average Memory Value #

1. Create a yaml manifest file hpa-avg-memory-value.yaml with the following content:

   apiVersion: autoscaling/v2beta2
   kind: HorizontalPodAutoscaler
   metadata:
     name: avg-memory-value 1
     namespace: kube-system 2
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment 3
       name: example 4
     minReplicas: 1 5
     maxReplicas: 10 6
     metrics:
     - type: Resource
       resource:
         name: memory
         target:
           type: AverageValue
           averageValue: 500Mi 7

   1	Name of the HPA.
   2	Namespace of the HPA.
   3	Specifies the kind of object to scale (a replication controller, deployment, replica set or stateful set).
   4	Specifies the name of the object to scale.
   5	Specifies the minimum number of replicas.
   6	Specifies the maximum number of replicas.
   7	The average value of the requested memory that each pod uses.

2. Apply the yaml manifest by running:

   kubectl apply -f hpa-avg-memory-value.yaml

3. Check the current status of the HPA:

   kubectl get hpa
   
   NAME                     REFERENCE            TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
   avg-memory-value         Deployment/example   11603968/500Mi   1         10        1          6m24s