Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE CaaS Platform 4.5.2

7 Logging

7.1 Introduction

Logging is ubiquitous throughout SUSE CaaS Platform. Some tools will only print their outputs to the currently running session shell and not create a "log file".

If you need to retain the output of these files you can tee them into a separate file (refer to Section 7.2, “Logging in skuba”).

Many other service components will produce log files or other log info streams. You can collect, store and evaluate these logs via Section 7.4, “Centralized Logging” for use with the Section 8.1, “Monitoring Stack”.

Note
Note

If you are looking for troubleshooting logs please refer to Section 15.3, “Log collection”.

7.2 Logging in skuba

One important part of deploying and maintaining a product is to have reliable logs. Tools like skuba take the approach of printing the output to the standard output directly. This is not just common practice, but it also has the advantage that then the user has more flexibility on how to manage said output.

Thus, whenever throughout this guide we write a skuba command, take into account that the output will be printed into the standard output. If you would also like to have the logs stored somewhere else for later inspection, you can use tools like tee. For example:

skuba node bootstrap --user sles --sudo --target <IP/FQDN> <NODE_NAME> | tee <NODE_NAME>-skuba-node-bootstrap.log

Otherwise, you might want to use other tools to manage the logs for later inspection. The point being that this guide will never consider how to manage these logs because skuba itself does not. It’s up to you to manage these logs in any way you find desirable.

Moreover, skuba has also various levels of log verbosity. This is managed by the -v, --verbosity flag. This flag accepts an integer argument, ranging from 0 to 5, where a higher number means a higher level of verbosity. If you don’t pass any arguments, then 0 is assumed. We recommend using the default argument, since it will already log warnings and errors, among other relevant output, whereas 5 can be a bit overwhelming. Thus, for the above example, we would recommend something like:

skuba node bootstrap -v --user sles --sudo --target <IP/FQDN> <NODE_NAME> | tee <NODE_NAME>-skuba-node-bootstrap.log

Now the <NODE_NAME>-skuba-node-bootstrap.log will have more useful information than without the -v flag. We strongly recommend using this flag in order to get as much useful information as possible from a single run.

7.3 Audit Log

To track actions that have been performed on the cluster, you can enable the Kubernetes audit log during cluster bootstrap or on a running cluster.

This allows the audit logs to be written on the Kubernetes master nodes at /var/log/kube-apiserver/audit.log or the given path.

For more information on the audit log and its contents, see: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

7.3.1 Limitations

The Kubernetes audit log only collects and stores actions performed on the level of the cluster. This does not include any resulting actions of application services.

7.3.2 Enable Auditing During Cluster Bootstrap

  1. Create audit policy file - audit.yaml. Here uses a simple policy for demonstration.

    apiVersion: audit.k8s.io/v1beta1
    kind: Policy
    rules:
      - level: Metadata 1

    1

    The audit level of the event. This sample will log all requests at the Metadata level. For detailed information, refer to: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy

  2. Create audit policy file directory on all master nodes.

    sudo mkdir -p /etc/kubernetes/policies
  3. Copy audit policy file - audit.yaml to /etc/kubernetes/policies/audit.yaml on all master nodes.

  4. Edit kubeadm-init.conf file in skuba init folder to add audit related configurations.

    vi <my_cluster>/kubeadm-init.conf
     ...
    apiServer:
      extraArgs:
        audit-log-path: /var/log/kube-apiserver/audit.log
        audit-policy-file: /etc/kubernetes/policies/audit.yaml 1
        audit-log-maxage: "30" 2
        audit-log-maxsize: "100" 3
        audit-log-maxbackup: "5" 4
        audit-log-format: json 5
      extraVolumes:
      - name: audit-policy
        hostPath: /etc/kubernetes/policies/audit.yaml 6
        mountPath: /etc/kubernetes/policies/audit.yaml 7
        readOnly: true
        pathType: File
      - name: audit-logs
        hostPath: /var/log/kube-apiserver 8
        mountPath: /var/log/kube-apiserver 9
        pathType: DirectoryOrCreate
     ...

    1

    Path to the YAML file that defines the audit policy configuration.

    2

    The maximum number of days to retain old audit log files based on the timestamp encoded in their filename. (Default: 15)

    3

    The maximum size in megabytes of the audit log file before it gets rotated. (Default: 10)

    4

    The maximum number of old audit log files to retain. (Default: 20)

    5

    Format of saved audits. Known formats are "legacy", "json". "legacy" indicates 1-line text format for each event. "json" indicates structured json format.

    6

    The audit policy configuration file path from the host node’s filesystem.

    7

    The audit policy configuration file path on the api-server pod.

    8

    The audit log file directory from the host node’s filesystem.

    9

    The audit log file directory on the api-server pod.

  5. Proceed with Cluster Bootstrap.

  6. If everything is setup correctly, you should be able to see audit logs are written to /var/log/kube-apiserver/audit.log.

7.3.3 Enable Auditing On Running Cluster

Note
Note

The following steps take effect only on the updated master nodes. You need to repeat the following steps on every master node in the cluster.

  1. Create audit policy file - audit.yaml. Here uses a simple policy for demonstration.

    apiVersion: audit.k8s.io/v1beta1
    kind: Policy
    rules:
      - level: Metadata 1

    1

    The audit level of the event. This sample will log all requests at the Metadata level. For detailed information, refer to: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy

  2. Create audit policy file directory on master node.

    sudo mkdir -p /etc/kubernetes/policies
  3. Copy audit policy file - audit.yaml to /etc/kubernetes/policies/audit.yaml on master node.

  4. Edit /etc/kubernetes/manifests/kube-apiserver.yaml.

     ...
    spec:
      containers:
      - command:
        - kube-apiserver
        - --audit-log-path=/var/log/kube-apiserver/audit.log
        - --audit-policy-file=/etc/kubernetes/policies/audit.yaml 1
        - --audit-log-maxage=30 2
        - --audit-log-maxsize=100 3
        - --audit-log-maxbackup=5 4
        - --audit-log-format=json 5
     ...
        volumeMounts:
        - mountPath: /etc/kubernetes/policies/audit.yaml 6
          name: audit-policy
          readOnly: true
        - mountPath: /var/log/kube-apiserver 7
          name: audit-logs
     ...
      volumes:
      - hostPath:
          path: /etc/kubernetes/policies/audit.yaml 8
          type: File
        name: audit-policy
      - hostPath:
          path: /var/log/kube-apiserver 9
          type: DirectoryOrCreate
        name: audit-logs
     ...

    1

    Path to the YAML file that defines the audit policy configuration.

    2

    The maximum number of days to retain old audit log files based on the timestamp encoded in their filename. (Default: 15)

    3

    The maximum size in megabytes of the audit log file before it gets rotated. (Default: 10)

    4

    The maximum number of old audit log files to retain. (Default: 20)

    5

    Format of saved audits. Known formats are "legacy", "json". "legacy" indicates 1-line text format for each event. "json" indicates structured json format.

    6

    The audit policy configuration file path on the api-server pod.

    7

    The audit log file directory on the api-server pod.

    8

    The audit policy configuration file path from the host node’s filesystem.

    9

    The audit log file directory from the host node’s filesystem.

  5. Restart kubelet.

    sudo systemctl restart kubelet
  6. If everything is set up correctly, you should be able to see audit logs being written to /var/log/kube-apiserver/audit.log.

7.3.4 Disable Auditing

Note
Note

The following steps take effect only on the updated master nodes. You need to repeat the following steps on every master node in the cluster.

  1. Remote access to the master node.

ssh sles@<master_node>
  1. Edit /etc/kubernetes/manifests/kube-apiserver.yaml and remove the following lines.

    ...
       - --audit-log-path=/var/log/kube-apiserver/audit.log
       - --audit-policy-file=/etc/kubernetes/policies/audit.yaml
       - --audit-log-maxage=30
       - --audit-log-maxsize=100
       - --audit-log-maxbackup=5
       - --audit-log-format=json
    ...
       - mountPath: /etc/kubernetes/policies/audit.yaml
         name: audit-policy
         readOnly: true
       - mountPath: /var/log/kube-apiserver
         name: audit-logs
    ...
     - hostPath:
         path: /etc/kubernetes/policies/audit.yaml
         type: File
       name: audit-policy
     - hostPath:
         path: /var/log/kube-apiserver
         type: DirectoryOrCreate
       name: audit-logs
  2. Restart kubelet.

    sudo systemctl restart kubelet

7.4 Centralized Logging

Centralized Logging is a means of collecting logs from the SUSE CaaS Platform for centralized management. It forwards system and Kubernetes cluster logs to a specified external logging service, for example, Rsyslog server.

Collecting logs in a central location can be useful for audit or debug purposes or to analyze and visually present data.

7.4.1 Prerequisites

In order to successfully use Centralized Logging, you first need to install Helm. Helm is used to install the log agents and provide custom logging settings.

Refer to Section 3.1.2.1, “Installing Helm”.

7.4.2 Types of Logs

You can log the following groups of services. See Section 7.4.4, “Deployment” for more information on how to select and customize the logs.

Kubernetes System Components
  • Kubelet

  • Cri-o

Kubernetes Control Plane Components
  • API Server

  • Controller Manager

  • Scheduler

  • Cilium

  • Kube-proxy

  • All resources in the kube-system namespaces

Kubernetes Namespaces Pods
  • All namespaces in cluster except kube-system

Kubernetes Audit Log
OS Components
  • Kernel

  • Audit

  • Zypper

  • Network (wicked)

Centralized Logging is also restricted to the following protocols: UDP, TCP, TCP + TLS, TCP + mTLS.

7.4.3 Log Formats

The two supported syslog message formats are RFC 5424 and RFC 3164.

Note
Note

The Kubernetes cluster metadata is included in the RFC 5424 message.

Example RFC 3164

2019-05-30T09:11:21.968458+00:00 worker1 k8s.system/crio[12080] level=debug msg="Endpoint successfully created" containerID=caa46f14a68e766b877af01442e58731845bb45d8ce1f856553440a02c958b2f eventUUID=e2405f2a-82ba-11e9-9a06-fa163eebdfd6 subsys=cilium-cni

Example RFC 5424

<133>1 2019-05-30T08:28:38.784214+00:00 master0 k8s.pod/kube-system/kube-apiserver-master0/kube-apiserver - - [kube_meta namespace_id="1e030def-81db-11e9-a62b-fa163e1876c9" container_name="kube-apiserver" creation_timestamp="2019-05-29T06:29:31Z" host="master0" namespace_name="kube-system" master_url="https://kubernetes.default.svc.cluster.local:443" pod_id="4aaf10f9-81db-11e9-a62b-fa163e1876c9" pod_name="kube-apiserver-master0"] 2019-05-30T08:28:38.783780355+00:00 stderr F I0530 08:28:38.783710       1 log.go:172] http: TLS handshake error from 172.28.0.19:45888: tls: client offered only unsupported versions: [300]

7.4.4 Deployment

After you have successfully installed it, use Helm CLI to install log agents on each node, and provide customized settings via specific command options.

The only three mandatory parameters for a successful deployment of Centralized Logging are:

  • server.host, default value = rsyslog-server.default.svc.cluster.local

  • server.port, default value = 514

  • server.protocol, default value = TCP

See Section 7.4.6, “Optional settings” for the optional parameters and their default values.

  • Running the following will create the minimal working setup:

helm repo add suse https://kubernetes-charts.suse.com
helm install <RELEASE_NAME> suse/log-agent-rsyslog --namespace kube-system --set server.host=${SERVER_HOST} --set server.port=${SERVER_PORT}
Note
Note

If not specified otherwise, Helm will install log agents with TCP as the default value for server.protocol.

Warning
Warning

Currently, Rsyslog can cause a segfault (as described in https://github.com/rsyslog/rsyslog/issues/4200) as result of a conflict when both imjournal and imfile input modules are enabled. To avoid crashing Rsyslog, imfile and imjournal need to be mutually exclusive, which means they need to be configured in the Rsyslog helm chart to one of these:

--set logs.osSystem.enabled=true \ 1
--set logs.kubernetesSystem.enabled=false \ 2
--set logs.kubernetesControlPlane.enabled=false \ 3
--set logs.kubernetesUserNamespaces.enabled=false \ 4

1

Enables imfile

2 3 4

Disables imjournal

--set logs.osSystem.enabled=false \ 1
--set logs.kubernetesSystem.enabled=true \ 2
--set logs.kubernetesControlPlane.enabled=true \ 3
--set logs.kubernetesUserNamespaces.enabled=true \ 4

1

Disables imfile

2 3 4

Enables imjournal

If it is required to send both imjournal and imfile, Rsyslog needs to be installed directly on the host nodes to send the imjournal log and again deployed via the helm chart on the Kubernetes cluster to send imfile logs.

After this step, all of the log agents will initialize then start to forward logs from each node to the configured remote Rsyslog server.

  • To check the installation progress, use the helm status command:

helm status <RELEASE_NAME> --namespace kube-system
  • To uninstall log agents, use the helm delete command:

helm uninstall <RELEASE_NAME> --namespace kube-system

7.4.5 Queuing

Centralized Logging supports a configurable buffered queue. This can be used to improve log processing throughput and eliminate possible data loss, for instance after log agents shutdown, restart or in case of an unresponsive remote server. The queue files are located under /var/log/containers/{RELEASE_NAME}-log-agent-rsyslog on every node in the cluster. Queue files remain even after the log agents are deleted.

The buffered queue can be enabled/disabled with the following parameter:

queue.enabled, default value = false

Setting queue.enabled to false means that data will be stored in-memory only. Setting the parameter to true will set the data store to a mixture of in-memory and in-disk. Data will then be stored in memory until the queue is filled up, after which storing is switched to disk. Enabling the queue also automatically saves the queue to disk at service shutdown.

Additional parameters to define queue size and its disk usage are:

queue.size, default value = 50000

This option sets the number of messages allowed for the in-memory queue. This setting affects the Kubernetes cluster logs (kubernetes-control-plane and kubernetes-USER_NAME-space).

queue.maxDiskSpace, default value = 2147483648

This option sets the maximum size allowed for disk storage (in bytes). The storage is divided so that 20 percent of it is for journal logs and 80 percent for the remaining logs.

7.4.6 Optional settings

Note
Note

Options with empty default values are set as not specified.

ParameterFunctionDefault value

image.repository

specifies image repository to pull from

registry.suse.com/caasp/v4.5/rsyslog

image.tag

specifies image tag to pull

8.39.0

kubernetesPodAnnotationsEnabled

enables kubernetes meta annotations in pod logs

false

kubernetesPodLabelsEnabled

enables kubernetes meta labels in pod logs

false

logs.kubernetesAudit.enabled

enables Kubernetes audit logs

true

logs.kubernetesAudit.logDir

Kubernetes audit log directory

/var/log/kube-apiserver

logs.kubernetesAudit.logFile

Kubernetes audit log filename

audit.log

logs.kubernetesControlPlane.enabled

enables Kubernetes control plane logs

true

logs.kubernetesSystem.enabled

enables Kubernetes system logs (kubelet, crio)

true

logs.kubernetesUserNamespaces.enabled

enables Kubernetes user namespaces logs

false

logs.kubernetesUserNamespaces.exclude

excludes Kubernetes logs for specific namespaces

- ""

logs.osSystem.enabled

enables OS logs (auditd, kernel, wicked, zypper)

true

persistStateInterval

sets interval (number-of-messages) for data state persistency

100

queue.enabled

enables Rsyslog queue

false

queue.maxDiskSpace

sets maximum Rsyslog queue disk space in bytes

2147483648

queue.size

sets Rsyslog queue size in bytes

50000

resources.limits.cpu

sets CPU limits

 

resources.limits.memory

sets memory limits

512 Mi

resources.requests.cpu

sets CPU for requests

100m

resources.requests.memory

sets memory for requests

512 Mi

resumeInterval

specifies time (seconds) after failure before retry is attempted

30

resumeRetryCount

sets number of retries after first failure before the log is discarded. -1 is unlimited

-1

server.tls.clientCert

sets TLS client certificate

 

server.tls.clientKey

sets TLS client key

 

server.tls.enabled

enables TLS

false

server.tls.permittedPeer

sets a list of TLS/fingerprints or TLS/names with permission to access the server

 

server.tls.rootCa

specifies TLS root certificate authority

 
Print this page