Applies to SUSE Linux Enterprise Micro 5.4

10 Monitoring performance #

For performance monitoring purposes, SLE Micro provides a container image that enables you to run the Performance Co-Pilot (PCP) analysis toolkit in a container. The toolkit comprises tools for gathering and processing performance information collected either in real time or from PCP archive logs.

The performance data are collected by performance metrics domain agents and passed to the pmcd daemon. The daemon coordinates the gathering and exporting of performance statistics in response to requests from the PCP monitoring tools. pmlogger is then used to log the metrics. For details, refer to the PCP documentation.

10.1 Getting the PCP container image #

The PCP container image is based on the BCI-Init container that utilizes systemd used to manage the PCP services.

You can pull the container image using podman or from the Cockpit web management console. To pull the image by using podman, run the following command:

# podman pull registry.suse.com/suse/pcp:latest

To get the container image using Cockpit, go to Podman containers, click Get new image, and search for pcp. Then select the image from the registry.suse.com for SLE 15 SP4 and download it.

10.2 Running the PCP container #

The following command shows minimal options that you need to use to run a PCP container:

# podman run -d  \
  --systemd always \
  -p HOST_IP:HOST_PORT:CONTAINER_PORT \
  -v HOST_DIR:/var/log/pcp/pmlogger \
  PCP_CONTAINER_IMAGE

where the options have the following meaning:

-d: The container will run in a detached mode without tty.
--systemd always: Runs the container in the systemd mode. All services needed to run in the PCP container will be started automatically by systemd in the container.
--privileged: The container runs with extended privileges. Use this option if your system has SELinux enabled, otherwise the collected metrics will be incomplete.
-v HOST_DIR:/var/log/pcp/pmlogger: Creates a bind mount so that pmlogger archives are written to the HOST_DIR on the host. By default, pmlogger stores the collected metrics in /var/log/pcp/pmlogger.
PCP_CONTAINER_IMAGE: Is the downloaded PCP container image.

Other useful options of the podman run command follow:

Other options #

-p HOST_IP:HOST_PORT:CONTAINER_PORT

Publishes ports of the container by mapping a container port onto a host port. If you do not specify HOST_IP, the ports will be mapped on the local host. If you omit the HOST_PORT value, a random port number will be used. By default, the pmcd daemon listens and exposes the PMAPI to receive metrics on the port 44321, so it is recommended to map this port on the same port number on the host. The pmproxy daemon listens on and exposes the REST PMWEBAPI to access metrics on the 44322 port by default, so it is recommended to map this port on the same host port number.

--net host

The container uses the host's network. Use this option if you want to collect metrics from the host's network interfaces.

-e

The option enables you to set the following environment variables:

PCP_SERVICES

Is a comma-separated list of services to start by systemd in the container.

Default services are: pmcd, pmie, pmlogger, pmproxy.

You can use this variable, if you want to run a container with a list of services that is different from the default one, for example, only with pmlogger:

# podman run -d \
  --name pmlogger \
  --systemd always \
  -e PCP_SERVICES=pmlogger  \
  -v pcp-archives:/var/log/pcp/pmlogger  \
  registry.suse.com/suse/pcp:latest

HOST_MOUNT

Is a path inside the container to the bind mount of the host's root file system. The default value is not set.

REDIS_SERVERS

Specifies a connection to a Redis server. In a non-clustered setup, provide a comma-separated list of host specs. In a clustered setup, provide any individual cluster host, other hosts in the cluster are discovered automatically. The default value is: localhost:6379.

If you need to use a different configuration to the one provided by the environment variables, proceed as described in Section 10.3, “Configuring PCP services”.

10.3 Configuring PCP services #

All services that run inside the PCP container have a default configuration that might not suit your needs. If you need a custom configuration that cannot be covered by the environment variables described above, create configuration files for the PCP services and pass them to the PCP using a bind mount as follows:

# podman run -d \
  --name CONTAINER_NAME \
  --systemd always \
  -v $HOST_CONFIG:CONTAINER_CONFIG_PATH:z \
  -v HOST_LOGS_PATH:/var/log/pcp/pmlogger  \
  registry.suse.com/suse/pcp:latest

Where:

CONTAINER_NAME: Is an optional container name.
HOST_CONFIG: Is an absolute path to the config you created on the host machine. You can choose any file name you want.
CONTAINER_CONFIG_PATH: Is an absolute path to a particular configuration file inside the container. Each available configuration file is described in the corresponding sections further.
HOST_LOGS_PATH: Is a directory that should be a bind mount to the container logs.

For example, a container called pcp, with the configuration file pmcd on the host machine and the pcp-archives directory for logs on the host machine, is run by the following command:

# podman run -d \
  --name pcp  \
  --systemd always \
  -v $(pwd)/pcp-archives:/var/log/pcp/pmlogger \
  -v $(pwd)/pmcd:/etc/sysconfig/pmcd \
registry.suse.com/suse/pcp:latest

10.3.1 Custom `pmcd` daemon configuration #

The pmcd daemon configuration is stored in the /etc/sysconfig/pmcd file. The file stores environment variables that modify the behavior of the pmcd daemon.

10.3.1.1 The `/etc/sysconfig/pmcd` file #

You can add the following variables to the file to configure the pmcd daemon:

PMCD_LOCAL: Defines whether the remote host can connect to the pmcd daemon. If set to 0, remote connections to the daemon are allowed. If set to 1, the daemon listens only on the local host. The default value is 0.
PMCD_MAXPENDING: Defines the maximum count of pending connections to the agent. The default value is 5.
PMCD_ROOT_AGENT: If the pmdaroot is enabled (the value is set to 1), adding a new PDMA does not trigger restarting of other PMDAs. If pmdaroot is not enabled, pmcd will require to restart all PMDAs when a new PMDA is added. The default value is 1.
PMCD_RESTART_AGENTS: If set to 1, the pmcd daemon tries to restart any exited PMDA. Enable this option only if you have enabled pmdaroot, as pmcd itself does not have privileges to restart PMDA.
PMCD_WAIT_TIMEOUT: Defines the maximum time in seconds pmcd can wait to accept a connection. After this time, the connection is reported as failed. The default value is 60.
PCP_NSS_INIT_MODE: Defines the mode in which pmcd initializes the NSS certificate database when secured connections are used. The default value is readonly. You can set the mode to readwrite, but if the initialization fails, the default value is used as a fallback.

An example follows:

      PMCD_LOCAL=0
      PMCD_MAXPENDING=5
      PMCD_ROOT_AGENT=1
      PMCD_RESTART_AGENTS=1
      PMCD_WAIT_TIMEOUT=70
      PCP_NSS_INIT_MODE=readwrite

10.3.2 Custom `pmlogger` configuration #

The custom configuration for the pmlogger is stored in the following configuration files:

/etc/sysconfig/pmlogger
/etc/pcp/pmlogger/control.d/local

10.3.2.1 The `/etc/sysconfig/pmlogger` file #

You can use the following attributes to configure the pmlogger:

PMLOGGER_LOCAL: Defines whether pmlogger allows connections from remote hosts. If set to 1, pmlogger allows connections from local host only.
PMLOGGER_MAXPENDING: Defines the maximum count of pending connections. The default value is 5.
PMLOGGER_INTERVAL: Defines the default sampling interval pmlogger uses. The default value is 60 s. Keep in mind that this value can be overridden by the pmlogger command line.
PMLOGGER_CHECK_SKIP_LOGCONF: Setting this option to yes disables the regeneration and checking of the pmlogger configuration if the configuration pmlogger comes from pmlogconf. The default behavior is to regenerate configuration files and check for changes every time pmlogger is started.

An example follows:

PMLOGGER_LOCAL=1
PMLOGGER_MAXPENDING=5
PMLOGGER_INTERVAL=10
PMLOGGER_CHECK_SKIP_LOGCONF=yes

10.3.2.2 The `/etc/pcp/pmlogger/control.d/local` file #

The file /etc/pcp/pmlogger/control.d/local stores specifications of the host, which metrics should be logged, the logging frequency (default is 24 hours), and pmlogger options. For example:

# === VARIABLE ASSIGNMENTS ===
#
# DO NOT REMOVE OR EDIT THE FOLLOWING LINE
$version=1.1

# Uncomment one of the lines below to enable/disable compression behaviour
# that is different to the pmlogger_daily default.
# Value is days before compressing archives, 0 is immediate compression,
# "never" or "forever" suppresses compression.
#
#$PCP_COMPRESSAFTER=0 
#$PCP_COMPRESSAFTER=3
#$PCP_COMPRESSAFTER=never
    
# === LOGGER CONTROL SPECIFICATIONS ===
#   
#Host           P?  S?  directory                       args

# local primary logger
LOCALHOSTNAME   y   n   PCP_ARCHIVE_DIR/LOCALHOSTNAME   -r -T24h10m -c config.default -v 100Mb

Note: Defaults point to local host

If you run the pmlogger in a container on a different machine than the one that runs the pmcd (a client), change the following line to point to the client:

# local primary logger
CLIENT_HOSTNAME   y   n   PCP_ARCHIVE_DIR/CLIENT_HOSTNAME   -r -T24h10m -c config.default -v 100Mb

For example, for the slemicro_1 host name, the line should look as follows:

# local primary logger
slemicro_1   y   n   PCP_ARCHIVE_DIR/slemicro_1   -r -T24h10m -c config.default -v 100Mb

10.4 Starting the PCP container automatically on boot #

After you run the PCP container, you can configure systemd to start the container on boot. To do so, follow the procedure below:

Create a unit file for the container by using the podman generate systemd command:
```
# podman generate systemd --name CONTAINER_NAME > /etc/systemd/system/container-CONTAINER_NAME.service
```
where CONTAINER_NAME is the name of the PCP container you used when running the container from the container image.

Enable the service in systemd:

# systemctl enable container-CONTAINER_NAME

10.5 Metrics management #

10.5.1 Listing available performance metrics #

From within the container, you can use the command pminfo to list metrics. For example, to list all available performance metrics, run:

# pminfo

You can list a group of related metrics by specifying the metrics prefix:

# pminfo METRIC_PREFIX

For example, to list all metrics related to kernel, use:

# pminfo disk

disk.dev.r_await
disk.dm.await
disk.dm.r_await
disk.md.await
disk.md.r_await
...

You can also specify additional strings to narrow down the list of metrics, for example:

# piminfo disk.dev

disk.dev.read
disk.dev.write
disk.dev.total
disk.dev.blkread
disk.dev.blkwrite
disk.dev.blktotal
...

To get online help text of a particular metric, use the -t option followed by the metric, for example:

# pminfo -t kernel.cpu.util.user

kernel.cpu.util.user [percentage of user time across all CPUs, including guest CPU time]

To display a description text of a particular metric, use the -T option followed by the metric, for example:

# pminfo -T kernel.cpu.util.user

Help:
percentage of user time across all CPUs, including guest CPU time

10.5.2 Checking local metrics #

After you start the PCP container, you can verify that metrics are being recorded properly by running the following command inside the container:

# pcp

Performance Co-Pilot configuration on localhost:

 platform: Linux localhost 5.3.18-150300.59.68-default #1 SMP Wed May 4 11:29:09 UTC 2022 (ea30951) x86_64
 hardware: 1 cpu, 1 disk, 1 node, 1726MB RAM
 timezone: UTC
 services: pmcd pmproxy
     pmcd: Version 5.2.2-1, 9 agents, 4 clients
     pmda: root pmcd proc pmproxy xfs linux mmv kvm jbd2
 pmlogger: primary logger: /var/log/pcp/pmlogger/localhost/20220607.09.24
     pmie: primary engine: /var/log/pcp/pmie/localhost/pmie.log

Now check if the logs are written to a proper destination:

# ls PATH_TO_PMLOGGER_LOGS

where PATH_TO_PMLOGGER_LOGS should be /var/log/pcp/pmlogger/localhost/ in this case.

10.5.3 Recording metrics from remote systems #

You can deploy collector containers that collect metrics from different remote systems than the ones where the pmlogger containers are running. Each remote collector system needs the pmcd daemon and a set of pmda. To deploy several collectors with a centralized monitoring system, proceed as follows.

On each system you want to collect metrics from (clients), run a container with the pmcd daemon:

# podman run -d \
    --name pcp-pmcd \
    --privileged \
    --net host \
    --systemd always \
    -e PCP_SERVICES=pmcd \
    -e HOST_MOUNT=/host \
    -v /:/host:ro,rslave \
    registry.suse.com/suse/pcp:latest

On the monitoring system, create a pmlogger configuration file for each client control.CLIENT with the following content:
```
$version=1.1
 
CLIENT_HOSTNAME n n PCP_ARCHIVE_DIR/CLIENT -N -r -T24h10m -c config.default -v 100Mb
```
Keep in mind that the CLIENT_HOSTNAME must be resolvable in DNS. You can use IP addresses or fully qualified domain names (FQDN) instead.
On the monitoring system, create a directory for each client to store the recorded logs:
```
# mkdir /root/pcp-archives/CLIENT
```
For example, for slemicro_1:
```
# mkdir /root/pcp-archives/slemicro_1
```
On the monitoring system, run a container with pmlogger for each client:
```
# podman run -d \
    --name pcp-pmlogger-CLIENT \
    --systemd always \
    -e PCP_SERVICES=pmlogger \
    -v /root/pcp-archives/CLIENT:/var/log/pcp/pmlogger:z \
    -v $(pwd)/control.CLIENT:/etc/pcp/pmlogger/control.d/local:z \
    registry.suse.com/suse/pcp:latest
```
For example, for a client called slemicro_1:
```
# podman run -d \
    --name pcp-pmlogger-slemicro_1 \
    --systemd always \
    -e PCP_SERVICES=pmlogger \
    -v /root/pcp-archives:/var/log/pcp/pmlogger:z \
    -v $(pwd)/control.slemicro_1:/etc/pcp/pmlogger/control.d/local:z \
    registry.suse.com/suse/pcp:latest
```
Note
The second bind mount points to the configuration file created in Step 2 and replaces the default pmlogger configuration. If you do not create this bind mount, pmlogger uses the default /etc/pcp/pmlogger/control.d/local file and logging from clients fails as the default configuration points to a local host. For details about the configuration file, refer to Section 10.3.2.2, “The /etc/pcp/pmlogger/control.d/local file”.

To check if the log collection is working properly, run:

# ls -l pcp-archives/CLIENT/CLIENT

For example:

# ls -l pcp-archives/slemicro_1/slemicro_1

total 1076
-rw-r--r--. 1 systemd-network systemd-network 876372 Jun  8 11:24 20220608.10.58.0
-rw-r--r--. 1 systemd-network systemd-network    312 Jun  8 11:22 20220608.10.58.index
-rw-r--r--. 1 systemd-network systemd-network 184486 Jun  8 10:58 20220608.10.58.meta
-rw-r--r--. 1 systemd-network systemd-network    246 Jun  8 10:58 Latest
-rw-r--r--. 1 systemd-network systemd-network  24595 Jun  8 10:58 pmlogger.log