Analyzing Performance Metrics Using the Performance Co-Pilot Analysis Toolkit

The PCP container image is based on the BCI-Init container that utilizes systemd used to manage the PCP services.

You can pull the container image using Podman or from the Cockpit Web management console. To pull the image by using Podman, run the following command:

# podman pull registry.suse.com/suse/pcp:latest

To get the container image using Cockpit, go to Podman containers, click Get new image, and search for pcp. Then select the image from the registry.suse.com for SLE 15 SP4 and download it.

After you run the PCP container, you can configure systemd to start the container on boot. To do so, follow the procedure below:

The pmcd daemon configuration is stored in the /etc/sysconfig/pmcd file. The file stores environment variables that modify the behavior of the pmcd daemon.

You can add the following variables to the /etc/sysconfig/pmcd file to configure the pmcd daemon:

An example follows:

      PMCD_LOCAL=0
      PMCD_MAXPENDING=5
      PMCD_ROOT_AGENT=1
      PMCD_RESTART_AGENTS=1
      PMCD_WAIT_TIMEOUT=70
      PCP_NSS_INIT_MODE=readwrite

The custom configuration for the pmlogger is stored in the following configuration files:

PMLOGGER_LOCAL=1
PMLOGGER_MAXPENDING=5
PMLOGGER_INTERVAL=10
PMLOGGER_CHECK_SKIP_LOGCONF=yes

3.2.2 The /etc/pcp/pmlogger/control.d/local file #

 

The file /etc/pcp/pmlogger/control.d/local stores specifications of the host, which metrics should be logged, the logging frequency (default is 24 hours), and pmlogger options. For example:

# === VARIABLE ASSIGNMENTS ===
#
# DO NOT REMOVE OR EDIT THE FOLLOWING LINE
$version=1.1

# Uncomment one of the lines below to enable/disable compression behaviour
# that is different to the pmlogger_daily default.
# Value is days before compressing archives, 0 is immediate compression,
# "never" or "forever" suppresses compression.
#
#$PCP_COMPRESSAFTER=0
#$PCP_COMPRESSAFTER=3
#$PCP_COMPRESSAFTER=never

# === LOGGER CONTROL SPECIFICATIONS ===
#
#Host           P?  S?  directory                       args

# local primary logger
LOCALHOSTNAME   y   n   PCP_ARCHIVE_DIR/LOCALHOSTNAME   -r -T24h10m -c config.default -v 100Mb

Note: Defaults point to local host

If you run the pmlogger in a container on a different machine than the one that runs the pmcd (a client), change the following line to point to the client:

# local primary logger
CLIENT_HOSTNAME   y   n   PCP_ARCHIVE_DIR/CLIENT_HOSTNAME   -r -T24h10m -c config.default -v 100Mb

For example, for the slemicro_1 host name, the line should look as follows:

# local primary logger
slemicro_1   y   n   PCP_ARCHIVE_DIR/slemicro_1   -r -T24h10m -c config.default -v 100Mb

From within the container, you can use the command pminfo to list metrics. For example, to list all available performance metrics, run:

# pminfo

You can list a group of related metrics by specifying the metrics prefix:

# pminfo METRIC_PREFIX

For example, to list all metrics related to kernel, use:

# pminfo disk

disk.dev.r_await
disk.dm.await
disk.dm.r_await
disk.md.await
disk.md.r_await
...

You can also specify additional strings to narrow down the list of metrics, for example:

# piminfo disk.dev

disk.dev.read
disk.dev.write
disk.dev.total
disk.dev.blkread
disk.dev.blkwrite
disk.dev.blktotal
...

To get online help text of a particular metric, use the -t option followed by the metric, for example:

# pminfo -t kernel.cpu.util.user

kernel.cpu.util.user [percentage of user time across all CPUs, including guest CPU time]

To display a description text of a particular metric, use the -T option followed by the metric, for example:

# pminfo -T kernel.cpu.util.user

Help:
percentage of user time across all CPUs, including guest CPU time

After you start the PCP container, you can verify that metrics are being recorded properly by running the following command inside the container:

# pcp

Performance Co-Pilot configuration on localhost:

 platform: Linux localhost 5.3.18-150300.59.68-default #1 SMP Wed May 4 11:29:09 UTC 2022 (ea30951) x86_64
 hardware: 1 cpu, 1 disk, 1 node, 1726MB RAM
 timezone: UTC
 services: pmcd pmproxy
     pmcd: Version 5.2.2-1, 9 agents, 4 clients
     pmda: root pmcd proc pmproxy xfs linux mmv kvm jbd2
 pmlogger: primary logger: /var/log/pcp/pmlogger/localhost/20220607.09.24
     pmie: primary engine: /var/log/pcp/pmie/localhost/pmie.log

Now check if the logs are written to a proper destination:

# ls PATH_TO_PMLOGGER_LOGS

where PATH_TO_PMLOGGER_LOGS should be /var/log/pcp/pmlogger/localhost/ in this case.

You can deploy collector containers that collect metrics from different remote systems than the ones where the pmlogger containers are running. Each remote collector system needs the pmcd daemon and a set of pmda. To deploy several collectors with a centralized monitoring system, proceed as follows.