Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / Analyzing Performance Metrics Using the Performance Co-Pilot Analysis Toolkit

Analyzing Performance Metrics Using the Performance Co-Pilot Analysis Toolkit

Publication Date: 28 Sep 2024
WHAT?

For performance monitoring purposes, SUSE Linux Enterprise Micro provides a container image that enables you to run the Performance Co-Pilot (PCP) analysis toolkit in a container.

WHY?

You want to monitor performance on your system, and this article provides information on how to configure and use the toolkit.

EFFORT

It takes approximately 40 minutes of reading time.

GOAL

You will be able to start the PCP container with a configuration that suits your needs.

REQUIREMENTS
  • A running instance of SLE Micro.

1 Performance Co-Pilot analysis toolkit

The toolkit comprises tools for gathering and processing performance information collected either in real time or from PCP archive logs.

The performance data is collected by performance metrics domain agents and passed to the pmcd daemon. The daemon coordinates the gathering and exporting of performance statistics in response to requests from the PCP monitoring tools. pmlogger is then used to log the metrics. For details, refer to the PCP documentation.

1.1 Getting the PCP container image

The PCP container image is based on the BCI-Init container that utilizes systemd used to manage the PCP services.

You can pull the container image using Podman or from the Cockpit Web management console. To pull the image by using Podman, run the following command:

# podman pull registry.suse.com/suse/pcp:latest

To get the container image using Cockpit, go to Podman containers, click Get new image, and search for pcp. Then select the image from the registry.suse.com for SLE 15 SP4 and download it.

2 Running the PCP container

The following command shows minimal options that you need to use to run a PCP container:

# podman run -d  \
  --systemd always \
  -p HOST_IP:HOST_PORT:CONTAINER_PORT \
  -v HOST_DIR:/var/log/pcp/pmlogger \
  PCP_CONTAINER_IMAGE

where the options have the following meaning:

-d

The container runs in a detached mode without tty.

--systemd always

Runs the container in the systemd mode. All services needed to run in the PCP container are started automatically by systemd in the container.

--privileged

The container runs with extended privileges. Use this option if your system has SELinux enabled, otherwise the collected metrics are incomplete.

-v HOST_DIR:/var/log/pcp/pmlogger

Creates a bind mount so that pmlogger archives are written to the HOST_DIR on the host. By default, pmlogger stores the collected metrics in /var/log/pcp/pmlogger.

PCP_CONTAINER_IMAGE

Is the downloaded PCP container image.

Other useful options of the podman run command follow:

Other options
-p HOST_IP:HOST_PORT:CONTAINER_PORT

Publishes ports of the container by mapping a container port onto a host port. If you do not specify HOST_IP, the ports are mapped on the local host. If you omit the HOST_PORT value, a random port number is used. By default, the pmcd daemon listens on and exposes the PMAPI to receive metrics on the port 44321, so we recommend mapping this port on the same port number on the host. The pmproxy daemon listens on and exposes the REST PMWEBAPI to access metrics on the 44322 port by default, so it is recommended to map this port on the same host port number.

--net host

The container uses the host's network. Use this option to collect metrics from the host's network interfaces.

-e

The option enables you to set the following environment variables:

PCP_SERVICES

Is a comma-separated list of services to start by systemd in the container.

Default services are: pmcd, pmie, pmlogger, pmproxy.

You can use this variable to run a container with a list of services that is different from the default one, for example, only with pmlogger:

# podman run -d \
  --name pmlogger \
  --systemd always \
  -e PCP_SERVICES=pmlogger  \
  -v pcp-archives:/var/log/pcp/pmlogger  \
  registry.suse.com/suse/pcp:latest
HOST_MOUNT

Is a path inside the container to the bind mount of the host's root file system. The default value is not set.

REDIS_SERVERS

Specifies a connection to a Redis server. In a non-clustered setup, provide a comma-separated list of host specs. In a clustered setup, provide any individual cluster host, other hosts in the cluster are discovered automatically. The default value is: localhost:6379.

If you need to use a different configuration than the one provided by the environment variables, proceed as described in Section 3, “Configuring PCP services”.

2.1 Starting the PCP container automatically on boot

After you run the PCP container, you can configure systemd to start the container on boot. To do so, follow the procedure below:

  1. Create a unit file for the container by using the podman generate systemd command:

    # podman generate systemd --name CONTAINER_NAME > /etc/systemd/system/container-CONTAINER_NAME.service

    where CONTAINER_NAME is the name of the PCP container you used when running the container from the container image.

  2. Enable the service in systemd:

    # systemctl enable container-CONTAINER_NAME

3 Configuring PCP services

All services that run inside the PCP container have a default configuration that might not suit your needs. If you need a custom configuration that cannot be covered by the environment variables, create configuration files for the PCP services and pass them to the PCP using a bind mount as follows:

# podman run -d \
  --name CONTAINER_NAME \
  --systemd always \
  -v $HOST_CONFIG:CONTAINER_CONFIG_PATH:z \
  -v HOST_LOGS_PATH:/var/log/pcp/pmlogger  \
  registry.suse.com/suse/pcp:latest

Where:

CONTAINER_NAME

Is an optional container name.

HOST_CONFIG

Is an absolute path to the config you created on the host machine. You can choose any file name you want.

CONTAINER_CONFIG_PATH

Is an absolute path to a particular configuration file inside the container. Each available configuration file is described in the corresponding sections further.

HOST_LOGS_PATH

Is a directory that should be a bind mount to the container logs.

For example, a container called pcp, with the configuration file pmcd on the host machine and the pcp-archives directory for logs on the host machine, is run by the following command:

# podman run -d \
  --name pcp  \
  --systemd always \
  -v $(pwd)/pcp-archives:/var/log/pcp/pmlogger \
  -v $(pwd)/pmcd:/etc/sysconfig/pmcd \
registry.suse.com/suse/pcp:latest

3.1 Custom pmcd daemon configuration

The pmcd daemon configuration is stored in the /etc/sysconfig/pmcd file. The file stores environment variables that modify the behavior of the pmcd daemon.

You can add the following variables to the /etc/sysconfig/pmcd file to configure the pmcd daemon:

PMCD_LOCAL

Defines whether the remote host can connect to the pmcd daemon. If set to 0, remote connections to the daemon are allowed. If set to 1, the daemon listens only on the local host. The default value is 0.

PMCD_MAXPENDING

Defines the maximum count of pending connections to the agent. The default value is 5.

PMCD_ROOT_AGENT

If the pmdaroot is enabled (the value is set to 1), adding a new PDMA does not trigger restarting of other PMDAs. If pmdaroot is not enabled, pmcd will require restarting all PMDAs when a new PMDA is added. The default value is 1.

PMCD_RESTART_AGENTS

If set to 1, the pmcd daemon tries to restart any exited PMDA. Enable this option only if you have enabled pmdaroot, as pmcd itself does not have privileges to restart PMDA.

PMCD_WAIT_TIMEOUT

Defines the maximum time in seconds pmcd can wait to accept a connection. After this time, the connection is reported as failed. The default value is 60.

PCP_NSS_INIT_MODE

Defines the mode in which pmcd initializes the NSS certificate database when secured connections are used. The default value is readonly. You can set the mode to readwrite, but if the initialization fails, the default value is used as a fallback.

An example follows:

      PMCD_LOCAL=0
      PMCD_MAXPENDING=5
      PMCD_ROOT_AGENT=1
      PMCD_RESTART_AGENTS=1
      PMCD_WAIT_TIMEOUT=70
      PCP_NSS_INIT_MODE=readwrite

3.2 Custom pmlogger configuration

The custom configuration for the pmlogger is stored in the following configuration files:

  • /etc/sysconfig/pmlogger

  • /etc/pcp/pmlogger/control.d/local

3.2.1 The /etc/sysconfig/pmlogger file

You can use the following attributes to configure the pmlogger:

PMLOGGER_LOCAL

Defines whether pmlogger allows connections from remote hosts. If set to 1, pmlogger allows connections from a local host only.

PMLOGGER_MAXPENDING

Defines the maximum count of pending connections. The default value is 5.

PMLOGGER_INTERVAL

Defines the default sampling interval pmlogger uses. The default value is 60 s. Keep in mind that this value can be overridden by the pmlogger command line.

PMLOGGER_CHECK_SKIP_LOGCONF

Setting this option to yes disables the regeneration and checking of the pmlogger configuration if the configuration pmlogger comes from pmlogconf. The default behavior is to regenerate configuration files and check for changes every time pmlogger is started.

An example follows:

PMLOGGER_LOCAL=1
PMLOGGER_MAXPENDING=5
PMLOGGER_INTERVAL=10
PMLOGGER_CHECK_SKIP_LOGCONF=yes

3.2.2 The /etc/pcp/pmlogger/control.d/local file

The file /etc/pcp/pmlogger/control.d/local stores specifications of the host, which metrics should be logged, the logging frequency (default is 24 hours), and pmlogger options. For example:

# === VARIABLE ASSIGNMENTS ===
#
# DO NOT REMOVE OR EDIT THE FOLLOWING LINE
$version=1.1

# Uncomment one of the lines below to enable/disable compression behaviour
# that is different to the pmlogger_daily default.
# Value is days before compressing archives, 0 is immediate compression,
# "never" or "forever" suppresses compression.
#
#$PCP_COMPRESSAFTER=0
#$PCP_COMPRESSAFTER=3
#$PCP_COMPRESSAFTER=never

# === LOGGER CONTROL SPECIFICATIONS ===
#
#Host           P?  S?  directory                       args

# local primary logger
LOCALHOSTNAME   y   n   PCP_ARCHIVE_DIR/LOCALHOSTNAME   -r -T24h10m -c config.default -v 100Mb
Note
Note: Defaults point to local host

If you run the pmlogger in a container on a different machine than the one that runs the pmcd (a client), change the following line to point to the client:

# local primary logger
CLIENT_HOSTNAME   y   n   PCP_ARCHIVE_DIR/CLIENT_HOSTNAME   -r -T24h10m -c config.default -v 100Mb

For example, for the slemicro_1 host name, the line should look as follows:

# local primary logger
slemicro_1   y   n   PCP_ARCHIVE_DIR/slemicro_1   -r -T24h10m -c config.default -v 100Mb

4 Managing PCP metrics

4.1 Listing PCP metrics

From within the container, you can use the command pminfo to list metrics. For example, to list all available performance metrics, run:

# pminfo

You can list a group of related metrics by specifying the metrics prefix:

# pminfo METRIC_PREFIX

For example, to list all metrics related to kernel, use:

# pminfo disk

disk.dev.r_await
disk.dm.await
disk.dm.r_await
disk.md.await
disk.md.r_await
...

You can also specify additional strings to narrow down the list of metrics, for example:

# piminfo disk.dev

disk.dev.read
disk.dev.write
disk.dev.total
disk.dev.blkread
disk.dev.blkwrite
disk.dev.blktotal
...

To get online help text of a particular metric, use the -t option followed by the metric, for example:

# pminfo -t kernel.cpu.util.user

kernel.cpu.util.user [percentage of user time across all CPUs, including guest CPU time]

To display a description text of a particular metric, use the -T option followed by the metric, for example:

# pminfo -T kernel.cpu.util.user

Help:
percentage of user time across all CPUs, including guest CPU time

4.2 Checking local metrics

After you start the PCP container, you can verify that metrics are being recorded properly by running the following command inside the container:

# pcp

Performance Co-Pilot configuration on localhost:

 platform: Linux localhost 5.3.18-150300.59.68-default #1 SMP Wed May 4 11:29:09 UTC 2022 (ea30951) x86_64
 hardware: 1 cpu, 1 disk, 1 node, 1726MB RAM
 timezone: UTC
 services: pmcd pmproxy
     pmcd: Version 5.2.2-1, 9 agents, 4 clients
     pmda: root pmcd proc pmproxy xfs linux mmv kvm jbd2
 pmlogger: primary logger: /var/log/pcp/pmlogger/localhost/20220607.09.24
     pmie: primary engine: /var/log/pcp/pmie/localhost/pmie.log

Now check if the logs are written to a proper destination:

# ls PATH_TO_PMLOGGER_LOGS

where PATH_TO_PMLOGGER_LOGS should be /var/log/pcp/pmlogger/localhost/ in this case.

4.3 Recording metrics from remote systems

You can deploy collector containers that collect metrics from different remote systems than the ones where the pmlogger containers are running. Each remote collector system needs the pmcd daemon and a set of pmda. To deploy several collectors with a centralized monitoring system, proceed as follows.

  1. On each system you want to collect metrics from (clients), run a container with the pmcd daemon:

    # podman run -d \
        --name pcp-pmcd \
        --privileged \
        --net host \
        --systemd always \
        -e PCP_SERVICES=pmcd \
        -e HOST_MOUNT=/host \
        -v /:/host:ro,rslave \
        registry.suse.com/suse/pcp:latest
  2. On the monitoring system, create a pmlogger configuration file for each client control.CLIENT with the following content:

    $version=1.1
    
    CLIENT_HOSTNAME n n PCP_ARCHIVE_DIR/CLIENT -N -r -T24h10m -c config.default -v 100Mb

    Keep in mind that the CLIENT_HOSTNAME must be resolvable in DNS. You can use IP addresses or fully qualified domain names (FQDN) instead.

  3. On the monitoring system, create a directory for each client to store the recorded logs:

    # mkdir /root/pcp-archives/CLIENT

    For example, for slemicro_1:

    # mkdir /root/pcp-archives/slemicro_1
  4. On the monitoring system, run a container with pmlogger for each client:

    # podman run -d \
        --name pcp-pmlogger-CLIENT \
        --systemd always \
        -e PCP_SERVICES=pmlogger \
        -v /root/pcp-archives/CLIENT:/var/log/pcp/pmlogger:z \
        -v $(pwd)/control.CLIENT:/etc/pcp/pmlogger/control.d/local:z \
        registry.suse.com/suse/pcp:latest

    For example, for a client called slemicro_1:

    # podman run -d \
        --name pcp-pmlogger-slemicro_1 \
        --systemd always \
        -e PCP_SERVICES=pmlogger \
        -v /root/pcp-archives:/var/log/pcp/pmlogger:z \
        -v $(pwd)/control.slemicro_1:/etc/pcp/pmlogger/control.d/local:z \
        registry.suse.com/suse/pcp:latest
    Note
    Note

    The second bind mount points to the configuration file created in Step 2 and replaces the default pmlogger configuration. If you do not create this bind mount, pmlogger uses the default /etc/pcp/pmlogger/control.d/local file and logging from clients fails as the default configuration points to a local host. For details about the configuration file, refer to Section 3.2.2, “The /etc/pcp/pmlogger/control.d/local file”.

  5. To check if the log collection is working properly, run:

    # ls -l pcp-archives/CLIENT/CLIENT

    For example:

    # ls -l pcp-archives/slemicro_1/slemicro_1
    
    total 1076
    -rw-r--r--. 1 systemd-network systemd-network 876372 Jun  8 11:24 20220608.10.58.0
    -rw-r--r--. 1 systemd-network systemd-network    312 Jun  8 11:22 20220608.10.58.index
    -rw-r--r--. 1 systemd-network systemd-network 184486 Jun  8 10:58 20220608.10.58.meta
    -rw-r--r--. 1 systemd-network systemd-network    246 Jun  8 10:58 Latest
    -rw-r--r--. 1 systemd-network systemd-network  24595 Jun  8 10:58 pmlogger.log