10 Monitoring performance #
For performance monitoring purposes, SLE Micro provides a container image that enables you to run the Performance Co-Pilot (PCP) analysis toolkit in a container. The toolkit comprises tools for gathering and processing performance information collected either in real time or from PCP archive logs.
The performance data are collected by performance metrics domain
agents and passed to the pmcd daemon. The
daemon coordinates the gathering and exporting of performance statistics in
response to requests from the PCP monitoring tools.
pmlogger
is then used to log the metrics. For details,
refer to the
PCP
documentation.
10.1 Getting the PCP container image #
The PCP container image is based on the BCI-Init
container that utilizes systemd
used to manage the PCP services.
You can pull the container image using podman or from the Cockpit web management console. To pull the image by using podman, run the following command:
#
podman pull registry.suse.com/suse/pcp:latest
To get the container image using Cockpit, go to pcp. Then select the image from the
registry.suse.com
for SLE 15 SP4 and download it.
10.2 Running the PCP container #
The following command shows minimal options that you need to use to run a PCP container:
#
podman run -d \
--systemd always \
-p HOST_IP:HOST_PORT:CONTAINER_PORT \
-v HOST_DIR:/var/log/pcp/pmlogger \
PCP_CONTAINER_IMAGE
where the options have the following meaning:
-d
The container will run in a detached mode without tty.
--systemd always
Runs the container in the
systemd
mode. All services needed to run in the PCP container will be started automatically bysystemd
in the container.--privileged
The container runs with extended privileges. Use this option if your system has SELinux enabled, otherwise the collected metrics will be incomplete.
-v HOST_DIR:/var/log/pcp/pmlogger
Creates a bind mount so that
pmlogger
archives are written to the HOST_DIR on the host. By default,pmlogger
stores the collected metrics in/var/log/pcp/pmlogger
.- PCP_CONTAINER_IMAGE
Is the downloaded PCP container image.
Other useful options of the podman run
command follow:
-p HOST_IP:HOST_PORT:CONTAINER_PORT
Publishes ports of the container by mapping a container port onto a host port. If you do not specify HOST_IP, the ports will be mapped on the local host. If you omit the HOST_PORT value, a random port number will be used. By default, the
pmcd
daemon listens and exposes the PMAPI to receive metrics on the port 44321, so it is recommended to map this port on the same port number on the host. Thepmproxy
daemon listens on and exposes the REST PMWEBAPI to access metrics on the 44322 port by default, so it is recommended to map this port on the same host port number.--net host
The container uses the host's network. Use this option if you want to collect metrics from the host's network interfaces.
-e
The option enables you to set the following environment variables:
- PCP_SERVICES
Is a comma-separated list of services to start by
systemd
in the container.Default services are:
pmcd
,pmie
,pmlogger
,pmproxy
.You can use this variable, if you want to run a container with a list of services that is different from the default one, for example, only with
pmlogger
:#
podman run -d \ --name pmlogger \ --systemd always \ -e PCP_SERVICES=pmlogger \ -v pcp-archives:/var/log/pcp/pmlogger \ registry.suse.com/suse/pcp:latest- HOST_MOUNT
Is a path inside the container to the bind mount of the host's root file system. The default value is not set.
- REDIS_SERVERS
Specifies a connection to a Redis server. In a non-clustered setup, provide a comma-separated list of host specs. In a clustered setup, provide any individual cluster host, other hosts in the cluster are discovered automatically. The default value is:
localhost:6379
.
If you need to use a different configuration to the one provided by the environment variables, proceed as described in Section 10.3, “Configuring PCP services”.
10.3 Configuring PCP services #
All services that run inside the PCP container have a default configuration that might not suit your needs. If you need a custom configuration that cannot be covered by the environment variables described above, create configuration files for the PCP services and pass them to the PCP using a bind mount as follows:
#
podman run -d \
--name CONTAINER_NAME \
--systemd always \
-v $HOST_CONFIG:CONTAINER_CONFIG_PATH:z \
-v HOST_LOGS_PATH:/var/log/pcp/pmlogger \
registry.suse.com/suse/pcp:latest
Where:
- CONTAINER_NAME
Is an optional container name.
- HOST_CONFIG
Is an absolute path to the config you created on the host machine. You can choose any file name you want.
- CONTAINER_CONFIG_PATH
Is an absolute path to a particular configuration file inside the container. Each available configuration file is described in the corresponding sections further.
- HOST_LOGS_PATH
Is a directory that should be a bind mount to the container logs.
For example, a container called pcp
, with the
configuration file pmcd
on the host machine and the
pcp-archives
directory for logs on the host machine, is
run by the following command:
#
podman run -d \
--name pcp \
--systemd always \
-v $(pwd)/pcp-archives:/var/log/pcp/pmlogger \
-v $(pwd)/pmcd:/etc/sysconfig/pmcd \
registry.suse.com/suse/pcp:latest
10.3.1 Custom pmcd
daemon configuration #
The pmcd
daemon configuration is stored in the
/etc/sysconfig/pmcd
file. The file stores environment
variables that modify the behavior of the pmcd
daemon.
10.3.1.1 The /etc/sysconfig/pmcd
file #
You can add the following variables to the file to configure the
pmcd
daemon:
- PMCD_LOCAL
Defines whether the remote host can connect to the
pmcd
daemon. If set to 0, remote connections to the daemon are allowed. If set to 1, the daemon listens only on the local host. The default value is 0.- PMCD_MAXPENDING
Defines the maximum count of pending connections to the agent. The default value is 5.
- PMCD_ROOT_AGENT
If the
pmdaroot
is enabled (the value is set to 1), adding a new PDMA does not trigger restarting of other PMDAs. Ifpmdaroot
is not enabled,pmcd
will require to restart all PMDAs when a new PMDA is added. The default value is 1.- PMCD_RESTART_AGENTS
If set to 1, the
pmcd
daemon tries to restart any exited PMDA. Enable this option only if you have enabledpmdaroot
, aspmcd
itself does not have privileges to restart PMDA.- PMCD_WAIT_TIMEOUT
Defines the maximum time in seconds
pmcd
can wait to accept a connection. After this time, the connection is reported as failed. The default value is 60.- PCP_NSS_INIT_MODE
Defines the mode in which
pmcd
initializes the NSS certificate database when secured connections are used. The default value isreadonly
. You can set the mode toreadwrite
, but if the initialization fails, the default value is used as a fallback.
An example follows:
PMCD_LOCAL=0 PMCD_MAXPENDING=5 PMCD_ROOT_AGENT=1 PMCD_RESTART_AGENTS=1 PMCD_WAIT_TIMEOUT=70 PCP_NSS_INIT_MODE=readwrite
10.3.2 Custom pmlogger
configuration #
The custom configuration for the pmlogger
is stored in
the following configuration files:
/etc/sysconfig/pmlogger
/etc/pcp/pmlogger/control.d/local
10.3.2.1 The /etc/sysconfig/pmlogger
file #
You can use the following attributes to configure the
pmlogger
:
- PMLOGGER_LOCAL
Defines whether
pmlogger
allows connections from remote hosts. If set to 1,pmlogger
allows connections from local host only.- PMLOGGER_MAXPENDING
Defines the maximum count of pending connections. The default value is 5.
- PMLOGGER_INTERVAL
Defines the default sampling interval
pmlogger
uses. The default value is 60 s. Keep in mind that this value can be overridden by thepmlogger
command line.- PMLOGGER_CHECK_SKIP_LOGCONF
Setting this option to yes disables the regeneration and checking of the
pmlogger
configuration if the configurationpmlogger
comes frompmlogconf
. The default behavior is to regenerate configuration files and check for changes every timepmlogger
is started.
An example follows:
PMLOGGER_LOCAL=1 PMLOGGER_MAXPENDING=5 PMLOGGER_INTERVAL=10 PMLOGGER_CHECK_SKIP_LOGCONF=yes
10.3.2.2 The /etc/pcp/pmlogger/control.d/local
file #
The file /etc/pcp/pmlogger/control.d/local
stores
specifications of the host, which metrics should be logged, the logging
frequency (default is 24 hours), and pmlogger
options.
For example:
# === VARIABLE ASSIGNMENTS === # # DO NOT REMOVE OR EDIT THE FOLLOWING LINE $version=1.1 # Uncomment one of the lines below to enable/disable compression behaviour # that is different to the pmlogger_daily default. # Value is days before compressing archives, 0 is immediate compression, # "never" or "forever" suppresses compression. # #$PCP_COMPRESSAFTER=0 #$PCP_COMPRESSAFTER=3 #$PCP_COMPRESSAFTER=never # === LOGGER CONTROL SPECIFICATIONS === # #Host P? S? directory args # local primary logger LOCALHOSTNAME y n PCP_ARCHIVE_DIR/LOCALHOSTNAME -r -T24h10m -c config.default -v 100Mb
If you run the pmlogger
in a container on a different
machine than the one that runs the pmcd
(a client),
change the following line to point to the client:
# local primary logger CLIENT_HOSTNAME y n PCP_ARCHIVE_DIR/CLIENT_HOSTNAME -r -T24h10m -c config.default -v 100Mb
For example, for the slemicro_1
host name, the line
should look as follows:
# local primary logger slemicro_1 y n PCP_ARCHIVE_DIR/slemicro_1 -r -T24h10m -c config.default -v 100Mb
10.4 Starting the PCP container automatically on boot #
After you run the PCP container, you can configure systemd
to start the
container on boot. To do so, follow the procedure below:
Create a unit file for the container by using the
podman generate systemd
command:#
podman generate systemd --name CONTAINER_NAME > /etc/systemd/system/container-CONTAINER_NAME.servicewhere CONTAINER_NAME is the name of the PCP container you used when running the container from the container image.
Enable the service in
systemd
:#
systemctl enable container-CONTAINER_NAME
10.5 Metrics management #
10.5.1 Listing available performance metrics #
From within the container, you can use the command
pminfo
to list metrics. For example, to list all
available performance metrics, run:
#
pminfo
You can list a group of related metrics by specifying the metrics prefix:
#
pminfo METRIC_PREFIX
For example, to list all metrics related to kernel, use:
#
pminfo disk
disk.dev.r_await
disk.dm.await
disk.dm.r_await
disk.md.await
disk.md.r_await
...
You can also specify additional strings to narrow down the list of metrics, for example:
#
piminfo disk.dev
disk.dev.read
disk.dev.write
disk.dev.total
disk.dev.blkread
disk.dev.blkwrite
disk.dev.blktotal
...
To get online help text of a particular metric, use the -t
option followed by the metric, for example:
#
pminfo -t kernel.cpu.util.user
kernel.cpu.util.user [percentage of user time across all CPUs, including guest CPU time]
To display a description text of a particular metric, use the
-T
option followed by the metric, for example:
#
pminfo -T kernel.cpu.util.user
Help:
percentage of user time across all CPUs, including guest CPU time
10.5.2 Checking local metrics #
After you start the PCP container, you can verify that metrics are being recorded properly by running the following command inside the container:
#
pcp
Performance Co-Pilot configuration on localhost:
platform: Linux localhost 5.3.18-150300.59.68-default #1 SMP Wed May 4 11:29:09 UTC 2022 (ea30951) x86_64
hardware: 1 cpu, 1 disk, 1 node, 1726MB RAM
timezone: UTC
services: pmcd pmproxy
pmcd: Version 5.2.2-1, 9 agents, 4 clients
pmda: root pmcd proc pmproxy xfs linux mmv kvm jbd2
pmlogger: primary logger: /var/log/pcp/pmlogger/localhost/20220607.09.24
pmie: primary engine: /var/log/pcp/pmie/localhost/pmie.log
Now check if the logs are written to a proper destination:
#
ls PATH_TO_PMLOGGER_LOGS
where PATH_TO_PMLOGGER_LOGS should be
/var/log/pcp/pmlogger/localhost/
in this case.
10.5.3 Recording metrics from remote systems #
You can deploy collector containers that collect metrics from different
remote systems than the ones where the pmlogger
containers are running. Each remote collector system needs the
pmcd
daemon and a set of pmda. To
deploy several collectors with a centralized monitoring system, proceed as
follows.
On each system you want to collect metrics from (clients), run a container with the
pmcd
daemon:#
podman run -d \ --name pcp-pmcd \ --privileged \ --net host \ --systemd always \ -e PCP_SERVICES=pmcd \ -e HOST_MOUNT=/host \ -v /:/host:ro,rslave \ registry.suse.com/suse/pcp:latestOn the monitoring system, create a
pmlogger
configuration file for each clientcontrol.CLIENT
with the following content:$version=1.1 CLIENT_HOSTNAME n n PCP_ARCHIVE_DIR/CLIENT -N -r -T24h10m -c config.default -v 100Mb
Keep in mind that the CLIENT_HOSTNAME must be resolvable in DNS. You can use IP addresses or fully qualified domain names (FQDN) instead.
On the monitoring system, create a directory for each client to store the recorded logs:
#
mkdir /root/pcp-archives/CLIENTFor example, for
slemicro_1
:#
mkdir /root/pcp-archives/slemicro_1On the monitoring system, run a container with
pmlogger
for each client:#
podman run -d \ --name pcp-pmlogger-CLIENT \ --systemd always \ -e PCP_SERVICES=pmlogger \ -v /root/pcp-archives/CLIENT:/var/log/pcp/pmlogger:z \ -v $(pwd)/control.CLIENT:/etc/pcp/pmlogger/control.d/local:z \ registry.suse.com/suse/pcp:latestFor example, for a client called
slemicro_1
:#
podman run -d \ --name pcp-pmlogger-slemicro_1 \ --systemd always \ -e PCP_SERVICES=pmlogger \ -v /root/pcp-archives:/var/log/pcp/pmlogger:z \ -v $(pwd)/control.slemicro_1:/etc/pcp/pmlogger/control.d/local:z \ registry.suse.com/suse/pcp:latestNoteThe second bind mount points to the configuration file created in Step 2 and replaces the default
pmlogger
configuration. If you do not create this bind mount,pmlogger
uses the default/etc/pcp/pmlogger/control.d/local
file and logging from clients fails as the default configuration points to a local host. For details about the configuration file, refer to Section 10.3.2.2, “The/etc/pcp/pmlogger/control.d/local
file”.To check if the log collection is working properly, run:
#
ls -l pcp-archives/CLIENT/CLIENTFor example:
#
ls -l pcp-archives/slemicro_1/slemicro_1 total 1076 -rw-r--r--. 1 systemd-network systemd-network 876372 Jun 8 11:24 20220608.10.58.0 -rw-r--r--. 1 systemd-network systemd-network 312 Jun 8 11:22 20220608.10.58.index -rw-r--r--. 1 systemd-network systemd-network 184486 Jun 8 10:58 20220608.10.58.meta -rw-r--r--. 1 systemd-network systemd-network 246 Jun 8 10:58 Latest -rw-r--r--. 1 systemd-network systemd-network 24595 Jun 8 10:58 pmlogger.log