Securing SUSE Linux Enterprise Micro Using Keylime

1 Remote attestation using Keylime #

With the growing demand on securing devices against unauthorized changes, the use of the security mechanism called remote attestation (RA) has been experiencing significant growth. Using RA, a host (client) can authenticate its boot chain status and running software on a remote host (verifier). RA is often combined with public-key encryption (using TPM2), thus the sent information can only be read by the services that requested the attestation, and the validity of the data can be verified.

Remote attestation on SLE Micro is implemented by Keylime.

1.1 Terminology #

Remote attestation technology uses the following terms:

Attestation key (AK): A data signing key that proves that the data comes from a real TPM and has not been tampered with.
Core root of trust for measurement: Calculates its own hash and the hash of the next step in the boot process, initiating the chain of measurements.
Endorsement key (EK): An encryption key that is permanently embedded in the TPM when it is manufactured. The public part of the key and the certification stored in the TPM are used to recognize a genuine TPM.
Integrity management architecture (IMA): A kernel integrity subsystem that provides a means of detecting malicious changes to files.
Measured boot: A method with which each component in the booting sequence calculates a hash of the next one before delegating the execution of the next component. The hash extends one or several PCRs of the TPM. An event is created with the information about where the measurement took place and what was measured. Such events are collected in an event log, and, along with the extended PCR values, the events can be compared with the expected values representing a healthy system.
Platform Configuration Register (PCR): A memory location in TPM that, for example, stores hashes of booting layers. PCR can be updated only by using the non-reversible operation: extend. A signed list of current PCR values can be obtained by the quote command on TPM, and this quote can be verified by a third party during the attestation process.
Secure boot: Each step of the booting process checks a cryptographic signature on the executable of the next step before launching it.
Trusted Platform Module (TPM): A self-contained security cryptographic processor present in the system as hardware or implemented in the firmware that serves as a root of trust. TPM provides a PCR for storing the hashes of booting layers. A typical TPM provides several functions, like a random number generator, counters or a local clock. It also stores 24 PCRs grouped by banks per each supported cryptographic hash function (SHA1, SHA256, SHA384 or SHA512).
Note
By default, TPM usage is disabled. Therefore, the measured boot does not take place. To enable the remote attestation, enable TPM in the EFI/BIOS menu.
Secure payload: A mechanism to deliver encrypted data to healthy agents. Payloads are used to provide keys, passwords, certificates, configurations or scripts that are further used by the agent.

1.2 What is Keylime? #

Keylime is a remote attestation solution that enables you to monitor the health of remote nodes using a TPM as a root of trust for measurement. With Keylime, you can perform multiple tasks, for example:

Validate of the PCRs extended during the measured boot.
Create analysis and make assertions of the event log.
Make assertion of the value of any PCR in the remote system.
Monitor the validity of open or executed files.
Deliver encrypted data to verified nodes via secure payloads.
Execute custom scripts that are triggered when a machine fails the attested measurements.

1.3 Architecture #

Keylime consists of an agent, a verifier, a registrar and a command-line tool (tenant). Agents are on those systems that need to be attested. The verifier and registrar are on remote systems that perform the registration and attestation of agents. Keep in mind that only the agent role is available on SLE Micro. For details about each component, refer to the following sections.

1.3.1 Keylime agent #

The agent is a service that runs on the system that needs to be attested. The agent sends the event log, IMA hashes, and information about the measured boot to the verifier, using the local TPM as a certifier of the data validity.

When a new agent is started, it needs to register itself in the registrar first. To do so, the agent needs a TLS certificate to establish the connection. The TLS certificate is generated by the registrar, but it needs to be installed manually to the agent. After the registration, the agent sends its attestation key and the public part of the endorsement key to the registrar. The registrar responds to the agent with a challenge in a process called credential activation, which validates the TPM of the agent. Once the agent has been registered, it is ready to be enrolled for attestation.

1.3.2 Keylime registrar #

The registrar is used to register agents that should be attested. The registrar collects the agent's attestation key, the public part of the endorsement key and the endorsement key certification, and verifies that the agent attestation key belongs to the endorsement key.

1.3.3 Keylime verifier #

The verifier performs the actual attestation of agents and continuously pulls the required attestation data from agents (among others, the PCR values, IMA logs, and UEFI event logs).

2 Running the Keylime workload using Podman #

Keylime is a remote attestation solution that enables you to monitor the health of remote nodes. The verifier and registrar are essential components of Keylime on remote systems to perform the registration and attestation of Keylime agents.

Note

The container described in this article delivers control plane services verifier and registrar and a tenant command-line tool (CLI) that are part of the Keylime project.

Before you start installing and registering agents, prepare the verifier and the registrar on remote hosts, as described in the following procedure.

Identify the Keylime workload image.

# podman search keylime
[...]
registry.opensuse.org/devel/microos/containers/containerfile/opensuse/keylime-control-plane

Pull the image from the registry.

# podman pull\
  registry.opensuse.org/devel/microos/containers/containerfile/opensuse/keylime-control-plane:latest

Create the keylime-control-plane volume to persist the database and certificates required during the attestation process.

# podman container runlabel install \
  registry.opensuse.org/devel/microos/containers/containerfile/opensuse/keylime-control-plane:latest

Start the container and related services.
```
# podman container runlabel run \
  registry.opensuse.org/devel/microos/containers/containerfile/opensuse/keylime-control-plane:latest
```
The keylime-control-plane container is created. It includes configured and running registrar and verifier services. Internally, the container exposes ports 8881, 8890 and 8891 to the host using the default values. Validate the firewall configuration to allow access to the ports and to allow communication between containers, because the tenant CLI requires it.

Tip

If you need to stop Keylime services, run the following command:

# podman kill keylime-control-plane-container

2.1 Monitoring Keylime services #

To get the status of running containers on the host, run the following command:

# podman ps

To view the logs of Keylime services, run the following command:

# podman logs keylime-control-plane-container

2.2 Executing the tenant CLI #

The tenant CLI tool is included in the container, and if the host firewall does not interfere with the ports exposed by Keylime services, you can execute it using the same image, for example:

# podman run --rm \
-v keylime-control-plane-volume:/var/lib/keylime/ \
keylime-control-plane:latest \
keylime_tenant -v 10.88.0.1 -r 10.88.0.1 --cert default -c reglist

2.3 Extracting the Keylime certificate #

The first time that the Keylime container is executed, its services create a certificate required by several agents. You need to extract the certificate from the container and copy it to the agent's /var/lib/keylime/cv_ca/ directory.

# podman cp \
keylime-control-plane-container:/var/lib/keylime/cv_ca/cacert.crt
.
# scp cacert.crt
AGENT_HOST:/var/lib/keylime/cv_ca/

Tip

Find more details about installing the agent in Section 3, “Installing the Keylime agent”.

3 Installing the Keylime agent #

Keylime is a remote attestation solution that enables you to monitor the health of remote nodes. The Keylime agent is a service that runs on the system that needs to be attested and sends the event log, IMA hashes, and information about the measured boot to the verifier.

The Keylime agent is not present on SLE Micro by default, and you need to install it manually. To install the agent, proceed as follows:

Install the rust-keylime package as follows:
```
# transactional-update pkg in rust-keylime
```
Then reboot the system.

Adjust the default agent's configuration.

Create a directory to store a new configuration file for your changes in /etc/keylime/agent.conf.d/. The default configuration is stored in /usr/etc/keylime/agent.conf, but we do not recommend editing this file because it may be overwritten in upcoming system updates.
```
# mkdir -p /etc/keylime/agent.conf.d
```

Create a new file /etc/keylime/agent.conf.d/agent.conf:

# cat << EOF > /etc/keylime/agent.conf.d/agent.conf
 [agent]
 
 uuid = "d111ec46-34d8-41af-ad56-d560bc97b2e8"1
 registrar_ip = "<REMOTE_IP>"2
 revocation_notification_ip = "<REMOTE_IP>"3
 EOF

1	The unique identifier is generated each time the agent is run. However, you can define a specific value by this option.
2	IP address of the registrar.
3	IP address of the verifier.

Change the owner of the /etc/keylime/ directory to keylime:tss:
```
# chown -R keylime:tss /etc/keylime
```
Change permissions on the /etc/keylime/ directory:
```
# chmod -R 600 /etc/keylime
```

Copy the certificates generated by the CA to the agent node. On the agent node, do the following:
1. Prepare a directory for the certificate:
```
# mkdir -p /var/lib/keylime/cv_ca
```
2. Copy the certificate to the agent:
```
# scp CERT_SERVER_ADDRESS:/var/lib/keylime/cv_ca/cacert.crt /var/lib/keylime/cv_ca
```
3. Change the owner of the certificate to keylime:tss:
```
# chown -R keylime:tss /var/lib/keylime/cv_ca
```

Start and enable the keylime_agent.service:

# systemctl enable --now keylime_agent.service

4 Registering the Keylime agent #

Keylime is a remote attestation solution that enables you to monitor the health of remote nodes. The Keylime agent is a service that runs on the system that needs to be attested and sends the event log, IMA hashes, and information about the measured boot to the verifier.

You can register a new agent either by using the CLI tenant or by editing the configuration of the verifier. Using the tenant on the verifier host, run the following:

# keylime_tenant -v 127.0.0.1 \
  -t AGENT \1
  -u UUID \2
  --cert default \
  -c add
  [--include PATH_TO_ZIP_FILE]3

1	AGENT is an IP address of the agent to be registered.
2	UUID is the agent's unique identifier.
3	The file passed by the `include` option is used to deliver secret payload data to the agent. For details, refer to Section 5, “Keylime secure payloads”.

You can list registered agents by using the reglist command on the verifier host as follows:

# keylime_tenant -v 127.0.0.1 \
  --cert default \
  -c reglist

To remove a registered agent, specify the agent using the -t and -u options and the -c delete command as follows:

# keylime_tenant -v 127.0.0.1 \
  -t AGENT \
  -u UUID \
  -c delete

5 Keylime secure payloads #

Keylime is a remote attestation solution that enables you to monitor the health of remote nodes.

5.1 What is a secure payload? #

A Keylime secure payload enables you to deliver encrypted data to healthy agents. Payloads are used to provide keys, passwords, certificates, configurations or scripts that are used by the Keylime agent at a later stage.

5.2 How does a secure payload work? #

A secure payload is delivered to the agent in a zip file that must contain a shell script named autorun.sh. The script is executed only if the agent has been properly registered and verified. To deliver the zip file, use the --include option of the keylime_tenant command.

For example, the following autorun.sh script creates a directory structure and copies SSH keys there. The related zip archive must include these SSH keys.

> cat autorun.sh
#!/bin/bash
 
 mkdir -p /root/.ssh/
 cp id_rsa* /root/.ssh/
 chmod 600 /root/.ssh/id_rsa*
 cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys

6 Enabling IMA tracking for Keylime #

Keylime is a remote attestation solution that enables you to monitor the health of remote nodes. Integrity management architecture (IMA) is a kernel integrity subsystem that provides a means of detecting malicious changes to files.

When using IMA, the kernel calculates a hash of accessed files. The hash is then used to extend the PCR 10 in the TPM and also log a list of accessed files. The verifier can request a signed quote to the agent for PCR 10 to get the logs of all accessed files including the file hashes. Verifiers then compare the accessed files with a local allowlist of approved files. If any of the hashes are not recognized, the system is considered unsafe, and a revocation event is triggered.

Before Keylime can collect information, IMA/EVM needs to be enabled. To enable the process, boot a kernel of the agent with the ima_appraise=log and ima_policy=tcb parameters:

Update the GRUB_CMDLINE_LINUX_DEFAULT option with the parameters in /etc/default/grub:
```
GRUB_CMDLINE_LINUX_DEFAULT="ima_appraise=log ima_policy=tcb"
```
Regenerate grub.cfg by running:
```
# transactional-update grub.cfg
```
Reboot your system.

The procedure above uses the default kernel IMA policy. To avoid monitoring too many files and therefore creating long logs, create a new custom policy. Find more details in the Keylime documentation.

To indicate the expected hashes, use the --allowlistoption of the keylime_tenant command when registering the agent. To view the excluded or ignored files, use the --exclude option of the keylime_tenant command:

# keylime_tenant --allowlist
    -v 127.0.0.1 \
    -u UUID

7 For more information #

Keylime home page is at https://keylime.dev.
Latest Keylime documentation is at https://keylime.readthedocs.io/en/latest/.
For a high-level overview of IMA/EVM, refer to https://en.opensuse.org/SDB:Ima_evm#Introduction.
Find more details about creating a new kernel IMA policy in https://keylime-docs.readthedocs.io/en/latest/user_guide/runtime_ima.html.

8 Legal Notice #

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”.

For SUSE trademarks, see https://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.