Applies to SUSE Linux Enterprise Server 15 SP3

13 Storage encryption for hosted applications with cryptctl #

Databases and similar applications are often hosted on external servers that are serviced by third-party staff. Certain data center maintenance tasks require third-party staff to directly access affected systems. In such cases, privacy requirements necessitate disk encryption.

cryptctl allows encrypting sensitive directories using LUKS and offers the following additional features:

Encryption keys are located on a central server, which can be located on customer premises.
Encrypted partitions are automatically remounted after an unplanned reboot.

cryptctl consists of two components:

A client is a machine that has one or more encrypted partitions but does not permanently store the necessary key to decrypt those partitions. For example, clients can be cloud or otherwise hosted machines.
The server holds encryption keys that can be requested by clients to unlock encrypted partitions.
You can also set up the cryptctl server to store encryption keys on a KMIP 1.3-compatible (Key Management Interoperability Protocol) server. In that case, the cryptctl server will not store the encryption keys of clients and is dependent upon the KMIP-compatible server to provide these.

Warning: cryptctl Server maintenance

Since the cryptctl server manages timeouts for the encrypted disks and, depending on the configuration, can also hold encryption keys, it should be under your direct control and managed only by trusted personnel.

Additionally, it should be backed up regularly. Losing the server's data means losing access to encrypted partitions on the clients.

To handle encryption, cryptctl uses LUKS with aes-xts-256 encryption and 512-bit keys. Encryption keys are transferred using TLS with certificate verification.

The client asks the server for the disk decryption key, the server responds

Figure 13.1: Key retrieval with cryptctl (model without connection to KMIP server) #

Note: Install cryptctl

Before continuing, make sure the package cryptctl is installed on all machines you intend to set up as servers or clients.

13.1 Setting up a `cryptctl` server #

Before you can define a machine as a cryptctl client, you need to set up a machine as a cryptctl server.

Before beginning, choose whether to use a self-signed certificate to secure communication between the server and clients. If not, generate a TLS certificate for the server and have it signed by a certificate authority.

Additionally, you can have clients authenticate to the server using certificates signed by a certificate authority. To use this extra security measure, make sure to have a CA certificate at hand before starting this procedure.

As root, run:
```
# cryptctl init-server
```
Answer each of the following prompts and press Enter after every answer. If there is a default answer, it is shown in square brackets at the end of the prompt.
1. Create a very strong password, and protect it well. This password will unlock all partitions that are registered on the server.
2. Specify the path to a PEM-encoded TLS certificate or certificate chain file or leave the field empty to create a self-signed certificate. If you specify a path, use an absolute path.
3. If you want the server to be identified by a host name other than the default shown, specify a host name. cryptctl will then generate certificates which include the host name.
4. Specify the IP address that belongs to the network interface that you want to listen on for decryption requests from the clients, then set a port number (the default is port 3737).
  The default IP address setting, 0.0.0.0 means that cryptctl will listen on all network interfaces for client requests using IPv4.
5. Specify a directory on the server that will hold the decryption keys for clients.
6. Specify whether clients need to authenticate to the server using a TLS certificate. If you choose No, this means that clients authenticate using disk UUIDs only. (However, communication will be encrypted using the server certificate in any case.)
  If you choose Yes, pick a PEM-encoded certificate authority to use for signing client certificates.
7. Specify whether to use a KMIP 1.3-compatible server (or multiple such servers) to store encryption keys of clients. If you choose this option, provide the host names and ports for one or multiple KMIP-compatible servers.
  Additionally, provide a user name, password, a CA certificate for the KMIP server, and a client identity certificate for the cryptctl server.
  Important: No easy reconfiguration of KMIP setting
  The setting to use a KMIP server cannot easily be changed later. To change this setting, both the cryptctl server and its clients need to be configured afresh.
8. Finally, configure an SMTP server for e-mail notifications for encryption and decryption requests or leave the prompt empty to skip setting up e-mail notifications.
  Note: Password-protected servers
  cryptctl currently cannot send e-mail using authentication-protected SMTP servers. If that is necessary, set up a local SMTP proxy.
9. When asked whether to start the cryptctl server, enter y.
To check the status of the service cryptctl-server, use:
```
# systemctl status cryptctl-server
```

To reconfigure the server later, do either of the following:

Run the command cryptctl init-server again. cryptctl will then propose the existing settings as the defaults, so that you only need to specify the values that you want to change.
Make changes directly in the configuration file /etc/sysconfig/cryptctl-server.
However, to avoid issues, do not change the settings AUTH_PASSWORD_HASH and AUTH_PASSWORD_SALT manually. The values of these options need to be calculated correctly.

13.2 Setting up a `cryptctl` client #

The following interactive setup of cryptctl is currently the only setup method.

Make sure the following preconditions are fulfilled:

A cryptctl server is available over the network.
There is a directory to encrypt.
The client machine has an empty partition available that is large enough to fit the directory to encrypt.
When using a self-signed certificate, the certificate (*.crt file) generated on the server is available locally on the client. Otherwise, the certificate authority of the server certificate must be trusted by the client.
If you set up the server to require clients to authenticate using a client certificate, prepare a TLS certificate for the client which is signed by the CA certificate you chose for the server.

As root, run:
```
# cryptctl encrypt
```
Answer each of the following prompts and press Enter after every answer. If there is a default answer, it is shown in square brackets at the end of the prompt.
1. Specify the host name and port to connect to on the cryptctl server.
2. If you configured the server to have clients authenticate to it using a TLS certificate, specify a certificate and a key file for the client. The client certificate must be signed by the certificate authority chosen when setting up the server.
3. Specify the absolute path to the server certificate (the *.crt file).
4. Enter the encryption password that you specified when setting up the server.
5. Specify the path to the directory to encrypt. Specify the path to the empty partition that will contain the encrypted content of the directory.
6. Specify the number of machines that are allowed to decrypt the partition simultaneously.
  Then specify the timeout in seconds before additional machines are allowed to decrypt the partition after the last vital sign was received from the client or clients.
  When a machine unexpectedly stops working and then reboots, it needs to be able to unlock its partitions again. That means this timeout should be set to a time slightly shorter than the reboot time of the client.
  Important: Timeout length
  If the time is set too long, the machine cannot decrypt encrypted partitions on the first try. cryptctl will then continue to periodically check whether the encryption key has become available. However, this will introduce a delay.
  If the timeout is set too short, machines with a copy of the encrypted partition have an increased chance of unlocking the partition first.
To start encryption, enter yes.
cryptctl will now encrypt the specified directory to the previously empty partition and then mount the newly encrypted partition. The file system type will be of the same type as the original unencrypted file system.
Before creating the encrypted partition, cryptctl moves the unencrypted content of the original directory to a location prefixed with cryptctl-moved-.

To check that the directory is indeed mounted correctly, use:

> lsblk -o NAME,MOUNTPOINT,UUID
NAME                        MOUNTPOINT          UUID
[...]
sdc
└─sdc1                                          PARTITION_UUID
  └─cryptctl-unlocked-sdc1  /secret-partition   UNLOCKED_UUID

cryptctl identifies the encrypted partition by its UUID. For the previous example, that is the UUID displayed next to sdc1.

On the server, you can check whether the directory was decrypted using cryptctl.

# cryptctl list-keys

For a successfully decrypted partition, you will see output like:

2019/06/06 15:50:00 ReloadDB: successfully loaded database of 1 records
Total: 1 records (date and time are in zone EDT)
Used By     When                 UUID  Max.Users  Num.Users  Mount Point
IP_ADDRESS  2019-06-06 15:00:50  UUID  1          1          /secret-partition

For a partition not decrypted successfully, you will see output like:

2019/06/06 15:50:00 ReloadDB: successfully loaded database of 1 records
Total: 1 records (date and time are in zone EDT)
Used By      When                 UUID  Max.Users  Num.Users  Mount Point
             2019-06-06 15:00:50  UUID  1          1          /secret-partition

See the difference in the empty Used by column.

Verify that the UUID shown is that of the previously encrypted partition.

After verifying that the encrypted partition works, delete the unencrypted content from the client. For example, use rm. For more safety, overwrite the content of the files before deleting them, for example, using shred -u.
Important: shred does not guarantee that data is completely erased
Depending on the type of storage media, using shred is not a guarantee that all data is completely removed. In particular, SSDs usually employ wear leveling strategies that render shred ineffective.

The configuration for the connection from client to server is stored in /etc/sysconfig/cryptctl-client and can be edited manually.

The server stores an encryption key for the client partition in /var/lib/cryptctl/keydb/PARTITION_UUID.

13.3 Checking partition unlock status using server-side commands #

When a cryptctl client is active, it will send a “heartbeat” to the cryptctl server every 10 seconds. If the server does not receive a heartbeat from the client for the length of the timeout configured during the client setup, the server will assume that the client is offline. It will then allow another client to connect (or allow the same client to reconnect after a reboot).

To see the usage status of all keys, use:

# cryptctl list-keys

The information under Num. Users shows whether the key is currently in use. To see more detail on a single key, use:

# cryptctl show-key UUID

This command will show information about mount point, mount options, usage options, the last retrieval of the key, and the last three heartbeats from clients.

Additionally, you can use journalctl to find logs of when keys were retrieved.

13.4 Unlocking encrypted partitions manually #

There are two ways of unlocking a partition manually, both of which are run on a client:

Online unlocking. Online unlocking allows circumventing timeout or user limitations. This method can be used when there is a network connection between client and server but the client could not (yet) unlock the partition automatically. This method will unlock all encrypted partitions on a machine.
To use it, run cryptctl online-unlock. Be prepared to enter the password specified when setting up the server.
Offline unlocking. This method can be used when a client cannot or must not be brought online to communicate with its server. The encryption key from the server must still be available. This method is meant as a last resort only and can only unlock a single partition at a time.
To use it, run cryptctl offline-unlock. The server's key file for the requisite partition (/var/lib/cryptctl/keydb/PARTITION_UUID) needs to be available on the client.

13.5 Maintenance downtime procedure #

To ensure that partitions cannot be decrypted during a maintenance downtime, turn off the client and disable the cryptctl server. You can do so by either:

Stopping the service cryptctl-server:
```
# systemctl stop cryptctl-server
```
Unplugging the cryptctl server from the network.

13.6 More information #

For more information, also see the project home page https://github.com/SUSE/cryptctl/.