Applies to SUSE Linux Enterprise Server 15 SP6

13 Storage encryption for hosted applications with cryptctl #

Databases and similar applications are often hosted on external servers that are serviced by third-party staff. Certain data center maintenance tasks require third-party staff to directly access affected systems. In such cases, privacy requirements necessitate disk encryption.

cryptctl allows encrypting sensitive directories using LUKS and offers the following additional features:

Encryption keys are located on a central server, which can be located on customer premises.
Encrypted partitions are automatically remounted after an unplanned reboot.

cryptctl consists of two components:

A client is a machine that has one or more encrypted partitions but does not permanently store the necessary key to decrypt those partitions. For example, clients can be cloud or otherwise hosted machines.
The server holds encryption keys that can be requested by clients to unlock encrypted partitions.
You can also set up the cryptctl server to store encryption keys on a KMIP 1.3-compatible (Key Management Interoperability Protocol) server. In that case, the cryptctl server does not store the encryption keys of clients and is dependent upon the KMIP-compatible server to provide these.

Warning: cryptctl Server maintenance

Since the cryptctl server manages timeouts for the encrypted disks and, depending on the configuration, can also hold encryption keys, it should be under your direct control and managed by trusted personnel.

Additionally, it should be backed up regularly. Losing the server's data means losing access to encrypted partitions on the clients.

To handle encryption, cryptctl uses LUKS with aes-xts-256 encryption and 512-bit keys. Encryption keys are transferred using TLS with certificate verification.

The client asks the server for the disk decryption key, the server responds

Figure 13.1: Key retrieval with cryptctl (model without connection to KMIP server) #

Note: Install cryptctl

Before continuing, make sure the package cryptctl is installed on all machines you intend to set up as servers or clients.

13.1 Setting up a `cryptctl` server #

Before you can define a machine as a cryptctl client, you need to set up a machine as a cryptctl server.

Before beginning, choose whether to use a self-signed certificate to secure communication between the server and clients. If not, generate a TLS certificate for the server and have it signed by a certificate authority.

Additionally, you can have clients authenticate to the server using certificates signed by a certificate authority. To use this extra security measure, make sure to have a CA certificate at hand before starting this procedure.

As root, run:
```
# cryptctl init-server
```
Answer each of the following prompts and press Enter after every answer. If there is a default answer, it is shown in square brackets at the end of the prompt.
1. Create a strong password, and protect it well. This password unlocks all partitions that are registered on the server.
2. Specify the path to a PEM-encoded TLS certificate or certificate chain file or leave the field empty to create a self-signed certificate. If you specify a path, use an absolute path.
3. If you want the server to be identified by a host name other than the default shown, specify a host name. cryptctl generates certificates which include the host name.
4. Specify the IP address that belongs to the network interface that you want to listen on for decryption requests from the clients, then set a port number (the default is port 3737).
  The default IP address setting, 0.0.0.0 means that cryptctl listens on all network interfaces for client requests using IPv4.
5. Specify a directory on the server that holds the decryption keys for clients.
6. Specify whether clients need to authenticate to the server using a TLS certificate. If you choose No, this means that clients authenticate using disk UUIDs. (However, communication is encrypted using the server certificate in any case.)
  If you choose Yes, pick a PEM-encoded certificate authority to use for signing client certificates.
7. Specify whether to use a KMIP 1.3-compatible server (or multiple such servers) to store encryption keys of clients. If you choose this option, provide the host names and ports for one or multiple KMIP-compatible servers.
  Additionally, provide a user name, password, a CA certificate for the KMIP server, and a client identity certificate for the cryptctl server.
  Important: No easy reconfiguration of KMIP setting
  The setting to use a KMIP server cannot easily be changed later. To change this setting, both the cryptctl server and its clients need to be configured afresh.
8. Finally, configure an SMTP server for e-mail notifications for encryption and decryption requests or leave the prompt empty to skip setting up e-mail notifications.
  Note: Password-protected servers
  cryptctl currently cannot send e-mail using authentication-protected SMTP servers. If that is necessary, set up a local SMTP proxy.
9. When asked whether to start the cryptctl server, enter y.
To check the status of the service cryptctl-server, use:
```
# systemctl status cryptctl-server
```

To reconfigure the server later, do either of the following:

Run the command cryptctl init-server again. cryptctl proposes the existing settings as the defaults, so that you only need to specify the values that you want to change.
Make changes directly in the configuration file /etc/sysconfig/cryptctl-server.
However, to avoid issues, do not change the settings AUTH_PASSWORD_HASH and AUTH_PASSWORD_SALT manually. The values of these options need to be calculated correctly.

13.2 Setting up a `cryptctl` client #

The following interactive setup of cryptctl is currently the only setup method.

Make sure the following preconditions are fulfilled:

A cryptctl server is available over the network.
There is a directory to encrypt.
The client machine has an empty partition available that is large enough to fit the directory to encrypt.
When using a self-signed certificate, the certificate (*.crt file) generated on the server is available locally on the client. Otherwise, the certificate authority of the server certificate must be trusted by the client.
If you set up the server to require clients to authenticate using a client certificate, prepare a TLS certificate for the client which is signed by the CA certificate you chose for the server.

As root, run:
```
# cryptctl encrypt
```
Answer each of the following prompts and press Enter after every answer. If there is a default answer, it is shown in square brackets at the end of the prompt.
1. Specify the host name and port to connect to on the cryptctl server.
2. If you configured the server to have clients authenticate to it using a TLS certificate, specify a certificate and a key file for the client. The client certificate must be signed by the certificate authority chosen when setting up the server.
3. Specify the absolute path to the server certificate (the *.crt file).
4. Enter the encryption password that you specified when setting up the server.
5. Specify the path to the directory to encrypt. Specify the path to the empty partition that contains the encrypted content of the directory.
6. Specify the number of machines that are allowed to decrypt the partition simultaneously.
  Then specify the timeout in seconds before additional machines are allowed to decrypt the partition after the last vital sign was received from the client or clients.
  When a machine unexpectedly stops working and then reboots, it needs to be able to unlock its partitions again. That means this timeout should be set to a time slightly shorter than the reboot time of the client.
  Important: Timeout length
  If the time is set too long, the machine cannot decrypt encrypted partitions on the first try. cryptctl then continues to periodically check whether the encryption key has become available. However, this introduces a delay.
  If the timeout is set too short, machines with a copy of the encrypted partition have an increased chance of unlocking the partition first.
To start encryption, enter yes.
cryptctl encrypts the specified directory to the previously empty partition and then mounts the newly encrypted partition. The file system type is the same type as the original unencrypted file system.
Before creating the encrypted partition, cryptctl moves the unencrypted content of the original directory to a location prefixed with cryptctl-moved-.

To check that the directory is indeed mounted correctly, use:

> lsblk -o NAME,MOUNTPOINT,UUID
NAME                        MOUNTPOINT          UUID
[...]
sdc
└─sdc1                                          PARTITION_UUID
  └─cryptctl-unlocked-sdc1  /secret-partition   UNLOCKED_UUID

cryptctl identifies the encrypted partition by its UUID. For the previous example, that is the UUID displayed next to sdc1.

On the server, you can check whether the directory was decrypted using cryptctl.

# cryptctl list-keys

For a successfully decrypted partition, you see output like:

2019/06/06 15:50:00 ReloadDB: successfully loaded database of 1 records
Total: 1 records (date and time are in zone EDT)
Used By     When                 UUID  Max.Users  Num.Users  Mount Point
IP_ADDRESS  2019-06-06 15:00:50  UUID  1          1          /secret-partition

For a partition not decrypted successfully, you see output like:

2019/06/06 15:50:00 ReloadDB: successfully loaded database of 1 records
Total: 1 records (date and time are in zone EDT)
Used By      When                 UUID  Max.Users  Num.Users  Mount Point
             2019-06-06 15:00:50  UUID  1          1          /secret-partition

See the difference in the empty Used by column.

Verify that the UUID shown is that of the previously encrypted partition.

After verifying that the encrypted partition works, delete the unencrypted content from the client. For example, use rm. For more safety, overwrite the content of the files before deleting them, for example, using shred -u.
Important: shred does not guarantee that data is erased
Depending on the type of storage media, using shred is not a guarantee that all data is removed. In particular, SSDs employ wear leveling strategies that render shred ineffective.

The configuration for the connection from client to server is stored in /etc/sysconfig/cryptctl-client and can be edited manually.

The server stores an encryption key for the client partition in /var/lib/cryptctl/keydb/PARTITION_UUID.

13.3 Configuring /etc/fstab for LUKS volumes #

When configuring the mount point for a new file system encrypted with LUKS, YaST uses, by default, the name of the encrypted device in /etc/fstab. (For example, /dev/mapper/cr_sda1.) Using the device name, rather than the UUID or volume label, results in a more robust operation of systemd generators and other related tools.

You have the option to adjust that default behavior for each device, either with the Expert Partitioner in the installer, or via AutoYaST.

This change does not affect upgrades or any other scenario in which the mount points are already defined in /etc/fstab. Newly created mount points are affected, such as during the installation of a new system, or creating new partitions on running systems.

13.4 Checking partition unlock status using server-side commands #

When a cryptctl client is active, it sends a “heartbeat” to the cryptctl server every 10 seconds. If the server does not receive a heartbeat from the client for the length of the timeout configured during the client setup, the server assumes that the client is offline. It then allows another client to connect (or allow the same client to reconnect after a reboot).

To see the usage status of all keys, use:

# cryptctl list-keys

The information under Num. Users shows whether the key is currently in use. To see more detail on a single key, use:

# cryptctl show-key UUID

This command shows information about mount point, mount options, usage options, the last retrieval of the key and the last three heartbeats from clients.

Additionally, you can use journalctl to find logs of when keys were retrieved.

13.5 Unlocking encrypted partitions manually #

There are two ways of unlocking a partition manually, both of which are run on a client:

Online unlocking. Online unlocking allows circumventing timeout or user limitations. This method can be used when there is a network connection between client and server but the client could not (yet) unlock the partition automatically. This method unlocks all encrypted partitions on a machine.
To use it, run cryptctl online-unlock. Be prepared to enter the password specified when setting up the server.
Offline unlocking. This method can be used when a client cannot or must not be brought online to communicate with its server. The encryption key from the server must still be available. This method is meant as a last resort and can only unlock a single partition at a time.
To use it, run cryptctl offline-unlock. The server's key file for the requisite partition (/var/lib/cryptctl/keydb/PARTITION_UUID) needs to be available on the client.

13.6 Maintenance downtime procedure #

To ensure that partitions cannot be decrypted during a maintenance downtime, turn off the client and disable the cryptctl server. You can do so by either:

Stopping the service cryptctl-server:
```
# systemctl stop cryptctl-server
```
Unplugging the cryptctl server from the network.

13.7 Setting up an HA environment for cryptctl-server service #

To avoid downtimes when the cryptctl-server needs to be stopped for maintenance or suffers a damage, it is strongly recommended to set up the cryptctl-server in an HA environment. You need at least a two-node High Availability cluster for this. The following setup shows how to create a two-node HA cluster for cryptctl-server using self-signed certificates.

Make sure the following preconditions are fulfilled:

At least two servers which have SUSE Linux Enterprise Server and the High Availability extension installed are available. All servers must also have the cryptctl package installed. All servers can reach each other via SSH.
If you set up a new cluster, you need an additional IP address for the HA Web Console of the cluster (AdminIP).
A separate IP address (CrypServerIP) is reserved for the cryptctl-server.
A separate dns-name (CrypServerHostName) is reserved for the cryptctl-server and is resolved to the above IP address.
An HA-enabled block device or NFS share is available to store the keys.
In our example, we use an NFS share: nfs-server.example.org/data/cryptctl-keys. It is mounted to the standard location /var/lib/cryptctl/keydb.
It is strongly recommended to use an SBD device.

Procedure 13.1: Setting up a cryptctl two-node HA cluster #

Log in to Node1 as root.
Set up a cryptctl-server as described in Section 13.1, “Setting up a cryptctl server”. Use the following parameters:
1. To create the certificate, use the dedicated hostname CrypServerHostName of the cryptctl server. Do not use the host name of the host.
2. Use the dedicated IP address CrypServerIP of the cryptctl server. Do not use the default IP address setting.
3. Do not configure a KMIP server.
4. When asked whether to start the cryptctl server, enter n.
Set up a two-node HA cluster:
1. Important
  Node1 must be the server where you have configured the cryptctl server.
  On the machine where you have configured the cryptctl server, set up the first node as follows:
```
# crm cluster init -i NetDev -A AdminIP -n ClusterName
```
2. Log in via SSH to Node2 and join the cluster from there:
```
# ssh Node2
# crm cluster join -y Node1
```
3. For more information, also see the Installation and Setup Quick Start.
Set up the resource group for the cryptctl server:

You can set up all needed resource agents and copy all files to all nodes with the cryptcl crm-shell-script in one step. We strongly recommend that you verify the setup in the first step:

# crm script verify cryptctl  \
cert-path=/etc/cryptctl/servertls/CertificateFileName \
cert-key-path=/etc/cryptctl/servertls/CertificateKeyFileName \
virtual-ip:ip=CrypServerIP \
filesystem:device=DevicePath
filesystem:fstype=FileSystemType

If the check was successful, set up the cluster group by running the script as follows:

# crm script verify cryptctl  \
cert-path=/etc/cryptctl/servertls/CertificateFileName \
cert-key-path=/etc/cryptctl/servertls/CertificateKeyFileName \
virtual-ip:ip=CrypServerIP \
filesystem:device=DevicePath
filesystem:fstype=FileSystemType

Table 13.1: List of all parameters to define the resource group with the cryptctl crm script. #

Name	Mandatory	Default Value	Description
id	no	cryptctl	Name of the resource group.
cert-path	yes		The full path to the created certificate.
cert-key-path	yes		The full path to the created certificate key.
virtual-ip:id	no	cryptctl-vip	The ID of the virtual IP resource for the cryptctl server.
virtual-ip:ip	yes		The IP address of the cryptctl server.
virtual-ip:nic	no	Detected by the virtual-ip resource agent.	The network device the cryptctl server should be listening on. Only required if the device cannot be detected from the IP address.
virtual-ip:cidr_netmask	no	Detected by the virtual-ip resource agent.	The numeric netmask of the IP address of the cryptctl server. Only required if the netmask cannot be detected from the IP address.
virtual-ip:broadcast	no	Detected by the virtual-ip resource agent.	The broadcast address of the IP address of the cryptctl server. Only required if this cannot be detected from the IP address.
filesystem:id	no	cryptctl-filesystem	The ID of the file system resource containing the disk encryption keys and records.
filesystem:device	yes		The device containing the file system. This can be a block device like `/dev/sda...` or an NFS share path `server:/path`.
filesystem:directory	no	/var/lib/cryptctl/keydb	The directory where the device containing the file system is located. This can be a block device like `/dev/sda...` or an NFS share path `server:/path`.
filesystem:fstype	yes		The file system type (for example, NFS, XFS, EXT4).
filesystem:options	no	The default options of the selected file system.	Mount options for the file system.

13.8 More information #

For more information, also see the project home page https://github.com/SUSE/cryptctl/.