13 Storage encryption for hosted applications with cryptctl #
Databases and similar applications are often hosted on external servers that are serviced by third-party staff. Certain data center maintenance tasks require third-party staff to directly access affected systems. In such cases, privacy requirements necessitate disk encryption.
cryptctl
allows encrypting sensitive directories using
LUKS and offers the following additional features:
Encryption keys are located on a central server, which can be located on customer premises.
Encrypted partitions are automatically remounted after an unplanned reboot.
cryptctl
consists of two components:
A client is a machine that has one or more encrypted partitions but does not permanently store the necessary key to decrypt those partitions. For example, clients can be cloud or otherwise hosted machines.
The server holds encryption keys that can be requested by clients to unlock encrypted partitions.
You can also set up the
cryptctl
server to store encryption keys on a KMIP 1.3-compatible (Key Management Interoperability Protocol) server. In that case, thecryptctl
server does not store the encryption keys of clients and is dependent upon the KMIP-compatible server to provide these.
cryptctl
Server maintenance
Since the cryptctl
server manages timeouts for the
encrypted disks and, depending on the configuration, can also hold
encryption keys, it
should be under your direct control and managed by trusted
personnel.
Additionally, it should be backed up regularly. Losing the server's data means losing access to encrypted partitions on the clients.
To handle encryption, cryptctl
uses LUKS with
aes-xts-256 encryption and 512-bit keys. Encryption keys are transferred
using TLS with certificate verification.
cryptctl
(model without connection to KMIP server) #cryptctl
Before continuing, make sure the package cryptctl is installed on all machines you intend to set up as servers or clients.
13.1 Setting up a cryptctl
server #
Before you can define a machine as a cryptctl
client,
you need to set up a machine as a cryptctl
server.
Before beginning, choose whether to use a self-signed certificate to secure communication between the server and clients. If not, generate a TLS certificate for the server and have it signed by a certificate authority.
Additionally, you can have clients authenticate to the server using certificates signed by a certificate authority. To use this extra security measure, make sure to have a CA certificate at hand before starting this procedure.
As
root
, run:#
cryptctl init-server
Answer each of the following prompts and press Enter after every answer. If there is a default answer, it is shown in square brackets at the end of the prompt.
Create a strong password, and protect it well. This password unlocks all partitions that are registered on the server.
Specify the path to a PEM-encoded TLS certificate or certificate chain file or leave the field empty to create a self-signed certificate. If you specify a path, use an absolute path.
If you want the server to be identified by a host name other than the default shown, specify a host name.
cryptctl
generates certificates which include the host name.Specify the IP address that belongs to the network interface that you want to listen on for decryption requests from the clients, then set a port number (the default is port 3737).
The default IP address setting,
0.0.0.0
means thatcryptctl
listens on all network interfaces for client requests using IPv4.Specify a directory on the server that holds the decryption keys for clients.
Specify whether clients need to authenticate to the server using a TLS certificate. If you choose
, this means that clients authenticate using disk UUIDs. (However, communication is encrypted using the server certificate in any case.)If you choose
, pick a PEM-encoded certificate authority to use for signing client certificates.Specify whether to use a KMIP 1.3-compatible server (or multiple such servers) to store encryption keys of clients. If you choose this option, provide the host names and ports for one or multiple KMIP-compatible servers.
Additionally, provide a user name, password, a CA certificate for the KMIP server, and a client identity certificate for the
cryptctl
server.Important: No easy reconfiguration of KMIP settingThe setting to use a KMIP server cannot easily be changed later. To change this setting, both the
cryptctl
server and its clients need to be configured afresh.Finally, configure an SMTP server for e-mail notifications for encryption and decryption requests or leave the prompt empty to skip setting up e-mail notifications.
Note: Password-protected serverscryptctl
currently cannot send e-mail using authentication-protected SMTP servers. If that is necessary, set up a local SMTP proxy.When asked whether to start the
cryptctl
server, entery
.
To check the status of the service
cryptctl-server
, use:#
systemctl status cryptctl-server
To reconfigure the server later, do either of the following:
Run the command
cryptctl init-server
again.cryptctl
proposes the existing settings as the defaults, so that you only need to specify the values that you want to change.Make changes directly in the configuration file
/etc/sysconfig/cryptctl-server
.However, to avoid issues, do not change the settings
AUTH_PASSWORD_HASH
andAUTH_PASSWORD_SALT
manually. The values of these options need to be calculated correctly.
13.2 Setting up a cryptctl
client #
The following interactive setup of cryptctl
is
currently the only setup method.
Make sure the following preconditions are fulfilled:
A
cryptctl
server is available over the network.There is a directory to encrypt.
The client machine has an empty partition available that is large enough to fit the directory to encrypt.
When using a self-signed certificate, the certificate (
*.crt
file) generated on the server is available locally on the client. Otherwise, the certificate authority of the server certificate must be trusted by the client.If you set up the server to require clients to authenticate using a client certificate, prepare a TLS certificate for the client which is signed by the CA certificate you chose for the server.
As
root
, run:#
cryptctl encrypt
Answer each of the following prompts and press Enter after every answer. If there is a default answer, it is shown in square brackets at the end of the prompt.
Specify the host name and port to connect to on the
cryptctl
server.If you configured the server to have clients authenticate to it using a TLS certificate, specify a certificate and a key file for the client. The client certificate must be signed by the certificate authority chosen when setting up the server.
Specify the absolute path to the server certificate (the
*.crt
file).Enter the encryption password that you specified when setting up the server.
Specify the path to the directory to encrypt. Specify the path to the empty partition that contains the encrypted content of the directory.
Specify the number of machines that are allowed to decrypt the partition simultaneously.
Then specify the timeout in seconds before additional machines are allowed to decrypt the partition after the last vital sign was received from the client or clients.
When a machine unexpectedly stops working and then reboots, it needs to be able to unlock its partitions again. That means this timeout should be set to a time slightly shorter than the reboot time of the client.
Important: Timeout lengthIf the time is set too long, the machine cannot decrypt encrypted partitions on the first try.
cryptctl
then continues to periodically check whether the encryption key has become available. However, this introduces a delay.If the timeout is set too short, machines with a copy of the encrypted partition have an increased chance of unlocking the partition first.
To start encryption, enter
yes
.cryptctl
encrypts the specified directory to the previously empty partition and then mounts the newly encrypted partition. The file system type is the same type as the original unencrypted file system.Before creating the encrypted partition,
cryptctl
moves the unencrypted content of the original directory to a location prefixed withcryptctl-moved-
.To check that the directory is indeed mounted correctly, use:
>
lsblk -o NAME,MOUNTPOINT,UUID
NAME MOUNTPOINT UUID [...] sdc └─sdc1 PARTITION_UUID └─cryptctl-unlocked-sdc1 /secret-partition UNLOCKED_UUIDcryptctl
identifies the encrypted partition by its UUID. For the previous example, that is the UUID displayed next tosdc1
.On the server, you can check whether the directory was decrypted using
cryptctl
.#
cryptctl list-keys
For a successfully decrypted partition, you see output like:
2019/06/06 15:50:00 ReloadDB: successfully loaded database of 1 records Total: 1 records (date and time are in zone EDT) Used By When UUID Max.Users Num.Users Mount Point IP_ADDRESS 2019-06-06 15:00:50 UUID 1 1 /secret-partition
For a partition not decrypted successfully, you see output like:
2019/06/06 15:50:00 ReloadDB: successfully loaded database of 1 records Total: 1 records (date and time are in zone EDT) Used By When UUID Max.Users Num.Users Mount Point 2019-06-06 15:00:50 UUID 1 1 /secret-partition
See the difference in the empty
Used by
column.Verify that the UUID shown is that of the previously encrypted partition.
After verifying that the encrypted partition works, delete the unencrypted content from the client. For example, use
rm
. For more safety, overwrite the content of the files before deleting them, for example, usingshred -u
.Important:shred
does not guarantee that data is erasedDepending on the type of storage media, using
shred
is not a guarantee that all data is removed. In particular, SSDs employ wear leveling strategies that rendershred
ineffective.
The configuration for the connection from client to server is stored in
/etc/sysconfig/cryptctl-client
and can be edited
manually.
The server stores an encryption key for the client partition in
/var/lib/cryptctl/keydb/PARTITION_UUID
.
13.3 Configuring /etc/fstab for LUKS volumes #
When configuring the mount point for a new file system encrypted with
LUKS, YaST uses, by default, the name of the encrypted device
in /etc/fstab
.
(For example, /dev/mapper/cr_sda1
.) Using the
device name, rather than the UUID or volume label, results in a more robust
operation of systemd generators and other related tools.
You have the option to adjust that default behavior for each device, either with the Expert Partitioner in the installer, or via AutoYaST.
This change does not affect upgrades or any other scenario in which the
mount points are already defined in /etc/fstab
.
Newly created mount points are affected, such as during the
installation of a new system, or creating new partitions on running
systems.
13.4 Checking partition unlock status using server-side commands #
When a cryptctl
client is active, it sends a
“heartbeat” to the cryptctl
server every
10 seconds. If the server does not receive a heartbeat from the client
for the length of the timeout configured during the client setup, the
server assumes that the client is offline. It then allows another
client to connect (or allow the same client to reconnect after a reboot).
To see the usage status of all keys, use:
#
cryptctl list-keys
The information under Num. Users
shows whether the key
is currently in use. To see more detail on a single key, use:
#
cryptctl show-key UUID
This command shows information about mount point, mount options, usage options, the last retrieval of the key and the last three heartbeats from clients.
Additionally, you can use journalctl
to find logs of
when keys were retrieved.
13.5 Unlocking encrypted partitions manually #
There are two ways of unlocking a partition manually, both of which are run on a client:
Online unlocking. Online unlocking allows circumventing timeout or user limitations. This method can be used when there is a network connection between client and server but the client could not (yet) unlock the partition automatically. This method unlocks all encrypted partitions on a machine.
To use it, run
cryptctl online-unlock
. Be prepared to enter the password specified when setting up the server.Offline unlocking. This method can be used when a client cannot or must not be brought online to communicate with its server. The encryption key from the server must still be available. This method is meant as a last resort and can only unlock a single partition at a time.
To use it, run
cryptctl offline-unlock
. The server's key file for the requisite partition (/var/lib/cryptctl/keydb/PARTITION_UUID
) needs to be available on the client.
13.6 Maintenance downtime procedure #
To ensure that partitions cannot be decrypted during a maintenance
downtime, turn off the client and disable the
cryptctl
server. You can do so by either:
Stopping the service
cryptctl-server
:#
systemctl stop cryptctl-server
Unplugging the
cryptctl
server from the network.
13.7 Setting up an HA environment for cryptctl-server service #
To avoid downtimes when the cryptctl-server needs to be stopped for maintenance or suffers a damage, it is strongly recommended to set up the cryptctl-server in an HA environment. You need at least a two-node High Availability cluster for this. The following setup shows how to create a two-node HA cluster for cryptctl-server using self-signed certificates.
Make sure the following preconditions are fulfilled:
At least two servers which have SUSE Linux Enterprise Server and the High Availability extension installed are available. All servers must also have the cryptctl package installed. All servers can reach each other via SSH.
If you set up a new cluster, you need an additional IP address for the HA Web Console of the cluster (AdminIP).
A separate IP address (CrypServerIP) is reserved for the cryptctl-server.
A separate dns-name (CrypServerHostName) is reserved for the cryptctl-server and is resolved to the above IP address.
An HA-enabled block device or NFS share is available to store the keys.
In our example, we use an NFS share:
nfs-server.example.org/data/cryptctl-keys
. It is mounted to the standard location/var/lib/cryptctl/keydb
.It is strongly recommended to use an SBD device.
Log in to Node1 as
root
.Set up a cryptctl-server as described in Section 13.1, “Setting up a
cryptctl
server”. Use the following parameters:To create the certificate, use the dedicated hostname CrypServerHostName of the cryptctl server. Do not use the host name of the host.
Use the dedicated IP address CrypServerIP of the cryptctl server. Do not use the default IP address setting.
Do not configure a KMIP server.
When asked whether to start the
cryptctl
server, entern
.
Set up a two-node HA cluster:
- Important
Node1 must be the server where you have configured the cryptctl server.
On the machine where you have configured the cryptctl server, set up the first node as follows:
#
crm cluster init -i NetDev -A AdminIP -n ClusterName
Log in via SSH to Node2 and join the cluster from there:
#
ssh Node2
#
crm cluster join -y Node1
For more information, also see the Installation and Setup Quick Start.
Set up the resource group for the cryptctl server:
You can set up all needed resource agents and copy all files to all nodes with the cryptcl crm-shell-script in one step. We strongly recommend that you verify the setup in the first step:
#
crm script verify cryptctl \ cert-path=/etc/cryptctl/servertls/CertificateFileName \ cert-key-path=/etc/cryptctl/servertls/CertificateKeyFileName \ virtual-ip:ip=CrypServerIP \ filesystem:device=DevicePath filesystem:fstype=FileSystemType
If the check was successful, set up the cluster group by running the script as follows:
#
crm script verify cryptctl \ cert-path=/etc/cryptctl/servertls/CertificateFileName \ cert-key-path=/etc/cryptctl/servertls/CertificateKeyFileName \ virtual-ip:ip=CrypServerIP \ filesystem:device=DevicePath filesystem:fstype=FileSystemType
Name |
Mandatory |
Default Value |
Description |
---|---|---|---|
id | no | cryptctl | Name of the resource group. |
cert-path | yes | The full path to the created certificate. | |
cert-key-path | yes | The full path to the created certificate key. | |
virtual-ip:id | no | cryptctl-vip | The ID of the virtual IP resource for the cryptctl server. |
virtual-ip:ip | yes | The IP address of the cryptctl server. | |
virtual-ip:nic | no | Detected by the virtual-ip resource agent. | The network device the cryptctl server should be listening on. Only required if the device cannot be detected from the IP address. |
virtual-ip:cidr_netmask | no | Detected by the virtual-ip resource agent. | The numeric netmask of the IP address of the cryptctl server. Only required if the netmask cannot be detected from the IP address. |
virtual-ip:broadcast | no | Detected by the virtual-ip resource agent. | The broadcast address of the IP address of the cryptctl server. Only required if this cannot be detected from the IP address. |
filesystem:id | no | cryptctl-filesystem | The ID of the file system resource containing the disk encryption keys and records. |
filesystem:device | yes |
The device containing the file system. This can be a block device
like /dev/sda... or an NFS share path
server:/path .
| |
filesystem:directory | no | /var/lib/cryptctl/keydb |
The directory where the device containing the file system is located. This can be a block device
like /dev/sda... or an NFS share path
server:/path .
|
filesystem:fstype | yes | The file system type (for example, NFS, XFS, EXT4). | |
filesystem:options | no | The default options of the selected file system. | Mount options for the file system. |
13.8 More information #
For more information, also see the project home page https://github.com/SUSE/cryptctl/.