Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Cloud Application Platform 1.5.2

20 Cloud Controller Database Secret Rotation

The Cloud Controller Database (CCDB) encrypts sensitive information like passwords. By default, the encryption key is generated by SCF, for details see https://github.com/SUSE/scf/blob/2d095a71008c33a23ca39d2ab9664e5602f8707e/container-host-files/etc/scf/config/role-manifest.yml#L1656-L1662. If it is compromised and needs to be rotated, new keys can be added. Note that existing encrypted information will not be updated. The encrypted information must be set again to have them re-encrypted with the new key. The old key cannot be dropped until all references to it are removed from the database.

Updating these secrets is a manual process. The following procedure outlines how this is done.

  1. Add env.CC_DB_CURRENT_KEY_LABEL and secrets.CC_DB_ENCRYPTION_KEYS to your scf-config-values.yaml. Replace the example values with your own.

    env:
      CC_DB_CURRENT_KEY_LABEL: new_key
    
    secrets:
      CC_DB_ENCRYPTION_KEYS:
        new_key: "new_key_value"
  2. Use the helm upgrade command to import the above data into the cluster. This restarts relevant pods with the new information from the previous step:

    Note
    Note: Setting UAA_CA_CERT

    Starting with SUSE Cloud Application Platform 1.5.2, you no longer need to set UAA_CA_CERT when using an external UAA with a certificate signed by a well known Certificate Authority (CA). It is only needed when you use an external UAA with either a certificate generated by the secret-generator or a self-signed certificate.

    If you need to set UAA_CA_CERT:

    1. Obtain your UAA secret and certificate:

      tux > SECRET=$(kubectl get pods --namespace uaa \
      --output jsonpath='{.items[?(.metadata.name=="uaa-0")].spec.containers[?(.name=="uaa")].env[?(.name=="INTERNAL_CA_CERT")].valueFrom.secretKeyRef.name}')
      
      tux > CA_CERT="$(kubectl get secret $SECRET --namespace uaa \
      --output jsonpath="{.data['internal-ca-cert']}" | base64 --decode -)"
    2. Then pass --set "secrets.UAA_CA_CERT=${CA_CERT}" as part of your helm command.

    tux > helm upgrade susecf-scf suse/cf \
    --values scf-config-values.yaml \
    --set "secrets.UAA_CA_CERT=${CA_CERT}" \
    --version 2.20.3
  3. Perform the rotation:

    1. Run the rotation for the encryption keys. A series of JSON-formatted log entries describing the key rotation progress for various Cloud Controller models will be displayed:

      tux > kubectl exec --namespace scf api-group-0 -- bash -c \
      'source /var/vcap/jobs/cloud_controller_ng/bin/ruby_version.sh; \
      export CLOUD_CONTROLLER_NG_CONFIG=/var/vcap/jobs/cloud_controller_ng/config/cloud_controller_ng.yml; \
      cd /var/vcap/packages/cloud_controller_ng/cloud_controller_ng; \
      bundle exec rake rotate_cc_database_key:perform'
    2. Restart the api-group pod.

      tux > kubectl delete pod api-group-0 --namespace scf --force --grace-period=0

    Note that keys should be appended to the existing secret to ensure existing environment variables can be decoded. Any operator can check which keys are in use by accessing the CCDB. If the encryption_key_label is empty, the default generated key is still being used

    tux > kubectl exec --stdin --tty mysql-0 --namespace scf -- /bin/bash -c 'mysql -p${MYSQL_ADMIN_PASSWORD} --socket /var/vcap/sys/run/pxc-mysql/mysqld.sock'
    MariaDB [(none)]> select name, encrypted_environment_variables, encryption_key_label from ccdb.apps;
    +--------+--------------------------------------------------------------------------------------------------------------+----------------------+
    | name   | encrypted_environment_variables                                                                              | encryption_key_label |
    +--------+--------------------------------------------------------------------------------------------------------------+----------------------+
    | go-env | XF08q9HFfDkfxTvzgRoAGp+oci2l4xDeosSlfHJUkZzn5yvr0U/+s5LrbQ2qKtET0ssbMm3L3OuSkBnudZLlaCpFWtEe5MhUe2kUn3A6rUY= | key0                 |
    +--------+--------------------------------------------------------------------------------------------------------------+----------------------+
    1 row in set (0.00 sec)

    For example, if keys were being rotated again, the secret would become:

    SECRET_DATA=$(echo "{key0: abc-123, key1: def-456}" | base64)

    and the CC_DB_CURRENT_KEY_LABEL would be updated to match the new key.

20.1 Tables with Encrypted Information

The CCDB contains several tables with encrypted information as follows:

apps

Environment variables

buildpack_lifecycle_buildpacks

Buildpack URLs may contain passwords

buildpack_lifecycle_data

Buildpack URLs may contain passwords

droplets

May contain Docker registry passwords

env_groups

Environment variables

packages

May contain Docker registry passwords

service_bindings

Contains service credentials

service_brokers

Contains service credentials

service_instances

Contains service credentials

service_keys

Contains service credentials

tasks

Environment variables

20.1.1 Update Existing Data with New Encryption Key

To ensure the encryption key is updated for existing data, the command (or its update- equivalent) can be run again with the same parameters. Some commands need to be deleted/recreated to update the label.

apps

Run cf set-env again

buildpack_lifecycle_buildpacks, buildpack_lifecycle_data, droplets

cf restage the app

packages

cf delete, then cf push the app (Docker apps with registry password)

env_groups

Run cf set-staging-environment-variable-group or cf set-running-environment-variable-group again

service_bindings

Run cf unbind-service and cf bind-service again

service_brokers

Run cf update-service-broker with the appropriate credentials

service_instances

Run cf update-service with the appropriate credentials

service_keys

Run cf delete-service-key and cf create-service-key again

tasks

While tasks have an encryption key label, they are generally meant to be a one-off event, and left to run to completion. If there is a task still running, it could be stopped with cf terminate-task, then run again with cf run-task.

Print this page