7 Transport Layer Security (TLS) Overview

Read the following if you are using the default cert-name: my-public-cert in your model.

The bundled test cert for public endpoints, located at ~/openstack/my_cloud/config/tls/certs/my-public-cert, is now expired but was left in the product in case you changed the content with your valid cert. Please verify if the certificate is expired and generate your own by following the guidelines further down on this page or by using a generic instruction from the web.

You can verify the expiry by running this command:

openssl x509 -in ~/openstack/my_cloud/config/tls/certs/my-public-cert -noout -enddate
notAfter=Feb 12 01:18:46 2019 GMT

Before you begin, the following list of terms will be helpful when generating and installing certificates.

The major difference between an external VIP certificate and an internal VIP certificate is that the internal VIP has approximately 40 domain names in the SAN. This is because each service has a different domain name in HPE Helion OpenStack 8. So it is unlikely that you can create an internal server certificate before running the configuration processor. But after a configuration processor run, a certificate request would be created for each of your cert-names.

For this example certificate configuration, let us assume there is no FQDN for the external VIP and that you are going to use the default IP address provided by HPE Helion OpenStack 8. Let's also assume that for the internal VIP you will use the defaults as well. If you were to call your certificate authority "example-CA," the CA certificate would then be called "example-CA.crt" and the key would be called "example-CA.key." In the following examples, the external VIP certificate will be named "example-public-cert" and the internal VIP certificate will be named "example-internal-cert."

Any time you make a cert change when using your own CA:

Important

Be sure to install your own certificate for all production clouds after installing and testing your cloud. If you ever want to test or troubleshoot later, you will be able to revert to the sample certificate to get back to a stable state for testing.

Note

Unless this is a new deployment, do not update both the certificate and the CA together. Add the CA first and then run a site deploy. Then update the certificate and run tls-reconfigure, FND-CLU-stop, FND-CLU-start and then hlm-reconfigure. If a playbook has failed, rerun it with -vv to get detailed error information. The configure, HAproxy restart, and reconfigure steps are included below. If this is a new deployment and you are adding your own certs/CA before running site.yml this caveat does not apply.

You can add your own certificate by following the instructions below. All changes must go into the following file:

~/openstack/my_cloud/definition/data/network_groups.yml

The entries for TLS for the internal and admin load balancers are:

- provider: ip-cluster
        name: lb
        tls-components:
        - default
        components:
        # These services do not currently support TLS so they are not listed
        # under tls-components
        - nova-metadata
        roles:
        - internal
        - admin
        cert-file: openstack-internal-cert
        # The openstack-internal-cert is a reserved name and
        # this certificate will be autogenerated. You
        # can bring in your own certificate with a different name

        # cert-file: customer-provided-internal-cert
        # replace this with name of file in "config/tls/certs/"

The configuration processor will also create a request template for each named certificate under info/cert_reqs/, which looks like:

info/cert_reqs/customer-provided-internal-cert

These request templates contain the subject Alt-names that the certificates need. You can add to this template before generating your certificate signing request.

You would then send the CSR to your CA to be signed, and once you receive the certificate, place it in config/tls/certs

When you bring in your own certificate, you may want to bring in the trust chains (or CA certificate) for this certificate. This is usually not required if the CA is a public signer that is typically bundled with the operating system. However, we suggest you include it anyway by copying the file into the directory config/cacerts/.

HPE Helion OpenStack generates its own internal certificates but is designed to allow you to bring in your own certificates for the VIPs. Here is the general process.

Note

In a production setting you will not perform this step. You will use your company's CA or a valid public CA.

This section demonstrates to how you can create your own self-signed CA and then use this CA to sign server certificates. This CA can be your organization's IT internal CA that is self-signed and whose CA certificates are deployed on your organization's machines. This way the server certificate becomes legitimate.

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[ req_distinguished_name ]
CN = "openstack-vip"

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = "deployerincloud-ccp-c0-m1-mgmt"
DNS.2 = "deployerincloud-ccp-vip-CEI-API-mgmt"
DNS.3 = "deployerincloud-ccp-vip-CND-API-mgmt"
[...]
DNS.47 = "192.168.245.5"
IP.1 = "192.168.245.5"

=============end of certificate request file.

Note

In the case of a public VIP certificate, please add all the FQDNs you want it to support Currently, HPE Helion OpenStack does not add the hostname for the external-name specified in network_groups.yml to the certificate request file. However, you can add it to the certificate request file manually. Here we assume that myopenstack.test is your external-name. In that case you would add this line (to the full sample certificate request file shown in Example 7.1, “Certificate request file”):

DNS.48 = "myopenstack.test"

Note

Any attempt to use IP addresses rather than FQDNs in certificates must use subject alternate name entries that list both the IP address (needed for Google) and DNS with an IP (needed for a Python bug workaround). Failure to create the certificates in this manner will cause future installations of Go-based tools (such as Cloud Foundry, Stackato and other PaaS components) to fail.

In the case of a public VIP certificate, add all the FQDNs you want it to support. Openstack does not add the hostname for the external-name specified in network_groups.yml to the certificate request file. However, you can add it to the certificate request file manually. Assume that myopenstack.test is your external-name. In that case you would add this line to the full sample certificate request file shown above.

DNS.48 = "myopenstack.test"

Note

Any attempt to use IP addresses rather than FQDNs in certificates must use subject alternate name entries that list both the IP address (needed for Google) and DNS with an IP (needed for a Python bug workaround). Failure to create the certificates in this manner will cause future installations of Go-based tools (such as Cloud Foundry, Stackato and other PaaS components) to fail.

Note

In a production setting you will not perform this step. You will use your company's CA or a valid public CA.

Now that you have a CA and a certificate request file, it is time to generate a CSR.

Note

Please use a unique CN for your example Certificate Authority and do not install multiple CA certificates with the same CN into your cloud.

export EXAMPLE_SERVER_KEY_FILE='example-internal-cert.key'
export EXAMPLE_SERVER_CSR_FILE='example-internal-cert.csr'
export EXAMPLE_SERVER_REQ_FILE=example-internal-cert.req
openssl req -newkey rsa:2048 -nodes -keyout "$EXAMPLE_SERVER_KEY_FILE" \
-out "$EXAMPLE_SERVER_CSR_FILE" -extensions v3_req -config "$EXAMPLE_SERVER_REQ_FILE"

In production you would usually send the generated example-internal-cert.csr file to your IT department. But in this example you are your own CA, so sign and generate a server certificate.

Note

In a production setting you will not perform this step. You will send the CSR created in the previous section to your company CA or a to a valid public CA and have them sign and send you back the certificate.

This section demonstrates how you would use the self-signed CA that you created earlier to sign and generate a server certificate. A server certificate is essentially a signed public key, the signer being a CA and trusted by a client. When you install this signed CA's certificate (called CA certificate or trust chain) on the client machine, you are telling the client to trust this CA, and to implicitly trust any server certificates that are signed by this CA. This creates a trust anchor.

CA configuration file

When the CA signs the certificate, it uses a configuration file that tells it to verify the CSR. Note that in a production scenario the CA takes care of this for you.

By default, the cipher suite is set to: HIGH:!aNULL:!eNULL:!DES:!3DES. This setting is recommended in the OpenStack documentation. You may override this by editing config/haproxy/defaults.yml. The parameters can be found under the haproxy_globals list.

- "ssl-default-bind-ciphers HIGH:!aNULL:!eNULL:!DES:!3DES"
- "ssl-default-server-ciphers HIGH:!aNULL:!eNULL:!DES:!3DES"

Make the changes as needed. Keep the two options identical.

You can determine if an endpoint is behind TLS by running the following command, which probes a Keystone identity service endpoint that is behind TLS:

tux > echo | openssl s_client -connect 192.168.245.5:5000 | \
  openssl x509 -fingerprint -noout
depth=0 CN = openstack-vip
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = openstack-vip
verify error:num=27:certificate not trusted
verify return:1
depth=0 CN = openstack-vip
verify error:num=21:unable to verify the first certificate
verify return:1
DONE
SHA1 Fingerprint=C6:46:1E:59:C6:11:BF:72:5E:DD:FC:FF:B0:66:A7:A2:CC:32:1C:B8

The next command probes a MariaDB endpoint that is not behind TLS:

echo | openssl s_client -connect 192.168.245.5:3306 | openssl x509 -fingerprint -noout
140448358213264:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:795:
unable to load certificate
140454148159120:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE

You can determine if the trust chain is correctly deployed by running the following commands:

tux > echo | openssl s_client -connect 192.168.245.9:5000 2>/dev/null \
  | grep code
Verify return code: 21 (unable to verify the first certificate)
tux > echo | openssl s_client -connect 192.168.245.9:5000 -CAfile \
  /usr/local/share/ca-certificates/openstack_frontend_cacert.crt 2>/dev/null \
  | grep code
Verify return code: 0 (ok)

The first command produces error 21, which is then fixed by providing the CA certificate file. This verifies that the CA certificate matches the server certificate.

You should leave TLS enabled in production. However, if you need to disable it for any reason, you must change tls-components to components in network_groups.yml (as shown earlier) and comment out the cert-file. Additionally, if you have a network_groups.yml file from a previous installation, TLS will not be enabled unless you change components to tls-components in that file. By default, Horizon is configured with TLS in the input model. You should not disable TLS in the input model for Horizon as that is a public endpoint and is required. Additionally, you should keep all services behind TLS, but using the input model file network_groups.yml you may turn TLS off for a service for troubleshooting or debugging. TLS should always be enabled for production environments.

If you are using an example input model on a clean install, all supported TLS services will be enabled before deployment of your cloud. If you want to change this setting later, such as when upgrading, you can change the input model and reconfigure the system. The process is as follows:

MySQL replication over TLS is disabled. This is true even if you followed the instruction to turn on Mysql TLS in the previous section. Those steps turn on the service interactions to the database.

Turning on MySQL replication over TLS

Note

Using TLS connections for MySQL replication will incur a performance cost.

You should have already enabled TLS for MySQL client interactions in the previous section. If not, read Section 7.3.1, “Enabling TLS on the database server for client access”.

TLS for MySQL replication is not turned on by default. Therefore, you will need to follow a manual process. Again, the steps are different for new systems and upgrades.

If your cluster is already up, perform these steps to enable MySQL replication over TLS:

Almost all services that have a database are able to communicate over TLS. You can test whether a service, in this example the Identity service (Keystone), is communicating with MySQL over TLS by executing the following steps:

On the one of the rabbitmq nodes you can list the clients and their TLS status by running:

$ sudo rabbitmqctl -q list_connections ssl state ssl_protocol user name

You will see output like this where true indicates the client is using TLS for the connection, and false, as shown here, indicates the connection is over TCP:

Listing connections ...

rmq_barbican_user       false

Other indicators will be rabbit_use_ssl = True in the Oslo messaging section of client configurations. The list of clients that support TLS are as follows:

Has the deployer been restarted after the original site installation or is this a new deployer? If so, TLS certificates need to be bootstrapped before a playbook is run with limits. You can do this by running the following command.

cd ~/scratch/ansible/next/ardana/ansible
ansible-playbook -i hosts/verb_hosts tls-reconfigure.yml --limit TLS-CA

In general, if a certificate update fails, it is because of the following: Haproxy has not restarted or the Trust chain is not installed. This is the certificate of the CA that signed the server certificate.

It is important to note that while HPE Helion OpenStack 8 allows you to add new trust chains, it would be better if you add all the required trust chains during the initial deploy. Trust chain changes can impact services.

However, this does not apply to certificates. There is a certificate-related issue whereby haproxy is not restarted if certificate content has been changed but the certificate file name remained the same. If you are having issues and you have replaced the content of existing CA file with new content, create another CA file with a new name. Also make sure the CA file has a .crt extension.

Do not update both certificate and the CA together. Add the CA first and then run a site deploy. Then update the certificate and run tls-reconfigure, FND-CLU-stop, FND-CLU-start and then ardana-reconfigure. If you know which playbook failed, rerun it with -vv to get detaled error information. The configure, HAproxy restart, and reconfigure steps are included in Section 7.2, “TLS Configuration”.

You can run the following commands to see if client libraries see the CA you have added:

~/scratch/ansible/next/ardana/ansible$ ansible -i hosts/verb_hosts FND-STN -a 'sudo keytool -list -alias \
    debian:username-internal-cacert-001.pem -keystore /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/security/cacerts -storepass changeit'
  padawan-ccp-c0-m1-mgmt | FAILED | rc=1 >>
  sudo: keytool: command not found

  padawan-ccp-comp0001-mgmt | FAILED | rc=1 >>
  sudo: keytool: command not found

  padawan-ccp-comp0003-mgmt | FAILED | rc=1 >>
  sudo: keytool: command not found

  padawan-ccp-comp0002-mgmt | FAILED | rc=1 >>
  sudo: keytool: command not found

  padawan-ccp-c1-m1-mgmt | success | rc=0 >>
  debian:username-internal-cacert-001.pem, May 9, 2016, trustedCertEntry,
  Certificate fingerprint (SHA1): E7:B2:6E:9E:00:FB:86:0F:E5:46:CD:B8:C5:67:13:53:4E:3D:8F:43

  padawan-ccp-c1-m2-mgmt | success | rc=0 >>
  debian:username-internal-cacert-001.pem, May 9, 2016, trustedCertEntry,
  Certificate fingerprint (SHA1): E7:B2:6E:9E:00:FB:86:0F:E5:46:CD:B8:C5:67:13:53:4E:3D:8F:43

  padawan-ccp-c1-m3-mgmt | success | rc=0 >>
  debian:username-internal-cacert-001.pem, May 9, 2016, trustedCertEntry,
  Certificate fingerprint (SHA1): E7:B2:6E:9E:00:FB:86:0F:E5:46:CD:B8:C5:67:13:53:4E:3D:8F:43

Java client libraries are used by Monasca, so compute nodes will not have them. So the first three errors are expected. Check that the fingerprint is correct by checking the CA:

~/scratch/d002-certs/t002$ openssl x509 -in example-CA.crt -noout -fingerprint
  SHA1 Fingerprint=E7:B2:6E:9E:00:FB:86:0F:E5:46:CD:B8:C5:67:13:53:4E:3D:8F:43

If they do not match, there likely was a name collision. Add the CA cert again with a new file name. If you get Monasca errors but find that the fingerprints match, try stopping and restarting Monasca.

ansible-playbook -i hosts/verb_hosts monasca-stop.yml
ansible-playbook -i hosts/verb_hosts monasca-start.yml

Use the following steps to re-create expired TLS certificates for MySQL Percona clusters.

Use the following steps to re-create expired TLS certificates for RabbitMQ.

Certificates can fail in HPE Helion OpenStack 8 due to the following.