Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE OpenStack Cloud 9

8 Transport Layer Security (TLS) Overview Edit source

The Transport Layer Security (TLS) protocol, successor to SSL, provides the mechanisms to ensure authentication, non-repudiation, confidentiality, and integrity of user communications to and between the SUSE OpenStack Cloud services from internal and public endpoints.

OpenStack endpoints are HTTP (REST) services providing APIs to other OpenStack services on the management network. All traffic to OpenStack services coming in on the public endpoints and some traffic between services can be secured using TLS connections.

In SUSE OpenStack Cloud 9, the following are enabled for TLS

  • API endpoints in the internal and admin VIPs can now be secured by TLS.

  • API endpoints can be provided with their own certificates (this is shown in the model examples) or they can simply use the default certificate.

  • The barbican key management service API can be secured by TLS from the load balancer to the service endpoint.

  • You can add multiple trust chains (certificate authority (CA) certificates).

  • Fully qualified domain names (FQDNs) can be used for public endpoints and now they can be changed. The external name in the input model files (in ~/openstack/my_cloud/definition/data/network_groups.yml) is where the domain name is indicated and changed.

  • There are two monitoring alarms specific to certificates, 14-days to certificate expiration and 1-day to expiration.

  • TLS can be turned off/on for individual endpoints.

8.1 Comparing Clean Installation and Upgrade of SUSE OpenStack Cloud Edit source

Clean install: all TLS-encrypted services are already listed under tls-components in network_groups.yml

You just have to:

  • Add your self-signed CA cert and server cert (for testing)

  • Or add your public (or company) CA-signed server cert and the public (or company) CA cert (for production)

Upgrade: you do not have TLS enabled already on the internal endpoints so you need to

  • Add your self-signed CA cert and server cert (for testing)

  • Or add your public (or company) CA-signed server cert and the public (or company) CA cert (for production)

  • Add all the services to tls-components in network_groups.yml

For information on enabling and disabling TLS, see Section 8.2, “TLS Configuration”.

For instructions on installing certificates, see Section 8.2, “TLS Configuration”.

8.2 TLS Configuration Edit source

In SUSE OpenStack Cloud 9, you can provide your own certificate authority and certificates for internal and public virtual IP addresses (VIPs), and you should do so for any production cloud. The certificates automatically generated by SUSE OpenStack Cloud are useful for testing and setup, but you should always install your own for production use. For further information, see Book “Deployment Guide using Cloud Lifecycle Manager”, Chapter 41 “Configuring Transport Layer Security (TLS)”

8.3 Enabling TLS for MySQL Traffic Edit source

MySQL traffic can be encrypted using TLS. For completely new SUSE OpenStack Cloud deployments using the supplied input model example files, you will have to uncomment the commented entries for tls-component-endpoints:. For upgrades from a previous version, you will have to add the entries to your input model files if you have not already done so. This topic explains how to do both.

8.3.1 Enabling TLS on the database server for client access Edit source

  1. Edit network_groups.yml to either add mysql under tls-component-endpoints in your existing file from a previous version, or uncomment it if installing from scratch.

    tls-component-endpoints:
      - mysql
  2. After making the necessary changes, commit the changed file to git and run the config-processor-run and reconfigure Ansible playbooks:

    cd ~/openstack
    git add -A
    git commit -m "My changed config"
    cd ~/openstack/ardana/ansible/
    ansible-playbook -i hosts/localhost config-processor-run.yml -e encrypt="<encryption key>" -e rekey=""
    ansible-playbook -i hosts/localhost ready-deployment.yml
    cd ~/scratch/ansible/next/ardana/ansible
  3. Next, either run site.yml if you are installing a new system:

    ansible-playbook -i hosts/verb_hosts site.yml
  4. or ardana-reconfigure if you are reconfiguring an existing one:

    ansible-playbook -i hosts/verb_hosts ardana-reconfigure.yml

8.3.2 MySQL replication over TLS Edit source

MySQL replication over TLS is disabled. This is true even if you followed the instruction to turn on Mysql TLS in the previous section. Those steps turn on the service interactions to the database.

Turning on MySQL replication over TLS

Note
Note

Using TLS connections for MySQL replication will incur a performance cost.

You should have already enabled TLS for MySQL client interactions in the previous section. If not, read Section 8.3.1, “Enabling TLS on the database server for client access”.

TLS for MySQL replication is not turned on by default. Therefore, you will need to follow a manual process. Again, the steps are different for new systems and upgrades.

8.3.3 Enabling TLS for MySQL replication on a new deployment Edit source

  1. Log in to the Cloud Lifecycle Manager node and before running the config processor, edit the ~/openstack/my_cloud/config/mariadb/defaults.yml file.

  2. Search for mysql_gcomms_bind_tls. You should find this section:

    # TLS disabled for cluster
    #mysql_gcomms_bind_tls: "{{ host.bind['FND_MDB'].mysql_gcomms.tls }}"
    mysql_gcomms_bind_tls: False
  3. Uncomment the appropriate line so the file looks like this:

    # TLS enabled for cluster
    mysql_gcomms_bind_tls: "{{ host.bind['FND_MDB'].mysql_gcomms.tls }}"
    #mysql_gcomms_bind_tls: False
  4. Follow the steps to deploy or reconfigure your cloud: Step 2 in Section 8.3.1, “Enabling TLS on the database server for client access”.

8.3.4 Enabling TLS for MySQL replication on an existing system Edit source

If your cluster is already up, perform these steps to enable MySQL replication over TLS:

  1. Edit the following two files: ~/openstack/my_cloud/config/mariadb/defaults.yml and ~/scratch/ansible/next/ardana/ansible/roles/FND-MDB/defaults/main.yml. Note that these files are identical. The first is a master file and the second is a scratch version that is used for the current deployment. Make the same changes as explained in Section 8.3.3, “Enabling TLS for MySQL replication on a new deployment”.

  2. Then run the following command:

    ansible-playbook -i hosts/verb_hosts tls-percona-reconfigure.yml

    After this your MySQL should come up and replicate over TLS. You need to follow this section again if you ever want to switch TLS off for MySQL replication. You also must repeat these steps if any lifecycle operation changes the mysql_gcomms_bind_tls option.

8.3.5 Testing whether a service is using TLS Edit source

Almost all services that have a database are able to communicate over TLS. You can test whether a service, in this example the Identity service (keystone), is communicating with MySQL over TLS by executing the following steps:

  1. Log into a node member of the database cluster, change to the root user (such as by using sudo -i) and run the mysql command.

    root@<server>:~# mysql

    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

  2. Run:

    mysql> select * from information_schema.user_statistics where user='keystone'\G
  3. Note the results. TOTAL_SSL_CONNECTIONS should not be zero:

    *************************** 1. row ***************************
                      USER: keystone
         TOTAL_CONNECTIONS: 316
    CONCURRENT_CONNECTIONS: 0
            CONNECTED_TIME: 905790
                 BUSY_TIME: 205
                  CPU_TIME: 141
            BYTES_RECEIVED: 197137617
                BYTES_SENT: 801964
      BINLOG_BYTES_WRITTEN: 0
              ROWS_FETCHED: 972421
              ROWS_UPDATED: 6893
           TABLE_ROWS_READ: 1025866
           SELECT_COMMANDS: 660209
           UPDATE_COMMANDS: 3039
            OTHER_COMMANDS: 299746
       COMMIT_TRANSACTIONS: 0
     ROLLBACK_TRANSACTIONS: 295200
        DENIED_CONNECTIONS: 0
          LOST_CONNECTIONS: 83
             ACCESS_DENIED: 0
             EMPTY_QUERIES: 71778
     TOTAL_SSL_CONNECTIONS: 298
    1 row in set (0.00 sec)
    
    mysql>

8.4 Enabling TLS for RabbitMQ Traffic Edit source

RabbitMQ traffic can be encrypted using TLS. To enable it, you will have to add entries for tls-component-endpoints: in your input model files if you have not already done so. This topic explains how.

  1. Edit openstack/my_cloud/definition/data/network_groups.yml, adding rabbitmq to the tls-component-endpoints section:

    tls-component-endpoints:
         - barbican-api
         - mysql
         - rabbitmq
  2. Commit the changes:

    cd ~/openstack
    git add -A
    git commit -m "My changed config"
  3. Then run the typical deployment steps:

    cd ~/openstack/ardana/ansible/
    ansible-playbook -i hosts/localhost config-processor-run.yml -e encrypt="<encryption key>" -e rekey=""
    ansible-playbook -i hosts/localhost ready-deployment.yml
  4. Change directories:

    cd ~/scratch/ansible/next/ardana/ansible
  5. Then for a fresh TLS install run:

    ansible-playbook -i hosts/verb_hosts site.yml
  6. Or, to reconfigure an existing system run:

    ansible-playbook -i hosts/verb_hosts ardana-reconfigure.yml

8.4.1 Testing Edit source

On the one of the rabbitmq nodes you can list the clients and their TLS status by running:

$ sudo rabbitmqctl -q list_connections ssl state ssl_protocol user name

You will see output like this where true indicates the client is using TLS for the connection, and false, as shown here, indicates the connection is over TCP:

Listing connections ...

rmq_barbican_user       false

Other indicators will be rabbit_use_ssl = True in the Oslo messaging section of client configurations. The list of clients that support TLS are as follows:

  • barbican

  • cinder

  • designate

  • Eon

  • glance

  • heat

  • ironic

  • keystone

  • monasca

  • neutron

  • nova

  • Octavia

8.5 Troubleshooting TLS Edit source

8.5.1 Troubleshooting TLS certificate errors when running playbooks with a limit Edit source

Has the deployer been restarted after the original site installation or is this a new deployer? If so, TLS certificates need to be bootstrapped before a playbook is run with limits. You can do this by running the following command.

cd ~/scratch/ansible/next/ardana/ansible
ansible-playbook -i hosts/verb_hosts tls-reconfigure.yml --limit TLS-CA

8.5.2 Certificate Update Failure Edit source

In general, if a certificate update fails, it is because of the following: Haproxy has not restarted or the Trust chain is not installed. This is the certificate of the CA that signed the server certificate.

8.5.3 Troubleshooting trust chain installation Edit source

It is important to note that while SUSE OpenStack Cloud 9 allows you to add new trust chains, it would be better if you add all the required trust chains during the initial deploy. Trust chain changes can impact services.

However, this does not apply to certificates. There is a certificate-related issue whereby haproxy is not restarted if certificate content has been changed but the certificate file name remained the same. If you are having issues and you have replaced the content of existing CA file with new content, create another CA file with a new name. Also make sure the CA file has a .crt extension.

Do not update both certificate and the CA together. Add the CA first and then run a site deploy. Then update the certificate and run tls-reconfigure, FND-CLU-stop, FND-CLU-start and then ardana-reconfigure. If you know which playbook failed, rerun it with -vv to get detaled error information. The configure, HAproxy restart, and reconfigure steps are included in Section 8.2, “TLS Configuration”.

You can run the following commands to see if client libraries see the CA you have added:

~/scratch/ansible/next/ardana/ansible$ ansible -i hosts/verb_hosts FND-STN -a 'sudo keytool -list -alias \
    debian:username-internal-cacert-001.pem -keystore /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/security/cacerts -storepass changeit'
  padawan-ccp-c0-m1-mgmt | FAILED | rc=1 >>
  sudo: keytool: command not found

  padawan-ccp-comp0001-mgmt | FAILED | rc=1 >>
  sudo: keytool: command not found

  padawan-ccp-comp0003-mgmt | FAILED | rc=1 >>
  sudo: keytool: command not found

  padawan-ccp-comp0002-mgmt | FAILED | rc=1 >>
  sudo: keytool: command not found

  padawan-ccp-c1-m1-mgmt | success | rc=0 >>
  debian:username-internal-cacert-001.pem, May 9, 2016, trustedCertEntry,
  Certificate fingerprint (SHA1): E7:B2:6E:9E:00:FB:86:0F:E5:46:CD:B8:C5:67:13:53:4E:3D:8F:43

  padawan-ccp-c1-m2-mgmt | success | rc=0 >>
  debian:username-internal-cacert-001.pem, May 9, 2016, trustedCertEntry,
  Certificate fingerprint (SHA1): E7:B2:6E:9E:00:FB:86:0F:E5:46:CD:B8:C5:67:13:53:4E:3D:8F:43

  padawan-ccp-c1-m3-mgmt | success | rc=0 >>
  debian:username-internal-cacert-001.pem, May 9, 2016, trustedCertEntry,
  Certificate fingerprint (SHA1): E7:B2:6E:9E:00:FB:86:0F:E5:46:CD:B8:C5:67:13:53:4E:3D:8F:43

Java client libraries are used by monasca, so compute nodes will not have them. So the first three errors are expected. Check that the fingerprint is correct by checking the CA:

~/scratch/d002-certs/t002$ openssl x509 -in example-CA.crt -noout -fingerprint
  SHA1 Fingerprint=E7:B2:6E:9E:00:FB:86:0F:E5:46:CD:B8:C5:67:13:53:4E:3D:8F:43

If they do not match, there likely was a name collision. Add the CA cert again with a new file name. If you get monasca errors but find that the fingerprints match, try stopping and restarting monasca.

ansible-playbook -i hosts/verb_hosts monasca-stop.yml
ansible-playbook -i hosts/verb_hosts monasca-start.yml

8.5.4 Expired TLS Certificates Edit source

Use the following steps to re-create expired TLS certificates for MySQL Percona clusters.

  1. Determine if the TLS certificates for MySQL / Percsona have expired.

    ardana > cd /etc/mysql/
    ardana > openssl x509 -noout -enddate -in control-plane-1-mysql-internal-cert.pem
  2. Regenerate the TLS certificates on the deployer.

    ardana > cd ~/scratch/ansible/next/hos/ansible
    ardana > ansible-playbook -i hosts/verb_hosts tls-reconfigure.yml --limit DEPLOYER_HOST
  3. Distribute the regenerated TLS certificates to the MySQL Percona clusters.

    ardana > cd ~/scratch/ansible/next/hos/ansible
    ardana > ansible-playbook -i hosts/verb_hosts --extra-vars "mysql_certs_needs_regeneration=true" tls-percona-reconfigure.yml
  4. Verify Percona cluster status on a controller node

    ardana > sudo mysql -e 'show status'

Use the following steps to re-create expired TLS certificates for RabbitMQ.

  1. Determine if SSL certificate for RabbitMQ is expired

    root # cd /etc/rabbitmq
    root # openssl x509 -noout -text -in control-plane-1-rabbitmq.pem | grep After
    Not After : Nov 6 15:15:38 2018 GMT
  2. Regenerate the TLS certificates on the deployer.

    ardana > cd ~/scratch/ansible/next/hos/ansible
    ardana > ansible-playbook -i hosts/verb_hosts tls-reconfigure.yml --limit DEPLOYER_HOST
  3. Reconfigure RabbitMQ. Certificate will be re-created if the input model is correct.

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts --extra-vars "rabbitmq_tls_certs_force_regeneration=true" rabbitmq-reconfigure.yml

8.5.5 Troubleshooting certificates Edit source

Certificates can fail in SUSE OpenStack Cloud 9 due to the following.

  • Trust chain issue. This is dealt with in the previous section

  • Wrong certificate: Compare the fingerprints. If they differ, then you have a wrong certificate somewhere.

  • Date range of the certificate is either in the future or expired: Check the dates and change certificates as necessary, observing the naming cautions above.

  • TLS handshake fails because the client does not support the ciphers the server offers. It is possible that you reused a certificate created for a different network model. Make sure the request file found under info/cert_req/ are used to create the certificate. If not, the service VIP names may not match.