HPE Helion OpenStack 8

Migration Guide

Publication Date: April 11, 2024
Important
Important: Use Latest Documentation

Always use the latest online version of the documentation for the migration process.

1 Migration from HPE Helion OpenStack 5.0.x to HPE Helion OpenStack 5.SCP

The migration procedure is based on the following assumptions:

  • Skilled customer staff should be readily available to assist in the migration, with expertise in the areas of cloud administration, network administration, storage administration, and knowledge of data models and configuration settings used by the Cloud Lifecycle Manager.

    Ideally, systems involved in the upgrade should be physically accessible in case any operations become necessary that cannot be administered via their remote network interfaces.

    The customer is responsible for backing up all the control plane and deployer nodes to an external data store before starting the migration.

    Important
    Important: No Migration Rollback

    We do not support rollback of migration after the process has been started.

  • Metering, monitoring, and logging (MML) data will not be migrated (a clean installation will be done using a new MML back-end included with HPE Helion OpenStack 5.SCP).

  • Upgrading will require re-imaging the operating system on the physical disks containing the root partition of your controller nodes. Existing data on these disks, which may include audit logs and Swift objects, may be lost as a result. Ensure that your model files and any other important files on the deployer are copied to a safe location. (Note that the primary OpenStack MySQL database will undergo a backup/restore as part of the upgrade process to ensure that its contents are preserved.)

  • Review your existing partitioning model to understand how your data will be affected by re-imaging. Any data that is not on a disk separate from the primary root partition (i.e. Swift objects or audit logs) will be lost during re-imaging. Note that the control plane nodes must be booted from physical drives and must not have any attached SAN disks.

  • All Compute Nodes are running supported Linux distributions, and any prior hLinux Compute Nodes have been removed from the cloud. (For details about migrating instances from hLinux to SLES or RHEL Compute Nodes, please review the following HOS 5 documentation: Release Notes, Installation Guide, and Operations Guide.)

    • SUSE Linux Enterprise Server 12 SP3 for SLES Compute Nodes

    • RHEL 7.3 for RHEL Compute Nodes

  • There are some limitations to migrating instances. For details about migrating instances from hLinux to SLES or RHEL Compute Nodes, please review the following HOS 5 documentation: Release Notes, Installation Guide and Operations Guide.

  • HPE Helion OpenStack 8 dropped support for internally deployed storage such as Ceph and VSA. Data must be moved out of existing internal storage backends to an external SES cluster before performing an upgrade.

  • The cloud data model does not contain services which are not included in HPE Helion OpenStack 8, such as Ceph and VSA.

  • In order to migrate ESX compute VMs as part of this process, existing ESX hosts must be capable of migrating VMs between them, as rebalancing workloads will be necessary while ESX hosts are individually removed and restored as members of the clouds.

  • Any customizations to virtual environments used by OpenStack components will be lost during the upgrade process. These may include third party extensions, regardless of origin, that have not been included in HPE Helion OpenStack 8.

  • No significant issues or alarms are actively being reported by Monasca and Operations Console.

Preparation Phase

  1. Because all Freezer content will be obsolete upon migrating to the new platform, it is recommended to stop Freezer jobs and eliminate historic backups.

    (old-deployer)> source ~/backup.osrc
    (old-deployer)> for id in $(freezer job-list -f value | awk '{print $1}')
      do freezer job-stop $id
    done
    (old-deployer)> for container in $(openstack container list -f value); do
      for backup in $(openstack object list $container -f value); do
        openstack object delete $container $backup
        echo "Deleting $backup"
      done
    done
  2. If your cloud includes VMware ESX compute clusters, save the EON service activation template for future reference, as the EON service will no longer exist once migration is complete.

    (old-deployer)> source ~/service.osrc
    (old-deployer)> eon get-activation-template \
      --filename ~/activationtemplate.json --type esxcluster
  3. Obtain the HPE Helion OpenStack 5.SCP installation ISO and use the mount command to make its contents available on the existing deployer:

    (old-deployer)> mkdir hos-5.scp
    (old-deployer)> sudo mount HELION5SCP_ISO_FILE hos-5.scp
    (old-deployer)> tar -xzvf hos-5.scp/hos/hlm-migration-tools.tar.gz
    (old-deployer)> sudo umount hos-5.scp
  4. Run the validation playbook to ensure that the current cloud load and configuration is compatible with the migration process:

    (old-deployer)> cd hlm-migration-tools/ansible
    (old-deployer)> ansible-playbook \
      -i ~/scratch/ansible/next/hos/ansible/hosts/verb_hosts \
      pre-migration-validations.yml

    The validation playbook checks that:

    • The HPE Helion OpenStack version number is either 5.0.1 or 5.0.3

    • HPE Helion OpenStack 8-compatible Linux distributions on Compute Nodes

    • System clocks are synchronized across cloud nodes

    • HA capacity of Control Nodes (L3/DHCP agent)

    • Configuration and capacity of Control Nodes (Swift, Cinder)

    • Data model compatibility with HPE Helion OpenStack 8 (no vsa, ceph, eon-*, or cmc-service)

    Warning
    Warning: Check Validation Results

    Review the contents of /var/log/pre-migration-validations.log immediately to discover if any errors were identified while running the validation playbook. Any identified issues must be resolved before proceeding with the migration.

  5. Set up SMT repository hosting on a SLES server external to your cloud so that operating system and software updates can be applied during migration without waiting for additional download steps.

  6. Identify a candidate seed Control Node in the primary control plane. This node must meet the following requirements:

    • Networking connectivity is equivalent to that of the existing deployer.

    • It is a member of the database cluster (running the MySQL service).

    • It is not actively hosting any singleton OpenStack services (cinder-volume or nova-consoleauth). Test for empty output with the following command:

      (each controller)> ps -ef | egrep 'cinder-volume|nova-consoleauth'
  7. Consider the effect of control plane downtime on existing DHCP leases:

    • Default DHCP lease time is 48 hours as defined in the ~/helion/hos/ansible/roles/neutron-common/templates/neutron.conf.j2 file by the dhcp_lease_duration parameter.

    • DHCP leases are administered by Neutron and dnsmasq. This functionality will not be available during the control plane downtime.

    • DHCP clients typically renew their leases either after 50% of the lease time or upon reboot.

    • Normally, this leaves a 24-hour window for control plane downtime at any point before workload's DHCP leases run out and cannot be renewed. While this is typically enough for performing a migration, increasing the DHCP lease time to 144 hours (518400 seconds) is recommended to ensure that lease expiration will not be a concern that affects your migration, even if unexpected delays in the process are encountered.

    • To view the upcoming timeouts of DHCP leases:

      1. Determine which servers are running the DHCP agents

        (old-deployer)> neutron agent-list  | awk '/DHCP/{print $7}'
        mig-cp1-neut-m2-mgmt
        mig-cp1-neut-m1-mgmt
      2. From the old deployer, SSH to each host listed above, and display the lease times for active VMs.

        (old-deployer)> ssh mig-cp1-neut-m2-mgmt
                (mig-cp1-neut-m2)> for i in $(cat /var/run/neutron/dhcp/*/leases | awk '{print $1}'); do
                             date -d @$i
                           done
        (mig-cp1-neut-m2)> exit
        ...
    • To increase the DHCP lease time before migrating, open the roles/neutron-common/templates/neutron.conf.j2 file from the ~/helion/hos/ansible directory, and edit the dhcp_lease_duration parameter. Then, apply the change with:

      (old-deployer)> git commit -a -m "DHCP lease adjustment"
      (old-deployer)> ansible-playbook -i hosts/localhost config-processor-run.yml
      (old-deployer)> ansible-playbook -i hosts/localhost ready-deployment.yml
      (old-deployer)> cd ~/scratch/ansible/next/hos/ansible
      (old-deployer)> ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml

      This will reset the duration for new leases, but the existing IP assignments to VMs will wait until their original DHCP lease expires before being refreshed with the updated lease duration. Thus, these steps should be performed at least 2/3rds of the lease time prior to the planned start of migration.

      Continue to monitor the upcoming DHCP lease expirations as described above until all existing leases have been refreshed to a lengthier period, and then continue with the migration.

  8. Prepare the cloud data model for migration:

    • In the ~/helion/my_cloud/definition/data/disks_*.yml files used by your server role that includes monitoring services, replace the existing Vertica partition definition with two Cassandra partitions:

      - name: cassandra_db
        size: (original size used by vertica partition, minus 1%)
        mount: /var/cassandra/data
        fstype: ext4
        mkfs-opts: -O large_file
        consumer:
          name: vertica
      - name: cassandra_log
        size: 1%
        mount: /var/cassandra/commitlog
        fstype: ext4
        mkfs-opts: -O large_file
    • Edit ~/helion/my_cloud/definition/data/control_plane.yml and identify the cluster definition that already includes an entry for mysql in its service-components.

    • Ensure that lifecycle-manager is included in the same cluster's service-components list. If it was necessary to add this as a new entry, it will also need to be removed from its original position in the deployer.

    • Update the servers.yml entry for all nodes in the control plane to specify distro-id: sles12sp3-x86_64 in each one. (Otherwise, they will be re-imaged with hLinux.)

      Note
      Note: VMware ESX Nodes

      All ESX proxy and OVSvApp nodes should be included in the control plane updates to the new distro-id as described above.

    • If your environment includes a 3PAR storage backend, edit the file ~/helion/my_cloud/config/cinder/cinder.conf.j2 and update your driver to its new name (from HP3PARFCDriver to HPE3PARFCDriver, or from HP3PARISCSIDriver to HPE3PARISCSIDriver.)

    • Commit your changes to the Git model:

      (old-deployer)> git commit -a -m "Prepare for HOS 5.1 installation"
  9. If the intended seed node also currently hosts Neutron L3 routing services, evacuate the routers from the node so that it can be re-imaged without interruption to tenant networks.

    • In Distributed Virtual Router (DVR) enabled clouds, responsibility for network routing is shared across the compute nodes, and downtime of the seed node as a controller will have no adverse effect on workload network traffic. To see if DVR is enabled in your cloud, run the following command and check the entries in the "Distributed" column.

      (old-deployer)> source ~/service.osrc
      (old-deployer)> neutron router-list

      If you observe non-distributed routers, then the next step will ensure that any responsibility for those routers' function is migrated away from the seed node to other nodes in the controller cluster, so that routing remains available even while the seed node is taken offline. (This step is not necessary if all your routers are distributed.)

      (old-deployer)> cd ~/scratch/ansible/next/hos/ansible
      (old-deployer)> cp -r ~/hlm-migration-tools/ansible/* ./
      (old-deployer)> ansible-playbook -i hosts/verb_hosts \
        neutron-router-evacuate.yml --limit=SEED_HOST

      SEED_HOST represents the name of the seed node, for example, hos-cp1-dbmq-m3.

      Important
      Important: Host Selection

      For correct behavior, you must use the --limit argument to select the target seed host. This will output a success or error message at the end of its run.

  10. Re-image the seed controller as a SLES-based system:

    • Display the current network connections of the seed controller (before re-imaging) for reference and save it locally.

      (seed)> ip a
    • Reinstall the operating system on the seed node's primary disk using an ISO, cobbler (from the existing deployer), or other provisioner. Note that support for provisioning with cobbler will be available only if the seed node is configured to use Legacy Boot Mode; a setting of UEFI Boot Mode will require manual installation.

    • If using cobbler, begin the seed node re-imaging by running:

      (old-deployer)> cd ~/helion/hos/ansible
      (old-deployer)> sudo cobbler system remove --name SEED_NODE_ID
      (old-deployer)> ansible-playbook -i hosts/localhost cobbler-deploy.yml
      (old-deployer)> ansible-playbook -i hosts/localhost bm-reimage.yml -e nodelist=SEED_NODE_ID

      Replace SEED_NODE_ID with the ID of the seed node from the servers.yml file.

      Cobbler will initialize the seed node's networking to use the IP address of your original deployer as the default gateway. This will negatively impact the network's operation later on in the migration, and should be updated after the system is up and running. On the new seed node, edit or create the file /etc/sysconfig/network/routes and change the address in the line starting with "default" to point to your environment's actual default gateway. It has the format:

      default 192.168.1.254 - -

      You may be able to copy this from the value specified in that file on your existing deployer. Then, restart networking with:

      (seed)> sudo systemctl restart network
    • Otherwise, if installing manually or using a proprietary tool for provisioning, reformat the primary partition (preserving the overall partitioning scheme based on the data model) and install the SUSE Linux Enterprise Server 12 SP3 operating system. Follow the instructions as described in Book “Installing with Cloud Lifecycle Manager”, Chapter 3 “Installing the Cloud Lifecycle Manager server”, but when creating the user account and group, use the name stack instead of ardana.

      It is very important to avoid changing partitioning layouts or reformatting any volumes other than the primary partition during the installation. This is necessary to preserve any Swift object data that may be present on the server.

  11. Transfer configuration and files from your existing cloud to the new seed node

    • The disk volume on the new seed may be smaller than the corresponding volume on the old deployer. It may be necessary to firstremove extraneous files from your deployer (such as old qcow2 or ISO images), then transfer configuration and files from the old deployer to the new seed node:

      (old-deployer)> rsync -avP ~ SEED_IP:/home
      (old-deployer)> scp -r /tmp/hel* SEED_IP:/tmp
    • If cobbler will be used from the new seed for provisioning the remaining cloud Control Nodes:

      • Ensure that the seed node is able to reach IPMI interfaces referenced in the servers.yml file:

        (seed)> for s in `grep "ilo-ip" ~/helion/my_cloud/definition/data/servers.yml | grep -v "#" | cut -d ":" -f 2 | tr -d " " `
          do ping -q -c 2 $s
        done
      • If additional routes are required for the seed to reach the IPMI interfaces, add these network routes to the seed. For example:

        (seed)> sudo ip route add IPMI_NETWORK via IPMI_GATEWAY_FROM_SEED
      • Kill the DHCP services running on the old deployer:

        (old-deployer)> sudo pkill dhcpd
  12. On the new seed node, remove the package repository that was configured when its operating system was first installed. Then, mount the operating system image to make it available to the HPE Helion OpenStack installer.

    • Important
      Important: ISO filenames

      For correct behavior, the filenames of the images described below must match exactly their names in the user's home directory.

      If your cloud includes RHEL Compute Nodes, ensure that ~/rhel7.iso was either transferred from the original deployer or is downloaded to the user's home directory.

      Ensure that ~/sles12sp3sdk.iso was either transferred from the original deployer or is downloaded to the user's home directory.

      Ensure that ~/sles12sp3.iso was either transferred from the original deployer or is downloaded to the home directory on the seed node and named sles12sp3.iso. Then, mount it:

      (seed)> sudo mkdir -p /media/cdrom
      (seed)> sudo mount /home/stack/sles12sp3.iso /media/cdrom

      If you installed the seed node using cobbler from the original deployer, replace the primary package repository on the new seed node with a link to the content delivered on the ISO media instead of a remote network location.

      (seed)> sudo zypper rr SLES12-SP3-12.3-0
      (seed)> sudo zypper ar file:/media/cdrom SLES12-SP3-12.3-0
      (seed)> sudo zypper ref
  13. Install HPE Helion OpenStack 5.SCP on the seed node by downloading and unpacking HPE Helion OpenStack 5.SCP installation content:

    (seed)> cd ~
    (seed)> mkdir -p hos-5.scp
    (seed)> sudo mount HELION5SCP_ISO_FILE hos-5.scp
    (seed)> tar -xf hos-5.scp/hos/hos-5.1.0-*.tar
    (seed)> sudo mkdir -p /var/log/apache2
    (seed)> cd ~/hos-5.1.0
    (seed)> ./hos-init.bash
  14. Update repository references of Compute Node as appropriate to point to the seed node.

    • On each SLES compute node, remove all the repositories except for SLES-OS and SLES-SDK. Use zypper lr, to list them and zypper rr <REPO_NAME> to remove them. If your compute nodes were kept up to date using the SUSE Updates repository via an SMT server, they will get future updates via the SMT server running on or accessible by the future deployer.

    • On each RHEL compute node, edit the repositories located in either /etc/yum.repos.d or /etc/yum/repos.d to redirect any IP addresses from the old deployer to point to the new seed node. (These references are likely to be found in the files cobbler-config.repo or rhel73.repo.)

      Then, refresh the yum cache by running:

      (rhel-compute)> sudo yum clean all
  15. If VMware ESX compute hosts are present in the cloud, encrypt your ESX login credentials for embedding in the data model. This will require use of the HOS_USER_PASSWORD_ENCRYPT_KEY environment variable and the hosencrypt.py script, which is also used for encrypting the following other data model elements:

    • Ironic OneView password

    • Cinder back-end passwords (optional)

    • Barbican master key for simple_crypto_plugin backend (optional)

    • HPE DCN Nuage VSD password

    The use of hosencrypt.py for encryption is required for VMware ESX credentials (unlike some of the items above, which may optionally be stored in plaintext.)

    If you have used hosencrypt.py previously for any of the items above, then you have already selected an encryption key for use with your cloud. This encryption key must be exported into the environment variable HOS_USER_PASSWORD_ENCRYPT_KEY when the password is encrypted, and again before running playbooks such as site.yml. If this is your first time using such encryption, choose a new, unique encryption key and proceed:

    (seed)> export HOS_USER_PASSWORD_ENCRYPT_KEY=ENCRYPTIONKEY
    (seed)> ./hosencrypt.py
    unencrypted value? <VCENTERPASSWORD>
    @hos@U2FsdGVkX180/6dJIPPt49epntxAOFP0MtpjWsOaYzc=

    Capture the resulting encrypted value and paste it into the password field of ~/helion/my_cloud/definition/data/pass_through.yml, replacing the value previously set there by EON. If you have questions about the structure or content in pass_through.yml, comments regarding its use are provided with the HPE Helion OpenStack 5.SCP example data models such as in ~/helion/examples/entry-scale-esx-kvm-vsa/data/pass_through.yml.

  16. In your cloud's data model, edit the control plane definition and add a new data model entry for nova-placement-api into the cluster which already includes nova-api.

    (seed)> cd ~/helion/my_cloud/definition/data
    (seed)> vi control_plane.yml
    (seed)> git commit -a -m "Adding nova-placement-api"
  17. Prepare for running the migration playbook:

    (seed)> cd ~/helion/hos/ansible
    (seed)> ansible-playbook -i hosts/localhost config-processor-run.yml
    (seed)> ansible-playbook -i hosts/localhost ready-deployment.yml
  18. Initialize ansible host facts and ensure that the existing cloud is fully addressable by the new deployer:

    (seed)> cd ~/scratch/ansible/next/hos/ansible
    (seed)> ansible-playbook -i hosts/verb_hosts hlm-refresh-facts.yml
  19. Note
    Note: OpenStack Clouds Only

    The following step is only necessary on OpenStack clouds and should not be used in standalone Swift deployments.

    Flush the Keystone token cache by running the following command on any node hosting the Keystone API service. (This may be the seed node in an entry-scale environment, or a dedicated controller node in a mid-scale environment.)

    (controller)> sudo /opt/stack/service/keystone/venv/bin/keystone-manage token_flush

    Then, prune the primary database to remove data that is no longer relevant:

    (seed)> cd ~/scratch/ansible/next/hos/ansible
    (seed)> ansible-playbook -i hosts/verb_hosts percona-prune.yml
Procedure 1: Migration Phase
Warning
Warning: Downtime

The cloud will not be available until all of the following steps are completed.

  1. If your cloud's data model is password-protected, use the following method to save the password in a local file before proceeding:

    (seed)> touch ~/.vault_pass.txt
    (seed)>  chmod 600 ~/.vault_pass.txt
    (seed)>  vi ~/.vault_pass.txt

    Ensure that the contents of the file are set to include only the plaintext vault password. Then, after saving it, run export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txt in preparation for upcoming steps.

  2. Stop or pause OpenStack services on the control plane (excluding the seed that has been freshly imaged):

    (seed)> cd ~/scratch/ansible/next/hos/ansible
    (seed)> ./hlm-quiesce.sh --limit '!SEED_HOST'

    The SEED_HOST name must exactly match the host name that is referenced in hosts/verb_hosts, such as ardana-cp1-c1-m1.

  3. Create a database backup by retrieving MySQL database contents from a running database node in the database cluster.

    (seed)> cd ~/scratch/ansible/next/hos/ansible
    (seed)> ansible-playbook -i hosts/verb_hosts percona-vertica-removal-cleanup.yml
    (seed)> ansible-playbook -i hosts/verb_hosts percona-export.yml \
      -e dbcontent=~/mysql_dump.sql
  4. Re-image the remaining servers in the control plane.

    • Re-image the remaining servers in the control plane. If the original deployer is a standalone deployer, then it can be preserved until later, to assist with recovery (if needed). If the deployer function is combined with a controller, then ensure that the previous section's steps to transfer configuration and files using rsync has completed so that no important data is lost.

      Reinstall the operating systems on the control plane primary disks using an ISO, cobbler, or other provisioner. If using cobbler, you will need to create the grub config files for all the SLES nodes using the following commands:

      (seed)> cd ~/helion/hos/ansible
      (seed)> ansible-playbook -i hosts/localhost cobbler-deploy.yml
      (seed)> ansible-playbook -i hosts/localhost prepare-sles-grub2.yml \
        -e nodelist=NODE_IDs
      (seed)> ansible-playbook -i hosts/localhost bm-reimage.yml \
      -e nodelist=NODE_IDs
      Note
      Note

      Do not reimage the Compute Nodes, an original standalone deployer, or seed. Identify the set of remaining control plane nodes from the full list of systems addressable by cobbler, as shown by cobbler system list, and provide them as a comma-separated list in the nodelist parameter.

    • If installing the operating system manually (which can be done by booting from the HPE Helion OpenStack 5.SCP installation ISO) or using a proprietary tool for provisioning, reformat the primary partition (preserving the overall partitioning scheme based on the data model) and install the SUSE Linux Enterprise Server 12 SP3 operating system. Follow the instructions as described in Book “Installing with Cloud Lifecycle Manager”, Chapter 3 “Installing the Cloud Lifecycle Manager server”, but when creating the user account and group, use the name stack instead of ardana.

    • ESX (nova-proxy) nodes must also be rebuilt at this time, and must be done using the above-mentioned manual process (not using cobbler). For these virtual machines, reinstall the operating system using the SUSE Linux Enterprise Server 12 SP3 installation ISO. Note that, when installing, you must select a partitioning method using logical volumes and establish a primary logical volume which is reasonably small in size (i.e. 50GB). It will be automatically expanded according to your data model during upcoming steps in the migration process. Again, the proper system user account name and group for these VMs is stack.

    • In a mid-scale cloud topology, some data files may have been stored on the non-primary drive(s) in the system. If this was the case, the wipe_disk.yml playbook should be used (after the nodes have been rebuilt) to purge the old data.

      Make a note of the output of the following command:

      (seed)> cd ~/scratch/ansible/next/hos/ansible
      (seed)> ansible-playbook -i hosts/verb_hosts hlm-refresh-facts.yml

      Specify your own list of nodes matching the identifiers shown in the results of the previous command. However, do not run the wipe_disk.yml playbook on any Compute Nodes or nodes containing Swift object storage.

      (seed)> ansible-playbook -i hosts/verb_hosts wipe_disks.yml \
        --limit dbmq-m1,dbmq-m2,dbmq-m3,mtrmon-m1,mtrmon-m2,mtrmon-m3
  5. The repos should now be set up on all servers. Verify that this is the case with the following:

    (seed)> cd ~/scratch/ansible/next/hos/ansible
    (seed)> ansible resources -a "zypper ref" --become

    All issues with the repositories must be resolved before continuing. All servers except RHEL Compute Nodes should complete successfully. All SLES servers should reference the repo on port 79 on the new seed.

  6. Create the log directory for the configuration processor, and then run the HPE Helion OpenStack 5.SCP installation. Enable database import from the snapshot that was captured from your cloud by using the dbcontent flag:

    (seed)> mkdir -p /var/log/configuration_processor
    (seed)> cd ~/scratch/ansible/next/hos/ansible
    (seed)> ansible-playbook -i hosts/verb_hosts -vvv site.yml -e dbcontent=~/mysql_dump.sql
  7. This will be a time-consuming installation, and should be allowed to complete without interruption. Once it is done, the cloud should again be operational, with the exception of monitoring components that will remain offline until the upgrade to HPE Helion OpenStack 8 is fully complete.

  8. If OVSvApp nodes are present in the environment, they should now be re-imaged with an operating system installed from the SUSE Linux Enterprise Server 12 SP3 ISO. Follow the directions above regarding ESX (nova-proxy) nodes exactly, doing the same for OVSvApp nodes.

    Note
    Note

    Note that, while a VMware ESX host's OVSvApp node is offline, other running VMs on the same host will experience degraded network connectivity. To ensure maximum uptime, migrate all other VMs away from a host before reinstalling its OVSvApp node. Once the node is fully online again, the affected VMs can be returned to their original host.

    After re-imaging an OVSvApp node, install HPE Helion OpenStack 5.SCP components on it from your seed using:

    (seed)> ansible-playbook -i hosts/verb_hosts site.yml \
      -l 'OVSVAPP_HOSTS'

    More than one OVSvApp host can be installed at once by separating their names using commas. Note that each host should be identified by its Ansible-friendly name, matching the way it is referred to in the hosts/verb_hosts file.

Procedure 2: Cleaning Up
Note
Note: No Downtime

These steps do not cause service downtime and can be performed while the cloud is up and running.

  1. Deployer nodes offer limited disk space, and it is appropriate at this time to transfer the ~/mysql_dump.sql file to an external location outside the cloud (where it can be available in case of any recovery needs) and remove it from the seed node:

    (seed)> rm ~/mysql_dump.sql
  2. If desired, transfer the deployer role in the HPE Helion OpenStack 5.SCP cloud to a different node, such as one that originally served as a dedicated deployer.

    • If the deployer will be moving to an entirely different cluster (for example, Database cluster to Controller cluster or standalone Deployer cluster), additional software packages need to be installed.

      • Edit ~/helion/my_cloud/definition/data/control_plane.yml and move the lifecycle-manager from its current location into the new desired cluster's service-components list. Commit the change:

        (seed)> git commit -a -m "Enable lifecycle-manager for new deployer"
      • Execute these steps to prepare for running playbooks:

        (seed)> cd ~/helion/hos/ansible
        (seed)> ansible-playbook -i hosts/localhost config-processor-run.yml
        (seed)> ansible-playbook -i hosts/localhost ready-deployment.yml
      • At this point, it is appropriate to re-image a standalone deployer that would have been left intact through the upgrade steps to this point, to ensure that the SLES operating system is loaded.

        (seed)>  cd ~/helion/hos/ansible
        (seed)> ansible-playbook -i hosts/localhost bm-reimage.yml \
          -e nodelist=ORIGINAL_DEPLOYER
      • Install the lifecycle-manager services on the new deployer and remove cloud-wide references to the seed:

        (seed)>  cd ~/scratch/ansible/next/hos/ansible
        (seed)> ansible-playbook -i hosts/verb_hosts site.yml -l NEW_DEPLOYER_HOST
        (seed)> ansible resources -i hosts/verb_hosts -m shell \
          -a "zypper rr SLES12-SP3-12.3-0 || true" --become

        NEW_DEPLOYER_HOST must match the node's identity in the file hosts/verb_hosts (for example, ardana-cp1-c0-m1)

    • Stop the deployer-related services and repositories running on the current seed:

      (seed)> sudo systemctl stop dhcpd
      (seed)> sudo systemctl disable dhcpd
      (seed)> sudo systemctl stop cobblerd
      (seed)> sudo systemctl disable cobblerd
      (seed)> sudo sed -i /sles12sp3.iso/d /etc/fstab
      (seed)> sudo umount /media/cdrom
    • If you used the cobbler mechanism from your seed node earlier to re-image the node which will now be serving as your new deployer, its zypper repos should be changed to use local sources instead of retrieving content from the seed node via the network:

      (new-deployer)> sudo zypper rr SLES-OS
      (new-deployer)> sudo zypper ar file:/media/cdrom SLES12-SP3-12.3-0

      The old repos on other SLES systems will automatically be reset to the new deployer later when the osconfig-run.yml playbook is run.

    • Unmount any ISOs that are no longer necessary on the seed node:

      (seed)> sudo umount /home/stack/hos5.scp
    • Copy the user's home directory and certificates from the seed to the new deployer:

      (seed)> rsync -avP ~/ NEW_DEPLOYER_IP:/home/stack
      (seed)> scp -r /tmp/hel* NEW_DEPLOYER_IP:/tmp
    • Connect to the new deployer and initialize it for use:

      (new-deployer)> cd ~
      (new-deployer)> rm -rf .ansible_fact_cache
      (new-deployer)> sudo mkdir -p /media/cdrom
      (new-deployer)> sudo mount /home/stack/sles12sp3.iso /media/cdrom
      (new-deployer)> sudo mount HELION5SCP_ISO_FILE hos-5.scp
      (new deployer)> tar -xf hos-5.scp/hos/hos-5.1.0-*.tar
      (new deployer)> cd ~/hos-5.1.0
      (new deployer)> ./hos-init.bash
      (new deployer)> cd ~/helion/hos/ansible
      (new deployer)> ansible-playbook -i hosts/localhost cobbler-deploy.yml
      (new-deployer)> ansible-playbook -i hosts/localhost config-processor-run.yml
      (new-deployer)> ansible-playbook -i hosts/localhost ready-deployment.yml
    • On each RHEL Compute Node, edit the repositories located in either /etc/yum.repos.d or /etc/yum/repos.d to redirect any IP addresses from the seed node to point to the new deployer. (These references are likely to be found in the files cobbler-config.repo or rhel73.repo.)

      Then, refresh the yum cache by running:

      (rhel-compute)> sudo yum clean all
    • Use the new deployer to inform all other cloud nodes of its new role.

      (new-deployer)> cd ~/scratch/ansible/next/hos/ansible
      (new-deployer)> ansible-playbook -i hosts/verb_hosts osconfig-run.yml

2 Migration from HPE Helion OpenStack 5.SCP to Ardana.Newton

The following procedure describes the process of migrating from HPE Helion OpenStack 5.SCP to Ardana.Newton.

From this point forward, all commands should be executed on the new deployer node (which will be the same as the earlier seed node unless this role was explicitly transferred using the instructions at the end of the prior section).

Procedure 3: Preparation
  1. Convert the helion Git repository to an openstack Git repository for use by Ardana:

    (stack)> cd ~
    (stack)> cp -r helion openstack
    (stack)> cd openstack
    (stack)> git checkout site
  2. If your environment included a SES storage backend, remove all references to it from your data model. This may be done by using git revert COMMIT-ID to revert a specific commit including the SES backend definition, or git checkout hos -- FILENAME to restore an entire file to its original version (followed by git diff --staged to inspect the difference between files and ensure that you ar satisfied with the final condition of the configuration file.) Committing these edits will be necessary for the following files in the ~/openstack git repository:

    • hos/ansible/roles/_CND-CMN/templates/cinder.conf.j2

    • hos/ansible/roles/NOV-CMP-KVM/templates/hypervisor.conf.j2

    • hos/ansible/roles/GLA-API/templates/glance-api.conf.j2 (if the SES backend was additionally configured for use with Glance)

  3. Save a record of all other manual changes that have been previously committed to the hos/ansible directory, as you will need to reapply them later:

    (stack)> git diff hos -- hos/ansible > /tmp/hos.diff

    Forcibly undo all the changes identified:

    (stack)> rm -rf hos/ansible
    (stack)> git checkout hos -- hos/ansible
  4. Reset the codebase to make it ready for Ardana, then transform the rest of the repository:

    (stack)> git commit -a -m "Resetting codebase in prep for Ardana"
    (stack)> git branch -m hos ardana
    (stack)> git checkout ardana
    (stack)> sed -i 's/hos/ardana/' .gitignore
    (stack)> git commit -a -m "Prep for Ardana"
    (stack)> git mv hos ardana
    (stack)> git mv hos_extensions ardana_extensions
    (stack)> for i in $(find -L my_cloud/config -xtype l)
      do ln -sfn $(readlink $i | sed 's/\/hos\//\/ardana\//') $i
    done
    (stack)> git commit -a -m "Conversion to Ardana"
    (stack)> git checkout cp-persistent
    (stack)> sed -i 's/component: helion-ca/component: ardana-ca/' \
      my_cloud/persistent_state/private_data_control-plane-1.yml
    (stack)> sed -i 's/helion_internal_ca/ardana_internal_ca/' \
      my_cloud/persistent_state/private_data_control-plane-1.yml
    (stack)> git commit -a -m "Conversion to Ardana"
    (stack)> git checkout site
    (stack)> git merge ardana -m "Conversion to Ardana"

    Any merge conflicts that arise through this procedure can be addressed in the manner described in Book “Installing with Cloud Lifecycle Manager”, Chapter 10 “Using Git for Configuration Management”, Section 10.3 “Resolving Git merge conflicts”.

  5. If the file /tmp/hos.diff contains changes that were saved prior to the repo conversion above, reinstate them in the ardana/ansible directory.

    (stack)> if [ -s /tmp/hos.diff ]; then
      sed 's/ \([ab]\?[\/]\?\)hos\/ansible/ \1ardana\/ansible/g' /tmp/hos.diff | git apply
    fi

    If merge conflicts are reported, some files may need to be edited manually to reconcile the updates to file paths and directory structure that make the codebase suitable for upgrading to Ardana.Newton. Further information about merge operations can be found at Book “Installing with Cloud Lifecycle Manager”, Chapter 10 “Using Git for Configuration Management”, Section 10.3 “Resolving Git merge conflicts”.

    (stack)> git add -A
    (stack)> git commit -a -m "Reinstated hos playbook customizations (batch)"
  6. Remove and archive the HPE Helion OpenStack 5.SCP-related content:

    (stack)> mv ~/scratch ~/scratch.scp
    (stack)> sudo rm /etc/apache2/sites-enabled/deployer_venv_server.conf
  7. Prepare the ardana_packager working directory to include existing SUSE repositories:

    (stack)> sudo mkdir -p /opt/ardana_packager/ardana/sles12/zypper
    (stack)> sudo ln -s /opt/hlm_packager/hlm /opt/ardana_packager/hlm
    (stack)> sudo ln -s /opt/hlm_packager/hlm/sles12/zypper/OS \
      /opt/ardana_packager/ardana/sles12/zypper/OS
  8. Install Ardana.Newton content onto the deployer by mounting the installation media and running the following commands:

    (stack)> cd ~
    (stack)> sudo mkdir -p /media/ardana-0.35.0
    (stack)> sudo mount ~/HELION5CLM_ISO_FILE /media/ardana-0.35.0
    (stack)> tar -xf /media/ardana-0.35.0/ardana/ardana-0.35.0*.tar
    (stack)> cd ardana-0.35.0
    (stack)> ./ardana-init.bash

    If merge conflicts are reported, some files may need to be edited manually to reconcile updates being introduced to the playbooks by the Ardana.Newton codebase. Use caution when evaluating these merge conflicts, and ensure that the only changes given priority over the incoming playbooks are those which are specifically relevant to your cloud's configuration and should be retained into the future. Further information about merge operations can be found at Book “Installing with Cloud Lifecycle Manager”, Chapter 10 “Using Git for Configuration Management”, Section 10.3 “Resolving Git merge conflicts”.

  9. Install Ardana.Newton:

    (stack)> cd ~/openstack/ardana/ansible
    (stack)> ansible-playbook -i hosts/localhost config-processor-run.yml
    (stack)> ansible-playbook -i hosts/localhost ready-deployment.yml
    
    (stack)> cd ~/scratch.scp/ansible/next/hos/ansible/
    (stack)> ansible-playbook -i hosts/verb_hosts osconfig-iptables-rename.yml
    (stack)> ansible -i hosts/verb_hosts resources --become \
      -a "sed -i 's/helion/ardana/' /etc/iproute2/rt_tables"

    Because the OpenStack service versions are comparable between HPE Helion OpenStack 5.0.x and Ardana.Newton, it is not necessary to run the entire deploy/upgrade playbook for Ardana.Newton at this time.

3 Migration from Ardana.Newton to Ardana.Pike

  1. Create the ardana user throughout all nodes in the cloud:

    (stack)> cd ~/scratch/ansible/next/ardana/ansible
    (stack)> ansible-playbook -i hosts/verb_hosts ardana-user-create.yml
  2. Mount the HPE Helion OpenStack 8 installation media on the host (after copying the ISO file into the user's home directory)

    (stack)> sudo mkdir -p /media/cloud
    (stack)> echo /home/stack/HPE-HELION-OPENSTACK-8-x86_64-GM-DVD1.iso \
      /media/cloud iso9660 loop 0 0 | sudo tee -a /etc/fstab
    (stack)> sudo mount -a
  3. Add the contents of the ISO as a Zypper repository:

    (stack)> sudo zypper ar -G -f file:/media/cloud Cloud
    (stack)> sudo zypper refresh
  4. Begin the installation:

    (stack)> sudo zypper -n in patterns-cloud-ardana
  5. Before continuing the installation, add a password of your choosing to the new ardana user:

    (stack)> sudo passwd ardana
  6. Then, exit your current session on the deployer and open a new session as the ardana user, logging in with the password that you just established. From this point on, the ardana user will be used for all operations, and the stack user should be considered deprecated.

    To ensure that the stack user does not continue to be used, locking its account with the following command is highly recommended:

    ardana > sudo usermod -L stack
    ardana > sudo rm -rf ~stack/.ssh/authorized_keys
    ardana > sudo mv ~stack/scratch ~stack/scratch.newton
    ardana > sudo mv ~stack/helion ~stack/helion.newton
  7. Continue the installation as the ardana user:

    ardana > ARDANA_INIT_AUTO=1 /usr/bin/ardana-init

    If merge conflicts are reported, some files may need to be edited manually to reconcile the difference between playbook versions with respect to manual changes previously made for your cloud. Use caution when evaluating these merge conflicts, and ensure that the only changes given priority over the incoming playbooks are those which are specifically relevant to your cloud's configuration and should be retained into the future. Further information about merge operations can be found at Book “Installing with Cloud Lifecycle Manager”, Chapter 10 “Using Git for Configuration Management”, Section 10.3 “Resolving Git merge conflicts”

  8. The installation process creates the directory /srv/www/suse-12.3/x86_64/repos/PTF and adds it as a software repository. To enable a successful migration, you must add required Program Temporary Fix (PTF) RPM packages to this repository manually. These packages are provided to you by SUSE via an active L3 support contract.

    To add the PTF RPMs manually, the RPMs should be copied into the /srv/www/suse-12.3/x86_64/repos/PTF directory. For example, if the downloaded RPMs are located in /tmp/PTF:

    ardana > sudo cp /tmp/PTF/*.rpm /srv/www/suse-12.3/x86_64/repos/PTF
  9. Apply the PTF packages obtained in the previous step and confirm that they install correctly:

    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost deployer-init.yml
    ardana > sudo zypper refresh PTF
    ardana > sudo zypper dup --from PTF
    ardana > sudo zypper in ardana-ses
    ardana > rpm -qa | grep PTF

    Ensure that all migration RPMs starting with the prefix "ardana" available via your SUSE support contract are present in the resulting list. If any are missing, repeat the above process or contact technical support before continuing.

  10. Apply the updates received via PTF packages to your active cloud:

    ardana > /usr/bin/ardana-init
  11. Set the default Glance store to hlm-default-store so that your existing image content remains accessible following the upgrade.

    ardana > cd ~/openstack/ardana/ansible
    ardana > sed -i 's/ardana-default-store/hlm-default-store/' \
      roles/GLA-API/defaults/main.yml

    Update certificate references to use the new "ardana" name:

    ardana > sed -i 's/cert-file: helion/cert-file: ardana/' \
      ../../my_cloud/definition/data/network_groups.yml \
      ../../my_cloud/definition/data/control_plane.yml

    Adjust data model to install cassandra instead of vertica. Manually replace vertica with cassandra in the service-components list, and remove any entry for ops-console-monitor:

    ardana > vi ../../my_cloud/definition/data/control_plane.yml

    In files where a cassandra_db partition exists, manually update its consumer name from vertica to cassandra and add an identical consumer block to the cassandra_log partition definition with:

    ardana > vi ../../my_cloud/definition/data/disks_*

    For example:

    - name: cassandra_db
      size: 20%
      mount: /var/cassandra/data
      fstype: ext4
      mkfs-opts: -O large_file
      consumer:
        name: cassandra
    
    - name: cassandra_log
      size: 1%
      mount: /var/cassandra/commitlog
      fstype: ext4
      mkfs-opts: -O large_file
      consumer:
        name: cassandra

    If your environment includes a SES backend, add configuration files to enable it based on the Book “Installing with Cloud Lifecycle Manager”, Chapter 23 “Integrations”, Section 23.3 “SUSE Enterprise Storage Integration” instructions. (However, the site.yml and ardana-reconfigure.yml playbooks should not be used at this time.)

    Save the updates to the data model:

    ardana > git commit -a -m 'Updated data model'
  12. Build the ~/scratch directory for the upgrade:

    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
  13. Prepare the servers for Pike infrastructure:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts FND-AP2-vhosts-transition.yml
    ardana > sudo rename helion ardana /tmp/helion_*
    ardana > sudo rename helion ardana /tmp/ardana_*/helion*
  14. Set up and verify package repositories for installation:

    ardana > sudo ln -s /media/cloud /srv/www/suse-12.3/x86_64/repos/Cloud
    ardana > sudo ln -s /opt/hlm_packager/hlm/sles12/zypper/extras/sles12sp3-ceph-updates \
      /srv/www/suse-12.3/x86_64/repos/sles12sp3-ceph-updates
    ardana > ansible-playbook -i hosts/verb_hosts ardana-replace-legacy-repos.yml

    Check that the repos have been properly set up, and that refreshing them succeeds for all SLES cloud nodes:

    ardana > ansible resources -m shell -a "zypper ref" --become
  15. If your environment includes RHEL Compute Nodes, prepare your deployer for provisioning OpenStack services on them by following the steps described in Book “Installing with Cloud Lifecycle Manager”, Chapter 22 “Installing RHEL Compute”, Section 22.4 “Using RHEL as a Compute Node”.

    Note
    Note

    If you opt to install an SMT server on your new deployer for mirroring the CentOS repo, you must first run the command sudo ln -s /var/lib/mysql/mysql.sock /var/run/mysql/mysql.sock to enable SMT access to the database. (This access is enabled permanently upon completion of the upgrade, but must be manually set up if needed at this time.)

    If the CA Management procedure is used to set up a root CA certificate in support of the deployer SMT server, you must immediately follow it by running sudo ln -s /etc/ssl/ca-bundle.pem /etc/ssl/certs/ca-certificates.crt before proceeding with the software upgrade.

  16. Run the HPE Helion OpenStack 8 upgrade playbook:

    ardana > ansible-playbook -i hosts/verb_hosts -vvv ardana-upgrade-from-legacy.yml

    Note that the ardana-upgrade-from-legacy.yml playbook is not guaranteed to be idempotent, and re-running it upon a failure may produce new errors. If this playbook fails, please contact customer support and be prepared to share details of the failure from the contents of ~/.ansible/ansible.log to determine appropriate next steps.

  17. Now that the upgrade has completed, it will be necessary to reboot the control plane nodes in your cloud for changes to take effect. At this time, reboot only the control plane nodes (not compute nodes) in your cloud per the instructions in Book “Operations Guide”, Chapter 13 “System Maintenance”, Section 13.1 “Planned System Maintenance”, Section 13.1.1 “Whole Cloud Maintenance”, Section 13.1.1.1 “Bringing Down Your Cloud: Services Down Method”.

  18. If your cloud includes OpenStack-provisioned load balancers, the following step is necessary to update the Amphora load balancer image used by the Octavia service:

    ardana > ansible-playbook -i hosts/verb_hosts -vvv service-guest-image \
      -e service_package=$(ls /srv/www/suse-12.3/x86_64/repos/PTF/openstack-octavia-amphora-image*.rpm)
  19. After updating the Amphora image with the above playbook, the following steps are recommended to confirm that load balancer functionality works as expected. (This will ensure that your workloads remain available through the upcoming reboots of compute nodes.)

    • Confirm using neutron lbaas-loadbalancer-list that any existing load balances are online and reporting with state ACTIVE.

    • Create a new load balancer with neutron lbaas-loadbalancer-create migration_test OCTAVIA-MGMT-NET-ID. (The ID of OCTAVIA-MGMT-NET is available from the list reported by openstack network list.)

    • Run neutron lbaas-loadbalancer-list to ensure that the new load balancer appears and becomes ACTIVE, then try using nova delete to remove its associated VM and ensure that it is recreated by the failover mechanism.

  20. You may now proceed to reboot the compute nodes in your cloud for all changes to fully take effect and re-enable normal behavior. Continue to refer to the steps described in Book “Operations Guide”, Chapter 13 “System Maintenance”, Section 13.1 “Planned System Maintenance”, Section 13.1.1 “Whole Cloud Maintenance”, Section 13.1.1.1 “Bringing Down Your Cloud: Services Down Method”.

  21. At this time, the PTF and other temporary packages can be removed:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible -i hosts/verb_hosts resources --become -m shell \
      -a "zypper rr sles12sp3-ceph-updates || true"
    ardana > sudo rm /srv/www/suse-12.3/x86_64/repos/sles12sp3-ceph-updates
    ardana > sudo rm -rf /srv/www/suse-12.3/x86_64/repos/PTF/*.rpm
    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost deployer-init.yml
    ardana > sudo zypper refresh PTF
    ardana > sudo zypper dup --from Cloud
    Note
    Note

    When running sudo zypper dup --from Cloud, warnings will appear about packages being downgraded. This is expected and should be acknowledged as you proceed.

  22. Determine how to handle modified policy files which restrict the admin user's privileges more tightly than in previous releases:

    With this upgrade, the policy.json files have been aligned with OpenStack Pike. When running OpenStack commands an operator may see messages saying Policy doesn't allow ... such as:

    Policy doesn't allow volume_extension:services:index to be performed

    In this case the operator can be given permission by adding a role to the user using the openstack role add command, for example:

    ardana > openstack role add --user USER_NAME \
      --project PROJECT_NAME cinder_admin

    Role assignments should be carefully considered. In an environment which requires separation of duties, each operator should be assigned only the roles they require to perform their duties and should use their own user account to perform their administrative tasks. For more information see the Book “Security Guide”, Chapter 4 “HPE Helion OpenStack®: Service Admin Role Segregation in the Identity Service”.

4 Applying Product Updates

  1. Before applying updates to your cloud, you should replace the repositories configured on each node during the migration with links to an SMT server that mirrors the latest versions of your software. To begin this process, register your deployer node and install the HPE Helion OpenStack extension as described in Book “Installing with Cloud Lifecycle Manager”, Chapter 3 “Installing the Cloud Lifecycle Manager server”, Section 3.5.2 “Installing the HPE Helion OpenStack Extension”.

    The easiest way to provide the required repositories on the Cloud Lifecycle Manager Server is to configure it as an SMT server as described in Book “Installing with Cloud Lifecycle Manager”, Chapter 4 “Installing and Setting Up an SMT Server on the Cloud Lifecycle Manager server (Optional)”. Alternatives to setting up an SMT server are described in Book “Installing with Cloud Lifecycle Manager”, Chapter 5 “Software Repository Setup”.

  2. Next, use the following commands to replace the original ISO repositories with the latest package sources provided by SUSE:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > sudo rm /opt/hlm_packager/hlm/sles12/zypper/OS /srv/www/suse-12.3/x86_64/repos/Cloud
    ardana > ansible resources -i hosts/verb_hosts -m shell \
      -a "zypper rr Cloud; zypper rr SLES-OS || true" --become
    ardana > ansible-playbook -i hosts/verb_hosts _osconfig-setup-repos.yml
  3. Ensure that the "Cloud" repository has been replaced by the following repositories on all SLES nodes:

    • SLES12-SP3-Pool

    • SLES12-SP3-Updates

    • HPE-Helion-OpenStack-8-Pool

    • HPE-Helion-OpenStack-8-Update

  4. Finally, apply the latest versions of software packages using the process described in Book “Operations Guide”, Chapter 13 “System Maintenance”, Section 13.3 “Cloud Lifecycle Manager Maintenance Update Procedure”.

Print this page