Bare Metal as a Service is enabled in this release for deployment of Nova instances on bare metal nodes using flat networking. HPE DL and SL line of servers are supported.
This page describes the installation step requirements for the HPE Helion OpenStack Entry-scale Cloud with Ironic Flat Network.
Prior to deploying an operational environment with Ironic, operators need to be aware of the nature of TLS certificate authentication. As pre-built deployment agent ramdisks images are supplied, these ramdisk images will only authenticate known third-party TLS Certificate Authorities in the interest of end-to-end security. As such, uses of self-signed certificates and private certificate authorities will be unable to leverage ironic without modifying the supplied ramdisk images.
Set up your configuration files, as follows:
See the sample sets of configuration files in the
~/openstack/examples/
directory. Each set will have an
accompanying README.md file that explains the contents of each of the
configuration files.
Copy the example configuration files into the required setup directory and edit them to contain the details of your environment:
cp -r ~/openstack/examples/entry-scale-ironic-flat-network/* \ ~/openstack/my_cloud/definition/
(Optional)
You can use the ardanaencrypt.py
script to
encrypt your IPMI passwords. This script uses OpenSSL.
Change to the Ansible directory:
ardana >
cd ~/openstack/ardana/ansible
Put the encryption key into the following environment variable:
export ARDANA_USER_PASSWORD_ENCRYPT_KEY=<encryption key>
Run the python script below and follow the instructions. Enter a password that you want to encrypt.
ardana >
./ardanaencrypt.py
Take the string generated and place it in the
ilo-password
field in your
~/openstack/my_cloud/definition/data/servers.yml
file, remembering to enclose it in quotes.
Repeat the above for each server.
Before you run any playbooks, remember that you need to export the
encryption key in the following environment variable: export
ARDANA_USER_PASSWORD_ENCRYPT_KEY=<encryption key>
Commit your configuration to the local git repo (Chapter 10, Using Git for Configuration Management), as follows:
ardana >
cd ~/openstack/ardana/ansibleardana >
git add -Aardana >
git commit -m "My config or other commit message"
This step needs to be repeated any time you make changes to your configuration files before you move on to the following steps. See Chapter 10, Using Git for Configuration Management for more information.
To provision the baremetal nodes in your cloud deployment you can either use the automated operating system installation process provided by HPE Helion OpenStack or you can use the 3rd party installation tooling of your choice. We will outline both methods below:
If you do not wish to use the automated operating system installation tooling included with HPE Helion OpenStack then the requirements that have to be met using the installation tooling of your choice are:
The operating system must be installed via the SLES ISO provided on the SUSE Customer Center.
Each node must have SSH keys in place that allows the same user from the Cloud Lifecycle Manager node who will be doing the deployment to SSH to each node without a password.
Passwordless sudo needs to be enabled for the user.
There should be a LVM logical volume as /root
on each
node.
If the LVM volume group name for the volume group holding the
root
LVM logical volume is
ardana-vg
, then it will align with the disk input
models in the examples.
Ensure that openssh-server
,
python
, python-apt
, and
rsync
are installed.
If you chose this method for installing your baremetal hardware, skip forward to the step Running the Configuration Processor.
If you would like to use the automated operating system installation tools provided by HPE Helion OpenStack, complete the steps below.
This phase of the install process takes the baremetal information that was
provided in servers.yml
and installs the Cobbler
provisioning tool and loads this information into Cobbler. This sets each
node to netboot-enabled: true
in Cobbler. Each node
will be automatically marked as netboot-enabled: false
when it completes its operating system install successfully. Even if the
node tries to PXE boot subsequently, Cobbler will not serve it. This is
deliberate so that you cannot reimage a live node by accident.
The cobbler-deploy.yml
playbook prompts for a password
- this is the password that will be encrypted and stored in Cobbler, which
is associated with the user running the command on the Cloud Lifecycle Manager, that you
will use to log in to the nodes via their consoles after install. The
username is the same as the user set up in the initial dialogue when
installing the Cloud Lifecycle Manager from the ISO, and is the same user that is running
the cobbler-deploy play.
When imaging servers with your own tooling, it is still necessary to have
ILO/IPMI settings for all nodes. Even if you are not using Cobbler, the
username and password fields in servers.yml
need to
be filled in with dummy settings. For example, add the following to
servers.yml
:
ilo-user: manual ilo-password: deployment
Run the following playbook which confirms that there is IPMI connectivity for each of your nodes so that they are accessible to be re-imaged in a later step:
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost bm-power-status.yml
Run the following playbook to deploy Cobbler:
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost cobbler-deploy.yml
This phase of the install process goes through a number of distinct steps:
Powers down the nodes to be installed
Sets the nodes hardware boot order so that the first option is a network boot.
Powers on the nodes. (The nodes will then boot from the network and be installed using infrastructure set up in the previous phase)
Waits for the nodes to power themselves down (this indicates a successful install). This can take some time.
Sets the boot order to hard disk and powers on the nodes.
Waits for the nodes to be reachable by SSH and verifies that they have the signature expected.
Deploying nodes has been automated in the Cloud Lifecycle Manager and requires the following:
All of your nodes using SLES must already be installed, either manually or via Cobbler.
Your input model should be configured for your SLES nodes, according to the instructions at Book “Planning an Installation with Cloud Lifecycle Manager”, Chapter 10 “Modifying Example Configurations for Compute Nodes”, Section 10.1 “SLES Compute Nodes”.
You should have run the configuration processor and the
ready-deployment.yml
playbook.
Execute the following steps to re-image one or more nodes after you have
run the ready-deployment.yml
playbook.
Run the following playbook, specifying your SLES nodes using the nodelist. This playbook will reconfigure Cobbler for the nodes listed.
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook prepare-sles-grub2.yml -e \ nodelist=node1[,node2,node3]
Re-image the node(s) with the following command:
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost bm-reimage.yml \ -e nodelist=node1[,node2,node3]
If a nodelist is not specified then the set of nodes in Cobbler with
netboot-enabled: True
is selected. The playbook pauses
at the start to give you a chance to review the set of nodes that it is
targeting and to confirm that it is correct.
You can use the command below which will list all of your nodes with the
netboot-enabled: True
flag set:
sudo cobbler system find --netboot-enabled=1
Once you have your configuration files setup, you need to run the configuration processor to complete your configuration.
When you run the configuration processor, you will be prompted for two
passwords. Enter the first password to make the configuration processor
encrypt its sensitive data, which consists of the random inter-service
passwords that it generates and the ansible group_vars
and host_vars
that it produces for subsequent deploy
runs. You will need this password for subsequent Ansible deploy and
configuration processor runs. If you wish to change an encryption password
that you have already used when running the configuration processor then
enter the new password at the second prompt, otherwise just press
Enter to bypass this.
Run the configuration processor with this command:
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost config-processor-run.yml
For automated installation (for example CI), you can specify the required passwords on the ansible command line. For example, the command below will disable encryption by the configuration processor:
ardana >
ansible-playbook -i hosts/localhost config-processor-run.yml \
-e encrypt="" -e rekey=""
If you receive an error during this step, there is probably an issue with one or more of your configuration files. Verify that all information in each of your configuration files is correct for your environment. Then commit those changes to Git using the instructions in the previous section before re-running the configuration processor again.
For any troubleshooting information regarding these steps, see Section 24.2, “Issues while Updating Configuration Files”.
Use the playbook below to create a deployment directory:
cd ~/openstack/ardana/ansible ansible-playbook -i hosts/localhost ready-deployment.yml
[OPTIONAL] - Run the wipe_disks.yml
playbook to ensure
all of your non-OS partitions on your nodes are completely wiped before
continuing with the installation. The wipe_disks.yml
playbook is only meant to be run on systems immediately after running
bm-reimage.yml
. If used for any other case, it may
not wipe all of the expected partitions.
If you are using fresh machines this step may not be necessary.
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts wipe_disks.yml
If you have used an encryption password when running the configuration processor use the command below and enter the encryption password when prompted:
ardana >
ansible-playbook -i hosts/verb_hosts wipe_disks.yml --ask-vault-pass
Run the site.yml
playbook below:
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts site.yml
If you have used an encryption password when running the configuration processor use the command below and enter the encryption password when prompted:
ardana >
ansible-playbook -i hosts/verb_hosts site.yml --ask-vault-pass
The step above runs osconfig
to configure the cloud
and ardana-deploy
to deploy the cloud. Therefore, this
step may run for a while, perhaps 45 minutes or more, depending on the
number of nodes in your environment.
Verify that the network is working correctly. Ping each IP in the
/etc/hosts
file from one of the controller nodes.
For any troubleshooting information regarding these steps, see Section 24.3, “Issues while Deploying the Cloud”.
Run the ironic-cloud-configure.yml
playbook below:
cd ~/scratch/ansible/next/ardana/ansible ansible-playbook -i hosts/verb_hosts ironic-cloud-configure.yml
This step configures ironic flat network, uploads glance images and sets the ironic configuration.
To see the images uploaded to glance, run:
$ source ~/service.osrc $ glance image list
This will produce output like the following example, showing three images that have been added by Ironic:
+--------------------------------------+--------------------------+ | ID | Name | +--------------------------------------+--------------------------+ | d4e2a0ff-9575-4bed-ac5e-5130a1553d93 | ir-deploy-iso-HOS3.0 | | b759a1f0-3b33-4173-a6cb-be5706032124 | ir-deploy-kernel-HOS3.0 | | ce5f4037-e368-46f2-941f-c01e9072676c | ir-deploy-ramdisk-HOS3.0 | +--------------------------------------+--------------------------+
To see the network created by Ironic, run:
$ neutron net-list
This returns details of the "flat-net" generated by the Ironic install:
+---------------+----------+-------------------------------------------------------+ | id | name | subnets | +---------------+----------+-------------------------------------------------------+ | f9474...11010 | flat-net | ca8f8df8-12c8-4e58-b1eb-76844c4de7e8 192.168.245.0/24 | +---------------+----------+-------------------------------------------------------+
Once booted, nodes obtain network configuration via DHCP. If multiple interfaces are to be utilized, you may want to pre-build images with settings to execute DHCP on all interfaces. An easy way to build custom images is with KIWI, the command line utility to build Linux system appliances.
For information about building custom KIWI images, see Section 17.3.11, “Building Glance Images Using KIWI”. For more information, see the KIWI documentation at https://osinside.github.io/kiwi/.
Configuration Drives are stored unencrypted and should not include any sensitive data.
You can use Configuration Drives to store metadata for initial boot
setting customization. Configuration Drives are extremely useful for
initial machine configuration. However, as a general security practice,
they should not include any
sensitive data. Configuration Drives should only be trusted upon the initial
boot of an instance. cloud-init
utilizes a lock file for
this purpose. Custom instance images should not rely upon the integrity of a
Configuration Drive beyond the initial boot of a host as an administrative
user within a deployed instance can potentially modify a configuration drive
once written to disk and released for use.
For more information about Configuration Drives, see http://docs.openstack.org/user-guide/cli_config_drive.html.
As part of HPE Helion OpenStack 8, Ironic Python Agent, better known as IPA in the
OpenStack community, images are supplied and loaded into Glance. Two types of
image exist. One is a traditional boot ramdisk which is used by the
agent_ipmitool
, pxe_ipmitool
, and
pxe_ilo
drivers. The other is an ISO image that is
supplied as virtual media to the host when using the
agent_ilo
driver.
As these images are built in advance, they are unaware of any private certificate authorities. Users attempting to utilize self-signed certificates or a private certificate authority will need to inject their signing certificate(s) into the image in order for IPA to be able to boot on a remote node, and ensure that the TLS endpoints being connected to in HPE Helion OpenStack can be trusted. This is not an issue with publicly signed certificates.
As two different types of images exist, below are instructions for
disassembling the image ramdisk file or the ISO image. Once this has been
done, you will need to re-upload the files to glance, and update any impacted
node's driver_info
, for example, the
deploy_ramdisk
and ilo_deploy_iso
settings that were set when the node was first defined. Respectively, this
can be done with the
ironic node-update <node> replace driver_info/deploy_ramdisk=<glance_id>
or
ironic node-update <node> replace driver_info/ilo_deploy_iso=<glance_id>
Perform the following steps.
To upload your trusted CA certificate to the Cloud Lifecycle Manager, follow the directions in Section 30.7, “Upload to the Cloud Lifecycle Manager”.
Delete the deploy images.
ardana >
openstack image delete ir-deploy-iso-ARDANA5.0ardana >
openstack image delete ir-deploy-ramdisk-ARDANA5.0
On the deployer, run ironic-reconfigure.yml
playbook
to re-upload the images that include the new trusted CA bundle.
ardana >
cd /var/lib/ardana/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts ironic-reconfigure.yml
Update the existing Ironic nodes with the new image IDs accordingly. For example,
ardana >
openstack baremetal node set --driver-info \
deploy_ramdisk=NEW_RAMDISK_ID NODE_ID
HPE Helion OpenStack 8 introduces the concept of multiple control planes - see the Input Model documentation for the relevant Book “Planning an Installation with Cloud Lifecycle Manager”, Chapter 5 “Input Model”, Section 5.2 “Concepts”, Section 5.2.2 “Control Planes”, Section 5.2.2.1 “Control Planes and Regions” and Book “Planning an Installation with Cloud Lifecycle Manager”, Chapter 6 “Configuration Objects”, Section 6.2 “Control Plane”, Section 6.2.3 “Multiple Control Planes”. This document covers the use of an Ironic region in a multiple control plane cloud model in HPE Helion OpenStack.
IRONIC-FLAT-NET is the network configuration for baremetal control plane.
You need to set the environment variable OS_REGION_NAME to the Ironic region in baremetal control plane. This will set up the Ironic flat networking in Neutron.
export OS_REGION_NAME=<ironic_region>
To see details of the IRONIC-FLAT-NETWORK
created during
configuration, use the following command:
neutron net-list
Referring to the diagram below, the Cloud Lifecycle Manager is a shared service that runs in a Core API Controller in a Core API Cluster. Ironic Python Agent (IPA) must be able to make REST API calls to the Ironic API (the connection is represented by the green line to Internal routing). The IPA connect to Swift to get user images (the gray line connecting to Swift routing).
Swift is very resource-intensive and as a result, it is now optional in the HPE Helion OpenStack control plane. A number of services depend on Swift, and if it is not present, they must provide a fallback strategy. For example, Glance can use the filesystem in place of Swift for its backend store.
In Ironic, agent-based drivers require Swift. If it is not present, it is
necessary to disable access to this Ironic feature in the control plane. The
enable_agent_driver
flag has been added to the Ironic
configuration data and can have values of true
or
false
. Setting this flag to false
will
disable Swift configurations and the agent based drivers in the Ironic
control plane.
In a multiple control plane cloud setup, changes for Glance container name
in the Swift namespace of ironic-conductor.conf
introduces a conflict with the one in glance-api.conf
.
Provisioning with agent-based drivers requires the container name to be the
same in Ironic and Glance. Hence, on instance provisioning with agent-based
drivers (Swift-enabled), the agent is not able to fetch the images from
Glance store and fails at that point.
You can resolve this issue using the following steps:
Copy the value of swift_store_container
from the file
/opt/stack/service/glance-api/etc/glance-api.conf
Log in to the Cloud Lifecycle Manager and use the value for
swift_container
in glance namespace of
~/scratch/ansible/next/ardana/ansible/roles/ironic-common/templates/ironic-conductor.conf.j2
Run the following playbook:
cd ~/scratch/ansible/next/ardana/ansible ansible-playbook -i hosts/verb_hosts ironic-reconfigure.yml
Providing bare-metal resources to an untrusted third party is not advised as a malicious user can potentially modify hardware firmware.
The steps outlined in Section 17.1.7, “TLS Certificates with Ironic Python Agent (IPA) Images” must be performed.
A number of drivers are available to provision and manage bare-metal machines. The drivers are named based on the deployment mode and the power management interface. HPE Helion OpenStack has been tested with the following drivers:
agent_ilo
agent_ipmi
pxe_ilo
pxe_ipmi
For HPE servers, agent_ipmitool
and
agent_ilo
drivers are supported.
Before you start, you should be aware that:
Node Cleaning is enabled for all the drivers in HPE Helion OpenStack 8.
Node parameter settings must have matching flavors in terms of
cpus
, local_gb
, and
memory_mb
, boot_mode
and
cpu_arch
.
It is advisable that nodes enrolled for ipmitool drivers are pre-validated in terms of BIOS settings, in terms of boot mode, prior to setting capabilities.
Network cabling and interface layout should also be pre-validated in any given particular boot mode or configuration that is registered.
The use of agent_
drivers is predicated upon Glance
images being backed by a Swift image store, specifically the need for the
temporary file access features. Using the file system as a Glance back-end
image store means that the agent_
drivers cannot be
used.
Manual Cleaning (RAID) and Node inspection is supported by ilo drivers
(agent_ilo
and pxe_ilo)
As part of the HPE Helion OpenStack Entry-scale Ironic Cloud installation, Ironic Python Agent (IPA) images are supplied and loaded into Glance. To see the images that have been loaded, execute the following commands on the deployer node:
$ source ~/service.osrc glance image list
This will display three images that have been added by Ironic:
Deploy_iso : openstack-ironic-image.x86_64-8.0.0.kernel.4.4.120-94.17-default Deploy_kernel : openstack-ironic-image.x86_64-8.0.0.xz Deploy_ramdisk : openstack-ironic-image.x86_64-8.0.0.iso
The ir-deploy-ramdisk
image is a traditional boot ramdisk
used by the agent_ipmitool
,
pxe_ipmitool
, and pxe_ilo
drivers
while ir-deploy-iso
is an ISO image that is supplied as
virtual media to the host when using the agent_ilo
driver.
The information required to provision a node varies slightly depending on the driver used. In general the following details are required.
Network access information and credentials to connect to the management interface of the node.
Sufficient properties to allow for Nova flavor matching.
A deployment image to perform the actual deployment of the guest operating system to the bare-metal node.
A combination of the ironic node-create
and
ironic node-update
commands are used for registering a
node's characteristics with the Ironic service. In particular,
ironic node-update <nodeid>
add
and ironic node-update
<nodeid> replace
can be used to
modify the properties of a node after it has been created while
ironic node-update <nodeid>
remove
will remove a property.
agent_ilo
#If you want to use a boot mode of BIOS as opposed to UEFI, then you need to ensure that the boot mode has been set correctly on the IPMI:
While the iLO driver can automatically set a node to boot in UEFI mode via
the boot_mode
defined capability, it cannot set BIOS boot
mode once UEFI mode has been set.
Use the ironic node-create
command to specify the
agent_ilo
driver, network access and credential
information for the IPMI, properties of the node and the Glance ID of the
supplied ISO IPA image. Note that memory size is specified in megabytes while
disk size is specified in gigabytes.
ironic node-create -d agent_ilo -i ilo_address=IP_ADDRESS -i \ ilo_username=Administrator -i ilo_password=PASSWORD \ -p cpus=2 -p cpu_arch=x86_64 -p memory_mb=64000 -p local_gb=99 \ -i ilo_deploy_iso=DEPLOY_UUID
This will generate output similar to the following:
+--------------+---------------------------------------------------------------+ | Property | Value | +--------------+---------------------------------------------------------------+ | uuid | NODE_UUID | | driver_info | {u'ilo_address': u'IP_ADDRESS', u'ilo_password': u'******', | | | u'ilo_deploy_iso': u'DEPLOY_UUID', | | | u'ilo_username': u'Administrator'} | | extra | {} | | driver | agent_ilo | | chassis_uuid | | | properties | {u'memory_mb': 64000, u'local_gb': 99, u'cpus': 2, | | | u'cpu_arch': u'x86_64'} | | name | None | +--------------+---------------------------------------------------------------+
Now update the node with boot_mode
and
boot_option
properties:
ironic node-update NODE_UUID add \ properties/capabilities="boot_mode:bios,boot_option:local"
The ironic node-update
command returns details for all of
the node's characteristics.
+------------------------+------------------------------------------------------------------+ | Property | Value | +------------------------+------------------------------------------------------------------+ | target_power_state | None | | extra | {} | | last_error | None | | updated_at | None | | maintenance_reason | None | | provision_state | available | | clean_step | {} | | uuid | NODE_UUID | | console_enabled | False | | target_provision_state | None | | provision_updated_at | None | | maintenance | False | | inspection_started_at | None | | inspection_finished_at | None | | power_state | None | | driver | agent_ilo | | reservation | None | | properties | {u'memory_mb': 64000, u'cpu_arch': u'x86_64', u'local_gb': 99, | | | u'cpus': 2, u'capabilities': u'boot_mode:bios,boot_option:local'}| | instance_uuid | None | | name | None | | driver_info | {u'ilo_address': u'10.1.196.117', u'ilo_password': u'******', | | | u'ilo_deploy_iso': u'DEPLOY_UUID', | | | u'ilo_username': u'Administrator'} | | created_at | 2016-03-11T10:17:10+00:00 | | driver_internal_info | {} | | chassis_uuid | | | instance_info | {} | +------------------------+------------------------------------------------------------------+
agent_ipmi
#
Use the ironic node-create
command to specify the
agent_ipmi
driver, network access and credential
information for the IPMI, properties of the node and the Glance IDs of the
supplied kernel and ramdisk images. Note that memory size is specified in
megabytes while disk size is specified in gigabytes.
ironic node-create -d agent_ipmitool \
-i ipmi_address=IP_ADDRESS \
-i ipmi_username=Administrator -i ipmi_password=PASSWORD \
-p cpus=2 -p memory_mb=64000 -p local_gb=99 -p cpu_arch=x86_64 \
-i deploy_kernel=KERNEL_UUID \
-i deploy_ramdisk=RAMDISK_UUID
This will generate output similar to the following:
+--------------+-----------------------------------------------------------------------+ | Property | Value | +--------------+-----------------------------------------------------------------------+ | uuid | NODE2_UUID | | driver_info | {u'deploy_kernel': u'KERNEL_UUID', | | | u'ipmi_address': u'IP_ADDRESS', u'ipmi_username': u'Administrator', | | | u'ipmi_password': u'******', | | | u'deploy_ramdisk': u'RAMDISK_UUID'} | | extra | {} | | driver | agent_ipmitool | | chassis_uuid | | | properties | {u'memory_mb': 64000, u'cpu_arch': u'x86_64', u'local_gb': 99, | | | u'cpus': 2} | | name | None | +--------------+-----------------------------------------------------------------------+
Now update the node with boot_mode
and
boot_option
properties:
ironic node-update NODE_UUID add \ properties/capabilities="boot_mode:bios,boot_option:local"
The ironic node-update
command returns details for all of
the node's characteristics.
+------------------------+-----------------------------------------------------------------+ | Property | Value | +------------------------+-----------------------------------------------------------------+ | target_power_state | None | | extra | {} | | last_error | None | | updated_at | None | | maintenance_reason | None | | provision_state | available | | clean_step | {} | | uuid | NODE2_UUID | | console_enabled | False | | target_provision_state | None | | provision_updated_at | None | | maintenance | False | | inspection_started_at | None | | inspection_finished_at | None | | power_state | None | | driver | agent_ipmitool | | reservation | None | | properties | {u'memory_mb': 64000, u'cpu_arch': u'x86_64', | | | u'local_gb': 99, u'cpus': 2, | | | u'capabilities': u'boot_mode:bios,boot_option:local'} | | instance_uuid | None | | name | None | | driver_info | {u'ipmi_password': u'******', u'ipmi_address': u'IP_ADDRESS', | | | u'ipmi_username': u'Administrator', u'deploy_kernel': | | | u'KERNEL_UUID', | | | u'deploy_ramdisk': u'RAMDISK_UUID'} | | created_at | 2016-03-11T14:19:18+00:00 | | driver_internal_info | {} | | chassis_uuid | | | instance_info | {} | +------------------------+-----------------------------------------------------------------+
For more information on node enrollment, see the OpenStack documentation at http://docs.openstack.org/developer/ironic/deploy/install-guide.html#enrollment.
Nova uses flavors when fulfilling requests for bare-metal nodes. The Nova
scheduler attempts to match the requested flavor against the properties of
the created Ironic nodes. So an administrator needs to set up flavors that
correspond to the available bare-metal nodes using the command
nova flavor-create
:
nova flavor-create bmtest auto 64000 99 2 +----------------+--------+--------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Mem_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +----------------+--------+--------+------+-----------+------+-------+-------------+-----------+ | 645de0...b1348 | bmtest | 64000 | 99 | 0 | | 2 | 1.0 | True | +----------------+--------+--------+------+-----------+------+-------+-------------+-----------+
To see a list of all the available flavors, run nova
flavor-list
:
nova flavor-list +-------------+--------------+--------+------+-----------+------+-------+--------+-----------+ | ID | Name | Mem_MB | Disk | Ephemeral | Swap | VCPUs | RXTX | Is_Public | | | | | | | | | Factor | | +-------------+--------------+--------+------+-----------+------+-------+--------+-----------+ | 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True | | 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True | | 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True | | 6 | m1.baremetal | 4096 | 80 | 0 | | 2 | 1.0 | True | | 645d...1348 | bmtest | 64000 | 99 | 0 | | 2 | 1.0 | True | +-------------+--------------+--------+------+-----------+------+-------+--------+-----------+
Now set the CPU architecture and boot mode and boot option capabilities:
nova flavor-key 645de08d-2bc6-43f1-8a5f-2315a75b1348 set cpu_arch=x86_64 nova flavor-key 645de08d-2bc6-43f1-8a5f-2315a75b1348 set capabilities:boot_option="local" nova flavor-key 645de08d-2bc6-43f1-8a5f-2315a75b1348 set capabilities:boot_mode="bios"
For more information on flavor creation, see the OpenStack documentation at http://docs.openstack.org/developer/ironic/deploy/install-guide.html#flavor-creation.
Register the MAC addresses of all connected physical network interfaces intended for use with the bare-metal node.
ironic port-create -a 5c:b9:01:88:f0:a4 -n ea7246fd-e1d6-4637-9699-0b7c59c22e67
You can create a complete disk image using the instructions at Section 17.3.11, “Building Glance Images Using KIWI”.
The image you create can then be loaded into Glance:
glance image-create --name='leap' --disk-format=raw \
--container-format=bare \
--file /tmp/myimage/LimeJeOS-Leap-42.3.x86_64-1.42.3.raw
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | 45a4a06997e64f7120795c68beeb0e3c |
| container_format | bare |
| created_at | 2018-02-17T10:42:14Z |
| disk_format | raw |
| id | 17e4915a-ada0-4b95-bacf-ba67133f39a7 |
| min_disk | 0 |
| min_ram | 0 |
| name | leap |
| owner | 821b7bb8148f439191d108764301af64 |
| protected | False |
| size | 372047872 |
| status | active |
| tags | [] |
| updated_at | 2018-02-17T10:42:23Z |
| virtual_size | None |
| visibility | private |
+------------------+--------------------------------------+
This image will subsequently be used to boot the bare-metal node.
Create a key pair that you will use when you login to the newly booted node:
nova keypair-add ironic_kp > ironic_kp.pem
neutron net-list
+---------------+----------+----------------------------------------------------+
| id | name | subnets |
+---------------+----------+----------------------------------------------------+
| c0102...1ca8c | flat-net | 709ee2a1-4110-4b26-ba4d-deb74553adb9 192.3.15.0/24 |
+---------------+----------+----------------------------------------------------+
Before booting, it is advisable to power down the node:
ironic node-set-power-state ea7246fd-e1d6-4637-9699-0b7c59c22e67 off
You can now boot the bare-metal node with the information compiled in the preceding steps, using the Neutron network ID, the whole disk image ID, the matching flavor and the key name:
nova boot --nic net-id=c010267c-9424-45be-8c05-99d68531ca8c \ --image 17e4915a-ada0-4b95-bacf-ba67133f39a7 \ --flavor 645de08d-2bc6-43f1-8a5f-2315a75b1348 \ --key-name ironic_kp leap
This command returns information about the state of the node that is booting:
+--------------------------------------+------------------------+ | Property | Value | +--------------------------------------+------------------------+ | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | instance-00000001 | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | adpHw3KKTjHk | | config_drive | | | created | 2018-03-11T11:00:28Z | | flavor | bmtest (645de...b1348) | | hostId | | | id | a9012...3007e | | image | leap (17e49...f39a7) | | key_name | ironic_kp | | metadata | {} | | name | leap | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | d53bcaf...baa60dd | | updated | 2016-03-11T11:00:28Z | | user_id | e580c64...4aaf990 | +--------------------------------------+------------------------+
The boot process can take up to 10 minutes. Monitor the progress with the
IPMI console or with nova list
, nova show
<nova_node_id>
, and ironic node-show
<ironic_node_id>
commands.
nova list +---------------+--------+--------+------------+-------------+----------------------+ | ID | Name | Status | Task State | Power State | Networks | +---------------+--------+--------+------------+-------------+----------------------+ | a9012...3007e | leap | BUILD | spawning | NOSTATE | flat-net=192.3.15.12 | +---------------+--------+--------+------------+-------------+----------------------+
During the boot procedure, a login prompt will appear for SLES:
Ignore this login screen and wait for the login screen of your target operating system to appear:
If you now run the command nova list
, it should show the
node in the running state:
nova list
+---------------+--------+--------+------------+-------------+----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+---------------+--------+--------+------------+-------------+----------------------+
| a9012...3007e | leap | ACTIVE | - | Running | flat-net=192.3.15.14 |
+---------------+--------+--------+------------+-------------+----------------------+
You can now log in to the booted node using the key you generated earlier. (You may be prompted to change the permissions of your private key files, so that they are not accessible by others).
ssh leap@192.3.15.14 -i ironic_kp.pem
The following sections show you how to create your own images using KIWI, the command line utility to build Linux system appliances. For information on installing KIWI, see https://osinside.github.io/kiwi/installation.html.
KIWI creates images in a two-step process:
The prepare
operation generates an unpacked image tree
using the information provided in the image description.
The create
operation creates the packed image based on
the unpacked image and the information provided in the configuration file
(config.xml
).
Instructions for installing KIWI are available at https://osinside.github.io/kiwi/installation.html.
Image creation with KIWI is automated and does not require any user interaction. The information required for the image creation process is provided by the image description.
To use and run KIWI requires:
A recent Linux distribution such as:
openSUSE Leap 42.3
SUSE Linux Enterprise 12 SP3
openSUSE Tumbleweed
Enough free disk space to build and store the image (a minimum of 10 GB is recommended).
Python version 2.7, 3.4 or higher. KIWI supports both Python 2 and 3 versions
Git (package git-core) to clone a repository.
Virtualization technology to start the image (QEMU is recommended).
The following example shows how to build an openSUSE Leap image that is ready to run in QEMU.
Retrieve the example image descriptions.
git clone https://github.com/SUSE/kiwi-descriptions
Build the image with KIWI:
sudo kiwi-ng --type vmx system build \ --description kiwi-descriptions/suse/x86_64/suse-leap-42.3-JeOS \ --target-dir /tmp/myimage
A .raw
image will be built in the
/tmp/myimage
directory.
Test the live image with QEMU:
qemu \ -drive file=LimeJeOS-Leap-42.3.x86_64-1.42.3.raw,format=raw,if=virtio \ -m 4096
With a successful test, the image is complete.
By default, KIWI generates a file in the .raw
format.
The .raw
file is a disk image with a structure
equivalent to a physical hard disk. .raw
images are
supported by any hypervisor, but are not compressed and do not offer the
best performance.
Virtualization systems support their own formats (such as
qcow2
or vmdk
) with compression and
improved I/O performance. To build an image in a format other than
.raw
, add the format attribute to the type definition
in the preferences section of config.xml
. Using
qcow2
for example:
<image ...> <preferences> <type format="qcow2" .../> ... </preferences> ... </image
More information about KIWI is at https://osinside.github.io/kiwi/.
To enable Ironic multi-tenancy, you must first manually install the
python-networking-generic-switch
package along with all
its dependents on all Neutron nodes.
To manually enable the genericswitch
mechanism driver in
Neutron, the networking-generic-switch
package must be
installed first. Do the following steps in each of the controllers where
Neutron is running.
Comment out the multi_tenancy_switch_config
section in
~/openstack/my_cloud/definition/data/ironic/ironic_config.yml
.
SSH into the controller node
Change to root
ardana >
sudo -i
Activate the neutron venv
tux >
sudo . /opt/stack/venv/neutron-20180528T093206Z/bin/activate
Install netmiko package
tux >
sudo pip install netmiko
Clone the networking-generic-switch
source code into
/tmp
tux >
sudo cd /tmptux >
sudo git clone https://github.com/openstack/networking-generic-switch.git
Install networking_generic_switch
package
tux >
sudo python setup.py install
After the networking_generic_switch
package is installed,
the genericswitch
settings must be enabled in the input
model. The following process must be run again any time a maintenance update
is installed that updates the Neutron venv.
SSH into the deployer node as the user ardana
.
Edit the Ironic configuration data in the input model
~/openstack/my_cloud/definition/data/ironic/ironic_config.yml
. Make
sure the multi_tenancy_switch_config:
section is
uncommented and has the appropriate settings. driver_type
should be genericswitch
and
device_type
should be
netmiko_hp_comware
.
multi_tenancy_switch_config: - id: switch1 driver_type: genericswitch device_type: netmiko_hp_comware ip_address: 192.168.75.201 username: IRONICSHARE password: 'k27MwbEDGzTm'
Run the configure process to generate the model
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost config-processor-run.ymlardana >
ansible-playbook -i hosts/localhost ready-deployment.yml
Run neutron-reconfigure.yml
ardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/localhost neutron-reconfigure.yml
Run neutron-status.yml
to make sure everything is OK
ardana >
ansible-playbook -i hosts/verb_hosts neutron-status.yml
With the networking-generic-switch
package installed and
enabled, you can proceed with provisioning baremetal nodes with multi-tenancy.
Create a network and a subnet:
$ neutron net-create guest-net-1 Created a new network: +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | availability_zone_hints | | | availability_zones | | | created_at | 2017-06-10T02:49:56Z | | description | | | id | 256d55a6-9430-4f49-8a4c-cc5192f5321e | | ipv4_address_scope | | | ipv6_address_scope | | | mtu | 1500 | | name | guest-net-1 | | project_id | 57b792cdcdd74d16a08fc7a396ee05b6 | | provider:network_type | vlan | | provider:physical_network | physnet1 | | provider:segmentation_id | 1152 | | revision_number | 2 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | | | tags | | | tenant_id | 57b792cdcdd74d16a08fc7a396ee05b6 | | updated_at | 2017-06-10T02:49:57Z | +---------------------------+--------------------------------------+ $ neutron subnet-create guest-net-1 200.0.0.0/24 Created a new subnet: +-------------------+----------------------------------------------+ | Field | Value | +-------------------+----------------------------------------------+ | allocation_pools | {"start": "200.0.0.2", "end": "200.0.0.254"} | | cidr | 200.0.0.0/24 | | created_at | 2017-06-10T02:53:08Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 200.0.0.1 | | host_routes | | | id | 53accf35-ae02-43ae-95d8-7b5efed18ae9 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | | | network_id | 256d55a6-9430-4f49-8a4c-cc5192f5321e | | project_id | 57b792cdcdd74d16a08fc7a396ee05b6 | | revision_number | 2 | | service_types | | | subnetpool_id | | | tenant_id | 57b792cdcdd74d16a08fc7a396ee05b6 | | updated_at | 2017-06-10T02:53:08Z | +-------------------+----------------------------------------------+
Review glance image list
$ glance image list +--------------------------------------+--------------------------+ | ID | Name | +--------------------------------------+--------------------------+ | 0526d2d7-c196-4c62-bfe5-a13bce5c7f39 | cirros-0.4.0-x86_64 | +--------------------------------------+--------------------------+
Create Ironic node
$ ironic --ironic-api-version 1.22 node-create -d agent_ipmitool \ -n test-node-1 -i ipmi_address=192.168.9.69 -i ipmi_username=ipmi_user \ -i ipmi_password=XXXXXXXX --network-interface neutron -p memory_mb=4096 \ -p cpu_arch=x86_64 -p local_gb=80 -p cpus=2 \ -p capabilities=boot_mode:bios,boot_option:local \ -p root_device='{"name":"/dev/sda"}' \ -i deploy_kernel=db3d131f-2fb0-4189-bb8d-424ee0886e4c \ -i deploy_ramdisk=304cae15-3fe5-4f1c-8478-c65da5092a2c +-------------------+-------------------------------------------------------------------+ | Property | Value | +-------------------+-------------------------------------------------------------------+ | chassis_uuid | | | driver | agent_ipmitool | | driver_info | {u'deploy_kernel': u'db3d131f-2fb0-4189-bb8d-424ee0886e4c', | | | u'ipmi_address': u'192.168.9.69', | | | u'ipmi_username': u'gozer', u'ipmi_password': u'******', | | | u'deploy_ramdisk': u'304cae15-3fe5-4f1c-8478-c65da5092a2c'} | | extra | {} | | name | test-node-1 | | network_interface | neutron | | properties | {u'cpu_arch': u'x86_64', u'root_device': {u'name': u'/dev/sda'}, | | | u'cpus': 2, u'capabilities': u'boot_mode:bios,boot_option:local', | | | u'memory_mb': 4096, u'local_gb': 80} | | resource_class | None | | uuid | cb4dda0d-f3b0-48b9-ac90-ba77b8c66162 | +-------------------+-------------------------------------------------------------------+
ipmi_address, ipmi_username and ipmi_password are IPMI access parameters for baremetal Ironic node. Adjust memory_mb, cpus, local_gb to your node size requirements. They also need to be reflected in flavor setting (see below). Use capabilities boot_mode:bios for baremetal nodes operating in Legacy BIOS mode. For UEFI baremetal nodes, use boot_mode:uefi lookup deploy_kernel and deploy_ramdisk in glance image list output above.
Since we are using Ironic API version 1.22, node is created initial state enroll. It needs to be explicitly moved to available state. This behavior changed in API version 1.11
Create port
$ ironic --ironic-api-version 1.22 port-create --address f0:92:1c:05:6c:40 \ --node cb4dda0d-f3b0-48b9-ac90-ba77b8c66162 -l switch_id=e8:f7:24:bf:07:2e -l \ switch_info=hp59srv1-a-11b -l port_id="Ten-GigabitEthernet 1/0/34" \ --pxe-enabled true +-----------------------+--------------------------------------------+ | Property | Value | +-----------------------+--------------------------------------------+ | address | f0:92:1c:05:6c:40 | | extra | {} | | local_link_connection | {u'switch_info': u'hp59srv1-a-11b', | | | u'port_id': u'Ten-GigabitEthernet 1/0/34', | | | u'switch_id': u'e8:f7:24:bf:07:2e'} | | node_uuid | cb4dda0d-f3b0-48b9-ac90-ba77b8c66162 | | pxe_enabled | True | | uuid | a49491f3-5595-413b-b4a7-bb6f9abec212 | +-----------------------+--------------------------------------------+
for --address
, use MAC of 1st NIC of ironic baremetal
node, which will be used for PXE boot
for --node
, use ironic node uuid (see above)
for -l switch_id
, use switch management interface MAC
address. It can be
retrieved by pinging switch management IP and looking up MAC address in
'arp -l -n' command output.
for -l switch_info
, use switch_id from
data/ironic/ironic_config.yml
file. If you have several switch config definitions, use the right switch
your baremetal node is connected to.
for -l port_id, use port ID on the switch
Move ironic node to manage and then available state
$ ironic node-set-provision-state test-node-1 manage $ ironic node-set-provision-state test-node-1 provide
Once node is successfully moved to available state, its resources should be included into Nova hypervisor statistics
$ nova hypervisor-stats +----------------------+-------+ | Property | Value | +----------------------+-------+ | count | 1 | | current_workload | 0 | | disk_available_least | 80 | | free_disk_gb | 80 | | free_ram_mb | 4096 | | local_gb | 80 | | local_gb_used | 0 | | memory_mb | 4096 | | memory_mb_used | 0 | | running_vms | 0 | | vcpus | 2 | | vcpus_used | 0 | +----------------------+-------+
Prepare a keypair, which will be used for logging into the node
$ nova keypair-add ironic_kp > ironic_kp.pem
Obtain user image and upload it to glance. Please refer to OpenStack documentation on user image creation: https://docs.openstack.org/project-install-guide/baremetal/draft/configure-glance-images.html.
Deployed images are already populated by HPE Helion OpenStack installer.
$ glance image-create --name='Ubuntu Trusty 14.04' --disk-format=qcow2 \ --container-format=bare --file ~/ubuntu-trusty.qcow2 +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | d586d8d2107f328665760fee4c81caf0 | | container_format | bare | | created_at | 2017-06-13T22:38:45Z | | disk_format | qcow2 | | id | 9fdd54a3-ccf5-459c-a084-e50071d0aa39 | | min_disk | 0 | | min_ram | 0 | | name | Ubuntu Trusty 14.04 | | owner | 57b792cdcdd74d16a08fc7a396ee05b6 | | protected | False | | size | 371508736 | | status | active | | tags | [] | | updated_at | 2017-06-13T22:38:55Z | | virtual_size | None | | visibility | private | +------------------+--------------------------------------+ $ glance image list +--------------------------------------+---------------------------+ | ID | Name | +--------------------------------------+---------------------------+ | 0526d2d7-c196-4c62-bfe5-a13bce5c7f39 | cirros-0.4.0-x86_64 | | 83eecf9c-d675-4bf9-a5d5-9cf1fe9ee9c2 | ir-deploy-iso-EXAMPLE | | db3d131f-2fb0-4189-bb8d-424ee0886e4c | ir-deploy-kernel-EXAMPLE | | 304cae15-3fe5-4f1c-8478-c65da5092a2c | ir-deploy-ramdisk- EXAMPLE | | 9fdd54a3-ccf5-459c-a084-e50071d0aa39 | Ubuntu Trusty 14.04 | +--------------------------------------+---------------------------+
Create a baremetal flavor and set flavor keys specifying requested node size, architecture and boot mode. A flavor can be re-used for several nodes having the same size, architecture and boot mode
$ nova flavor-create m1.ironic auto 4096 80 2 +-------------+-----------+--------+------+---------+------+-------+-------------+-----------+ | ID | Name | Mem_MB | Disk | Ephemrl | Swap | VCPUs | RXTX_Factor | Is_Public | +-------------+-----------+--------+------+---------+------+-------+-------------+-----------+ | ab69...87bf | m1.ironic | 4096 | 80 | 0 | | 2 | 1.0 | True | +-------------+-----------+--------+------+---------+------+-------+-------------+-----------+ $ nova flavor-key ab6988...e28694c87bf set cpu_arch=x86_64 $ nova flavor-key ab6988...e28694c87bf set capabilities:boot_option="local" $ nova flavor-key ab6988...e28694c87bf set capabilities:boot_mode="bios"
Parameters must match parameters of ironic node above. Use
capabilities:boot_mode="bios"
for Legacy BIOS nodes. For
UEFI nodes, use capabilities:boot_mode="uefi"
Boot the node
$ nova boot --flavor m1.ironic --image 9fdd54a3-ccf5-459c-a084-e50071d0aa39 \ --key-name ironic_kp --nic net-id=256d55a6-9430-4f49-8a4c-cc5192f5321e \ test-node-1 +--------------------------------------+-------------------------------------------------+ | Property | Value | +--------------------------------------+-------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | XXXXXXXXXXXX | | config_drive | | | created | 2017-06-14T21:25:18Z | | flavor | m1.ironic (ab69881...5a-497d-93ae-6e28694c87bf) | | hostId | | | id | f1a8c63e-da7b-4d9a-8648-b1baa6929682 | | image | Ubuntu Trusty 14.04 (9fdd54a3-ccf5-4a0...0aa39) | | key_name | ironic_kp | | metadata | {} | | name | test-node-1 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | 57b792cdcdd74d16a08fc7a396ee05b6 | | updated | 2017-06-14T21:25:17Z | | user_id | cc76d7469658401fbd4cf772278483d9 | +--------------------------------------+-------------------------------------------------+
for --image
, use the ID of user image created at
previous step
for --nic net-id
, use ID of
the tenant network created at the beginning
During the node provisioning, the following is happening in the background:
Neutron connects to switch management interfaces and assigns provisioning
VLAN to baremetal node port on the switch. Ironic powers up the node using
IPMI interface. Node is booting IPA image via PXE. IPA image is writing
provided user image onto specified root device
(/dev/sda
) and powers node
down. Neutron connects to switch management interfaces and assigns tenant
VLAN to baremetal node port on the switch. A VLAN ID is selected from
provided range. Ironic powers up the node using IPMI interface. Node is
booting user image from disk.
Once provisioned, node will join the private tenant network. Access to private network from other networks is defined by switch configuration.
nova show <nova-node-id>
#nova show a90122ce-bba8-496f-92a0-8a7cb143007e +--------------------------------------+-----------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------+ | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | ardana-cp1-ir-compute0001-mgmt | | OS-EXT-SRV-ATTR:hypervisor_hostname | ea7246fd-e1d6-4637-9699-0b7c59c22e67 | | OS-EXT-SRV-ATTR:instance_name | instance-0000000a | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2016-03-11T12:26:25.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2016-03-11T12:17:54Z | | flat-net network | 192.3.15.14 | | flavor | bmtest (645de08d-2bc6-43f1-8a5f-2315a75b1348) | | hostId | ecafa4f40eb5f72f7298...3bad47cbc01aa0a076114f | | id | a90122ce-bba8-496f-92a0-8a7cb143007e | | image | ubuntu (17e4915a-ada0-4b95-bacf-ba67133f39a7) | | key_name | ironic_kp | | metadata | {} | | name | ubuntu | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | ACTIVE | | tenant_id | d53bcaf15afb4cb5aea3adaedbaa60dd | | updated | 2016-03-11T12:26:26Z | | user_id | e580c645bfec4faeadef7dbd24aaf990 | +--------------------------------------+-----------------------------------------------+
ironic node-show <ironic-node-id>
#ironic node-show ea7246fd-e1d6-4637-9699-0b7c59c22e67 +------------------------+--------------------------------------------------------------------------+ | Property | Value | +------------------------+--------------------------------------------------------------------------+ | target_power_state | None | | extra | {} | | last_error | None | | updated_at | 2016-03-11T12:26:25+00:00 | | maintenance_reason | None | | provision_state | active | | clean_step | {} | | uuid | ea7246fd-e1d6-4637-9699-0b7c59c22e67 | | console_enabled | False | | target_provision_state | None | | provision_updated_at | 2016-03-11T12:26:25+00:00 | | maintenance | False | | inspection_started_at | None | | inspection_finished_at | None | | power_state | power on | | driver | agent_ilo | | reservation | None | | properties | {u'memory_mb': 64000, u'cpu_arch': u'x86_64', u'local_gb': 99, | | | u'cpus': 2, u'capabilities': u'boot_mode:bios,boot_option:local'} | | instance_uuid | a90122ce-bba8-496f-92a0-8a7cb143007e | | name | None | | driver_info | {u'ilo_address': u'10.1.196.117', u'ilo_password': u'******', | | | u'ilo_deploy_iso': u'b9499494-7db3-4448-b67f-233b86489c1f', | | | u'ilo_username': u'Administrator'} | | created_at | 2016-03-11T10:17:10+00:00 | | driver_internal_info | {u'agent_url': u'http://192.3.15.14:9999', | | | u'is_whole_disk_image': True, u'agent_last_heartbeat': 1457699159} | | chassis_uuid | | | instance_info | {u'root_gb': u'99', u'display_name': u'ubuntu', u'image_source': u | | | '17e4915a-ada0-4b95-bacf-ba67133f39a7', u'capabilities': u'{"boot_mode": | | | "bios", "boot_option": "local"}', u'memory_mb': u'64000', u'vcpus': | | | u'2', u'image_url': u'http://192.168.12.2:8080/v1/AUTH_ba121db7732f4ac3a | | | 50cc4999a10d58d/glance/17e4915a-ada0-4b95-bacf-ba67133f39a7?temp_url_sig | | | =ada691726337805981ac002c0fbfc905eb9783ea&temp_url_expires=1457699878', | | | u'image_container_format': u'bare', u'local_gb': u'99', | | | u'image_disk_format': u'qcow2', u'image_checksum': | | | u'2d7bb1e78b26f32c50bd9da99102150b', u'swap_mb': u'0'} | +------------------------+--------------------------------------------------------------------------+
ironic port-show <ironic-port-id>
#ironic port-show a17a4ef8-a711-40e2-aa27-2189c43f0b67 +------------+-----------------------------------------------------------+ | Property | Value | +------------+-----------------------------------------------------------+ | node_uuid | ea7246fd-e1d6-4637-9699-0b7c59c22e67 | | uuid | a17a4ef8-a711-40e2-aa27-2189c43f0b67 | | extra | {u'vif_port_id': u'82a5ab28-76a8-4c9d-bfb4-624aeb9721ea'} | | created_at | 2016-03-11T10:40:53+00:00 | | updated_at | 2016-03-11T12:17:56+00:00 | | address | 5c:b9:01:88:f0:a4 | +------------+-----------------------------------------------------------+
nova hypervisor-list
and nova hypervisor-show
#nova hypervisor-list +-----+--------------------------------------+-------+---------+ | ID | Hypervisor hostname | State | Status | +-----+--------------------------------------+-------+---------+ | 541 | ea7246fd-e1d6-4637-9699-0b7c59c22e67 | up | enabled | +-----+--------------------------------------+-------+---------+
nova hypervisor-show ea7246fd-e1d6-4637-9699-0b7c59c22e67 +-------------------------+--------------------------------------+ | Property | Value | +-------------------------+--------------------------------------+ | cpu_info | | | current_workload | 0 | | disk_available_least | 0 | | free_disk_gb | 0 | | free_ram_mb | 0 | | host_ip | 192.168.12.6 | | hypervisor_hostname | ea7246fd-e1d6-4637-9699-0b7c59c22e67 | | hypervisor_type | ironic | | hypervisor_version | 1 | | id | 541 | | local_gb | 99 | | local_gb_used | 99 | | memory_mb | 64000 | | memory_mb_used | 64000 | | running_vms | 1 | | service_disabled_reason | None | | service_host | ardana-cp1-ir-compute0001-mgmt | | service_id | 25 | | state | up | | status | enabled | | vcpus | 2 | | vcpus_used | 2 | +-------------------------+--------------------------------------+
nova service-list
#nova service-list +----+------------------+-----------------------+----------+---------+-------+------------+----------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled | | | | | | | | | Reason | +----+------------------+-----------------------+----------+---------+-------+------------+----------+ | 1 | nova-conductor | ardana-cp1-c1-m1-mgmt | internal | enabled | up | date:time | - | | 7 | nova-conductor | " -cp1-c1-m2-mgmt | internal | enabled | up | date:time | - | | 10 | nova-conductor | " -cp1-c1-m3-mgmt | internal | enabled | up | date:time | - | | 13 | nova-scheduler | " -cp1-c1-m1-mgmt | internal | enabled | up | date:time | - | | 16 | nova-scheduler | " -cp1-c1-m3-mgmt | internal | enabled | up | date:time | - | | 19 | nova-scheduler | " -cp1-c1-m2-mgmt | internal | enabled | up | date:time | - | | 22 | nova-consoleauth | " -cp1-c1-m1-mgmt | internal | enabled | up | date:time | - | | 25 | nova-compute | " -cp1-ir- | nova | | enabled | up | date:time | - | | | | compute0001-mgmt | | | | | | +----+------------------+-----------------------+----------+---------+-------+------------+----------+
Sometimes the nova boot
command does not succeed and when
you do a nova list
, you will see output like the
following:
ardana >
nova list
+------------------+--------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+------------------+--------------+--------+------------+-------------+----------+
| ee08f82...624e5f | OpenSUSE42.3 | ERROR | - | NOSTATE | |
+------------------+--------------+--------+------------+-------------+----------+
You should execute the nova show <nova-node-id>
and
ironic node-show <ironic-node-id>
commands to get
more information about the error.
The error No valid host was found. There are not enough
hosts.
is typically seen when performing the nova
boot
where there is a mismatch between the properties set on the
node and the flavor used. For example, the output from a nova
show
command may look like this:
ardana >
nova show ee08f82e-8920-4360-be51-a3f995624e5f
+------------------------+------------------------------------------------------------------------------+
| Property | Value |
+------------------------+------------------------------------------------------------------------------+
| OS-EXT-AZ: | |
| availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR: | |
| hypervisor_hostname | - |
| OS-EXT-SRV-ATTR: | |
| instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG: | |
| terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-03-11T11:00:28Z |
| fault | {"message": "No valid host was found. There are not enough hosts |
| | available.", "code": 500, "details": " File \"/opt/stack/ |
| | venv/nova-20160308T002421Z/lib/python2.7/site-packages/nova/ |
| | conductor/manager.py\", line 739, in build_instances |
| | request_spec, filter_properties) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/nova/scheduler/utils.py\", line 343, in wrapped |
| | return func(*args, **kwargs) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/nova/scheduler/client/__init__.py\", line 52, |
| | in select_destinations context, request_spec, filter_properties) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/nova/scheduler/client/__init__.py\",line 37,in __run_method |
| | return getattr(self.instance, __name)(*args, **kwargs) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/nova/scheduler/client/query.py\", line 34, |
| | in select_destinations context, request_spec, filter_properties) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/nova/scheduler/rpcapi.py\", line 120, in select_destinations |
| | request_spec=request_spec, filter_properties=filter_properties) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/oslo_messaging/rpc/client.py\", line 158, in call |
| | retry=self.retry) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/oslo_messaging/transport.py\", line 90, in _send |
| | timeout=timeout, retry=retry) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 462, in send |
| | retry=retry) |
| | File \"/opt/stack/venv/nova-20160308T002421Z/lib/python2.7/ |
| | site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 453, in _send |
| | raise result |
| | ", "created": "2016-03-11T11:00:29Z"} |
| flavor | bmtest (645de08d-2bc6-43f1-8a5f-2315a75b1348) |
| hostId | |
| id | ee08f82e-8920-4360-be51-a3f995624e5f |
| image | opensuse (17e4915a-ada0-4b95-bacf-ba67133f39a7) |
| key_name | ironic_kp |
| metadata | {} |
| name | opensuse |
| os-extended-volumes: | |
| volumes_attached | [] |
| status | ERROR |
| tenant_id | d53bcaf15afb4cb5aea3adaedbaa60dd |
| updated | 2016-03-11T11:00:28Z |
| user_id | e580c645bfec4faeadef7dbd24aaf990 |
+------------------------+------------------------------------------------------------------------------+
You can find more information about the error by inspecting the log file at
/var/log/nova/nova-scheduler.log
or alternatively by
viewing the error location in the source files listed in the stack-trace (in
bold above).
To find the mismatch, compare the properties of the Ironic node:
+------------------------+---------------------------------------------------------------------+ | Property | Value | +------------------------+---------------------------------------------------------------------+ | target_power_state | None | | extra | {} | | last_error | None | | updated_at | None | | maintenance_reason | None | | provision_state | available | | clean_step | {} | | uuid | ea7246fd-e1d6-4637-9699-0b7c59c22e67 | | console_enabled | False | | target_provision_state | None | | provision_updated_at | None | | maintenance | False | | inspection_started_at | None | | inspection_finished_at | None | | power_state | None | | driver | agent_ilo | | reservation | None | | properties | {u'memory_mb': 64000, u'local_gb': 99, u'cpus': 2, u'capabilities': | | | u'boot_mode:bios,boot_option:local'} | | instance_uuid | None | | name | None | | driver_info | {u'ilo_address': u'10.1.196.117', u'ilo_password': u'******', | | | u'ilo_deploy_iso': u'b9499494-7db3-4448-b67f-233b86489c1f', | | | u'ilo_username': u'Administrator'} | | created_at | 2016-03-11T10:17:10+00:00 | | driver_internal_info | {} | | chassis_uuid | | | instance_info | {} | +------------------------+---------------------------------------------------------------------+
with the flavor characteristics:
ardana >
nova flavor-show
+----------------------------+-------------------------------------------------------------------+
| Property | Value |
+----------------------------+-------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 99 |
| extra_specs | {"capabilities:boot_option": "local", "cpu_arch": "x86_64", |
| | "capabilities:boot_mode": "bios"} |
| id | 645de08d-2bc6-43f1-8a5f-2315a75b1348 |
| name | bmtest |
| os-flavor-access:is_public | True |
| ram | 64000 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 2 |
+----------------------------+-------------------------------------------------------------------+
In this instance, the problem is caused by the absence of the "cpu_arch": "x86_64" property on the Ironic node. This can be resolved by updating the Ironic node, adding the missing property:
ardana >
ironic node-update ea7246fd-e1d6-4637-9699-0b7c59c22e67 \
add properties/cpu_arch=x86_64
and then re-running the nova boot
command.
Possible cause: The Neutron API session timed out before port creation was completed.
Resolution: Switch response time varies
by vendor; the value of url_timeout
must be increased to
allow for switch response.
Check Ironic Conductor logs
(/var/log/ironic/ironic-conductor.log
) for
ConnectTimeout
errors while connecting to Neutron for
port creation. For example:
19-03-20 19:09:14.557 11556 ERROR ironic.conductor.utils [req-77f3a7b...1b10c5b - default default] Unexpected error while preparing to deploy to node 557316...84dbdfbe8b0: ConnectTimeout: Request to https://192.168.75.1:9696/v2.0/ports timed out
Use the following steps to increase the value of
url_timeout
.
Log in to the deployer node.
Edit ./roles/ironic-common/defaults/main.yml
,
increasing the value of url_timeout
.
ardana >
cd /var/lib/ardana/scratch/ansible/next/ardana/ansibleardana >
vi ./roles/ironic-common/defaults/main.yml
Increase the value of the url_timeout
parameter in the
ironic_neutron:
section. Increase the parameter from
the default (60 seconds) to 120 and then in increments of 60 seconds until
the node deploys successfully.
Reconfigure Ironic.
ardana >
ansible-playbook -i hosts/verb_hosts ironic-reconfigure.yml
Possible cause: The IPMI commands to the node take longer to change the power state of the server.
Resolution: Check if the node power state can be changed using the following command
ardana >
ironic node-set-power-state $NODEUUID on
If the above command succeeds and the power_state column is updated correctly, then the following steps are required to increase the power sync interval time.
On the first controller, reconfigure Ironic to increase the power sync interval time. In the example below, it is set to 120 seconds. This value may have to be tuned based on the setup.
Go to the ~/openstack/my_cloud/config/ironic/
directory
and edit ironic-conductor.conf.j2
to set the
sync_power_state_interval
value:
[conductor] sync_power_state_interval = 120
Save the file and then run the following playbooks:
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost config-processor-run.ymlardana >
ansible-playbook -i hosts/localhost ready-deployment.ymlardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts ironic-reconfigure.yml
If you encounter the error below during the deployment:
"u'message': u'Error downloading image: Download of image id 77700...96551 failed: Image download failed for all URLs.', u'code': 500, u'type': u'ImageDownloadError', u'details': u'Download of image id 77700b53-9e15-406c-b2d5-13e7d9b96551 failed: Image download failed for all URLs.'"
you should visit the Single Sign-On Settings in the Security page of IPMI and change the Single Sign-On Trust Mode setting from the default of "Trust None (SSO disabled)" to "Trust by Certificate".
node-inspection
can cause temporary claim of IP addresses #
Possible cause: Running
node-inspection
on a node discovers all the NIC ports
including the NICs that do not have any connectivity. This causes a
temporary consumption of the network IPs and increased usage of the
allocated quota. As a result, other nodes are deprived of IP addresses and
deployments can fail.
Resolution:You can add node properties manually added instead of using the inspection tool.
Note: Upgrade ipmitool
to a version >= 1.8.15 or it
may not return detailed information about the NIC interface for
node-inspection
.
Possible causes:
Ironic conductor service associated with the node could go down.
There might be a properties mismatch. MAC address registered for the node could be incorrect.
Resolution: To recover from this
condition, set the provision state of the node to Error
and maintenance to True
. Delete the node and re-register
again.
Possible causes: By default, the boot order of baremetal node is set as NIC1, HDD and NIC2. If NIC1 fails, the nodes starts booting from HDD and the provisioning fails.
Resolution: Set boot order so that all the NICs appear before the hard disk of the baremetal as NIC1, NIC2…, HDD.
Possible causes:Ironic periodically performs a power state sync with all the baremetal nodes. When the number of nodes increase, Ironic does not get sufficient time to perform power operations.
Resolution: The following procedure gives a
way to increase sync_power_state_interval
:
Edit the file
~/openstack/my_cloud/config/ironic/ironic-conductor.conf.j2
and navigate to the section for [conductor]
Increase the sync_power_state_interval
. For example,
for 100 nodes, set sync_power_state_interval = 90
and
save the file.
Execute the following set of commands to reconfigure Ironic:
ardana >
cd ~/openstack/ardana/ansibleardana >
ansible-playbook -i hosts/localhost config-processor-run.ymlardana >
ansible-playbook -i hosts/localhost ready-deployment.ymlardana >
cd ~/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts ironic-reconfigure.yml
If you see DHCP error "No configuration methods succeeded" in iPXE right after successful DHCP performed by embedded NIC firmware, there may be an issue with Spanning Tree Protocol on the switch.
To avoid this error, Rapid Spanning Tree Protocol needs to be enabled on the switch. If this is not an option due to conservative loop detection strategies, use the steps outlined below to install the iPXE binary with increased DHCP timeouts.
Clone iPXE source code
tux >
git clone git://git.ipxe.org/ipxe.gittux >
cd ipxe/src
Modify lines 22-25 in file config/dhcp.h
, which declare
reduced DHCP timeouts (1-10 secs). Comment out lines with reduced timeouts
and uncomment normal PXE timeouts (4-32)
//#define DHCP_DISC_START_TIMEOUT_SEC 1 //#define DHCP_DISC_END_TIMEOUT_SEC 10 #define DHCP_DISC_START_TIMEOUT_SEC 4 /* as per PXE spec */ #define DHCP_DISC_END_TIMEOUT_SEC 32 /* as per PXE spec */
Make undionly.kpxe
(BIOS) and
ipxe.efi
(UEFI) images
tux >
make bin/undionly.kpxetux >
make bin-x86_64-efi/ipxe.efi
Copy iPXE images to Cloud Lifecycle Manager
tux >
scp bin/undionly.kpxe bin-x86_64-efi/ipxe.efi stack@10.0.0.4:
stack@10.0.0.4's password:
undionly.kpxe 100% 66KB 65.6KB/s 00:00
ipxe.efi 100% 918KB 918.2KB/s 00:00
From deployer, distribute image files onto all 3 controllers
stack@ardana-cp1-c1-m1-mgmt:ardana >
cd ~/scratch/ansible/next/ardana/ansible/ stack@ardana-cp1-c1-m1-mgmt:ardana >
~/scratch/ansible/next/ardana/ansible$ ansible -i hosts/verb_hosts \ IRN-CND -m copy -b -a 'src=~/ipxe.efi dest=/tftpboot' ... stack@ardana-cp1-c1-m1-mgmt:ardana >
~/scratch/ansible/next/ardana/ansible$ ansible -i hosts/verb_hosts \ IRN-CND -m copy -b -a 'src=~/undionly.kpxe dest=/tftpboot' ...
There is no need to restart services. With next PXE boot attempt, iPXE binary with the increased timeout will be downloaded to the target node via TFTP.
The following drivers are supported and tested:
pxe_ipmitool
(UEFI and Legacy BIOS mode, flat-network)
pxe_ipmitool
(UEFI and Legacy BIOS mode, flat-network)
pxe_ilo
(UEFI and Legacy BIOS mode, flat-network)
agent_ipmitool
(UEFI and Legacy BIOS mode, flat-network)
agent_ilo
(UEFI and Legacy BIOS mode, flat-network)
ISO Image Exceeds Free Space
When using the agent_ilo
driver, provisioning will
fail if the size of the user ISO image exceeds the free space available on
the ramdisk partition. This will produce an error in the Ironic Conductor
logs that may look like as follows:
"ERROR root [-] Command failed: prepare_image, error: Error downloading image: Download of image id 0c4d74e4-58f1-4f8d-8c1d-8a49129a2163 failed: Unable to write image to /tmp/0c4d74e4-58f1-4f8d-8c1d-8a49129a2163. Error: [Errno 28] No space left on device: ImageDownloadError: Error downloading image: Download of image id 0c4d74e4-58f1-4f8d-8c1d-8a49129a2163 failed: Unable to write image to /tmp/0c4d74e4-58f1-4f8d-8c1d-8a49129a2163. Error: [Errno 28] No space left on device"
By default, the total amount of space allocated to ramdisk is 4GB. To increase the space allocated for the ramdisk, you can update the deploy ISO image using the following workaround.
Save the deploy ISO to a file:
tux >
openstack image save --file deploy.isoIMAGE_ID
Replace IMAGE_ID with the ID of the deploy ISO
stored in Glance. The ID can be obtained using the openstack image list
.
Mount the saved ISO:
tux >
mkdir /tmp/mnttux >
sudo mount -t iso9660 -o loop deploy.iso /tmp/mnt
Since the mount directory is read-only, it is necessary to copy its content to be able to make modifications.
Copy the content of the mount directory to a custom directory:
tux >
mkdir /tmp/customtux >
cp -aRvf /tmp/mnt/* /tmp/custom/
Modify the bootloader files to increase the size of the ramdisk:
/tmp/custom/boot/x86_64/loader/isolinux.cfg /tmp/custom/EFI/BOOT/grub.cfg /tmp/custom/boot/grub2/grub.cfg
Find the openstack-ironic-image
label and modify the
ramdisk_size
parameter in the append
property. The ramdisk_size
value must be specified in Kilobytes.
label openstack-ironic-image kernel linux append initrd=initrd ramdisk_size=10485760 ramdisk_blocksize=4096 \ boot_method=vmedia showopts
Make sure that your baremetal node has the
amount of RAM that equals or exceeds the ramdisk_size
value.
Repackage the ISO using the genisoimage tool:
tux >
cd /tmp/customtux >
genisoimage -b boot/x86_64/loader/isolinux.bin -R -J -pad -joliet-long \ -iso-level 4 -A '0xaa2dab53' -no-emul-boot -boot-info-table \ -boot-load-size 4 -c boot/x86_64/boot.catalog -hide boot/x86_64/boot.catalog \ -hide-joliet boot/x86_64/boot.catalog -eltorito-alt-boot -b boot/x86_64/efi \ -no-emul-boot -joliet-long -hide glump -hide-joliet glump -o /tmp/custom_deploy.iso ./
When repackaging the ISO, make sure that you use the same label. You can
find the label file in the /tmp/custom/boot/
directory. The label begins with 0x
. For example, 0x51e568cb
.
Delete the existing deploy ISO in Glance:
tux >
openstack image delete IMAGE_ID
Create a new image with custom_deploy.iso
:
tux >
openstack image create --container-format bare \
--disk-format iso --public --file custom_deploy.iso ir-deploy-iso-ARDANA5.0
Re-deploy the Ironic node.
Partition Image Exceeds Free Space
The previous procedure applies to ISO images. It does not apply to
partition images
, although there will be a similar error
in the Ironic logs. However the resolution is different. An option must be
added to the PXE
line in the
main.yml
file to increase the /tmp
disk size with the following workaround:
Edit
/openstack/ardana/ansible/roles/ironic-common/defaults/main.yml
.
Add suse.tmpsize=4G
to
pxe_append_params
. Adjust the size of
suse.tmpsize
as needed for the partition image.
pxe_append_params : "nofb nomodeset vga=normal elevator=deadline security=apparmor crashkernel=256M console=tty0 console=ttyS0 suse.tmpsize=4G"
Update Git and run playbooks:
ardana >
git add -Aardana >
git commit -m "Add suse.tmpsize variable"ardana >
ansible-playbook -i hosts/localhost config-processor-run.ymlardana >
ansible-playbook -i hosts/localhost ready-deployment.ymlardana >
cd /var/lib/ardana/scratch/ansible/next/ardana/ansibleardana >
ansible-playbook -i hosts/verb_hosts ironic-reconfigure.yml
Re-deploy the Ironic node.
Cleaning is the process by which data is removed after a previous tenant has used the node. Cleaning requires use of ironic's agent_ drivers. It is extremely important to note that if the pxe_ drivers are utilized, no node cleaning operations will occur, and a previous tenant's data could be found on the node. The same risk of a previous tenant's data possibly can occur if cleaning is explicitly disabled as part of the installation.
By default, cleaning attempts to utilize ATA secure erase to wipe the contents of the disk. If secure erase is unavailable, the cleaning functionality built into the Ironic Python Agent falls back to an operation referred to as "shred" where random data is written over the contents of the disk, and then followed up by writing "0"s across the disk. This can be a time-consuming process.
An additional feature of cleaning is the ability to update firmware or potentially assert new hardware configuration, however, this is an advanced feature that must be built into the Ironic Python Agent image. Due to the complex nature of such operations, and the fact that no one size fits all, this requires a custom Ironic Python Agent image to be constructed with an appropriate hardware manager. For more information on hardware managers, see http://docs.openstack.org/developer/ironic-python-agent/#hardware-managers
Ironic's upstream documentation for cleaning may be found here: http://docs.openstack.org/developer/ironic/deploy/cleaning.html
Cleaning is enabled by default in ironic when installed via the Cloud Lifecycle Manager. You can verify this by examining the ironic-conductor.conf file. Look for:
[conductor] clean_nodes=true
When enabled, cleaning will be run automatically when nodes go from active
to available state or from manageable to available. To monitor what step of
cleaning the node is in, run ironic node-show
:
stack@ardana-cp1-c1-m1-mgmt:~$ ironic node-show 4e6d4273-2535-4830-a826-7f67e71783ed
+------------------------+-----------------------------------------------------------------------+
| Property | Value |
+------------------------+-----------------------------------------------------------------------+
| target_power_state | None |
| extra | {} |
| last_error | None |
| updated_at | 2016-04-15T09:33:16+00:00 |
| maintenance_reason | None |
| provision_state | cleaning |
| clean_step | {} |
| uuid | 4e6d4273-2535-4830-a826-7f67e71783ed |
| console_enabled | False |
| target_provision_state | available |
| provision_updated_at | 2016-04-15T09:33:16+00:00 |
| maintenance | False |
| inspection_started_at | None |
| inspection_finished_at | None |
| power_state | power off |
| driver | agent_ilo |
| reservation | ardana-cp1-c1-m1-mgmt |
| properties | {u'memory_mb': 4096, u'cpu_arch': u'amd64', u'local_gb': 80, |
| | u'cpus': 2, u'capabilities': u'boot_mode:uefi,boot_option:local'} |
| instance_uuid | None |
| name | None |
| driver_info | {u'ilo_deploy_iso': u'249bf095-e741-441d-bc28-0f44a9b8cd80', |
| | u'ipmi_username': u'Administrator', u'deploy_kernel': |
| | u'3a78c0a9-3d8d-4764-9300-3e9c00e167a1', u'ilo_address': |
| | u'10.1.196.113', u'ipmi_address': u'10.1.196.113', u'deploy_ramdisk': |
| | u'd02c811c-e521-4926-9f26-0c88bbd2ee6d', u'ipmi_password': u'******', |
| | u'ilo_password': u'******', u'ilo_username': u'Administrator'} |
| created_at | 2016-04-14T08:30:08+00:00 |
| driver_internal_info | {u'clean_steps': None, |
| | u'hardware_manager_version': {u'generic_hardware_manager': u'1.0'}, |
| | u'is_whole_disk_image': True, u'agent_erase_devices_iterations': 1, |
| | u'agent_url': u'http://192.168.246.245:9999', |
| | u'agent_last_heartbeat': 1460633166} |
| chassis_uuid | |
| instance_info | {} |
+------------------------+-----------------------------------------------------------------------+
The status will be in the driver_internal_info
field. You
will also be able to see the clean_steps
list there.
If an error occurs during the cleaning process, the node will enter the clean failed state so that it is not deployed. The node remains powered on for debugging purposes. The node can be moved to the manageable state to attempt a fix using the following command:
ironic node-set-provision-state <node id> manage
Once you have identified and fixed the issue, you can return the node to available state by executing the following commands:
ironic node-set-maintenance <node id> false ironic node-set-provision-state <node id> provide
This will retry the cleaning steps and set the node to available state upon their successful completion.
To disable node cleaning, edit
~/openstack/my_cloud/definition/data/ironic/ironic_config.yml
and set enable_node_cleaning
to false
.
Commit your changes:
cd ~/openstack/ardana/ansible git add -A git commit -m "disable node cleaning"
Deploy these changes by re-running the configuration processor and reconfigure the ironic installation:
cd ~/openstack/ardana/ansible ansible-playbook -i hosts/localhost config-processor-run.yml ansible-playbook -i hosts/localhost ready-deployment.yml cd ~/scratch/ansible/next/ardana/ansible ansible-playbook -i hosts/verb_hosts ironic-reconfigure.yml
Edit the file
~/openstack/my_cloud/definition/data/ironic/ironicconfig.yml
and set the value
enable_oneview: true
This will enable the HPE OneView driver for Ironic in HPE Helion OpenStack.
manage_url: https://<Onview appliance URL> oneview_username: "<Appliance username>" oneview_encrypted_password: "<Encrypted password>" oneview_allow_insecure_connections: <true/false> tls_cacert_file: <CA certificate for connection>
Encryption can be applied using ardanaencrypt.py
or using
openssl
. On the Cloud Lifecycle Manager node, export the key
used for encryption as environment variable:
export ARDANA_USER_PASSWORD_ENCRYPT_KEY="ENCRYPTION_KEY"
And then execute the following commands:
cd ~/openstack/ardana/ansible python ardanaencrypt.py
Enter password to be encrypted when prompted. The script uses the key that
was exported in the ARDANA_USER_PASSWORD_ENCRYPT_KEY
to do
the encryption.
For more information, see Book “Security Guide”, Chapter 9 “Encryption of Passwords and Sensitive Data”.
Before running the site.yml
playbook, export the key used
for encryption as environment variable:
export ARDANA_USER_PASSWORD_ENCRYPT_KEY="ENCRYPTION_KEY"
The decryption of the password is then automatically handled in ironic-ansible playbooks.
ironic node-create -d agent_pxe_oneview
Update node driver-info:
ironic node-update $NODE_UUID add driver_info/server_hardware_uri=$SH_URI
ironic node-update $NODE_UUID add \ properties/capabilities=server_hardware_type_uri:$SHT_URI,\ enclosure_group_uri:$EG_URI,server_profile_template_uri=$SPT_URI
ironic port-create -n $NODE_UUID -a $MAC_ADDRESS
Create Node:
ironic node-create -n ovbay7 -d agent_pxe_oneview
Update driver info:
ironic node-update $ID add driver_info/server_hardware_uri="/rest/server-hardware/3037...464B" \ driver_info/deploy_kernel="$KERNELDISK" driver_info/deploy_ramdisk="$RAMDISK"
Update node properties:
ironic node-update $ID add properties/local_gb=10 ironic node-update $ID add properties/cpus=24 properties/memory_mb=262144 \ properties/cpu_arch=x86_64
ironic node-update \ $ID add properties/capabilities=server_hardware_type_uri:'/rest/server-hardware-types/B31...F69E',\ enclosure_group_uri:'/rest/enclosure-groups/80efe...b79fa',\ server_profile_template_uri:'/rest/server-profile-templates/faafc3c0-6c81-47ca-a407-f67d11262da5'
GET login session auth id:
curl -k https://ONEVIEW_MANAGER_URL/rest/login-sessions \ -H "content-type:application/json" \ -X POST \ -d '{"userName":"USER_NAME", "password":"PASSWORD"}'
Get the complete node details in JSON format:
curl -k "https://ONEVIEW_MANAGER_URL;/rest/server-hardware/30373237-3132-4753-4835-32325652464B" -H "content-type:application/json" -H "Auth:<auth_session_id>"| python -m json.tool
ironic-oneview-cli
is already installed in
ironicclient
venv with a symbolic link to it. To generate
an rc
file for the HPE OneView CLI, follow these steps:
Run the following commands:
source ~/service.osrc glance image list
Note the deploy-kernel
and
deploy-ramdisk
UUID and then run the following command
to generate the rc
file:
ironic-oneview genrc
You will be prompted to enter:
HPE OneView Manager URL
Username
deploy-kernel
deploy-ramdisk
allow_insecure_connection
cacert file
The ironic-oneview.rc
file will be generated in the
current directory, by default. It is possible to specify a different
location.
Source the generated file:
source ironic-oneview.rc
Now enter the password of the HPE OneView appliance.
You can now use the CLI for node and flavor creation as follows:
ironic-oneview node-create ironic-oneview flavor-create
Node Creation:
Check the raid capabilities of the driver:
ironic --ironic-api-version 1.15 driver-raid-logical-disk-properties pxe_ilo
This will generate output similar to the following:
+----------------------+-------------------------------------------------------------------------+ | Property | Description | +----------------------+-------------------------------------------------------------------------+ | controller | Controller to use for this logical disk. If not specified, the | | | driver will choose a suitable RAID controller on the bare metal node. | | | Optional. | | disk_type | The type of disk preferred. Valid values are 'hdd' and 'ssd'. If this | | | is not specified, disk type will not be a selection criterion for | | | choosing backing physical disks. Optional. | | interface_type | The interface type of disk. Valid values are 'sata', 'scsi' and 'sas'. | | | If this is not specified, interface type will not be a selection | | | criterion for choosing backing physical disks. Optional. | | is_root_volume | Specifies whether this disk is a root volume. By default, this is False.| | | Optional. | | #_of_physical_disks | Number of physical disks to use for this logical disk. By default, the | | | driver uses the minimum number of disks required for that RAID level. | | | Optional. | | physical_disks | The physical disks to use for this logical disk. If not specified, the | | | driver will choose suitable physical disks to use. Optional. | | raid_level | RAID level for the logical disk. Valid values are '0', '1', '2', '5', | | | '6', '1+0', '5+0' and '6+0'. Required. | | share_physical_disks | Specifies whether other logical disks can share physical disks with this| | | logical disk. By default, this is False. Optional. | | size_gb | Size in GiB (Integer) for the logical disk. Use 'MAX' as size_gb if | | | this logical disk is supposed to use the rest of | | | the space available. Required. | | volume_name | Name of the volume to be created. If this is not specified, it will be | | | auto-generated. Optional. | +----------------------+-------------------------------------------------------------------------+
Node State will be Available
ironic node-create -d pxe_ilo -i ilo_address=<ip_address> \ -i ilo_username=<username> -i ilo_password=<password> \ -i ilo_deploy_iso=<iso_id> -i deploy_kernel=<kernel_id> \ -i deploy_ramdisk=<ramdisk_id> -p cpus=2 -p memory_mb=4096 \ -p local_gb=80 -p cpu_arch=amd64 \ -p capabilities="boot_option:local,boot_mode:bios"
ironic port-create -a <port> -n <node-uuid>
Apply the target raid configuration on the node:
See the OpenStack documentation for RAID configuration at http://docs.openstack.org/developer/ironic/deploy/raid.html.
Set the target RAID configuration by editing the file
raid_conf.json
and setting the appropriate values, for
example:
{ "logical_disks": [ {"size_gb": 5, "raid_level": "0", "is_root_volume": true} ] }
and then run the following command:
ironic --ironic-api-version 1.15 node-set-target-raid-config <node-uuid> raid_conf.json
The output produced should be similar to the following:
+-----------------------+-------------------------------------------------------------------------+ | Property | Value | +-----------------------+-------------------------------------------------------------------------+ | chassis_uuid | | | clean_step | {} | | console_enabled | False | | created_at | 2016-06-14T14:58:07+00:00 | | driver | pxe_ilo | | driver_info | {u'ilo_deploy_iso': u'd43e589a-07db-4fce-a06e-98e2f38340b4', | | | u'deploy_kernel': u'915c5c74-1ceb-4f78-bdb4-8547a90ac9c0', | | | u'ilo_address': u'10.1.196.116', u'deploy_ramdisk': | | | u'154e7024-bf18-4ad2-95b0-726c09ce417a', u'ilo_password': u'******', | | | u'ilo_username': u'Administrator'} | | driver_internal_info | {u'agent_cached_clean_steps_refreshed': u'2016-06-15 07:16:08.264091', | | | u'agent_cached_clean_steps': {u'raid': [{u'interface': u'raid', | | | u'priority': 0, u'step': u'delete_configuration'}, {u'interface': | | | u'raid', u'priority': 0, u'step': u'create_configuration'}], u'deploy': | | | [{u'priority': 10, u'interface': u'deploy', u'reboot_requested': False, | | | u'abortable': True, u'step': u'erase_devices'}]}, u'clean_steps': None, | | | u'hardware_manager_version': {u'generic_hardware_manager': u'3'}, | | | u'agent_erase_devices_iterations': 1, u'agent_url': | | | u'http://192.168.245.143:9999', u'agent_last_heartbeat': 1465974974} | | extra | {} | | inspection_finished_at| None | | inspection_started_at | None | | instance_info | {u'deploy_key': u'XXN2ON0V9ER429MECETJMUG5YHTKOQOZ'} | | instance_uuid | None | | last_error | None | | maintenance | False | | maintenance_reason | None | | name | None | | power_state | power off | | properties | {u'cpu_arch': u'amd64', u'root_device': {u'wwn': u'0x600508b1001ce286'},| | | u'cpus': 2, u'capabilities': | | | u'boot_mode:bios,raid_level:6,boot_option:local', u'memory_mb': 4096, | | | u'local_gb': 4} | | provision_state | available | | provision_updated_at | 2016-06-15T07:16:27+00:00 | | reservation | padawan-ironic-cp1-c1-m2-mgmt | | target_power_state | power off | | target_provision_state| None | | target_raid_config | {u'logical_disks': [{u'size_gb': 5, u'raid_level': u'6', | | | u'is_root_volume': True}]} | | updated_at | 2016-06-15T07:44:22+00:00 | | uuid | 22ab9f85-71a1-4748-8d6b-f6411558127e | +-----------------------+-------------------------------------------------------------------------+
Now set the state of the node to manageable:
ironic --ironic-api-version latest node-set-provision-state <node-uuid> manage
Manual cleaning steps:
Manual cleaning is enabled by default in production - the following are the steps to enable cleaning if the manual cleaning has been disabled.
Provide cleaning_network_uuid
in
ironic-conductor.conf
Edit the file
~/openstack/my_cloud/definition/data/ironic/ironic_config.yml
and set enable_node_cleaning
to
true
.
Then run the following set of commands:
cd ~/openstack/ardana/ansible git add -A git commit -m "enabling node cleaning" cd ~/openstack/ardana/ansible ansible-playbook -i hosts/localhost config-processor-run.yml ansible-playbook -i hosts/localhost ready-deployment.yml cd ~/scratch/ansible/next/ardana/ansible ansible-playbook -i hosts/verb_hosts ironic-reconfigure.yml
After performing these steps, the state of the node will become Cleaning.
Run the following command:
ironic --ironic-api-version latest node-set-provision-state 2123254e-8b31...aa6fd \ clean --clean-steps '[{ "interface": "raid","step": "delete_configuration"}, \ { "interface": "raid" ,"step": "create_configuration"}]'
Node-information after a Manual cleaning:
+-----------------------+-------------------------------------------------------------------------+ | Property | Value | +-----------------------+-------------------------------------------------------------------------+ | chassis_uuid | | | clean_step | {} | | console_enabled | False | | created_at | 2016-06-14T14:58:07+00:00 | | driver | pxe_ilo | | driver_info | {u'ilo_deploy_iso': u'd43e589a-07db-4fce-a06e-98e2f38340b4', | | | u'deploy_kernel': u'915c5c74-1ceb-4f78-bdb4-8547a90ac9c0', | | | u'ilo_address': u'10.1.196.116', u'deploy_ramdisk': | | | u'154e7024-bf18-4ad2-95b0-726c09ce417a', u'ilo_password': u'******', | | | u'ilo_username': u'Administrator'} | | driver_internal_info | {u'agent_cached_clean_steps_refreshed': u'2016-06-15 07:16:08.264091', | | | u'agent_cached_clean_steps': {u'raid': [{u'interface': u'raid', | | | u'priority': 0, u'step': u'delete_configuration'}, {u'interface': | | | u'raid', u'priority': 0, u'step': u'create_configuration'}], u'deploy': | | | [{u'priority': 10, u'interface': u'deploy', u'reboot_requested': False, | | | u'abortable': True, u'step': u'erase_devices'}]}, u'clean_steps': None, | | | u'hardware_manager_version': {u'generic_hardware_manager': u'3'}, | | | u'agent_erase_devices_iterations': 1, u'agent_url': | | | u'http://192.168.245.143:9999', u'agent_last_heartbeat': 1465974974} | | extra | {} | | inspection_finished_at| None | | inspection_started_at | None | | instance_info | {u'deploy_key': u'XXN2ON0V9ER429MECETJMUG5YHTKOQOZ'} | | instance_uuid | None | | last_error | None | | maintenance | False | | maintenance_reason | None | | name | None | | power_state | power off | | properties | {u'cpu_arch': u'amd64', u'root_device': {u'wwn': u'0x600508b1001ce286'},| | | u'cpus': 2, u'capabilities': | | | u'boot_mode:bios,raid_level:6,boot_option:local', u'memory_mb': 4096, | | | u'local_gb': 4} | | provision_state | manageable | | provision_updated_at | 2016-06-15T07:16:27+00:00 | | raid_config | {u'last_updated': u'2016-06-15 07:16:14.584014', u'physical_disks': | | | [{u'status': u'ready', u'size_gb': 1024, u'interface_type': u'sata', | | | u'firmware': u'HPGC', u'controller': u'Smart Array P440ar in Slot 0 | | | (Embedded)', u'model': u'ATA MM1000GBKAL', u'disk_type': u'hdd', | | | u'id': u'1I:3:3'}, {u'status': u'ready', u'size_gb': 1024, | | | u'interface_type': u'sata', u'firmware': u'HPGC', u'controller': u'Smart| | | Array P440ar in Slot 0 (Embedded)', u'model': u'ATA MM1000GBKAL', | | | u'disk_type': u'hdd', u'id': u'1I:3:1'}, {u'status': u'active', | | | u'size_gb': 1024, u'interface_type': u'sata', u'firmware': u'HPGC', | | | u'controller': u'Smart Array P440ar in Slot 0 (Embedded)', u'model': | | | u'ATA MM1000GBKAL', u'disk_type': u'hdd', u'id': u'1I:3:2'}, | | | {u'status': u'active', u'size_gb': 1024, u'interface_type': u'sata', | | | u'firmware': u'HPGC', u'controller': u'Smart Array P440ar in Slot 0 | | | (Embedded)', u'model': u'ATA MM1000GBKAL', u'disk_type': u'hdd', | | | u'id': u'2I:3:6'}, {u'status': u'active', u'size_gb': 1024, | | | u'interface_type': u'sata', u'firmware': u'HPGC', u'controller': u'Smart| | | Array P440ar in Slot 0 (Embedded)', u'model': u'ATA MM1000GBKAL', | | | u'disk_type': u'hdd', u'id': u'2I:3:5'}, {u'status': u'active', | | | u'size_gb': 1024, u'interface_type': u'sata', u'firmware': u'HPGC', | | | u'controller': u'Smart Array P440ar in Slot 0 (Embedded)', u'model': | | | u'ATA MM1000GBKAL', u'disk_type': u'hdd', u'id': u'1I:3:4'}], | | | u'logical_disks': [{u'size_gb': 4, u'physical_disks': [u'1I:3:2', | | | u'2I:3:6', u'2I:3:5', u'1I:3:4'], u'raid_level': u'6', | | | u'is_root_volume': True, u'root_device_hint': {u'wwn': | | | u'0x600508b1001ce286'}, u'controller': u'Smart Array P440ar in Slot 0 | | | (Embedded)', u'volume_name': u'015E795CPDNLH0BRH8N406AAB7'}]} | | reservation | padawan-ironic-cp1-c1-m2-mgmt | | target_power_state | power off | | target_provision_state| None | | target_raid_config | {u'logical_disks': [{u'size_gb': 5, u'raid_level': u'6', | | | u'is_root_volume': True}]} | | updated_at | 2016-06-15T07:44:22+00:00 | | uuid | 22ab9f85-71a1-4748-8d6b-f6411558127e | +-----------------------+-------------------------------------------------------------------------+
After the manual cleaning, run the following command to change the state of a node to available:
ironic --ironic-api-version latest node-set-provision-state <node-uuid> \ provide
Audit middleware supports delivery of CADF audit events via Oslo messaging
notifier capability. Based on notification_driver
configuration, audit events can be routed to messaging infrastructure
(notification_driver = messagingv2
) or can be routed to a
log file (notification_driver = log
).
Audit middleware creates two events per REST API interaction. The first event has information extracted from request data and the second one contains information on the request outcome (response).
You can enable audit logging for Ironic by changing the configuration in the
input model. Edit the file
~/openstack/my_cloud/definition/cloudConfig.yml
and in the
audit-settings
section, change the
default
value to enabled
. The
ironic-ansible playbooks will now enable audit support for Ironic.
API audit events will be logged in the corresponding audit directory, for
example, /var/audit/ironic/ironic-api-audit.log
. An audit
event will be logged in the log file for every request and response for an
API call.
The following output is an example of an audit event for an ironic
node-list
command:
{ "event_type":"audit.http.request", "timestamp":"2016-06-15 06:04:30.904397", "payload":{ "typeURI":"http://schemas.dmtf.org/cloud/audit/1.0/event", "eventTime":"2016-06-15T06:04:30.903071+0000", "target":{ "id":"ironic", "typeURI":"unknown", "addresses":[ { "url":"http://{ironic_admin_host}:6385", "name":"admin" }, { "url":"http://{ironic_internal_host}:6385", "name":"private" }, { "url":"http://{ironic_public_host}:6385", "name":"public" } ], "name":"ironic" }, "observer":{ "id":"target" }, "tags":[ "correlation_id?value=685f1abb-620e-5d5d-b74a-b4135fb32373" ], "eventType":"activity", "initiator":{ "typeURI":"service/security/account/user", "name":"admin", "credential":{ "token":"***", "identity_status":"Confirmed" }, "host":{ "agent":"python-ironicclient", "address":"10.1.200.129" }, "project_id":"d8f52dd7d9e1475dbbf3ba47a4a83313", "id":"8c1a948bad3948929aa5d5b50627a174" }, "action":"read", "outcome":"pending", "id":"061b7aa7-5879-5225-a331-c002cf23cb6c", "requestPath":"/v1/nodes/?associated=True" }, "priority":"INFO", "publisher_id":"ironic-api", "message_id":"2f61ebaa-2d3e-4023-afba-f9fca6f21fc2" }