10 Managing Networking #
Information about managing and configuring the Networking service.
10.1 SUSE OpenStack Cloud Firewall #
Firewall as a Service (FWaaS) provides the ability to assign network-level, port security for all traffic entering an existing tenant network. More information on this service can be found in the public OpenStack documentation located at http://specs.openstack.org/openstack/neutron-specs/specs/api/firewall_as_a_service__fwaas_.html. The following documentation provides command-line interface example instructions for configuring and testing a SUSE OpenStack Cloud firewall. FWaaS can also be configured and managed by the horizon web interface.
With SUSE OpenStack Cloud, FWaaS is implemented directly in the L3 agent (neutron-l3-agent). However if VPNaaS is enabled, FWaaS is implemented in the VPNaaS agent (neutron-vpn-agent). Because FWaaS does not use a separate agent process or start a specific service, there currently are no monasca alarms for it.
If DVR is enabled, the firewall service currently does not filter traffic between OpenStack private networks, also known as east-west traffic and will only filter traffic from external networks, also known as north-south traffic.
The L3 agent must be restarted on each compute node hosting a DVR router when removing the FWaaS or adding a new FWaaS. This condition only applies when updating existing instances connected to DVR routers. For more information, see the upstream bug.
10.1.1 Overview of the SUSE OpenStack Cloud Firewall configuration #
The following instructions provide information about how to identify and modify the overall SUSE OpenStack Cloud firewall that is configured in front of the control services. This firewall is administered only by a cloud admin and is not available for tenant use for private network firewall services.
During the installation process, the configuration processor will
automatically generate "allow" firewall rules for each server based on the
services deployed and block all other ports. These are populated in
~/openstack/my_cloud/info/firewall_info.yml, which includes
a list of all the ports by network, including the addresses on which the
ports will be opened. This is described in more detail in
Section 5.2.10.5, “Firewall Configuration”.
The firewall_rules.yml file in the input model allows you
to define additional rules for each network group. You can read more about
this in Section 6.15, “Firewall Rules”.
The purpose of this document is to show you how to make post-installation changes to the firewall rules if the need arises.
This process is not to be confused with Firewall-as-a-Service, which is a separate service that enables the ability for SUSE OpenStack Cloud tenants to create north-south, network-level firewalls to provide stateful protection to all instances in a private, tenant network. This service is optional and is tenant-configured.
10.1.2 SUSE OpenStack Cloud 9 FWaaS Configuration #
Check for an enabled firewall.
You should check to determine if the firewall is enabled. The output of the openstack extension list should contain a firewall entry.
openstack extension list
Assuming the external network is already created by the admin, this command will show the external network.
openstack network list
Create required assets.
Before creating firewalls, you will need to create a network, subnet, router, security group rules, start an instance and assign it a floating IP address.
Create the network, subnet and router.
openstack network create private openstack subnet create --name sub private 10.0.0.0/24 --gateway 10.0.0.1 openstack router create router openstack router add subnet router sub openstack router set router ext-net
Create security group rules. Security group rules filter traffic at VM level.
openstack security group rule create default --protocol icmp openstack security group rule create default --protocol tcp --port-range-min 22 --port-range-max 22 openstack security group rule create default --protocol tcp --port-range-min 80 --port-range-max 80
Boot a VM.
NET=$(openstack network list | awk '/private/ {print $2}') openstack server create --flavor 1 --image <image> --nic net-id=$NET vm1 --pollVerify if the instance is ACTIVE and is assigned an IP address.
openstack server list
Get the port id of the vm1 instance.
fixedip=$(openstack server list | awk '/vm1/ {print $12}' | awk -F '=' '{print $2}' | awk -F ',' '{print $1}') vmportuuid=$(openstack port list | grep $fixedip | awk '{print $2}')Create and associate a floating IP address to the vm1 instance.
openstack floating ip create ext-net --port-id $vmportuuid
Verify if the floating IP is assigned to the instance. The following command should show an assigned floating IP address from the external network range.
openstack server show vm1
Verify if the instance is reachable from the external network. SSH into the instance from a node in (or has route to) the external network.
ssh cirros@FIP-VM1 password: <password>
Create and attach the firewall.
By default, an internal "drop all" rule is enabled in IP tables if none of the defined rules match the real-time data packets.
Create new firewall rules using
firewall-rule-createcommand and providing the protocol, action (allow, deny, reject) and name for the new rule.Firewall actions provide rules in which data traffic can be handled. An allow rule will allow traffic to pass through the firewall, deny will stop and prevent data traffic from passing through the firewall and reject will reject the data traffic and return a destination-unreachable response. Using reject will speed up failure detection time dramatically for legitimate users, since they will not be required to wait for retransmission timeouts or submit retries. Some customers should stick with deny where prevention of port scanners and similar methods may be attempted by hostile attackers. Using deny will drop all of the packets, making it more difficult for malicious intent. The firewall action, deny is the default behavior.
The example below demonstrates how to allow icmp and ssh while denying access to http. See the
OpenStackClientcommand-line reference at https://docs.openstack.org/python-openstackclient/rocky/ on additional options such as source IP, destination IP, source port and destination port.NoteYou can create a firewall rule with an identical name and each instance will have a unique id associated with the created rule, however for clarity purposes this is not recommended.
neutron firewall-rule-create --protocol icmp --action allow --name allow-icmp neutron firewall-rule-create --protocol tcp --destination-port 80 --action deny --name deny-http neutron firewall-rule-create --protocol tcp --destination-port 22 --action allow --name allow-ssh
Once the rules are created, create the firewall policy by using the
firewall-policy-createcommand with the--firewall-rulesoption and rules to include in quotes, followed by the name of the new policy. The order of the rules is important.neutron firewall-policy-create --firewall-rules "allow-icmp deny-http allow-ssh" policy-fw
Finish the firewall creation by using the
firewall-createcommand, the policy name and the new name you want to give to your new firewall.neutron firewall-create policy-fw --name user-fw
You can view the details of your new firewall by using the
firewall-showcommand and the name of your firewall. This will verify that the status of the firewall is ACTIVE.neutron firewall-show user-fw
Verify the FWaaS is functional.
Since allow-icmp firewall rule is set you can ping the floating IP address of the instance from the external network.
ping <FIP-VM1>
Similarly, you can connect via ssh to the instance due to the allow-ssh firewall rule.
ssh cirros@<FIP-VM1> password: <password>
Run a web server on vm1 instance that listens over port 80, accepts requests and sends a WELCOME response.
$ vi webserv.sh #!/bin/bash MYIP=$(/sbin/ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print $1}'); while true; do echo -e "HTTP/1.0 200 OK Welcome to $MYIP" | sudo nc -l -p 80 done # Give it Exec rights $ chmod 755 webserv.sh # Execute the script $ ./webserv.shYou should expect to see curl fail over port 80 because of the deny-http firewall rule. If curl succeeds, the firewall is not blocking incoming http requests.
curl -vvv <FIP-VM1>
When using reference implementation, new networks, FIPs and routers created after the Firewall creation will not be automatically updated with firewall rules. Thus, execute the firewall-update command by passing the current and new router Ids such that the rules are reconfigured across all the routers (both current and new).
For example if router-1 is created before and router-2 is created after the firewall creation
$ neutron firewall-update —router <router-1-id> —router <router-2-id> <firewall-name>
10.1.3 Making Changes to the Firewall Rules #
Log in to your Cloud Lifecycle Manager.
Edit your
~/openstack/my_cloud/definition/data/firewall_rules.ymlfile and add the lines necessary to allow the port(s) needed through the firewall.In this example we are going to open up port range 5900-5905 to allow VNC traffic through the firewall:
- name: VNC network-groups: - MANAGEMENT rules: - type: allow remote-ip-prefix: 0.0.0.0/0 port-range-min: 5900 port-range-max: 5905 protocol: tcpNoteThe example above shows a
remote-ip-prefixof0.0.0.0/0which opens the ports up to all IP ranges. To be more secure you can specify your local IP address CIDR you will be running the VNC connect from.Commit those changes to your local git:
ardana >cd ~/openstack/ardana/ansibleardana >git add -Aardana >git commit -m "firewall rule update"Run the configuration processor:
ardana >cd ~/openstack/ardana/ansibleardana >ansible-playbook -i hosts/localhost config-processor-run.ymlCreate the deployment directory structure:
ardana >cd ~/openstack/ardana/ansibleardana >ansible-playbook -i hosts/localhost ready-deployment.ymlChange to the deployment directory and run the
osconfig-iptables-deploy.ymlplaybook to update your iptable rules to allow VNC:ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts osconfig-iptables-deploy.yml
You can repeat these steps as needed to add, remove, or edit any of these firewall rules.
10.1.4 More Information #
Firewalls are based in IPtable settings.
Each firewall that is created is known as an instance.
A firewall instance can be deployed on selected project routers. If no specific project router is selected, a firewall instance is automatically applied to all project routers.
Only 1 firewall instance can be applied to a project router.
Only 1 firewall policy can be applied to a firewall instance.
Multiple firewall rules can be added and applied to a firewall policy.
Firewall rules can be shared across different projects via the Share API flag.
Firewall rules supersede the Security Group rules that are applied at the Instance level for all traffic entering or leaving a private, project network.
For more information on the command-line interface (CLI) and firewalls, see the OpenStack networking command-line client reference: https://docs.openstack.org/python-openstackclient/rocky/
10.2 Using VPN as a Service (VPNaaS) #
SUSE OpenStack Cloud 9 VPNaaS Configuration
This document describes the configuration process and requirements for the SUSE OpenStack Cloud 9 Virtual Private Network (VPN) as a Service (VPNaaS) module.
10.2.1 Prerequisites #
SUSE OpenStack Cloud must be installed.
Before setting up VPNaaS, you will need to have created an external network and a subnet with access to the internet. Information on how to create the external network and subnet can be found in Section 10.2.4, “More Information”.
You should assume 172.16.0.0/16 as the ext-net CIDR in this document.
10.2.2 Considerations #
Using the neutron plugin-based VPNaaS causes additional processes to be run on the Network Service Nodes. One of these processes, the ipsec charon process from StrongSwan, runs as root and listens on an external network. A vulnerability in that process can lead to remote root compromise of the Network Service Nodes. If this is a concern customers should consider using a VPN solution other than the neutron plugin-based VPNaaS and/or deploying additional protection mechanisms.
10.2.3 Configuration #
Setup Networks You can setup VPN as a
Service (VPNaaS) by first creating networks, subnets and routers using the
neutron command line. The VPNaaS module enables the
ability to extend access between private networks across two different
SUSE OpenStack Cloud clouds or between a SUSE OpenStack Cloud cloud and a non-cloud network. VPNaaS
is based on the open source software application called StrongSwan.
StrongSwan (more information available
at http://www.strongswan.org/)
is an IPsec implementation and provides basic VPN gateway functionality.
You can execute the included commands from any shell with access to the service APIs. In the included examples, the commands are executed from the lifecycle manager, however you could execute the commands from the controller node or any other shell with aforementioned service API access.
The use of floating IP's is not possible with the current version of VPNaaS when DVR is enabled. Ensure that no floating IP is associated to instances that will be using VPNaaS when using a DVR router. Floating IP associated to instances are ok when using CVR router.
From the Cloud Lifecycle Manager, create first private network, subnet and router assuming that ext-net is created by admin.
openstack network create privateA openstack subnet create --name subA privateA 10.1.0.0/24 --gateway 10.1.0.1 openstack router create router1 openstack router add subnet router1 subA openstack router set router1 ext-net
Create second private network, subnet and router.
openstack network create privateB openstack subnet create --name subB privateB 10.2.0.0/24 --gateway 10.2.0.1 openstack router create router2 openstack router add subnet router2 subB openstack router set router2 ext-net
From the Cloud Lifecycle Manager run the following to start the virtual machines. Begin with adding secgroup rules for SSH and ICMP.
openstack security group rule create default --protocol icmp openstack security group rule create default --protocol tcp --port-range-min 22 --port-range-max 22
Start the virtual machine in the privateA subnet. Using nova images-list, use the image id to boot image instead of the image name. After executing this step, it is recommended that you wait approximately 10 seconds to allow the virtual machine to become active.
NETA=$(openstack network list | awk '/privateA/ {print $2}') openstack server create --flavor 1 --image <id> --nic net-id=$NETA vm1Start the virtual machine in the privateB subnet.
NETB=$(openstack network list | awk '/privateB/ {print $2}') openstack server create --flavor 1 --image <id> --nic net-id=$NETB vm2Verify private IP's are allocated to the respective vms. Take note of IP's for later use.
openstack server show vm1 openstack server show vm2
You can set up the VPN by executing the below commands from the lifecycle manager or any shell with access to the service APIs. Begin with creating the policies with
vpn-ikepolicy-createandvpn-ipsecpolicy-create.neutron vpn-ikepolicy-create ikepolicy neutron vpn-ipsecpolicy-create ipsecpolicy
Create the VPN service at router1.
neutron vpn-service-create --name myvpnA --description "My vpn service" router1 subA
Wait at least 5 seconds and then run
ipsec-site-connection-createto create a ipsec-site connection. Note that--peer-addressis the assign ext-net IP from router2 and--peer-cidris subB cidr.neutron ipsec-site-connection-create --name vpnconnection1 --vpnservice-id myvpnA \ --ikepolicy-id ikepolicy --ipsecpolicy-id ipsecpolicy --peer-address 172.16.0.3 \ --peer-id 172.16.0.3 --peer-cidr 10.2.0.0/24 --psk secret
Create the VPN service at router2.
neutron vpn-service-create --name myvpnB --description "My vpn serviceB" router2 subB
Wait at least 5 seconds and then run
ipsec-site-connection-createto create a ipsec-site connection. Note that--peer-addressis the assigned ext-net IP from router1 and--peer-cidris subA cidr.neutron ipsec-site-connection-create --name vpnconnection2 --vpnservice-id myvpnB \ --ikepolicy-id ikepolicy --ipsecpolicy-id ipsecpolicy --peer-address 172.16.0.2 \ --peer-id 172.16.0.2 --peer-cidr 10.1.0.0/24 --psk secret
On the Cloud Lifecycle Manager, run the
ipsec-site-connection-listcommand to see the active connections. Be sure to check that the vpn_services are ACTIVE. You can check this by runningvpn-service-listand then checking ipsec-site-connections status. You should expect that the time for both vpn-services and ipsec-site-connections to become ACTIVE could take as long as 1 to 3 minutes.neutron ipsec-site-connection-list +--------------------------------------+----------------+--------------+---------------+------------+-----------+--------+ | id | name | peer_address | peer_cidrs | route_mode | auth_mode | status | +--------------------------------------+----------------+--------------+---------------+------------+-----------+--------+ | 1e8763e3-fc6a-444c-a00e-426a4e5b737c | vpnconnection2 | 172.16.0.2 | "10.1.0.0/24" | static | psk | ACTIVE | | 4a97118e-6d1d-4d8c-b449-b63b41e1eb23 | vpnconnection1 | 172.16.0.3 | "10.2.0.0/24" | static | psk | ACTIVE | +--------------------------------------+----------------+--------------+---------------+------------+-----------+--------+
Verify VPN In the case of non-admin users, you can verify the VPN connection by pinging the virtual machines.
Check the VPN connections.
Notevm1-ip and vm2-ip denotes private IP's for vm1 and vm2 respectively. The private IPs are obtained, as described in of Step 4. If you are unable to SSH to the private network due to a lack of direct access, the VM console can be accessed through horizon.
ssh cirros@vm1-ip password: <password> # ping the private IP address of vm2 ping ###.###.###.###
In another terminal.
ssh cirros@vm2-ip password: <password> # ping the private IP address of vm1 ping ###.###.###.###
You should see ping responses from both virtual machines.
As the admin user, you should check to make sure that a route exists between the router gateways. Once the gateways have been checked, packet encryption can be verified by using traffic analyzer (tcpdump) by tapping on the respective namespace (qrouter-* in case of non-DVR and snat-* in case of DVR) and tapping the right interface (qg-***).
When using DVR namespaces, all the occurrences of qrouter-xxxxxx in the following commands should be replaced with respective snat-xxxxxx.
Check the if the route exists between two router gateways. You can get the right qrouter namespace id by executing sudo ip netns. Once you have the qrouter namespace id, you can get the interface by executing sudo ip netns qrouter-xxxxxxxx ip addr and from the result the interface can be found.
sudo ip netns sudo ip netns exec qrouter-<router1 UUID> ping <router2 gateway> sudo ip netns exec qrouter-<router2 UUID> ping <router1 gateway>
Initiate a tcpdump on the interface.
sudo ip netns exec qrouter-xxxxxxxx tcpdump -i qg-xxxxxx
Check the VPN connection.
ssh cirros@vm1-ip password: <password> # ping the private IP address of vm2 ping ###.###.###.###
Repeat for other namespace and right tap interface.
sudo ip netns exec qrouter-xxxxxxxx tcpdump -i qg-xxxxxx
In another terminal.
ssh cirros@vm2-ip password: <password> # ping the private IP address of vm1 ping ###.###.###.###
You will find encrypted packets containing ‘ESP’ in the tcpdump trace.
10.2.4 More Information #
VPNaaS currently only supports Pre-shared Keys (PSK) security between VPN gateways. A different VPN gateway solution should be considered if stronger, certificate-based security is required.
For more information on the neutron command-line interface (CLI) and VPN as a Service (VPNaaS), see the OpenStack networking command-line client reference: https://docs.openstack.org/python-openstackclient/rocky/
For information on how to create an external network and subnet, see the OpenStack manual: http://docs.openstack.org/user-guide/dashboard_create_networks.html
10.3 DNS Service Overview #
SUSE OpenStack Cloud DNS service provides multi-tenant Domain Name Service with REST API management for domain and records.
The DNS Service is not intended to be used as an internal or private DNS service. The name records in DNSaaS should be treated as public information that anyone could query. There are controls to prevent tenants from creating records for domains they do not own. TSIG provides a Transaction SIG nature to ensure integrity during zone transfer to other DNS servers.
10.3.1 For More Information #
For more information about designate REST APIs, see the OpenStack REST API Documentation at http://docs.openstack.org/developer/designate/rest.html.
For a glossary of terms for designate, see the OpenStack glossary at http://docs.openstack.org/developer/designate/glossary.html.
10.3.2 designate Initial Configuration #
After the SUSE OpenStack Cloud installation has been completed, designate requires initial configuration to operate.
10.3.2.1 Identifying Name Server Public IPs #
Depending on the back-end, the method used to identify the name servers' public IPs will differ.
10.3.2.1.1 InfoBlox #
InfoBlox will act as your public name servers, consult the InfoBlox management UI to identify the IPs.
10.3.2.1.2 BIND Back-end #
You can find the name server IPs in /etc/hosts by
looking for the ext-api addresses, which are the
addresses of the controllers. For example:
192.168.10.1 example-cp1-c1-m1-extapi 192.168.10.2 example-cp1-c1-m2-extapi 192.168.10.3 example-cp1-c1-m3-extapi
10.3.2.1.3 Creating Name Server A Records #
Each name server requires a public name, for example
ns1.example.com., to which designate-managed domains will
be delegated. There are two common locations where these may be registered,
either within a zone hosted on designate itself, or within a zone hosted on a
external DNS service.
If you are using an externally managed zone for these names:
For each name server public IP, create the necessary A records in the external system.
If you are using a designate-managed zone for these names:
Create the zone in designate which will contain the records:
ardana >openstack zone create --email hostmaster@example.com example.com. +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | action | CREATE | | created_at | 2016-03-09T13:16:41.000000 | | description | None | | email | hostmaster@example.com | | id | 23501581-7e34-4b88-94f4-ad8cec1f4387 | | masters | | | name | example.com. | | pool_id | 794ccc2c-d751-44fe-b57f-8894c9f5c842 | | project_id | a194d740818942a8bea6f3674e0a3d71 | | serial | 1457529400 | | status | PENDING | | transferred_at | None | | ttl | 3600 | | type | PRIMARY | | updated_at | None | | version | 1 | +----------------+--------------------------------------+For each name server public IP, create an A record. For example:
ardana >openstack recordset create --records 192.168.10.1 --type A example.com. ns1.example.com. +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | action | CREATE | | created_at | 2016-03-09T13:18:36.000000 | | description | None | | id | 09e962ed-6915-441a-a5a1-e8d93c3239b6 | | name | ns1.example.com. | | records | 192.168.10.1 | | status | PENDING | | ttl | None | | type | A | | updated_at | None | | version | 1 | | zone_id | 23501581-7e34-4b88-94f4-ad8cec1f4387 | +-------------+--------------------------------------+When records have been added, list the record sets in the zone to validate:
ardana >openstack recordset list example.com. +--------------+------------------+------+---------------------------------------------------+ | id | name | type | records | +--------------+------------------+------+---------------------------------------------------+ | 2d6cf...655b | example.com. | SOA | ns1.example.com. hostmaster.example.com 145...600 | | 33466...bd9c | example.com. | NS | ns1.example.com. | | da98c...bc2f | example.com. | NS | ns2.example.com. | | 672ee...74dd | example.com. | NS | ns3.example.com. | | 09e96...39b6 | ns1.example.com. | A | 192.168.10.1 | | bca4f...a752 | ns2.example.com. | A | 192.168.10.2 | | 0f123...2117 | ns3.example.com. | A | 192.168.10.3 | +--------------+------------------+------+---------------------------------------------------+Contact your domain registrar requesting Glue Records to be registered in the
com.zone for the nameserver and public IP address pairs above. If you are using a sub-zone of an existing company zone (for example,ns1.cloud.mycompany.com.), the Glue must be placed in themycompany.com.zone.
10.3.2.1.4 For More Information #
For additional DNS integration and configuration information, see the OpenStack designate documentation at https://docs.openstack.org/designate/rocky/.
For more information on creating servers, domains and examples, see the OpenStack REST API documentation at https://developer.openstack.org/api-ref/dns/.
10.3.3 DNS Service Monitoring Support #
10.3.3.1 DNS Service Monitoring Support #
Additional monitoring support for the DNS Service (designate) has been added to SUSE OpenStack Cloud.
In the Networking section of the Operations Console, you can see alarms for all of
the DNS Services (designate), such as designate-zone-manager, designate-api,
designate-pool-manager, designate-mdns, and designate-central after running
designate-stop.yml.
You can run designate-start.yml to start the DNS Services
back up and the alarms will change from a red status to green and be removed
from the New Alarms panel of the
Operations Console.
An example of the generated alarms from the Operations Console is provided below
after running designate-stop.yml:
ALARM: STATE: ALARM ID: LAST CHECK: DIMENSION: Process Check 0f221056-1b0e-4507-9a28-2e42561fac3e 2016-10-03T10:06:32.106Z hostname=ardana-cp1-c1-m1-mgmt, service=dns, cluster=cluster1, process_name=designate-zone-manager, component=designate-zone-manager, control_plane=control-plane-1, cloud_name=entry-scale-kvm Process Check 50dc4c7b-6fae-416c-9388-6194d2cfc837 2016-10-03T10:04:32.086Z hostname=ardana-cp1-c1-m1-mgmt, service=dns, cluster=cluster1, process_name=designate-api, component=designate-api, control_plane=control-plane-1, cloud_name=entry-scale-kvm Process Check 55cf49cd-1189-4d07-aaf4-09ed08463044 2016-10-03T10:05:32.109Z hostname=ardana-cp1-c1-m1-mgmt, service=dns, cluster=cluster1, process_name=designate-pool-manager, component=designate-pool-manager, control_plane=control-plane-1, cloud_name=entry-scale-kvm Process Check c4ab7a2e-19d7-4eb2-a9e9-26d3b14465ea 2016-10-03T10:06:32.105Z hostname=ardana-cp1-c1-m1-mgmt, service=dns, cluster=cluster1, process_name=designate-mdns, component=designate-mdns, control_plane=control-plane-1, cloud_name=entry-scale-kvm HTTP Status c6349bbf-4fd1-461a-9932-434169b86ce5 2016-10-03T10:05:01.731Z service=dns, cluster=cluster1, url=http://100.60.90.3:9001/, hostname=ardana-cp1-c1-m3-mgmt, component=designate-api, control_plane=control-plane-1, api_endpoint=internal, cloud_name=entry-scale-kvm, monitored_host_type=instance Process Check ec2c32c8-3b91-4656-be70-27ff0c271c89 2016-10-03T10:04:32.082Z hostname=ardana-cp1-c1-m1-mgmt, service=dns, cluster=cluster1, process_name=designate-central, component=designate-central, control_plane=control-plane-1, cloud_name=entry-scale-kvm
10.4 Networking Service Overview #
SUSE OpenStack Cloud Networking is a virtual Networking service that leverages the OpenStack neutron service to provide network connectivity and addressing to SUSE OpenStack Cloud Compute service devices.
The Networking service also provides an API to configure and manage a variety of network services.
You can use the Networking service to connect guest servers or you can define and configure your own virtual network topology.
10.4.1 Installing the Networking Service #
SUSE OpenStack Cloud Network Administrators are responsible for planning for the neutron Networking service, and once installed, to configure the service to meet the needs of their cloud network users.
10.4.2 Working with the Networking service #
To perform tasks using the Networking service, you can use the dashboard, API or CLI.
10.4.3 Reconfiguring the Networking service #
If you change any of the network configuration after installation, it is recommended that you reconfigure the Networking service by running the neutron-reconfigure playbook.
On the Cloud Lifecycle Manager:
ardana >cd ~/openstack/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml
10.4.4 For more information #
For information on how to operate your cloud we suggest you read the OpenStack Operations Guide. The Architecture section contains useful information about how an OpenStack Cloud is put together. However, SUSE OpenStack Cloud takes care of these details for you. The Operations section contains information on how to manage the system.
10.4.5 Neutron External Networks #
10.4.5.1 External networks overview #
This topic explains how to create a neutron external network.
External networks provide access to the internet.
The typical use is to provide an IP address that can be used to reach a VM from an external network which can be a public network like the internet or a network that is private to an organization.
10.4.5.2 Using the Ansible Playbook #
This playbook will query the Networking service for an existing external
network, and then create a new one if you do not already have one. The
resulting external network will have the name ext-net
with a subnet matching the CIDR you specify in the command below.
If you need to specify more granularity, for example specifying an allocation pool for the subnet, use the Section 10.4.5.3, “Using the python-neutronclient CLI”.
ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts neutron-cloud-configure.yml -e EXT_NET_CIDR=<CIDR>
The table below shows the optional switch that you can use as part of this playbook to specify environment-specific information:
| Switch | Description |
|---|---|
|
|
Optional. You can use this switch to specify the external network CIDR. If you choose not to use this switch, or use a wrong value, the VMs will not be accessible over the network.
This CIDR will be from the |
10.4.5.3 Using the python-neutronclient CLI #
For more granularity you can utilize the OpenStackClient tool to create your external network.
Log in to the Cloud Lifecycle Manager.
Source the Admin creds:
ardana >source ~/service.osrcCreate the external network and then the subnet using these commands below.
Creating the network:
ardana >openstack network create --router:external <external-network-name>Creating the subnet:
ardana >openstack subnet create EXTERNAL-NETWORK-NAME CIDR --gateway GATEWAY --allocation-pool start=IP_START,end=IP_END [--disable-dhcp]Where:
Value Description external-network-name This is the name given to your external network. This is a unique value that you will choose. The value
ext-netis usually used.CIDR Use this switch to specify the external network CIDR. If you do not use this switch or use a wrong value, the VMs will not be accessible over the network.
This CIDR will be from the EXTERNAL VM network.
--gateway Optional switch to specify the gateway IP for your subnet. If this is not included, it will choose the first available IP.
--allocation-pool start end
Optional switch to specify start and end IP addresses to use as the allocation pool for this subnet.
--disable-dhcp Optional switch if you want to disable DHCP on this subnet. If this is not specified, DHCP will be enabled.
10.4.5.4 Multiple External Networks #
SUSE OpenStack Cloud provides the ability to have multiple external networks, by using the Network Service (neutron) provider networks for external networks. You can configure SUSE OpenStack Cloud to allow the use of provider VLANs as external networks by following these steps.
Do NOT include the
neutron.l3_agent.external_network_bridgetag in the network_groups definition for your cloud. This results in thel3_agent.ini external_network_bridgebeing set to an empty value (rather than the traditional br-ex).Configure your cloud to use provider VLANs, by specifying the
provider_physical_networktag on one of the network_groups defined for your cloud.For example, to run provider VLANS over the EXAMPLE network group: (some attributes omitted for brevity)
network-groups: - name: EXAMPLE tags: - neutron.networks.vlan: provider-physical-network: physnet1After the cloud has been deployed, you can create external networks using provider VLANs.
For example, using the OpenStackClient:
Create external network 1 on vlan101
ardana >openstack network create --provider-network-type vlan --provider-physical-network physnet1 --provider-segment 101 --external ext-net1Create external network 2 on vlan102
ardana >openstack network create --provider-network-type vlan --provider-physical-network physnet1 --provider-segment 102 --external ext-net2
10.4.6 Neutron Provider Networks #
This topic explains how to create a neutron provider network.
A provider network is a virtual network created in the SUSE OpenStack Cloud cloud that is consumed by SUSE OpenStack Cloud services. The distinctive element of a provider network is that it does not create a virtual router; rather, it depends on L3 routing that is provided by the infrastructure.
A provider network is created by adding the specification to the SUSE OpenStack Cloud input model. It consists of at least one network and one or more subnets.
10.4.6.1 SUSE OpenStack Cloud input model #
The input model is the primary mechanism a cloud admin uses in defining a SUSE OpenStack Cloud installation. It exists as a directory with a data subdirectory that contains YAML files. By convention, any service that creates a neutron provider network will create a subdirectory under the data directory and the name of the subdirectory shall be the project name. For example, the Octavia project will use neutron provider networks so it will have a subdirectory named 'octavia' and the config file that specifies the neutron network will exist in that subdirectory.
├── cloudConfig.yml
├── data
│ ├── control_plane.yml
│ ├── disks_compute.yml
│ ├── disks_controller_1TB.yml
│ ├── disks_controller.yml
│ ├── firewall_rules.yml
│ ├── net_interfaces.yml
│ ├── network_groups.yml
│ ├── networks.yml
│ ├── neutron
│ │ └── neutron_config.yml
│ ├── nic_mappings.yml
│ ├── server_groups.yml
│ ├── server_roles.yml
│ ├── servers.yml
│ ├── swift
│ │ └── swift_config.yml
│ └── octavia
│ └── octavia_config.yml
├── README.html
└── README.md10.4.6.2 Network/Subnet specification #
The elements required in the input model for you to define a network are:
name
network_type
physical_network
Elements that are optional when defining a network are:
segmentation_id
shared
Required elements for the subnet definition are:
cidr
Optional elements for the subnet definition are:
allocation_pools which will require start and end addresses
host_routes which will require a destination and nexthop
gateway_ip
no_gateway
enable-dhcp
NOTE: Only IPv4 is supported at the present time.
10.4.6.3 Network details #
The following table outlines the network values to be set, and what they represent.
| Attribute | Required/optional | Allowed Values | Usage |
|---|---|---|---|
| name | Required | ||
| network_type | Required | flat, vlan, vxlan | The type of desired network |
| physical_network | Required | Valid | Name of physical network that is overlayed with the virtual network |
| segmentation_id | Optional | vlan or vxlan ranges | VLAN id for vlan or tunnel id for vxlan |
| shared | Optional | True | Shared by all projects or private to a single project |
10.4.6.4 Subnet details #
The following table outlines the subnet values to be set, and what they represent.
| Attribute | Req/Opt | Allowed Values | Usage |
|---|---|---|---|
| cidr | Required | Valid CIDR range | for example, 172.30.0.0/24 |
| allocation_pools | Optional | See allocation_pools table below | |
| host_routes | Optional | See host_routes table below | |
| gateway_ip | Optional | Valid IP addr | Subnet gateway to other nets |
| no_gateway | Optional | True | No distribution of gateway |
| enable-dhcp | Optional | True | Enable dhcp for this subnet |
10.4.6.5 ALLOCATION_POOLS details #
The following table explains allocation pool settings.
| Attribute | Req/Opt | Allowed Values | Usage |
|---|---|---|---|
| start | Required | Valid IP addr | First ip address in pool |
| end | Required | Valid IP addr | Last ip address in pool |
10.4.6.6 HOST_ROUTES details #
The following table explains host route settings.
| Attribute | Req/Opt | Allowed Values | Usage |
|---|---|---|---|
| destination | Required | Valid CIDR | Destination subnet |
| nexthop | Required | Valid IP addr | Hop to take to destination subnet |
Multiple destination/nexthop values can be used.
10.4.6.7 Examples #
The following examples show the configuration file settings for neutron and Octavia.
Octavia configuration
This file defines the mapping. It does not need to be edited unless you want to change the name of your VLAN.
Path:
~/openstack/my_cloud/definition/data/octavia/octavia_config.yml
---
product:
version: 2
configuration-data:
- name: OCTAVIA-CONFIG-CP1
services:
- octavia
data:
amp_network_name: OCTAVIA-MGMT-NETneutron configuration
Input your network configuration information for your provider VLANs in
neutron_config.yml found here:
~/openstack/my_cloud/definition/data/neutron/.
---
product:
version: 2
configuration-data:
- name: NEUTRON-CONFIG-CP1
services:
- neutron
data:
neutron_provider_networks:
- name: OCTAVIA-MGMT-NET
provider:
- network_type: vlan
physical_network: physnet1
segmentation_id: 2754
cidr: 10.13.189.0/24
no_gateway: True
enable_dhcp: True
allocation_pools:
- start: 10.13.189.4
end: 10.13.189.252
host_routes:
# route to MANAGEMENT-NET
- destination: 10.13.111.128/26
nexthop: 10.13.189.510.4.6.8 Implementing your changes #
Commit the changes to git:
ardana >git add -Aardana >git commit -a -m "configuring provider network"Run the configuration processor:
ardana >cd ~/openstack/ardana/ansibleardana >ansible-playbook -i hosts/localhost config-processor-run.ymlUpdate your deployment directory:
ardana >cd ~/openstack/ardana/ansibleardana >ansible-playbook -i hosts/localhost ready-deployment.ymlThen continue with your clean cloud installation.
If you are only adding a neutron Provider network to an existing model, then run the neutron-deploy.yml playbook:
ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts neutron-deploy.yml
10.4.6.9 Multiple Provider Networks #
The physical network infrastructure must be configured to convey the provider VLAN traffic as tagged VLANs to the cloud compute nodes and network service network nodes. Configuration of the physical network infrastructure is outside the scope of the SUSE OpenStack Cloud 9 software.
SUSE OpenStack Cloud 9 automates the server networking configuration and the Network Service configuration based on information in the cloud definition. To configure the system for provider VLANs, specify the neutron.networks.vlan tag with a provider-physical-network attribute on one or more network groups. For example (some attributes omitted for brevity):
network-groups:
- name: NET_GROUP_A
tags:
- neutron.networks.vlan:
provider-physical-network: physnet1
- name: NET_GROUP_B
tags:
- neutron.networks.vlan:
provider-physical-network: physnet2A network group is associated with a server network interface via an interface model. For example (some attributes omitted for brevity):
interface-models:
- name: INTERFACE_SET_X
network-interfaces:
- device:
name: bond0
network-groups:
- NET_GROUP_A
- device:
name: eth3
network-groups:
- NET_GROUP_BA network group used for provider VLANs may contain only a single SUSE OpenStack Cloud network, because that VLAN must span all compute nodes and any Network Service network nodes/controllers (that is, it is a single L2 segment). The SUSE OpenStack Cloud network must be defined with tagged-vlan false, otherwise a Linux VLAN network interface will be created. For example:
networks:
- name: NET_A
tagged-vlan: false
network-group: NET_GROUP_A
- name: NET_B
tagged-vlan: false
network-group: NET_GROUP_BWhen the cloud is deployed, SUSE OpenStack Cloud 9 will create the appropriate bridges on the servers, and set the appropriate attributes in the neutron configuration files (for example, bridge_mappings).
After the cloud has been deployed, create Network Service network objects for each provider VLAN. For example, using the Network Service CLI:
ardana >openstack network create --provider:network_type vlan --provider:physical_network physnet1 --provider-segment 101 mynet101ardana >openstack network create --provider:network_type vlan --provider:physical_network physnet2 --provider-segment 234 mynet234
10.4.6.10 More Information #
For more information on the Network Service command-line interface (CLI), see the OpenStack networking command-line client reference: http://docs.openstack.org/cli-reference/content/neutronclient_commands.html
10.4.7 Using IPAM Drivers in the Networking Service #
This topic describes how to choose and implement an IPAM driver.
10.4.7.1 Selecting and implementing an IPAM driver #
Beginning with the Liberty release, OpenStack networking includes a pluggable interface for the IP Address Management (IPAM) function. This interface creates a driver framework for the allocation and de-allocation of subnets and IP addresses, enabling the integration of alternate IPAM implementations or third-party IP Address Management systems.
There are three possible IPAM driver options:
Non-pluggable driver. This option is the default when the ipam_driver parameter is not specified in neutron.conf.
Pluggable reference IPAM driver. The pluggable IPAM driver interface was introduced in SUSE OpenStack Cloud 9 (OpenStack Liberty). It is a refactoring of the Kilo non-pluggable driver to use the new pluggable interface. The setting in neutron.conf to specify this driver is
ipam_driver = internal.Pluggable Infoblox IPAM driver. The pluggable Infoblox IPAM driver is a third-party implementation of the pluggable IPAM interface. the corresponding setting in neutron.conf to specify this driver is
ipam_driver = networking_infoblox.ipam.driver.InfobloxPool.NoteYou can use either the non-pluggable IPAM driver or a pluggable one. However, you cannot use both.
10.4.7.2 Using the Pluggable reference IPAM driver #
To indicate that you want to use the Pluggable reference IPAM driver, the
only parameter needed is "ipam_driver." You can set it by looking for the
following commented line in the
neutron.conf.j2 template (ipam_driver = internal)
uncommenting it, and committing the file. After following the standard
steps to deploy neutron, neutron will be configured to run using the
Pluggable reference IPAM driver.
As stated, the file you must edit is neutron.conf.j2 on
the Cloud Lifecycle Manager in the directory
~/openstack/my_cloud/config/neutron. Here is the relevant
section where you can see the ipam_driver parameter
commented out:
[DEFAULT] ... l3_ha_net_cidr = 169.254.192.0/18 # Uncomment the line below if the Reference Pluggable IPAM driver is to be used # ipam_driver = internal ...
After uncommenting the line ipam_driver = internal,
commit the file using git commit from the openstack/my_cloud
directory:
ardana > git commit -a -m 'My config for enabling the internal IPAM Driver'Then follow the steps to deploy SUSE OpenStack Cloud in the Chapter 13, Overview appropriate to your cloud configuration.
Currently there is no migration path from the non-pluggable driver to a pluggable IPAM driver because changes are needed to database tables and neutron currently cannot make those changes.
10.4.7.3 Using the Infoblox IPAM driver #
As suggested above, using the Infoblox IPAM driver requires changes to
existing parameters in nova.conf and
neutron.conf. If you want to use the infoblox appliance,
you will need to add the "infoblox service-component" to the service-role
containing the neutron API server. To use the infoblox appliance for IPAM,
both the agent and the Infoblox IPAM driver are
required. The infoblox-ipam-agent should be deployed on
the same node where the neutron-server component is running. Usually this is
a Controller node.
Have the Infoblox appliance running on the management network (the Infoblox appliance admin or the datacenter administrator should know how to perform this step).
Change the control plane definition to add i
nfoblox-ipam-agentas a service in the controller node cluster (see change in bold). Make the changes incontrol_plane.ymlfound here:~/openstack/my_cloud/definition/data/control_plane.yml--- product: version: 2 control-planes: - name: ccp control-plane-prefix: ccp ... clusters: - name: cluster0 cluster-prefix: c0 server-role: ARDANA-ROLE member-count: 1 allocation-policy: strict service-components: - lifecycle-manager - name: cluster1 cluster-prefix: c1 server-role: CONTROLLER-ROLE member-count: 3 allocation-policy: strict service-components: - ntp-server ... - neutron-server - infoblox-ipam-agent ... - designate-client - bind resources: - name: compute resource-prefix: comp server-role: COMPUTE-ROLE allocation-policy: anyModify the
~/openstack/my_cloud/config/neutron/neutron.conf.j2file on the controller node to comment and uncomment the lines noted below to enable use with the Infoblox appliance:[DEFAULT] ... l3_ha_net_cidr = 169.254.192.0/18 # Uncomment the line below if the Reference Pluggable IPAM driver is to be used # ipam_driver = internal # Comment out the line below if the Infoblox IPAM Driver is to be used # notification_driver = messaging # Uncomment the lines below if the Infoblox IPAM driver is to be used ipam_driver = networking_infoblox.ipam.driver.InfobloxPool notification_driver = messagingv2 # Modify the infoblox sections below to suit your cloud environment [infoblox] cloud_data_center_id = 1 # This name of this section is formed by "infoblox-dc:<infoblox.cloud_data_center_id>" # If cloud_data_center_id is 1, then the section name is "infoblox-dc:1" [infoblox-dc:0] http_request_timeout = 120 http_pool_maxsize = 100 http_pool_connections = 100 ssl_verify = False wapi_version = 2.2 admin_user_name = admin admin_password = infoblox grid_master_name = infoblox.localdomain grid_master_host = 1.2.3.4 [QUOTAS] ...Change
nova.conf.j2to replace the notification driver "messaging" to "messagingv2"... # Oslo messaging notification_driver = log # Note: # If the infoblox-ipam-agent is to be deployed in the cloud, change the # notification_driver setting from "messaging" to "messagingv2". notification_driver = messagingv2 notification_topics = notifications # Policy ...
Commit the changes:
ardana >cd ~/openstack/my_cloudardana >git commit –a –m 'My config for enabling the Infoblox IPAM driver'Deploy the cloud with the changes. Due to changes to the control_plane.yml, you will need to rerun the config-processor-run.yml playbook if you have run it already during the install process.
ardana >cd ~/openstack/ardana/ansibleardana >ansible-playbook -i hosts/localhost config-processor-run.ymlardana >ansible-playbook -i hosts/localhost ready-deployment.ymlardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts site.yml
10.4.7.4 Configuration parameters for using the Infoblox IPAM driver #
Changes required in the notification parameters in nova.conf:
| Parameter Name | Section in nova.conf | Default Value | Current Value | Description |
|---|---|---|---|---|
| notify_on_state_change | DEFAULT | None | vm_and_task_state |
Send compute.instance.update notifications on instance state changes. Vm_and_task_state means notify on vm and task state changes. Infoblox requires the value to be vm_state (notify on vm state change). Thus NO CHANGE is needed for infoblox |
| notification_topics | DEFAULT | empty list | notifications |
NO CHANGE is needed for infoblox. The infoblox installation guide requires the notifications to be "notifications" |
| notification_driver | DEFAULT | None | messaging |
Change needed. The infoblox installation guide requires the notification driver to be "messagingv2". |
Changes to existing parameters in neutron.conf
| Parameter Name | Section in neutron.conf | Default Value | Current Value | Description |
|---|---|---|---|---|
| ipam_driver | DEFAULT | None |
None (param is undeclared in neutron.conf) |
Pluggable IPAM driver to be used by neutron API server. For infoblox, the value is "networking_infoblox.ipam.driver.InfobloxPool" |
| notification_driver | DEFAULT | empty list | messaging |
The driver used to send notifications from the neutron API server to the neutron agents. The installation guide for networking-infoblox calls for the notification_driver to be "messagingv2" |
| notification_topics | DEFAULT | None | notifications |
No change needed. The row is here show the changes in the neutron parameters described in the installation guide for networking-infoblox |
Parameters specific to the Networking Infoblox Driver. All the parameters for the Infoblox IPAM driver must be defined in neutron.conf.
| Parameter Name | Section in neutron.conf | Default Value | Description |
|---|---|---|---|
| cloud_data_center_id | infoblox | 0 | ID for selecting a particular grid from one or more grids to serve networks in the Infoblox back end |
| ipam_agent_workers | infoblox | 1 | Number of Infoblox IPAM agent works to run |
| grid_master_host | infoblox-dc.<cloud_data_center_id> | empty string | IP address of the grid master. WAPI requests are sent to the grid_master_host |
| ssl_verify | infoblox-dc.<cloud_data_center_id> | False | Ensure whether WAPI requests sent over HTTPS require SSL verification |
| WAPI Version | infoblox-dc.<cloud_data_center_id> | 1.4 | The WAPI version. Value should be 2.2. |
| admin_user_name | infoblox-dc.<cloud_data_center_id> | empty string | Admin user name to access the grid master or cloud platform appliance |
| admin_password | infoblox-dc.<cloud_data_center_id> | empty string | Admin user password |
| http_pool_connections | infoblox-dc.<cloud_data_center_id> | 100 | |
| http_pool_maxsize | infoblox-dc.<cloud_data_center_id> | 100 | |
| http_request_timeout | infoblox-dc.<cloud_data_center_id> | 120 |
The diagram below shows nova compute sending notification to the infoblox-ipam-agent
10.4.7.5 Limitations #
There is no IPAM migration path from non-pluggable to pluggable IPAM driver (https://bugs.launchpad.net/neutron/+bug/1516156). This means there is no way to reconfigure the neutron database if you wanted to change neutron to use a pluggable IPAM driver. Unless you change the default of non-pluggable IPAM configuration to a pluggable driver at install time, you will have no other opportunity to make that change because reconfiguration of SUSE OpenStack Cloud 9from using the default non-pluggable IPAM configuration to SUSE OpenStack Cloud 9 using a pluggable IPAM driver is not supported.
Upgrade from previous versions of SUSE OpenStack Cloud to SUSE OpenStack Cloud 9 to use a pluggable IPAM driver is not supported.
The Infoblox appliance does not allow for overlapping IPs. For example, only one tenant can have a CIDR of 10.0.0.0/24.
The infoblox IPAM driver fails the creation of a subnet when a there is no gateway-ip supplied. For example, the command
openstack subnet create ... --no-gateway ...will fail.
10.4.8 Configuring Load Balancing as a Service (LBaaS) #
SUSE OpenStack Cloud 9 LBaaS Configuration
Load Balancing as a Service (LBaaS) is an advanced networking service that allows load balancing of multi-node environments. It provides the ability to spread requests across multiple servers thereby reducing the load on any single server. This document describes the installation steps and the configuration for LBaaS v2.
The LBaaS architecture is based on a driver model to support different load balancers. LBaaS-compatible drivers are provided by load balancer vendors including F5 and Citrix. A new software load balancer driver was introduced in the OpenStack Liberty release called "Octavia". The Octavia driver deploys a software load balancer called HAProxy. Octavia is the default load balancing provider in SUSE OpenStack Cloud 9 for LBaaS V2. Until Octavia is configured the creation of load balancers will fail with an error. Refer to Chapter 43, Configuring Load Balancer as a Service document for information on installing Octavia.
Before upgrading to SUSE OpenStack Cloud 9, contact F5 and SUSE to determine which F5 drivers have been certified for use with SUSE OpenStack Cloud. Loading drivers not certified by SUSE may result in failure of your cloud deployment.
LBaaS V2 offers with Chapter 43, Configuring Load Balancer as a Service a software load balancing solution that supports both a highly available control plane and data plane. However, should an external hardware load balancer be selected the cloud operation can achieve additional performance and availability.
LBaaS v2
Your vendor already has a driver that supports LBaaS v2. Many hardware load balancer vendors already support LBaaS v2 and this list is growing all the time.
You intend to script your load balancer creation and management so a UI is not important right now (horizon support will be added in a future release).
You intend to support TLS termination at the load balancer.
You intend to use the Octavia software load balancer (adding HA and scalability).
You do not want to take your load balancers offline to perform subsequent LBaaS upgrades.
You intend in future releases to need L7 load balancing.
Reasons not to select this version.
Your LBaaS vendor does not have a v2 driver.
You must be able to manage your load balancers from horizon.
You have legacy software which utilizes the LBaaS v1 API.
LBaaS v2 is installed by default with SUSE OpenStack Cloud and requires minimal configuration to start the service.
LBaaS V2 API currently supports load balancer failover with Octavia. LBaaS v2 API includes automatic failover of a deployed load balancer with Octavia. More information about this driver can be found in Chapter 43, Configuring Load Balancer as a Service.
10.4.8.1 Prerequisites #
SUSE OpenStack Cloud LBaaS v2
SUSE OpenStack Cloud must be installed for LBaaS v2.
Follow the instructions to install Chapter 43, Configuring Load Balancer as a Service
10.4.9 Load Balancer: Octavia Driver Administration #
This document provides the instructions on how to enable and manage various components of the Load Balancer Octavia driver if that driver is enabled.
Section 10.4.9.2, “Tuning Octavia Installation”
Homogeneous Compute Configuration
Octavia and Floating IP's
Configuration Files
Spare Pools
Section 10.4.9.3, “Managing Amphora”
Updating the Cryptographic Certificates
Accessing VM information in nova
Initiating Failover of an Amphora VM
10.4.9.1 Monasca Alerts #
The monasca-agent has the following Octavia-related plugins:
Process checks – checks if octavia processes are running. When it starts, it detects which processes are running and then monitors them.
http_connect check – checks if it can connect to octavia api servers.
Alerts are displayed in the Operations Console.
10.4.9.2 Tuning Octavia Installation #
Homogeneous Compute Configuration
Octavia works only with homogeneous compute node configurations. Currently, Octavia does not support multiple nova flavors. If Octavia needs to be supported on multiple compute nodes, then all the compute nodes should carry same set of physnets (which will be used for Octavia).
Octavia and Floating IPs
Due to a neutron limitation Octavia will only work with CVR routers. Another option is to use VLAN provider networks which do not require a router.
You cannot currently assign a floating IP address as the VIP (user facing) address for a load balancer created by the Octavia driver if the underlying neutron network is configured to support Distributed Virtual Router (DVR). The Octavia driver uses a neutron function known as allowed address pairs to support load balancer fail over.
There is currently a neutron bug that does not support this function in a DVR configuration
Octavia Configuration Files
The system comes pre-tuned and should not need any adjustments for most customers. If in rare instances manual tuning is needed, follow these steps:
Changes might be lost during SUSE OpenStack Cloud upgrades.
Edit the Octavia configuration files in
my_cloud/config/octavia. It is recommended that any
changes be made in all of the Octavia configuration files.
octavia-api.conf.j2
octavia-health-manager.conf.j2
octavia-housekeeping.conf.j2
octavia-worker.conf.j2
After the changes are made to the configuration files, redeploy the service.
Commit changes to git.
ardana >cd ~/openstackardana >git add -Aardana >git commit -m "My Octavia Config"Run the configuration processor and ready deployment.
ardana >cd ~/openstack/ardana/ansible/ardana >ansible-playbook -i hosts/localhost config-processor-run.ymlardana >ansible-playbook -i hosts/localhost ready-deployment.ymlRun the Octavia reconfigure.
ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts octavia-reconfigure.yml
Spare Pools
The Octavia driver provides support for creating spare pools of the HAProxy software installed in VMs. This means instead of creating a new load balancer when loads increase, create new load balancer calls will pull a load balancer from the spare pool. The spare pools feature consumes resources, therefore the load balancers in the spares pool has been set to 0, which is the default and also disables the feature.
Reasons to enable a load balancing spare pool in SUSE OpenStack Cloud
You expect a large number of load balancers to be provisioned all at once (puppet scripts, or ansible scripts) and you want them to come up quickly.
You want to reduce the wait time a customer has while requesting a new load balancer.
To increase the number of load balancers in your spares pool, edit
the Octavia configuration files by uncommenting the
spare_amphora_pool_size and adding the number of load
balancers you would like to include in your spares pool.
# Pool size for the spare pool # spare_amphora_pool_size = 0
10.4.9.3 Managing Amphora #
Octavia starts a separate VM for each load balancing function. These VMs are called amphora.
Updating the Cryptographic Certificates
Octavia uses two-way SSL encryption for communication between amphora and the control plane. Octavia keeps track of the certificates on the amphora and will automatically recycle them. The certificates on the control plane are valid for one year after installation of SUSE OpenStack Cloud.
You can check on the status of the certificate by logging into the controller node as root and running:
ardana > cd /opt/stack/service/octavia-SOME UUID/etc/certs/
openssl x509 -in client.pem -text –nooutThis prints the certificate out where you can check on the expiration dates.
To renew the certificates, reconfigure Octavia. Reconfiguring causes Octavia to automatically generate new certificates and deploy them to the controller hosts.
On the Cloud Lifecycle Manager execute octavia-reconfigure:
ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts octavia-reconfigure.yml
Accessing VM information in nova
You can use openstack project list as an administrative
user to obtain information about the tenant or project-id of the Octavia
project. In the example below, the Octavia project has a project-id of
37fd6e4feac14741b6e75aba14aea833.
ardana > openstack project list
+----------------------------------+------------------+
| ID | Name |
+----------------------------------+------------------+
| 055071d8f25d450ea0b981ca67f7ccee | glance-swift |
| 37fd6e4feac14741b6e75aba14aea833 | octavia |
| 4b431ae087ef4bd285bc887da6405b12 | swift-monitor |
| 8ecf2bb5754646ae97989ba6cba08607 | swift-dispersion |
| b6bd581f8d9a48e18c86008301d40b26 | services |
| bfcada17189e4bc7b22a9072d663b52d | cinderinternal |
| c410223059354dd19964063ef7d63eca | monitor |
| d43bc229f513494189422d88709b7b73 | admin |
| d5a80541ba324c54aeae58ac3de95f77 | demo |
| ea6e039d973e4a58bbe42ee08eaf6a7a | backup |
+----------------------------------+------------------+
You can then use openstack server list --tenant <project-id> to
list the VMs for the Octavia tenant. Take particular note of the IP address
on the OCTAVIA-MGMT-NET; in the example below it is
172.30.1.11. For additional nova command-line options see
Section 10.4.9.5, “For More Information”.
ardana > openstack server list --tenant 37fd6e4feac14741b6e75aba14aea833
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
| ID | Name | Tenant ID | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
| 1ed8f651-de31-4208-81c5-817363818596 | amphora-1c3a4598-5489-48ea-8b9c-60c821269e4c | 37fd6e4feac14741b6e75aba14aea833 | ACTIVE | - | Running | private=10.0.0.4; OCTAVIA-MGMT-NET=172.30.1.11 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+The Amphora VMs do not have SSH or any other access. In the rare case that there is a problem with the underlying load balancer the whole amphora will need to be replaced.
Initiating Failover of an Amphora VM
Under normal operations Octavia will monitor the health of the amphora constantly and automatically fail them over if there are any issues. This helps to minimize any potential downtime for load balancer users. There are, however, a few cases a failover needs to be initiated manually:
The Loadbalancer has become unresponsive and Octavia has not detected an error.
A new image has become available and existing load balancers need to start using the new image.
The cryptographic certificates to control and/or the HMAC password to verify Health information of the amphora have been compromised.
To minimize the impact for end users we will keep the existing load balancer working until shortly before the new one has been provisioned. There will be a short interruption for the load balancing service so keep that in mind when scheduling the failovers. To achieve that follow these steps (assuming the management ip from the previous step):
Assign the IP to a SHELL variable for better readability.
ardana >export MGM_IP=172.30.1.11Identify the port of the vm on the management network.
ardana >openstack port list | grep $MGM_IP | 0b0301b9-4ee8-4fb6-a47c-2690594173f4 | | fa:16:3e:d7:50:92 | {"subnet_id": "3e0de487-e255-4fc3-84b8-60e08564c5b7", "ip_address": "172.30.1.11"} |Disable the port to initiate a failover. Note the load balancer will still function but cannot be controlled any longer by Octavia.
NoteChanges after disabling the port will result in errors.
ardana >openstack port set --admin-state-up False 0b0301b9-4ee8-4fb6-a47c-2690594173f4 Updated port: 0b0301b9-4ee8-4fb6-a47c-2690594173f4You can check to see if the amphora failed over with
openstack server list --tenant <project-id>. This may take some time and in some cases may need to be repeated several times. You can tell that the failover has been successful by the changed IP on the management network.ardana >openstack server list --tenant 37fd6e4feac14741b6e75aba14aea833 +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+ | ID | Name | Tenant ID | Status | Task State | Power State | Networks | +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+ | 1ed8f651-de31-4208-81c5-817363818596 | amphora-1c3a4598-5489-48ea-8b9c-60c821269e4c | 37fd6e4feac14741b6e75aba14aea833 | ACTIVE | - | Running | private=10.0.0.4; OCTAVIA-MGMT-NET=172.30.1.12 | +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
Do not issue too many failovers at once. In a big installation you might be tempted to initiate several failovers in parallel for instance to speed up an update of amphora images. This will put a strain on the nova service and depending on the size of your installation you might need to throttle the failover rate.
10.4.9.4 Load Balancer: Octavia Administration #
10.4.9.4.1 Removing load balancers #
The following procedures demonstrate how to delete a load
balancer that is in the ERROR,
PENDING_CREATE, or
PENDING_DELETE state.
Query the Neutron service for the loadbalancer ID:
tux >neutron lbaas-loadbalancer-list neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+---------+----------------------------------+--------------+---------------------+----------+ | id | name | tenant_id | vip_address | provisioning_status | provider | +--------------------------------------+---------+----------------------------------+--------------+---------------------+----------+ | 7be4e4ab-e9c6-4a57-b767-da9af5ba7405 | test-lb | d62a1510b0f54b5693566fb8afeb5e33 | 192.168.1.10 | ERROR | haproxy | +--------------------------------------+---------+----------------------------------+--------------+---------------------+----------+Connect to the neutron database:
ImportantThe default database name depends on the life cycle manager. Ardana uses
ovs_neutronwhile Crowbar usesneutron.Ardana:
mysql> use ovs_neutron
Crowbar:
mysql> use neutron
Get the pools and healthmonitors associated with the loadbalancer:
mysql> select id, healthmonitor_id, loadbalancer_id from lbaas_pools where loadbalancer_id = '7be4e4ab-e9c6-4a57-b767-da9af5ba7405'; +--------------------------------------+--------------------------------------+--------------------------------------+ | id | healthmonitor_id | loadbalancer_id | +--------------------------------------+--------------------------------------+--------------------------------------+ | 26c0384b-fc76-4943-83e5-9de40dd1c78c | 323a3c4b-8083-41e1-b1d9-04e1fef1a331 | 7be4e4ab-e9c6-4a57-b767-da9af5ba7405 | +--------------------------------------+--------------------------------------+--------------------------------------+
Get the members associated with the pool:
mysql> select id, pool_id from lbaas_members where pool_id = '26c0384b-fc76-4943-83e5-9de40dd1c78c'; +--------------------------------------+--------------------------------------+ | id | pool_id | +--------------------------------------+--------------------------------------+ | 6730f6c1-634c-4371-9df5-1a880662acc9 | 26c0384b-fc76-4943-83e5-9de40dd1c78c | | 06f0cfc9-379a-4e3d-ab31-cdba1580afc2 | 26c0384b-fc76-4943-83e5-9de40dd1c78c | +--------------------------------------+--------------------------------------+
Delete the pool members:
mysql> delete from lbaas_members where id = '6730f6c1-634c-4371-9df5-1a880662acc9'; mysql> delete from lbaas_members where id = '06f0cfc9-379a-4e3d-ab31-cdba1580afc2';
Find and delete the listener associated with the loadbalancer:
mysql> select id, loadbalancer_id, default_pool_id from lbaas_listeners where loadbalancer_id = '7be4e4ab-e9c6-4a57-b767-da9af5ba7405'; +--------------------------------------+--------------------------------------+--------------------------------------+ | id | loadbalancer_id | default_pool_id | +--------------------------------------+--------------------------------------+--------------------------------------+ | 3283f589-8464-43b3-96e0-399377642e0a | 7be4e4ab-e9c6-4a57-b767-da9af5ba7405 | 26c0384b-fc76-4943-83e5-9de40dd1c78c | +--------------------------------------+--------------------------------------+--------------------------------------+ mysql> delete from lbaas_listeners where id = '3283f589-8464-43b3-96e0-399377642e0a';
Delete the pool associated with the loadbalancer:
mysql> delete from lbaas_pools where id = '26c0384b-fc76-4943-83e5-9de40dd1c78c';
Delete the healthmonitor associated with the pool:
mysql> delete from lbaas_healthmonitors where id = '323a3c4b-8083-41e1-b1d9-04e1fef1a331';
Delete the loadbalancer:
mysql> delete from lbaas_loadbalancer_statistics where loadbalancer_id = '7be4e4ab-e9c6-4a57-b767-da9af5ba7405'; mysql> delete from lbaas_loadbalancers where id = '7be4e4ab-e9c6-4a57-b767-da9af5ba7405';
Query the Octavia service for the loadbalancer ID:
tux >openstack loadbalancer list --column id --column name --column provisioning_status +--------------------------------------+---------+---------------------+ | id | name | provisioning_status | +--------------------------------------+---------+---------------------+ | d8ac085d-e077-4af2-b47a-bdec0c162928 | test-lb | ERROR | +--------------------------------------+---------+---------------------+Query the Octavia service for the amphora IDs (in this example we use
ACTIVE/STANDBYtopology with 1 spare Amphora):tux >openstack loadbalancer amphora list +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------------+ | id | loadbalancer_id | status | role | lb_network_ip | ha_ip | +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------------+ | 6dc66d41-e4b6-4c33-945d-563f8b26e675 | d8ac085d-e077-4af2-b47a-bdec0c162928 | ALLOCATED | BACKUP | 172.30.1.7 | 192.168.1.8 | | 1b195602-3b14-4352-b355-5c4a70e200cf | d8ac085d-e077-4af2-b47a-bdec0c162928 | ALLOCATED | MASTER | 172.30.1.6 | 192.168.1.8 | | b2ee14df-8ac6-4bb0-a8d3-3f378dbc2509 | None | READY | None | 172.30.1.20 | None | +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------------+Query the Octavia service for the loadbalancer pools:
tux >openstack loadbalancer pool list +--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+ | id | name | project_id | provisioning_status | protocol | lb_algorithm | admin_state_up | +--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+ | 39c4c791-6e66-4dd5-9b80-14ea11152bb5 | test-pool | 86fba765e67f430b83437f2f25225b65 | ACTIVE | TCP | ROUND_ROBIN | True | +--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+Connect to the octavia database:
mysql> use octavia
Delete any listeners, pools, health monitors, and members from the load balancer:
mysql> delete from listener where load_balancer_id = 'd8ac085d-e077-4af2-b47a-bdec0c162928'; mysql> delete from health_monitor where pool_id = '39c4c791-6e66-4dd5-9b80-14ea11152bb5'; mysql> delete from member where pool_id = '39c4c791-6e66-4dd5-9b80-14ea11152bb5'; mysql> delete from pool where load_balancer_id = 'd8ac085d-e077-4af2-b47a-bdec0c162928';
Delete the amphora entries in the database:
mysql> delete from amphora_health where amphora_id = '6dc66d41-e4b6-4c33-945d-563f8b26e675'; mysql> update amphora set status = 'DELETED' where id = '6dc66d41-e4b6-4c33-945d-563f8b26e675'; mysql> delete from amphora_health where amphora_id = '1b195602-3b14-4352-b355-5c4a70e200cf'; mysql> update amphora set status = 'DELETED' where id = '1b195602-3b14-4352-b355-5c4a70e200cf';
Delete the load balancer instance:
mysql> update load_balancer set provisioning_status = 'DELETED' where id = 'd8ac085d-e077-4af2-b47a-bdec0c162928';
The following script automates the above steps:
#!/bin/bash if (( $# != 1 )); then echo "Please specify a loadbalancer ID" exit 1 fi LB_ID=$1 set -u -e -x readarray -t AMPHORAE < <(openstack loadbalancer amphora list \ --format value \ --column id \ --column loadbalancer_id \ | grep ${LB_ID} \ | cut -d ' ' -f 1) readarray -t POOLS < <(openstack loadbalancer show ${LB_ID} \ --format value \ --column pools) mysql octavia --execute "delete from listener where load_balancer_id = '${LB_ID}';" for p in "${POOLS[@]}"; do mysql octavia --execute "delete from health_monitor where pool_id = '${p}';" mysql octavia --execute "delete from member where pool_id = '${p}';" done mysql octavia --execute "delete from pool where load_balancer_id = '${LB_ID}';" for a in "${AMPHORAE[@]}"; do mysql octavia --execute "delete from amphora_health where amphora_id = '${a}';" mysql octavia --execute "update amphora set status = 'DELETED' where id = '${a}';" done mysql octavia --execute "update load_balancer set provisioning_status = 'DELETED' where id = '${LB_ID}';"
10.4.9.5 For More Information #
For more information on the OpenStackClient and Octavia terminology, see the OpenStackClient guide.
10.4.10 Role-based Access Control in neutron #
This topic explains how to achieve more granular access control for your neutron networks.
Previously in SUSE OpenStack Cloud, a network object was either private to a project or could be used by all projects. If the network's shared attribute was True, then the network could be used by every project in the cloud. If false, only the members of the owning project could use it. There was no way for the network to be shared by only a subset of the projects.
neutron Role Based Access Control (RBAC) solves this problem for networks. Now the network owner can create RBAC policies that give network access to target projects. Members of a targeted project can use the network named in the RBAC policy the same way as if the network was owned by the project. Constraints are described in the section Section 10.4.10.10, “Limitations”.
With RBAC you are able to let another tenant use a network that you created, but as the owner of the network, you need to create the subnet and the router for the network.
10.4.10.1 Creating a Network #
ardana > openstack network create demo-net
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2018-07-25T17:43:59Z |
| description | |
| dns_domain | |
| id | 9c801954-ec7f-4a65-82f8-e313120aabc4 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | False |
| is_vlan_transparent | None |
| mtu | 1450 |
| name | demo-net |
| port_security_enabled | False |
| project_id | cb67c79e25a84e328326d186bf703e1b |
| provider:network_type | vxlan |
| provider:physical_network | None |
| provider:segmentation_id | 1009 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | |
| tags | |
| updated_at | 2018-07-25T17:43:59Z |
+---------------------------+--------------------------------------+10.4.10.2 Creating an RBAC Policy #
Here we will create an RBAC policy where a member of the project called 'demo' will share the network with members of project 'demo2'
To create the RBAC policy, run:
ardana > openstack network rbac create --target-project DEMO2-PROJECT-ID --type network --action access_as_shared demo-netHere is an example where the DEMO2-PROJECT-ID is 5a582af8b44b422fafcd4545bd2b7eb5
ardana > openstack network rbac create --target-tenant 5a582af8b44b422fafcd4545bd2b7eb5 \
--type network --action access_as_shared demo-net10.4.10.3 Listing RBACs #
To list all the RBAC rules/policies, execute:
ardana > openstack network rbac list
+--------------------------------------+-------------+--------------------------------------+
| ID | Object Type | Object ID |
+--------------------------------------+-------------+--------------------------------------+
| 0fdec7f0-9b94-42b4-a4cd-b291d04282c1 | network | 7cd94877-4276-488d-b682-7328fc85d721 |
+--------------------------------------+-------------+--------------------------------------+10.4.10.4 Listing the Attributes of an RBAC #
To see the attributes of a specific RBAC policy, run
ardana > openstack network rbac show POLICY-IDFor example:
ardana > openstack network rbac show 0fd89dcb-9809-4a5e-adc1-39dd676cb386Here is the output:
+---------------+--------------------------------------+ | Field | Value | +---------------+--------------------------------------+ | action | access_as_shared | | id | 0fd89dcb-9809-4a5e-adc1-39dd676cb386 | | object_id | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b | | object_type | network | | target_tenant | 5a582af8b44b422fafcd4545bd2b7eb5 | | tenant_id | 75eb5efae5764682bca2fede6f4d8c6f | +---------------+--------------------------------------+
10.4.10.5 Deleting an RBAC Policy #
To delete an RBAC policy, run openstack network rbac delete passing the policy id:
ardana > openstack network rbac delete POLICY-IDFor example:
ardana > openstack network rbac delete 0fd89dcb-9809-4a5e-adc1-39dd676cb386Here is the output:
Deleted rbac_policy: 0fd89dcb-9809-4a5e-adc1-39dd676cb386
10.4.10.6 Sharing a Network with All Tenants #
Either the administrator or the network owner can make a network shareable by all tenants.
The administrator can make a tenant's network shareable by all tenants.
To make the network demo-shareall-net accessible by all
tenants in the cloud:
To share a network with all tenants:
Get a list of all projects
ardana >~/service.osrcardana >openstack project listwhich produces the list:
+----------------------------------+------------------+ | ID | Name | +----------------------------------+------------------+ | 1be57778b61645a7a1c07ca0ac488f9e | demo | | 5346676226274cd2b3e3862c2d5ceadd | admin | | 749a557b2b9c482ca047e8f4abf348cd | swift-monitor | | 8284a83df4df429fb04996c59f9a314b | swift-dispersion | | c7a74026ed8d4345a48a3860048dcb39 | demo-sharee | | e771266d937440828372090c4f99a995 | glance-swift | | f43fb69f107b4b109d22431766b85f20 | services | +----------------------------------+------------------+
Get a list of networks:
ardana >openstack network listThis produces the following list:
+--------------------------------------+-------------------+----------------------------------------------------+ | id | name | subnets | +--------------------------------------+-------------------+----------------------------------------------------+ | f50f9a63-c048-444d-939d-370cb0af1387 | ext-net | ef3873db-fc7a-4085-8454-5566fb5578ea 172.31.0.0/16 | | 9fb676f5-137e-4646-ac6e-db675a885fd3 | demo-net | 18fb0b77-fc8b-4f8d-9172-ee47869f92cc 10.0.1.0/24 | | 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e | demo-shareall-net | 2bbc85a9-3ffe-464c-944b-2476c7804877 10.0.250.0/24 | | 73f946ee-bd2b-42e9-87e4-87f19edd0682 | demo-share-subset | c088b0ef-f541-42a7-b4b9-6ef3c9921e44 10.0.2.0/24 | +--------------------------------------+-------------------+----------------------------------------------------+
Set the network you want to share to a shared value of True:
ardana >openstack network set --share 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8eYou should see the following output:
Updated network: 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e
Check the attributes of that network by running the following command using the ID of the network in question:
ardana >openstack network show 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8eThe output will look like this:
+---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2018-07-25T17:43:59Z | | description | | | dns_domain | | | id | 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 1450 | | name | demo-net | | port_security_enabled | False | | project_id | cb67c79e25a84e328326d186bf703e1b | | provider:network_type | vxlan | | provider:physical_network | None | | provider:segmentation_id | 1009 | | qos_policy_id | None | | revision_number | 2 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | | | tags | | | updated_at | 2018-07-25T17:43:59Z | +---------------------------+--------------------------------------+
As the owner of the
demo-shareall-netnetwork, view the RBAC attributes fordemo-shareall-net(id=8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e) by first getting an RBAC list:ardana >echo $OS_USERNAME ; echo $OS_PROJECT_NAME demo demoardana >openstack network rbac listThis produces the list:
+--------------------------------------+--------------------------------------+ | id | object_id | +--------------------------------------+--------------------------------------+ | ... | | 3e078293-f55d-461c-9a0b-67b5dae321e8 | 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e | +--------------------------------------+--------------------------------------+
View the RBAC information:
ardana >openstack network rbac show 3e078293-f55d-461c-9a0b-67b5dae321e8 +---------------+--------------------------------------+ | Field | Value | +---------------+--------------------------------------+ | action | access_as_shared | | id | 3e078293-f55d-461c-9a0b-67b5dae321e8 | | object_id | 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e | | object_type | network | | target_tenant | * | | tenant_id | 1be57778b61645a7a1c07ca0ac488f9e | +---------------+--------------------------------------+With network RBAC, the owner of the network can also make the network shareable by all tenants. First create the network:
ardana >echo $OS_PROJECT_NAME ; echo $OS_USERNAME demo demoardana >openstack network create test-netThe network is created:
+---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2018-07-25T18:04:25Z | | description | | | dns_domain | | | id | a4bd7c3a-818f-4431-8cdb-fedf7ff40f73 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 1450 | | name | test-net | | port_security_enabled | False | | project_id | cb67c79e25a84e328326d186bf703e1b | | provider:network_type | vxlan | | provider:physical_network | None | | provider:segmentation_id | 1073 | | qos_policy_id | None | | revision_number | 2 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | | | tags | | | updated_at | 2018-07-25T18:04:25Z | +---------------------------+--------------------------------------+
Create the RBAC. It is important that the asterisk is surrounded by single-quotes to prevent the shell from expanding it to all files in the current directory.
ardana >openstack network rbac create --type network \ --action access_as_shared --target-project '*' test-netHere are the resulting RBAC attributes:
+---------------+--------------------------------------+ | Field | Value | +---------------+--------------------------------------+ | action | access_as_shared | | id | 0b797cc6-debc-48a1-bf9d-d294b077d0d9 | | object_id | a4bd7c3a-818f-4431-8cdb-fedf7ff40f73 | | object_type | network | | target_tenant | * | | tenant_id | 1be57778b61645a7a1c07ca0ac488f9e | +---------------+--------------------------------------+
10.4.10.7 Target Project (demo2) View of Networks and Subnets #
Note that the owner of the network and subnet is not the tenant named
demo2. Both the network and subnet are owned by tenant demo.
Demo2members cannot create subnets of the network. They also cannot
modify or delete subnets owned by demo.
As the tenant demo2, you can get a list of neutron networks:
ardana > openstack network list+--------------------------------------+-----------+--------------------------------------------------+ | id | name | subnets | +--------------------------------------+-----------+--------------------------------------------------+ | f60f3896-2854-4f20-b03f-584a0dcce7a6 | ext-net | 50e39973-b2e3-466b-81c9-31f4d83d990b | | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b | demo-net | d9b765da-45eb-4543-be96-1b69a00a2556 10.0.1.0/24 | ... +--------------------------------------+-----------+--------------------------------------------------+
And get a list of subnets:
ardana > openstack subnet list --network c3d55c21-d8c9-4ee5-944b-560b7e0ea33b+--------------------------------------+---------+--------------------------------------+---------------+ | ID | Name | Network | Subnet | +--------------------------------------+---------+--------------------------------------+---------------+ | a806f28b-ad66-47f1-b280-a1caa9beb832 | ext-net | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b | 10.0.1.0/24 | +--------------------------------------+---------+--------------------------------------+---------------+
To show details of the subnet:
ardana > openstack subnet show d9b765da-45eb-4543-be96-1b69a00a2556+-------------------+--------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------+
| allocation_pools | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.1.1 |
| host_routes | |
| id | d9b765da-45eb-4543-be96-1b69a00a2556 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | sb-demo-net |
| network_id | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b |
| subnetpool_id | |
| tenant_id | 75eb5efae5764682bca2fede6f4d8c6f |
+-------------------+--------------------------------------------+10.4.10.8 Target Project: Creating a Port Using demo-net #
The owner of the port is demo2. Members of the network owner project
(demo) will not see this port.
Running the following command:
ardana > openstack port create c3d55c21-d8c9-4ee5-944b-560b7e0ea33bCreates a new port:
+-----------------------+-----------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:vnic_type | normal |
| device_id | |
| device_owner | |
| dns_assignment | {"hostname": "host-10-0-1-10", "ip_address": "10.0.1.10", "fqdn": "host-10-0-1-10.openstacklocal."} |
| dns_name | |
| fixed_ips | {"subnet_id": "d9b765da-45eb-4543-be96-1b69a00a2556", "ip_address": "10.0.1.10"} |
| id | 03ef2dce-20dc-47e5-9160-942320b4e503 |
| mac_address | fa:16:3e:27:8d:ca |
| name | |
| network_id | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b |
| security_groups | 275802d0-33cb-4796-9e57-03d8ddd29b94 |
| status | DOWN |
| tenant_id | 5a582af8b44b422fafcd4545bd2b7eb5 |
+-----------------------+-----------------------------------------------------------------------------------------------------+10.4.10.9 Target Project Booting a VM Using Demo-Net #
Here the tenant demo2 boots a VM that uses the demo-net shared network:
ardana > openstack server create --flavor 1 --image $OS_IMAGE --nic net-id=c3d55c21-d8c9-4ee5-944b-560b7e0ea33b demo2-vm-using-demo-net-nic+--------------------------------------+------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------+
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | sS9uSv9PT79F |
| config_drive | |
| created | 2016-01-04T19:23:24Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 3a4dc44a-027b-45e9-acf8-054a7c2dca2a |
| image | cirros-0.3.3-x86_64 (6ae23432-8636-4e...1efc5) |
| key_name | - |
| metadata | {} |
| name | demo2-vm-using-demo-net-nic |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 5a582af8b44b422fafcd4545bd2b7eb5 |
| updated | 2016-01-04T19:23:24Z |
| user_id | a0e6427b036344fdb47162987cb0cee5 |
+--------------------------------------+------------------------------------------------+Run openstack server list:
ardana > openstack server listSee the VM running:
+-------------------+-----------------------------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +-------------------+-----------------------------+--------+------------+-------------+--------------------+ | 3a4dc...a7c2dca2a | demo2-vm-using-demo-net-nic | ACTIVE | - | Running | demo-net=10.0.1.11 | +-------------------+-----------------------------+--------+------------+-------------+--------------------+
Run openstack port list:
ardana > openstask port list --device-id 3a4dc44a-027b-45e9-acf8-054a7c2dca2aView the subnet:
+---------------------+------+-------------------+-------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+---------------------+------+-------------------+-------------------------------------------------------------------+
| 7d14ef8b-9...80348f | | fa:16:3e:75:32:8e | {"subnet_id": "d9b765da-45...00a2556", "ip_address": "10.0.1.11"} |
+---------------------+------+-------------------+-------------------------------------------------------------------+Run openstack port show:
ardana > openstack port show 7d14ef8b-9d48-4310-8c02-00c74d80348f+-----------------------+-----------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:vnic_type | normal |
| device_id | 3a4dc44a-027b-45e9-acf8-054a7c2dca2a |
| device_owner | compute:None |
| dns_assignment | {"hostname": "host-10-0-1-11", "ip_address": "10.0.1.11", "fqdn": "host-10-0-1-11.openstacklocal."} |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "d9b765da-45eb-4543-be96-1b69a00a2556", "ip_address": "10.0.1.11"} |
| id | 7d14ef8b-9d48-4310-8c02-00c74d80348f |
| mac_address | fa:16:3e:75:32:8e |
| name | |
| network_id | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b |
| security_groups | 275802d0-33cb-4796-9e57-03d8ddd29b94 |
| status | ACTIVE |
| tenant_id | 5a582af8b44b422fafcd4545bd2b7eb5 |
+-----------------------+-----------------------------------------------------------------------------------------------------+10.4.10.10 Limitations #
Note the following limitations of RBAC in neutron.
neutron network is the only supported RBAC neutron object type.
The "access_as_external" action is not supported – even though it is listed as a valid action by python-neutronclient.
The neutron-api server will not accept action value of 'access_as_external'. The
access_as_externaldefinition is not found in the specs.The target project users cannot create, modify, or delete subnets on networks that have RBAC policies.
The subnet of a network that has an RBAC policy cannot be added as an interface of a target tenant's router. For example, the command
openstack router add subnet tgt-tenant-router <sb-demo-net uuid>will error out.The security group rules on the network owner do not apply to other projects that can use the network.
A user in target project can boot up VMs using a VNIC using the shared network. The user of the target project can assign a floating IP (FIP) to the VM. The target project must have SG rules that allows SSH and/or ICMP for VM connectivity.
neutron RBAC creation and management are currently not supported in horizon. For now, the neutron CLI has to be used to manage RBAC rules.
A RBAC rule tells neutron whether a tenant can access a network (Allow). Currently there is no DENY action.
Port creation on a shared network fails if
--fixed-ipis specified in theopenstack port createcommand.
10.4.11 Configuring Maximum Transmission Units in neutron #
This topic explains how you can configure MTUs, what to look out for, and the results and implications of changing the default MTU settings. It is important to note that every network within a network group will have the same MTU.
An MTU change will not affect existing networks that have had VMs created on them. It will only take effect on new networks created after the reconfiguration process.
10.4.11.1 Overview #
A Maximum Transmission Unit, or MTU is the maximum packet size (in bytes) that a network device can or is configured to handle. There are a number of places in your cloud where MTU configuration is relevant: the physical interfaces managed and configured by SUSE OpenStack Cloud, the virtual interfaces created by neutron and nova for neutron networking, and the interfaces inside the VMs.
SUSE OpenStack Cloud-managed physical interfaces
SUSE OpenStack Cloud-managed physical interfaces include the physical interfaces
and the bonds, bridges, and VLANs created on top of them. The MTU for these
interfaces is configured via the 'mtu' property of a network group. Because
multiple network groups can be mapped to one physical interface, there may
have to be some resolution of differing MTUs between the untagged and tagged
VLANs on the same physical interface. For instance, if one untagged VLAN,
vlan101 (with an MTU of 1500) and a tagged VLAN vlan201 (with an MTU of
9000) are both on one interface (eth0), this means that eth0 can handle
1500, but the VLAN interface which is created on top of eth0 (that is,
vlan201@eth0) wants 9000. However, vlan201 cannot have a
higher MTU than eth0, so vlan201 will be limited to 1500 when it is brought
up, and fragmentation will result.
In general, a VLAN interface MTU must be lower than or equal to the base device MTU. If they are different, as in the case above, the MTU of eth0 can be overridden and raised to 9000, but in any case the discrepancy will have to be reconciled.
neutron/nova interfaces
neutron/nova interfaces include the virtual devices created by neutron and nova during the normal process of realizing a neutron network/router and booting a VM on it (qr-*, qg-*, tap-*, qvo-*, qvb-*, etc.). There is currently no support in neutron/nova for per-network MTUs in which every interface along the path for a particular neutron network has the correct MTU for that network. There is, however, support for globally changing the MTU of devices created by neutron/nova (see network_device_mtu below). This means that if you want to enable jumbo frames for any set of VMs, you will have to enable it for all your VMs. You cannot just enable them for a particular neutron network.
VM interfaces
VMs typically get their MTU via DHCP advertisement, which means that the dnsmasq processes spawned by the neutron-dhcp-agent actually advertise a particular MTU to the VMs. In SUSE OpenStack Cloud 9, the DHCP server advertises to all VMS a 1400 MTU via a forced setting in dnsmasq-neutron.conf. This is suboptimal for every network type (vxlan, flat, vlan, etc) but it does prevent fragmentation of a VM's packets due to encapsulation.
For instance, if you set the new *-mtu configuration options to a default of 1500 and create a VXLAN network, it will be given an MTU of 1450 (with the remaining 50 bytes used by the VXLAN encapsulation header) and will advertise a 1450 MTU to any VM booted on that network. If you create a provider VLAN network, it will have an MTU of 1500 and will advertise 1500 to booted VMs on the network. It should be noted that this default starting point for MTU calculation and advertisement is also global, meaning you cannot have an MTU of 8950 on one VXLAN network and 1450 on another. However, you can have provider physical networks with different MTUs by using the physical_network_mtus config option, but nova still requires a global MTU option for the interfaces it creates, thus you cannot really take advantage of that configuration option.
10.4.11.2 Network settings in the input model #
MTU can be set as an attribute of a network group in network_groups.yml. Note that this applies only to KVM. That setting means that every network in the network group will be assigned the specified MTU. The MTU value must be set individually for each network group. For example:
network-groups:
- name: GUEST
mtu: 9000
...
- name: EXTERNAL-API
mtu: 9000
...
- name: EXTERNAL-VM
mtu: 9000
...10.4.11.3 Infrastructure support for jumbo frames #
If you want to use jumbo frames, or frames with an MTU of 9000 or more, the physical switches and routers that make up the infrastructure of the SUSE OpenStack Cloud installation must be configured to support them. To realize the advantages, all devices in the same broadcast domain must have the same MTU.
If you want to configure jumbo frames on compute and controller nodes, then all switches joining the compute and controller nodes must have jumbo frames enabled. Similarly, the "infrastructure gateway" through which the external VM network flows, commonly known as the default route for the external VM VLAN, must also have the same MTU configured.
You can also consider anything in the same broadcast domain to be anything in the same VLAN or anything in the same IP subnet.
10.4.11.4 Enabling end-to-end jumbo frames for a VM #
Add an
mtuattribute to all the network groups in your model. Note that adding the MTU for the network groups will only affect the configuration for physical network interfaces.To add the mtu attribute, find the YAML file that contains your network-groups entry. We will assume it is network_groups.yml, unless you have changed it. Whatever the file is named, it will be found in ~/openstack/my_cloud/definition/data/.
To edit these files, begin by checking out the site branch on the Cloud Lifecycle Manager node. You may already be on that branch. If so, you will remain there.
ardana >cd ~/openstack/ardana/ansibleardana >git checkout siteThen begin editing the files. In
network_groups.yml, addmtu: 9000.network-groups: - name: GUEST hostname-suffix: guest mtu: 9000 tags: - neutron.networks.vxlanThis sets the physical interface managed by SUSE OpenStack Cloud 9 that has the GUEST network group tag assigned to it. This can be found in the
interfaces_set.ymlfile under theinterface-modelssection.Edit
neutron.conf.j2found in~/openstack/my_cloud/config/neutron/to setglobal_physnet_mtuto9000under[DEFAULT]:[DEFAULT] ... global_physnet_mtu = 9000
This allows neutron to advertise the optimal MTU to instances (based on
global_physnet_mtuminus the encapsulation size).Remove the
dhcp-option-force=26,1400line from~/openstack/my_cloud/config/neutron/dnsmasq-neutron.conf.j2.OvS will set
br-intto the value of the lowest physical interface. If you are using Jumbo frames on some of your networks,br-inton the controllers may be set to 1500 instead of 9000. Work around this condition by running:ovs-vsctl set int br-int mtu_request=9000
Commit your changes
ardana >git add -Aardana >git commit -m "your commit message goes here in quotes"If SUSE OpenStack Cloud has not been deployed yet, do normal deployment and skip to Step 8.
Assuming it has been deployed already, continue here:
Run the configuration processor:
ardana >ansible-playbook -i hosts/localhost config-processor-run.ymland ready the deployment:
ardana >ansible-playbook -i hosts/localhost ready-deployment.ymlThen run the network_interface-reconfigure.yml playbook, changing directories first:
ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts network_interface-reconfigure.ymlThen run neutron-reconfigure.yml:
ardana >ansible-playbook -i hosts/verb_hosts neutron-reconfigure.ymlThen nova-reconfigure.yml:
ardana >ansible-playbook -i hosts/verb_hosts nova-reconfigure.ymlNote: adding/changing network-group mtu settings will likely require a network restart when
network_interface-reconfigure.ymlis run.Follow the normal process for creating a neutron network and booting a VM or two. In this example, if a VXLAN network is created and a VM is booted on it, the VM will have an MTU of 8950, with the remaining 50 bytes used by the VXLAN encapsulation header.
Test and verify that the VM can send and receive jumbo frames without fragmentation. You can use
ping. For example, to test an MTU of 9000 using VXLAN:ardana >ping –M do –s 8950 YOUR_VM_FLOATING_IPSubstitute your actual floating IP address for the YOUR_VM_FLOATING_IP.
10.4.11.5 Enabling Optimal MTU Advertisement Feature #
To enable the optimal MTU feature, follow these steps:
Edit
~/openstack/my_cloud/config/neutron/neutron.conf.j2to removeadvertise_mtuvariable under [DEFAULT][DEFAULT] ... advertise_mtu = False #remove this
Remove the
dhcp-option-force=26,1400line from~/openstack/my_cloud/config/neutron/dnsmasq-neutron.conf.j2.If SUSE OpenStack Cloud has already been deployed, follow the remaining steps, otherwise follow the normal deployment procedures.
Commit your changes
ardana >git add -Aardana >git commit -m "your commit message goes here in quotes"Run the configuration processor:
ardana >ansible-playbook -i hosts/localhost config-processor-run.ymlRun ready deployment:
ardana >ansible-playbook -i hosts/localhost ready-deployment.ymlRun the
network_interface-reconfigure.ymlplaybook, changing directories first:ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts network_interface-reconfigure.ymlRun the
neutron-reconfigure.ymlplaybook:ardana >ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml
If you are upgrading an existing deployment, avoid creating MTU mismatch
between network interfaces in preexisting VMs and that of VMs created after
upgrade. If you do have an MTU mismatch, then the new VMs (having interface
with 1500 minus the underlay protocol overhead) will not be able to have L2
connectivity with preexisting VMs (with 1400 MTU due to
dhcp-option-force).
10.4.12 Improve Network Peformance with Isolated Metadata Settings #
In SUSE OpenStack Cloud, neutron currently sets enable_isolated_metadata =
True by default in dhcp_agent.ini because
several services require isolated networks (neutron networks without a
router). It also sets force_metadata = True if DVR is
enabled to improve the scalability on large environments with a high churn
rate. However, this has the effect of spawning a neutron-ns-metadata-proxy
process on one of the controller nodes for every active neutron network.
In environments that create many neutron networks, these extra
neutron-ns-metadata-proxy processes can quickly eat up a
lot of memory on the controllers, which does not scale up well.
For deployments that do not require isolated metadata (that is, they do not
require the Platform Services and will always create networks with an
attached router) and do not have a high churn rate, you can set
enable_isolated_metadata = False and force_metadata = False
in dhcp_agent.ini to reduce neutron memory usage on controllers,
allowing a greater number of active neutron networks.
Note that the dhcp_agent.ini.j2 template is found in
~/openstack/my_cloud/config/neutron on the Cloud Lifecycle Manager
node. The edit can be made there and the standard deployment can be run if
this is install time. In a deployed cloud, run the neutron reconfiguration
procedure outlined here:
First check out the site branch:
ardana >cd ~/openstack/my_cloud/config/neutronardana >git checkout siteEdit the
dhcp_agent.ini.j2file to change theenable_isolated_metadata = {{ neutron_enable_isolated_metadata }}force_metadata = {{ router_distributed }}line in the[DEFAULT]section to read:enable_isolated_metadata = False
force_metadata = False
Commit the file:
ardana >git add -Aardana >git commit -m "your commit message goes here in quotes"Run the
ready-deployment.ymlplaybook from~/openstack/ardana/ansible:ardana >cd ~/openstack/ardana/ansibleardana >ansible-playbook -i hosts/localhost ready-deployment.ymlThen run the
neutron-reconfigure.ymlplaybook, changing directories first:ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml
10.4.13 Moving from DVR deployments to non_DVR #
If you have an older deployment of SUSE OpenStack Cloud which is using DVR as a default and you are attempting to move to non_DVR, follow these steps:
Remove all your existing DVR routers and their workloads. Make sure to remove interfaces, floating ips and gateways, if applicable.
ardana >openstack router remove subnet ROUTER-NAME SUBNET-NAME/SUBNET-IDardana >openstack floating ip unset –port FLOATINGIP-ID PRIVATE-PORT-IDardana >openstack router unset ROUTER-NAME -NET-NAME/EXT-NET-IDThen delete the router.
ardana >openstack router delete ROUTER-NAMEBefore you create any non_DVR router make sure that l3-agents and metadata-agents are not running in any compute host. You can run the command
openstack network agent listto see if there are any neutron-l3-agent running in any compute-host in your deployment.You must disable
neutron-l3-agentandneutron-metadata-agenton every compute host by running the following commands:ardana >openstack network agent list +--------------------------------------+----------------------+--------------------------+-------------------+-------+----------------+---------------------------+ | id | agent_type | host | availability_zone | alive | admin_state_up | binary | +--------------------------------------+----------------------+--------------------------+-------------------+-------+----------------+---------------------------+ | 810f0ae7-63aa-4ee3-952d-69837b4b2fe4 | L3 agent | ardana-cp1-comp0001-mgmt | nova | :-) | True | neutron-l3-agent | | 89ac17ba-2f43-428a-98fa-b3698646543d | Metadata agent | ardana-cp1-comp0001-mgmt | | :-) | True | neutron-metadata-agent | | f602edce-1d2a-4c8a-ba56-fa41103d4e17 | Open vSwitch agent | ardana-cp1-comp0001-mgmt | | :-) | True | neutron-openvswitch-agent | ... +--------------------------------------+----------------------+--------------------------+-------------------+-------+----------------+---------------------------+ $ openstack network agent set --disable 810f0ae7-63aa-4ee3-952d-69837b4b2fe4 Updated agent: 810f0ae7-63aa-4ee3-952d-69837b4b2fe4 $ openstack network agent set --disable 89ac17ba-2f43-428a-98fa-b3698646543d Updated agent: 89ac17ba-2f43-428a-98fa-b3698646543dNoteOnly L3 and Metadata agents were disabled.
Once L3 and metadata neutron agents are stopped, follow steps 1 through 7 in the document Section 12.2, “Configuring SUSE OpenStack Cloud without DVR” and then run the
neutron-reconfigure.ymlplaybook:ardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml
10.4.14 OVS-DPDK Support #
SUSE OpenStack Cloud uses a version of Open vSwitch (OVS) that is built with the Data Plane Development Kit (DPDK) and includes a QEMU hypervisor which supports vhost-user.
The OVS-DPDK package modifes the OVS fast path, which is normally performed in kernel space, and allows it to run in userspace so there is no context switch to the kernel for processing network packets.
The EAL component of DPDK supports mapping the Network Interface Card (NIC) registers directly into userspace. The DPDK provides a Poll Mode Driver (PMD) that can access the NIC hardware from userspace and uses polling instead of interrupts to avoid the user to kernel transition.
The PMD maps the shared address space of the VM that is provided by the vhost-user capability of QEMU. The vhost-user mode causes neutron to create a Unix domain socket that allows communication between the PMD and QEMU. The PMD uses this in order to acquire the file descriptors to the pre-allocated VM memory. This allows the PMD to directly access the VM memory space and perform a fast zero-copy of network packets directly into and out of the VMs virtio_net vring.
This yields performance improvements in the time it takes to process network packets.
10.4.14.1 Usage considerations #
The target for a DPDK Open vSwitch is VM performance and VMs only run on compute nodes so the following considerations are compute node specific.
In order to use DPDK with VMs,
hugepagesmust be enabled; please see Section 10.4.14.3, “Configuring Hugepages for DPDK in Networks”. The memory to be used must be allocated at boot time so you must know beforehand how many VMs will be scheduled on a node. Also, for NUMA considerations, you want those hugepages on the same NUMA node as the NIC. A VM maps its entire address space into a hugepage.For maximum performance you must reserve logical cores for DPDK poll mode driver (PMD) usage and for hypervisor (QEMU) usage. This keeps the Linux kernel from scheduling processes on those cores. The PMD threads will go to 100% cpu utilization since it uses polling of the hardware instead of interrupts. There will be at least 2 cores dedicated to PMD threads. Each VM will have a core dedicated to it although for less performance VMs can share cores.
VMs can use the virtio_net or the virtio_pmd drivers. There is also a PMD for an emulated e1000.
Only VMs that use hugepages can be sucessfully launched on a DPDK-enabled NIC. If there is a need to support both DPDK and non-DPDK-based VMs, an additional port managed by the Linux kernel must exist.
10.4.14.2 For more information #
See the following topics for more information:
10.4.14.3 Configuring Hugepages for DPDK in Networks #
To take advantage of DPDK and its network performance enhancements, enable hugepages first.
With hugepages, physical RAM is reserved at boot time and dedicated to a virtual machine. Only that virtual machine and Open vSwitch can use this specifically allocated RAM. The host OS cannot access it. This memory is contiguous, and because of its larger size, reduces the number of entries in the memory map and number of times it must be read.
The hugepage reservation is made in /etc/default/grub,
but this is handled by the Cloud Lifecycle Manager.
In addition to hugepages, to use DPDK, CPU isolation is required. This is
achieved with the 'isolcups' command in
/etc/default/grub, but this is also managed by the
Cloud Lifecycle Manager using a new input model file.
The two new input model files introduced with this release to help you configure the necessary settings and persist them are:
memory_models.yml (for hugepages)
cpu_models.yml (for CPU isolation)
10.4.14.3.1 memory_models.yml #
In this file you set your huge page size along with the number of such huge-page allocations.
---
product:
version: 2
memory-models:
- name: COMPUTE-MEMORY-NUMA
default-huge-page-size: 1G
huge-pages:
- size: 1G
count: 24
numa-node: 0
- size: 1G
count: 24
numa-node: 1
- size: 1G
count: 4810.4.14.3.2 cpu_models.yml #
---
product:
version: 2
cpu-models:
- name: COMPUTE-CPU
assignments:
- components:
- nova-compute-kvm
cpu:
- processor-ids: 3-5,12-17
role: vm
- components:
- openvswitch
cpu:
- processor-ids: 0
role: eal
- processor-ids: 1-2
role: pmd10.4.14.3.3 NUMA memory allocation #
As mentioned above, the memory used for hugepages is locked down at boot
time by an entry in /etc/default/grub. As an admin, you
can specify in the input model how to arrange this memory on NUMA nodes. It
can be spread across NUMA nodes or you can specify where you want it. For
example, if you have only one NIC, you would probably want all the hugepages
memory to be on the NUMA node closest to that NIC.
If you do not specify the numa-node settings in the
memory_models.yml input model file and use only the last
entry indicating "size: 1G" and "count: 48" then this memory is spread
evenly across all NUMA nodes.
Also note that the hugepage service runs once at boot time and then goes to an inactive state so you should not expect to see it running. If you decide to make changes to the NUMA memory allocation, you will need to reboot the compute node for the changes to take effect.
10.4.14.4 DPDK Setup for Networking #
10.4.14.4.1 Hardware requirements #
Intel-based compute node. DPDK is not available on AMD-based systems.
The following BIOS settings must be enabled for DL360 Gen9:
Virtualization Technology
Intel(R) VT-d
PCI-PT (Also see Section 10.4.15.14, “Enabling PCI-PT on HPE DL360 Gen 9 Servers”)
Need adequate host memory to allow for hugepages. The examples below use 1G hugepages for the VMs
10.4.14.4.2 Limitations #
DPDK is supported on SLES only.
Applies to SUSE OpenStack Cloud 9 only.
Tenant network can be untagged vlan or untagged vxlan
DPDK port names must be of the form 'dpdk<portid>' where port id is sequential and starts at 0
No support for converting DPDK ports to non DPDK ports without rebooting compute node.
No security group support, need userspace conntrack.
No jumbo frame support.
10.4.14.4.3 Setup instructions #
These setup instructions and example model are for a three-host system. There is one controller with Cloud Lifecycle Manager in cloud control plane and two compute hosts.
After initial run of site.yml all compute nodes must be rebooted to pick up changes in grub for hugepages and isolcpus
Changes to non-uniform memory access (NUMA) memory, isolcpu, or network devices must be followed by a reboot of compute nodes
Run sudo reboot to pick up libvirt change and hugepage/isocpus grub changes
tux >sudo rebootUse the bash script below to configure nova aggregates, neutron networks, a new flavor, etc. And then it will spin up two VMs.
VM spin-up instructions
Before running the spin up script you need to get a copy of the cirros image to your Cloud Lifecycle Manager node. You can manually scp a copy of the cirros image to the system. You can copy it locallly with wget like so
ardana > wget http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.imgSave the following shell script in the home directory and run it. This should spin up two VMs, one on each compute node.
Make sure to change all network-specific information in the script to match your environment.
#!/usr/bin/env bash
source service.osrc
######## register glance image
openstack image create --name='cirros' --container-format=bare --disk-format=qcow2 < ~/cirros-0.3.4-x86_64-disk.img
####### create nova aggregate and flavor for dpdk
MI_NAME=dpdk
openstack aggregate create $MI_NAME nova
openstack aggregate add host $MI_NAME openstack-cp-comp0001-mgmt
openstack aggregate add host $MI_NAME openstack-cp-comp0002-mgmt
openstack aggregate set $MI_NAME pinned=true
openstack flavor create $MI_NAME 6 1024 20 1
openstack flavor set $MI_NAME set hw:cpu_policy=dedicated
openstack flavor set $MI_NAME set aggregate_instance_extra_specs:pinned=true
openstack flavor set $MI_NAME set hw:mem_page_size=1048576
######## sec groups NOTE: no sec groups supported on DPDK. This is in case we do non-DPDK compute hosts.
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
######## nova keys
openstack keypair create mykey >mykey.pem
chmod 400 mykey.pem
######## create neutron external network
openstack network create ext-net --router:external --os-endpoint-type internalURL
openstack subnet create ext-net 10.231.0.0/19 --gateway_ip=10.231.0.1 --ip-version=4 --disable-dhcp --allocation-pool start=10.231.17.0,end=10.231.17.255
######## neutron network
openstack network create mynet1
openstack subnet create mynet1 10.1.1.0/24 --name mysubnet1
openstack router create myrouter1
openstack router add subnet myrouter1 mysubnet1
openstack router set myrouter1 ext-net
export MYNET=$(openstack network list | grep mynet | awk '{print $2}')
######## spin up 2 VMs, 1 on each compute
openstack server create --image cirros --nic net-id=${MYNET} --key-name mykey --flavor dpdk --availability-zone nova:openstack-cp-comp0001-mgmt vm1
openstack server create --image cirros --nic net-id=${MYNET} --key-name mykey --flavor dpdk --availability-zone nova:openstack-cp-comp0002-mgmt vm2
######## create floating ip and attach to instance
export MYFIP1=$(nova floating-ip-create|grep ext-net|awk '{print $4}')
nova add-floating-ip vm1 ${MYFIP1}
export MYFIP2=$(nova floating-ip-create|grep ext-net|awk '{print $4}')
nova add-floating-ip vm2 ${MYFIP2}
openstack server list10.4.14.5 DPDK Configurations #
10.4.14.5.1 Base configuration #
The following is specific to DL360 Gen9 and BIOS configuration as detailed in Section 10.4.14.4, “DPDK Setup for Networking”.
EAL cores - 1, isolate: False in cpu-models
PMD cores - 1 per NIC port
Hugepages - 1G per PMD thread
Memory channels - 4
Global rx queues - based on needs
10.4.14.5.2 Performance considerations common to all NIC types #
Compute host core frequency
Host CPUs should be running at maximum performance. The following is a script to set that. Note that in this case there are 24 cores. This needs to be modified to fit your environment. For a HP DL360 Gen9, the BIOS should be configured to use "OS Control Mode" which can be found on the iLO Power Settings page.
for i in `seq 0 23`; do echo "performance" > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done
IO non-posted prefetch
The DL360 Gen9 should have the IO non-posted prefetch disabled. Experimental evidence shows this yields an additional 6-8% performance boost.
10.4.14.5.3 Multiqueue configuration #
In order to use multiqueue, a property must be applied to the glance image and a setting inside the resulting VM must be applied. In this example we create a 4 vCPU flavor for DPDK using 1G hugepages.
MI_NAME=dpdk openstack aggregate create $MI_NAME nova openstack aggregate add host $MI_NAME openstack-cp-comp0001-mgmt openstack aggregate add host $MI_NAME openstack-cp-comp0002-mgmt openstack aggregate set $MI_NAME pinned=true openstack flavor create $MI_NAME 6 1024 20 4 openstack flavor set $MI_NAME set hw:cpu_policy=dedicated openstack flavor set $MI_NAME set aggregate_instance_extra_specs:pinned=true openstack flavor set $MI_NAME set hw:mem_page_size=1048576
And set the hw_vif_multiqueue_enabled property on the glance image
ardana > openstack image set --property hw_vif_multiqueue_enabled=true IMAGE UUIDOnce the VM is booted using the flavor above, inside the VM, choose the number of combined rx and tx queues to be equal to the number of vCPUs
tux > sudo ethtool -L eth0 combined 4On the hypervisor you can verify that multiqueue has been properly set by looking at the qemu process
-netdev type=vhost-user,id=hostnet0,chardev=charnet0,queues=4 -device virtio-net-pci,mq=on,vectors=10,
Here you can see that 'mq=on' and vectors=10. The formula for vectors is 2*num_queues+2
10.4.14.6 Troubleshooting DPDK #
10.4.14.6.1 Hardware configuration #
Because there are several variations of hardware, it is up to you to verify that the hardware is configured properly.
Only Intel based compute nodes are supported. There is no DPDK available for AMD-based CPUs.
PCI-PT must be enabled for the NIC that will be used with DPDK.
When using Intel Niantic and the igb_uio driver, the VT-d must be enabled in the BIOS.
For DL360 Gen9 systems, the BIOS shared-memory Section 10.4.15.14, “Enabling PCI-PT on HPE DL360 Gen 9 Servers”.
Adequate memory must be available for Section 10.4.14.3, “Configuring Hugepages for DPDK in Networks” usage.
Hyper-threading can be enabled but is not required for base functionality.
Determine the PCI slot that the DPDK NIC(s) are installed in to determine the associated NUMA node.
Only the Intel Haswell, Broadwell, and Skylake microarchitectures are supported. Intel Sandy Bridge is not supported.
10.4.14.6.2 System configuration #
Only SLES12-SP4 compute nodes are supported.
If a NIC port is used with PCI-PT, SRIOV-only, or PCI-PT+SRIOV, then it cannot be used with DPDK. They are mutually exclusive. This is because DPDK depends on an OvS bridge which does not exist if you use any combination of PCI-PT and SRIOV. You can use DPDK, SRIOV-only, and PCI-PT on difference interfaces of the same server.
There is an association between the PCI slot for the NIC and a NUMA node. Make sure to use logical CPU cores that are on the NUMA node associated to the NIC. Use the following to determine which CPUs are on which NUMA node.
ardana >lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz Stepping: 2 CPU MHz: 1200.000 CPU max MHz: 1800.0000 CPU min MHz: 1200.0000 BogoMIPS: 3597.06 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 30720K NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47
10.4.14.6.3 Input model configuration #
If you do not specify a driver for a DPDK device, the igb_uio will be selected as default.
DPDK devices must be named
dpdk<port-id>where the port-id starts at 0 and increments sequentially.Tenant networks supported are untagged VXLAN and VLAN.
Jumbo Frames MTU is not supported with DPDK.
Sample VXLAN model
Sample VLAN model
10.4.14.6.4 Reboot requirements #
A reboot of a compute node must be performed when an input model change causes the following:
After the initial
site.ymlplay on a new OpenStack environmentChanges to an existing OpenStack environment that modify the
/etc/default/grubfile, such ashugepage allocations
CPU isolation
iommu changes
Changes to a NIC port usage type, such as
moving from DPDK to any combination of PCI-PT and SRIOV
moving from DPDK to kernel based eth driver
10.4.15 SR-IOV and PCI Passthrough Support #
SUSE OpenStack Cloud supports both single-root I/O virtualization (SR-IOV) and PCI passthrough (PCIPT). Both technologies provide for better network performance.
This improves network I/O, decreases latency, and reduces processor overhead.
10.4.15.1 SR-IOV #
A PCI-SIG Single Root I/O Virtualization and Sharing (SR-IOV) Ethernet interface is a physical PCI Ethernet NIC that implements hardware-based virtualization mechanisms to expose multiple virtual network interfaces that can be used by one or more virtual machines simultaneously. With SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF).
When compared with a PCI Passthtrough Ethernet interface, an SR-IOV Ethernet interface:
Provides benefits similar to those of a PCI Passthtrough Ethernet interface, including lower latency packet processing.
Scales up more easily in a virtualized environment by providing multiple VFs that can be attached to multiple virtual machine interfaces.
Shares the same limitations, including the lack of support for LAG, QoS, ACL, and live migration.
Has the same requirements regarding the VLAN configuration of the access switches.
The process for configuring SR-IOV includes creating a VLAN provider network and subnet, then attaching VMs to that network.
With SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF)
10.4.15.2 PCI passthrough Ethernet interfaces #
A passthrough Ethernet interface is a physical PCI Ethernet NIC on a compute node to which a virtual machine is granted direct access. PCI passthrough allows a VM to have direct access to the hardware without being brokered by the hypervisor. This minimizes packet processing delays but at the same time demands special operational considerations. For all purposes, a PCI passthrough interface behaves as if it were physically attached to the virtual machine. Therefore any potential throughput limitations coming from the virtualized environment, such as the ones introduced by internal copying of data buffers, are eliminated. However, by bypassing the virtualized environment, the use of PCI passthrough Ethernet devices introduces several restrictions that must be taken into consideration. They include:
no support for LAG, QoS, ACL, or host interface monitoring
no support for live migration
no access to the compute node's OVS switch
A passthrough interface bypasses the compute node's OVS switch completely, and is attached instead directly to the provider network's access switch. Therefore, proper routing of traffic to connect the passthrough interface to a particular tenant network depends entirely on the VLAN tagging options configured on both the passthrough interface and the access port on the switch (TOR).
The access switch routes incoming traffic based on a VLAN ID, which ultimately determines the tenant network to which the traffic belongs. The VLAN ID is either explicit, as found in incoming tagged packets, or implicit, as defined by the access port's default VLAN ID when the incoming packets are untagged. In both cases the access switch must be configured to process the proper VLAN ID, which therefore has to be known in advance
10.4.15.3 Leveraging PCI Passthrough #
Two parts are necessary to leverage PCI passthrough on a SUSE OpenStack Cloud 9 Compute Node: preparing the Compute Node, preparing nova and glance.
Preparing the Compute Node
There should be no kernel drivers or binaries with direct access to the PCI device. If there are kernel modules, they should be blacklisted.
For example, it is common to have a
nouveaudriver from when the node was installed. This driver is a graphics driver for Nvidia-based GPUs. It must be blacklisted as shown in this example.ardana >echo 'blacklist nouveau' >> /etc/modprobe.d/nouveau-default.confThe file location and its contents are important; the name of the file is your choice. Other drivers can be blacklisted in the same manner, possibly including Nvidia drivers.
On the host,
iommu_groupsis necessary and may already be enabled. To check if IOMMU is enabled:root #virt-host-validate ..... QEMU: Checking if IOMMU is enabled by kernel : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments) .....To modify the kernel cmdline as suggested in the warning, edit the file
/etc/default/gruband appendintel_iommu=onto theGRUB_CMDLINE_LINUX_DEFAULTvariable. Then runupdate-bootloader.A reboot will be required for
iommu_groupsto be enabled.After the reboot, check that IOMMU is enabled:
root #virt-host-validate ..... QEMU: Checking if IOMMU is enabled by kernel : PASS .....Confirm IOMMU groups are available by finding the group associated with your PCI device (for example Nvidia GPU):
ardana >lspci -nn | grep -i nvidia 08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [NVS 300] [10de:10d8] (rev a2) 08:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)In this example,
08:00.0and08:00.1are addresses of the PCI device. The vendorID is10de. The productIDs are10d8and0be3.Confirm that the devices are available for passthrough:
ardana >ls -ld /sys/kernel/iommu_groups/*/devices/*08:00.?/ drwxr-xr-x 3 root root 0 Feb 14 13:05 /sys/kernel/iommu_groups/20/devices/0000:08:00.0/ drwxr-xr-x 3 root root 0 Feb 19 16:09 /sys/kernel/iommu_groups/20/devices/0000:08:00.1/NoteWith PCI passthrough, only an entire IOMMU group can be passed. Parts of the group cannot be passed. In this example, the IOMMU group is
20.
Preparing nova and glance for passthrough
Information about configuring nova and glance is available in the documentation at https://docs.openstack.org/nova/rocky/admin/pci-passthrough.html. Both
nova-computeandnova-schedulermust be configured.
10.4.15.4 Supported Intel 82599 Devices #
| Vendor | Device | Title |
|---|---|---|
| Intel Corporation | 10f8 | 82599 10 Gigabit Dual Port Backplane Connection |
| Intel Corporation | 10f9 | 82599 10 Gigabit Dual Port Network Connection |
| Intel Corporation | 10fb | 82599ES 10-Gigabit SFI/SFP+ Network Connection |
| Intel Corporation | 10fc | 82599 10 Gigabit Dual Port Network Connection |
10.4.15.5 SRIOV PCIPT configuration #
If you plan to take advantage of SR-IOV support in SUSE OpenStack Cloud, plan in advance to meet the following requirements:
Use one of the supported NIC cards:
HP Ethernet 10Gb 2-port 560FLR-SFP+ Adapter (Intel Niantic). Product part number: 665243-B21 -- Same part number for the following card options:
FlexLOM card
PCI slot adapter card
Identify the NIC ports to be used for PCI Passthrough devices and SRIOV devices from each compute node
Ensure that:
SRIOV is enabled in the BIOS
HP Shared memory is disabled in the BIOS on the compute nodes.
The Intel boot agent is disabled on the compute (Section 10.4.15.11, “Intel bootutils” can be used to perform this)
NoteBecause of Intel driver limitations, you cannot use a NIC port as an SRIOV NIC as well as a physical NIC. Using the physical function to carry the normal tenant traffic through the OVS bridge at the same time as assigning the VFs from the same NIC device as passthrough to the guest VM is not supported.
If the above prerequisites are met, then SR-IOV or PCIPT can be reconfigured at any time. There is no need to do it at install time.
10.4.15.6 Deployment use cases #
The following are typical use cases that should cover your particular needs:
A device on the host needs to be enabled for both PCI-passthrough and PCI-SRIOV during deployment. At run time nova decides whether to use physical functions or virtual function depending on vnic_type of the port used for booting the VM.
A device on the host needs to be configured only for PCI-passthrough.
A device on the host needs to be configured only for PCI-SRIOV virtual functions.
10.4.15.7 Input model updates #
SUSE OpenStack Cloud 9 provides various options for the user to configure the network for tenant VMs. These options have been enhanced to support SRIOV and PCIPT.
the Cloud Lifecycle Manager input model changes to support SRIOV and PCIPT are as follows. If you were familiar with the configuration settings previously, you will notice these changes.
net_interfaces.yml: This file defines the interface details of the nodes. In it, the following fields have been added under the compute node interface section:
| Key | Value |
|---|---|
| sriov_only: |
Indicates that only SR-IOV be enabled on the interface. This should be set to true if you want to dedicate the NIC interface to support only SR-IOV functionality. |
| pci-pt: |
When this value is set to true, it indicates that PCIPT should be enabled on the interface. |
| vf-count: |
Indicates the number of VFs to be configured on a given interface. |
In control_plane.yml under Compute
resource, neutron-sriov-nic-agent has been
added as a service component.
under resources:
| Key | Value |
|---|---|
| name: | Compute |
| resource-prefix: | Comp |
| server-role: | COMPUTE-ROLE |
| allocation-policy: | Any |
| min-count: | 0 |
| service-components: | ntp-client |
| nova-compute | |
| nova-compute-kvm | |
| neutron-l3-agent | |
| neutron-metadata-agent | |
| neutron-openvswitch-agent | |
| - neutron-sriov-nic-agent* |
nic_device_data.yml: This is the new file
added with this release to support SRIOV and PCIPT configuration details. It
contains information about the specifics of a nic, and is found at
/usr/share/ardana/input-model/2.0/services/osconfig/nic_device_data.yml.
The fields in this file are as follows.
nic-device-types: The nic-device-types section contains the following key-value pairs:
Key Value name: The name of the nic-device-types that will be referenced in nic_mappings.yml
family: The name of the nic-device-families to be used with this nic_device_type
device_id: Device ID as specified by the vendor for the particular NIC
type: The value of this field can be
simple-portormulti-port. If a single bus address is assigned to more than one nic, The value will bemulti-port. If there is a one-to-one mapping between bus address and the nic, it will besimple-port.nic-device-families: The nic-device-families section contains the following key-value pairs:
Key Value name: The name of the device family that can be used for reference in nic-device-types.
vendor-id: Vendor ID of the NIC
config-script: A script file used to create the virtual functions (VF) on the Compute node.
driver: Indicates the NIC driver that needs to be used.
vf-count-type: This value can be either
portordriver.“port”: Indicates that the device supports per-port virtual function (VF) counts.
“driver:” Indicates that all ports using the same driver will be configured with the same number of VFs, whether or not the interface model specifies a vf-count attribute for the port. If two or more ports specify different vf-count values, the config processor errors out.
Max-vf-count: This field indicates the maximum VFs that can be configured on an interface as defined by the vendor.
control_plane.yml: This file provides the
information about the services to be run on a particular node. To support
SR-IOV on a particular compute node, you must run
neutron-sriov-nic-agent on that node.
Mapping the use cases with various fields in input model
| Vf-count | SR-IOV | PCIPT | OVS bridge | Can be NIC bonded | Use case | |
|---|---|---|---|---|---|---|
| sriov-only: true | Mandatory | Yes | No | No | No | Dedicated to SRIOV |
| pci-pt : true | Not Specified | No | Yes | No | No | Dedicated to PCI-PT |
| pci-pt : true | Specified | Yes | Yes | No | No | PCI-PT or SRIOV |
| pci-pt and sriov-only keywords are not specified | Specified | Yes | No | Yes | No | SRIOV with PF used by host |
| pci-pt and sriov-only keywords are not specified | Not Specified | No | No | Yes | Yes | Traditional/Usual use case |
10.4.15.8 Mappings between nic_mappings.yml and net_interfaces.yml #
The following diagram shows which fields in
nic_mappings.yml map to corresponding fields in
net_interfaces.yml:
10.4.15.9 Example Use Cases for Intel #
Nic-device-types and nic-device-families with Intel 82559 with ixgbe as the driver.
nic-device-types: - name: ''8086:10fb family: INTEL-82599 device-id: '10fb' type: simple-port nic-device-families: # Niantic - name: INTEL-82599 vendor-id: '8086' config-script: intel-82599.sh driver: ixgbe vf-count-type: port max-vf-count: 63net_interfaces.yml for the SRIOV-only use case:
- name: COMPUTE-INTERFACES - name: hed1 device: name: hed1 sriov-only: true vf-count: 6 network-groups: - GUEST1net_interfaces.yml for the PCIPT-only use case:
- name: COMPUTE-INTERFACES - name: hed1 device: name: hed1 pci-pt: true network-groups: - GUEST1net_interfaces.yml for the SRIOV and PCIPT use case
- name: COMPUTE-INTERFACES - name: hed1 device: name: hed1 pci-pt: true vf-count: 6 network-groups: - GUEST1net_interfaces.yml for SRIOV and Normal Virtio use case
- name: COMPUTE-INTERFACES - name: hed1 device: name: hed1 vf-count: 6 network-groups: - GUEST1net_interfaces.yml for PCI-PT (
hed1andhed4refer to the DUAL ports of the PCI-PT NIC)- name: COMPUTE-PCI-INTERFACES network-interfaces: - name: hed3 device: name: hed3 network-groups: - MANAGEMENT - EXTERNAL-VM forced-network-groups: - EXTERNAL-API - name: hed1 device: name: hed1 pci-pt: true network-groups: - GUEST - name: hed4 device: name: hed4 pci-pt: true network-groups: - GUEST
10.4.15.10 Launching Virtual Machines #
Provisioning a VM with SR-IOV NIC is a two-step process.
Create a neutron port with
vnic_type = direct.ardana >openstack port create --network $net_id --vnic-type direct sriov_portBoot a VM with the created
port-id.ardana >openstack server create --flavor m1.large --image opensuse --nic port-id=$port_id test-sriov
Provisioning a VM with PCI-PT NIC is a two-step process.
Create two neutron ports with
vnic_type = direct-physical.ardana >openstack port create --network net1 --vnic-type direct-physical pci-port1ardana >openstack port create --network net1 --vnic-type direct-physical pci-port2Boot a VM with the created ports.
ardana >openstack server create --flavor 4 --image opensuse --nic port-id pci-port1-port-id \ --nic port-id pci-port2-port-id vm1-pci-passthrough
If PCI-PT VM gets stuck (hangs) at boot time when using an Intel NIC, the boot agent should be disabled.
10.4.15.11 Intel bootutils #
When Intel cards are used for PCI-PT, a tenant VM can get stuck at boot time. When this happens, you should download Intel bootutils and use it to should disable bootagent.
Download
Preboot.tar.gzfrom https://downloadcenter.intel.com/download/19186/Intel-Ethernet-Connections-Boot-Utility-Preboot-Images-and-EFI-DriversUntar the
Preboot.tar.gzon the compute node where the PCI-PT VM is to be hosted.Go to
~/APPS/BootUtil/Linux_x64ardana >cd ~/APPS/BootUtil/Linux_x64and run following command:
ardana >./bootutil64e -BOOTENABLE disable -allBoot the PCI-PT VM; it should boot without getting stuck.
NoteEven though VM console shows VM getting stuck at PXE boot, it is not related to BIOS PXE settings.
10.4.15.12 Making input model changes and implementing PCI PT and SR-IOV #
To implement the configuration you require, log into the Cloud Lifecycle Manager node and update the Cloud Lifecycle Manager model files to enable SR-IOV or PCIPT following the relevant use case explained above. You will need to edit the following:
net_interfaces.ymlnic_device_data.ymlcontrol_plane.yml
To make the edits,
Check out the site branch of the local git repository and change to the correct directory:
ardana >git checkout siteardana >cd ~/openstack/my_cloud/definition/data/Open each file in vim or another editor and make the necessary changes. Save each file, then commit to the local git repository:
ardana >git add -Aardana >git commit -m "your commit message goes here in quotes"Have the Cloud Lifecycle Manager enable your changes by running the necessary playbooks:
ardana >cd ~/openstack/ardana/ansibleardana >ansible-playbook -i hosts/localhost config-processor-run.ymlardana >ansible-playbook -i hosts/localhost ready-deployment.ymlardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts site.yml
After running the site.yml playbook above, you must
reboot the compute nodes that are configured with Intel PCI devices.
When a VM is running on an SRIOV port on a given compute node, reconfiguration is not supported.
You can set the number of virtual functions that must be enabled on a compute node at install time. You can update the number of virtual functions after deployment. If any VMs have been spawned before you change the number of virtual functions, those VMs may lose connectivity. Therefore, it is always recommended that if any virtual function is used by any tenant VM, you should not reconfigure the virtual functions. Instead, you should delete/migrate all the VMs on that NIC before reconfiguring the number of virtual functions.
10.4.15.13 Limitations #
Security groups are not applicable for PCI-PT and SRIOV ports.
Live migration is not supported for VMs with PCI-PT and SRIOV ports.
Rate limiting (QoS) is not applicable on SRIOV and PCI-PT ports.
SRIOV/PCIPT is not supported for VxLAN network.
DVR is not supported with SRIOV/PCIPT.
For Intel cards, the same NIC cannot be used for both SRIOV and normal VM boot.
Current upstream OpenStack code does not support this hot plugin of SRIOV/PCIPT interface using the nova
attach_interfacecommand. See https://review.openstack.org/#/c/139910/ for more information.The
openstack port updatecommand will not work when admin state is down.SLES Compute Nodes with dual-port PCI-PT NICs, both ports should always be passed in the VM. It is not possible to split the dual port and pass through just a single port.
10.4.15.14 Enabling PCI-PT on HPE DL360 Gen 9 Servers #
The HPE DL360 Gen 9 and HPE ProLiant systems with Intel processors use a region of system memory for sideband communication of management information. The BIOS sets up Reserved Memory Region Reporting (RMRR) to report these memory regions and devices to the operating system. There is a conflict between the Linux kernel and RMRR which causes problems with PCI pass-through (PCI-PT). This is needed for IOMMU use by DPDK. Note that this does not affect SR-IOV.
In order to enable PCI-PT on the HPE DL360 Gen 9 you must have a version of firmware that supports setting this and you must change a BIOS setting.
To begin, get the latest firmware and install it on your compute nodes.
Once the firmware has been updated:
Reboot the server and press F9 (system utilities) during POST (power on self test)
Choose
Select the NIC for which you want to enable PCI-PT
Choose
Disable the shared memory feature in the BIOS.
Save the changes and reboot server
10.4.16 Setting up VLAN-Aware VMs #
Creating a VM with a trunk port will allow a VM to gain connectivity to one or more networks over the same virtual NIC (vNIC) through the use VLAN interfaces in the guest VM. Connectivity to different networks can be added and removed dynamically through the use of subports. The network of the parent port will be presented to the VM as the untagged VLAN, and the networks of the child ports will be presented to the VM as the tagged VLANs (the VIDs of which can be chosen arbitrarily as long as they are unique to that trunk). The VM will send/receive VLAN-tagged traffic over the subports, and neutron will mux/demux the traffic onto the subport's corresponding network. This is not to be confused with VLAN transparency where a VM can pass VLAN-tagged traffic transparently across the network without interference from neutron. VLAN transparency is not supported.
10.4.16.1 Terminology #
Trunk: a resource that logically represents a trunked vNIC and references a parent port.
Parent port: a neutron port that a Trunk is referenced to. Its network is presented as the untagged VLAN.
Subport: a resource that logically represents a tagged VLAN port on a Trunk. A Subport references a child port and consists of the <port>,<segmentation-type>,<segmentation-id> tuple. Currently only the
vlansegmentation type is supported.Child port: a neutron port that a Subport is referenced to. Its network is presented as a tagged VLAN based upon the segmentation-id used when creating/adding a Subport.
Legacy VM: a VM that does not use a trunk port.
Legacy port: a neutron port that is not used in a Trunk.
VLAN-aware VM: a VM that uses at least one trunk port.
10.4.16.2 Trunk CLI reference #
| Command | Action |
|---|---|
| network trunk create | Create a trunk. |
| network trunk delete | Delete a given trunk. |
| network trunk list | List all trunks. |
| network trunk show | Show information of a given trunk. |
| network trunk set | Add subports to a given trunk. |
| network subport list | List all subports for a given trunk. |
| network trunk unset | Remove subports from a given trunk. |
| network trunk set | Update trunk properties. |
10.4.16.3 Enabling VLAN-aware VM capability #
Edit
~/openstack/my_cloud/config/neutron/neutron.conf.j2to add thetrunkservice_plugin:service_plugins = {{ neutron_service_plugins }},trunkEdit
~/openstack/my_cloud/config/neutron/ml2_conf.ini.j2to enable the noop firewall driver:[securitygroup] firewall_driver = neutron.agent.firewall.NoopFirewallDriver
NoteThis is a manual configuration step because it must be made apparent that this step disables neutron security groups completely. The default SUSE OpenStack Cloud firewall_driver is
neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewall Driverwhich does not implement security groups for trunk ports. Optionally, the SUSE OpenStack Cloud default firewall_driver may still be used (this step can be skipped), which would provide security groups for legacy VMs but not for VLAN-aware VMs. However, this mixed environment is not recommended. For more information, see Section 10.4.16.6, “Firewall issues”.Commit the configuration changes:
ardana >git add -Aardana >git commit -m "Enable vlan-aware VMs"ardana >cd ~/openstack/ardana/ansible/If this is an initial deployment, continue the rest of normal deployment process:
ardana >ansible-playbook -i hosts/localhost config-processor-run.ymlardana >ansible-playbook -i hosts/localhost ready-deployment.ymlardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts site.ymlIf the cloud has already been deployed and this is a reconfiguration:
ardana >ansible-playbook -i hosts/localhost ready-deployment.ymlardana >cd ~/scratch/ansible/next/ardana/ansibleardana >ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml
10.4.16.4 Use Cases #
Creating a trunk port
Assume that a number of neutron networks/subnets already exist: private, foo-net, and bar-net. This will create a trunk with two subports allocated to it. The parent port will be on the "private" network, while the two child ports will be on "foo-net" and "bar-net", respectively:
Create a port that will function as the trunk's parent port:
ardana >openstack port create --name trunkparent privateCreate ports that will function as the child ports to be used in subports:
ardana >openstack port create --name subport1 foo-netardana >openstack port create --name subport2 bar-netCreate a trunk port using the
openstack network trunk createcommand, passing the parent port created in step 1 and child ports created in step 2:ardana >openstack network trunk create --parent-port trunkparent --subport port=subport1,segmentation-type=vlan,segmentation-id=1 --subport port=subport2,segmentation-type=vlan,segmentation-id=2 mytrunk +-----------------+-----------------------------------------------------------------------------------------------+ | Field | Value | +-----------------+-----------------------------------------------------------------------------------------------+ | admin_state_up | UP | | created_at | 2017-06-02T21:49:59Z | | description | | | id | bd822ebd-33d5-423e-8731-dfe16dcebac2 | | name | mytrunk | | port_id | 239f8807-be2e-4732-9de6-c64519f46358 | | project_id | f51610e1ac8941a9a0d08940f11ed9b9 | | revision_number | 1 | | status | DOWN | | sub_ports | port_id='9d25abcf-d8a4-4272-9436-75735d2d39dc', segmentation_id='1', segmentation_type='vlan' | | | port_id='e3c38cb2-0567-4501-9602-c7a78300461e', segmentation_id='2', segmentation_type='vlan' | | tenant_id | f51610e1ac8941a9a0d08940f11ed9b9 | | updated_at | 2017-06-02T21:49:59Z | +-----------------+-----------------------------------------------------------------------------------------------+ $ openstack network subport list --trunk mytrunk +--------------------------------------+-------------------+-----------------+ | Port | Segmentation Type | Segmentation ID | +--------------------------------------+-------------------+-----------------+ | 9d25abcf-d8a4-4272-9436-75735d2d39dc | vlan | 1 | | e3c38cb2-0567-4501-9602-c7a78300461e | vlan | 2 | +--------------------------------------+-------------------+-----------------+Optionally, a trunk may be created without subports (they can be added later):
ardana >openstack network trunk create --parent-port trunkparent mytrunk +-----------------+--------------------------------------+ | Field | Value | +-----------------+--------------------------------------+ | admin_state_up | UP | | created_at | 2017-06-02T21:45:35Z | | description | | | id | eb8a3c7d-9f0a-42db-b26a-ca15c2b38e6e | | name | mytrunk | | port_id | 239f8807-be2e-4732-9de6-c64519f46358 | | project_id | f51610e1ac8941a9a0d08940f11ed9b9 | | revision_number | 1 | | status | DOWN | | sub_ports | | | tenant_id | f51610e1ac8941a9a0d08940f11ed9b9 | | updated_at | 2017-06-02T21:45:35Z | +-----------------+--------------------------------------+A port that is already bound (that is, already in use by a VM) cannot be upgraded to a trunk port. The port must be unbound to be eligible for use as a trunk's parent port. When adding subports to a trunk, the child ports must be unbound as well.
Checking a port's trunk details
Once a trunk has been created, its parent port will show the
trunk_details attribute, which consists of the
trunk_id and list of subport dictionaries:
ardana > openstack port show -F trunk_details trunkparent
+---------------+-------------------------------------------------------------------------------------+
| Field | Value |
+---------------+-------------------------------------------------------------------------------------+
| trunk_details | {"trunk_id": "bd822ebd-33d5-423e-8731-dfe16dcebac2", "sub_ports": |
| | [{"segmentation_id": 2, "port_id": "e3c38cb2-0567-4501-9602-c7a78300461e", |
| | "segmentation_type": "vlan", "mac_address": "fa:16:3e:11:90:d2"}, |
| | {"segmentation_id": 1, "port_id": "9d25abcf-d8a4-4272-9436-75735d2d39dc", |
| | "segmentation_type": "vlan", "mac_address": "fa:16:3e:ff:de:73"}]} |
+---------------+-------------------------------------------------------------------------------------+
Ports that are not trunk parent ports will not have a
trunk_details field:
ardana > openstack port show -F trunk_details subport1
need more than 0 values to unpackAdding subports to a trunk
Assuming a trunk and new child port have been created already, the
trunk-subport-add command will add one or more subports
to the trunk.
Run
openstack network trunk setardana >openstack network trunk set --subport port=subport3,segmentation-type=vlan,segmentation-id=3 mytrunkRun
openstack network subport listardana >openstack network subport list --trunk mytrunk +--------------------------------------+-------------------+-----------------+ | Port | Segmentation Type | Segmentation ID | +--------------------------------------+-------------------+-----------------+ | 9d25abcf-d8a4-4272-9436-75735d2d39dc | vlan | 1 | | e3c38cb2-0567-4501-9602-c7a78300461e | vlan | 2 | | bf958742-dbf9-467f-b889-9f8f2d6414ad | vlan | 3 | +--------------------------------------+-------------------+-----------------+
The --subport option may be repeated multiple times in
order to add multiple subports at a time.
Removing subports from a trunk
To remove a subport from a trunk, use openstack network trunk
unset command:
ardana > openstack network trunk unset --subport subport3 mytrunkDeleting a trunk port
To delete a trunk port, use the openstack network trunk
delete command:
ardana > openstack network trunk delete mytrunk
Once a trunk has been created successfully, its parent port may be passed to
the openstack server create command, which will make the VM VLAN-aware:
ardana > openstack server create --image ubuntu-server --flavor 1 --nic port-id=239f8807-be2e-4732-9de6-c64519f46358 vlan-aware-vmA trunk cannot be deleted until its parent port is unbound. This means you must delete the VM using the trunk port before you are allowed to delete the trunk.
10.4.16.5 VLAN-aware VM network configuration #
This section illustrates how to configure the VLAN interfaces inside a VLAN-aware VM based upon the subports allocated to the trunk port being used.
Run
openstack network trunk subport listto see the VLAN IDs in use on the trunk port:ardana >openstack network subport list --trunk mytrunk +--------------------------------------+-------------------+-----------------+ | Port | Segmentation Type | Segmentation ID | +--------------------------------------+-------------------+-----------------+ | e3c38cb2-0567-4501-9602-c7a78300461e | vlan | 2 | +--------------------------------------+-------------------+-----------------+Run
openstack port showon the child port to get its mac_address:ardana >openstack port show -F mac_address 08848e38-50e6-4d22-900c-b21b07886fb7 +-------------+-------------------+ | Field | Value | +-------------+-------------------+ | mac_address | fa:16:3e:08:24:61 | +-------------+-------------------+Log into the VLAN-aware VM and run the following commands to set up the VLAN interface:
ardana >sudo ip link add link ens3 ens3.2 address fa:16:3e:11:90:d2 broadcast ff:ff:ff:ff:ff:ff type vlan id 2 $ sudo ip link set dev ens3.2 upNote the usage of the mac_address from step 2 and VLAN ID from step 1 in configuring the VLAN interface:
ardana >sudo ip link add link ens3 ens3.2 address fa:16:3e:11:90:d2 broadcast ff:ff:ff:ff:ff:ff type vlan id 2Trigger a DHCP request for the new vlan interface to verify connectivity and retrieve its IP address. On an Ubuntu VM, this might be:
ardana >sudo dhclient ens3.2ardana >sudo ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:8d:77:39 brd ff:ff:ff:ff:ff:ff inet 10.10.10.5/24 brd 10.10.10.255 scope global ens3 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe8d:7739/64 scope link valid_lft forever preferred_lft forever 3: ens3.2@ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:11:90:d2 brd ff:ff:ff:ff:ff:ff inet 10.10.12.7/24 brd 10.10.12.255 scope global ens3.2 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe11:90d2/64 scope link valid_lft forever preferred_lft forever
10.4.16.6 Firewall issues #
The SUSE OpenStack Cloud default firewall_driver is
neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver.
This default does not implement security groups for VLAN-aware VMs, but it
does implement security groups for legacy VMs. For this reason, it is
recommended to disable neutron security groups altogether when using
VLAN-aware VMs. To do so, set:
firewall_driver = neutron.agent.firewall.NoopFirewallDriver
Doing this will prevent having a mix of firewalled and non-firewalled VMs in the same environment, but it should be done with caution because all VMs would be non-firewalled.
10.5 Creating a Highly Available Router #
10.5.1 CVR and DVR High Available Routers #
CVR (Centralized Virtual Routing) and DVR (Distributed Virtual Routing) are two types of technologies which can be used to provide routing processes in SUSE OpenStack Cloud 9. You can create Highly Available (HA) versions of CVR and DVR routers by using the options in the table below when creating your router.
The neutron command for creating a router openstack router create
router_name --distributed=True|False --ha=True|False requires
administrative permissions. See the example in the next section, Section 10.5.2, “Creating a High Availability Router”.
| --distributed | --ha | Router Type | Description |
|---|---|---|---|
| False | False | CVR | Centralized Virtual Router |
| False | True | CVRHA | Centralized Virtual Router with L3 High Availablity |
| True | False | DVR | Distributed Virtual Router without SNAT High Availability |
| True | True | DVRHA | Distributed Virtual Router with SNAT High Availability |
10.5.2 Creating a High Availability Router #
You can create a highly available router using the OpenStackClient.
To create the HA router, add
--ha=Trueto theopenstack router createcommand. If you also want to make the router distributed, add--distributed=True. In this example, a DVR SNAT HA router is created with the namerouterHA.ardana >openstack router create routerHA --distributed=True --ha=TrueSet the gateway for the external network and add interface
ardana >openstack router set routerHA <ext-net-id>ardana >openstack router add subnet routerHA <private_subnet_id>When the router is created, the gateway is set, and the interface attached, you have a router with high availability.
10.5.3 Test Router for High Availability #
You can demonstrate that the router is HA by running a continuous ping from a VM instance that is running on the private network to an external server such as a public DNS. As the ping is running, list the l3 agents hosting the router and identify the agent that is responsible for hosting the active router. Induce the failover mechanism by creating a catastrophic event such as shutting down node hosting the l3 agent. Once the node is shut down, you will see that the ping from the VM to the external network continues to run as the backup l3 agent takes over. To verify the agent hosting the primary router has changed, list the agents hosting the router. You will see a different agent is now hosting the active router.
Boot an instance on the private network
ardana >openstack server create --image <image_id> --flavor <flavor_id> --nic net_id=<private_net_id> --key_name <key> VM1Log into the VM using the SSH keys
ardana >ssh -i <key> <ipaddress of VM1>Start a
pingto X.X.X.X. While pinging, make sure there is no packet loss and leave the ping running.ardana >ping X.X.X.XCheck which agent is hosting the active router.
ardana >openstack network agent list –routers <router_id>Shutdown the node hosting the agent.
Within 10 seconds, check again to see which L3 agent is hosting the active router.
ardana >openstack network agent list –routers <router_id>You will see a different agent.