Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE OpenStack Cloud 8

9 Managing Networking Edit source

Information about managing and configuring the Networking service.

9.1 Configuring the SUSE OpenStack Cloud Firewall Edit source

The following instructions provide information about how to identify and modify the overall SUSE OpenStack Cloud firewall that is configured in front of the control services. This firewall is administered only by a cloud admin and is not available for tenant use for private network firewall services.

During the installation process, the configuration processor will automatically generate "allow" firewall rules for each server based on the services deployed and block all other ports. These are populated in ~/openstack/my_cloud/info/firewall_info.yml, which includes a list of all the ports by network, including the addresses on which the ports will be opened. This is described in more detail in Book “Planning an Installation with Cloud Lifecycle Manager”, Chapter 5 “Input Model”, Section 5.2 “Concepts”, Section 5.2.10 “Networking”, Section 5.2.10.5 “Firewall Configuration”.

The firewall_rules.yml file in the input model allows you to define additional rules for each network group. You can read more about this in Book “Planning an Installation with Cloud Lifecycle Manager”, Chapter 6 “Configuration Objects”, Section 6.15 “Firewall Rules”.

The purpose of this document is to show you how to make post-installation changes to the firewall rules if the need arises.

Important
Important

This process is not to be confused with Firewall-as-a-Service (see Book “User Guide”, Chapter 14 “Using Firewall as a Service (FWaaS)”), which is a separate service that enables the ability for SUSE OpenStack Cloud tenants to create north-south, network-level firewalls to provide stateful protection to all instances in a private, tenant network. This service is optional and is tenant-configured.

9.1.1 Making Changes to the Firewall Rules Edit source

  1. Log in to your Cloud Lifecycle Manager.

  2. Edit your ~/openstack/my_cloud/definition/data/firewall_rules.yml file and add the lines necessary to allow the port(s) needed through the firewall.

    In this example we are going to open up port range 5900-5905 to allow VNC traffic through the firewall:

      - name: VNC
        network-groups:
      - MANAGEMENT
        rules:
         - type: allow
           remote-ip-prefix:  0.0.0.0/0
           port-range-min: 5900
           port-range-max: 5905
           protocol: tcp
    Note
    Note

    The example above shows a remote-ip-prefix of 0.0.0.0/0 which opens the ports up to all IP ranges. To be more secure you can specify your local IP address CIDR you will be running the VNC connect from.

  3. Commit those changes to your local git:

    ardana > cd ~/openstack/ardana/ansible
    ardana > git add -A
    ardana > git commit -m "firewall rule update"
  4. Run the configuration processor:

    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
  5. Create the deployment directory structure:

    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
  6. Change to the deployment directory and run the osconfig-iptables-deploy.yml playbook to update your iptable rules to allow VNC:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts osconfig-iptables-deploy.yml

You can repeat these steps as needed to add, remove, or edit any of these firewall rules.

9.2 DNS Service Overview Edit source

SUSE OpenStack Cloud DNS service provides multi-tenant Domain Name Service with REST API management for domain and records.

Warning
Warning

The DNS Service is not intended to be used as an internal or private DNS service. The name records in DNSaaS should be treated as public information that anyone could query. There are controls to prevent tenants from creating records for domains they do not own. TSIG provides a Transaction SIG nature to ensure integrity during zone transfer to other DNS servers.

9.2.1 For More Information Edit source

9.2.2 Designate Initial Configuration Edit source

After the SUSE OpenStack Cloud installation has been completed, Designate requires initial configuration to operate.

9.2.2.1 Identifying Name Server Public IPs Edit source

Depending on the back-end, the method used to identify the name servers' public IPs will differ.

9.2.2.1.1 InfoBlox Edit source

InfoBlox will act as your public name servers, consult the InfoBlox management UI to identify the IPs.

9.2.2.1.2 PowerDNS or BIND Back-end Edit source

You can find the name server IPs in /etc/hosts by looking for the ext-api addresses, which are the addresses of the controllers. For example:

192.168.10.1 example-cp1-c1-m1-extapi
192.168.10.2 example-cp1-c1-m2-extapi
192.168.10.3 example-cp1-c1-m3-extapi
9.2.2.1.3 Creating Name Server A Records Edit source

Each name server requires a public name, for example ns1.example.com., to which Designate-managed domains will be delegated. There are two common locations where these may be registered, either within a zone hosted on Designate itself, or within a zone hosted on a external DNS service.

If you are using an externally managed zone for these names:

  • For each name server public IP, create the necessary A records in the external system.

If you are using a Designate-managed zone for these names:

  1. Create the zone in Designate which will contain the records:

    ardana > openstack zone create --email hostmaster@example.com example.com.
    +----------------+--------------------------------------+
    | Field          | Value                                |
    +----------------+--------------------------------------+
    | action         | CREATE                               |
    | created_at     | 2016-03-09T13:16:41.000000           |
    | description    | None                                 |
    | email          | hostmaster@example.com               |
    | id             | 23501581-7e34-4b88-94f4-ad8cec1f4387 |
    | masters        |                                      |
    | name           | example.com.                         |
    | pool_id        | 794ccc2c-d751-44fe-b57f-8894c9f5c842 |
    | project_id     | a194d740818942a8bea6f3674e0a3d71     |
    | serial         | 1457529400                           |
    | status         | PENDING                              |
    | transferred_at | None                                 |
    | ttl            | 3600                                 |
    | type           | PRIMARY                              |
    | updated_at     | None                                 |
    | version        | 1                                    |
    +----------------+--------------------------------------+
  2. For each name server public IP, create an A record. For example:

    ardana > openstack recordset create --records 192.168.10.1 --type A example.com. ns1.example.com.
    +-------------+--------------------------------------+
    | Field       | Value                                |
    +-------------+--------------------------------------+
    | action      | CREATE                               |
    | created_at  | 2016-03-09T13:18:36.000000           |
    | description | None                                 |
    | id          | 09e962ed-6915-441a-a5a1-e8d93c3239b6 |
    | name        | ns1.example.com.                     |
    | records     | 192.168.10.1                         |
    | status      | PENDING                              |
    | ttl         | None                                 |
    | type        | A                                    |
    | updated_at  | None                                 |
    | version     | 1                                    |
    | zone_id     | 23501581-7e34-4b88-94f4-ad8cec1f4387 |
    +-------------+--------------------------------------+
  3. When records have been added, list the record sets in the zone to validate:

    ardana > openstack recordset list example.com.
    +--------------+------------------+------+---------------------------------------------------+
    | id           | name             | type | records                                           |
    +--------------+------------------+------+---------------------------------------------------+
    | 2d6cf...655b | example.com.     | SOA  | ns1.example.com. hostmaster.example.com 145...600 |
    | 33466...bd9c | example.com.     | NS   | ns1.example.com.                                  |
    | da98c...bc2f | example.com.     | NS   | ns2.example.com.                                  |
    | 672ee...74dd | example.com.     | NS   | ns3.example.com.                                  |
    | 09e96...39b6 | ns1.example.com. | A    | 192.168.10.1                                      |
    | bca4f...a752 | ns2.example.com. | A    | 192.168.10.2                                      |
    | 0f123...2117 | ns3.example.com. | A    | 192.168.10.3                                      |
    +--------------+------------------+------+---------------------------------------------------+
  4. Contact your domain registrar requesting Glue Records to be registered in the com. zone for the nameserver and public IP address pairs above. If you are using a sub-zone of an existing company zone (for example, ns1.cloud.mycompany.com.), the Glue must be placed in the mycompany.com. zone.

9.2.2.1.4 For More Information Edit source

For additional DNS integration and configuration information, see the OpenStack Designate documentation at https://docs.openstack.org/designate/pike/index.html.

For more information on creating servers, domains and examples, see the OpenStack REST API documentation at https://developer.openstack.org/api-ref/dns/.

9.2.3 DNS Service Monitoring Support Edit source

9.2.3.1 DNS Service Monitoring Support Edit source

Additional monitoring support for the DNS Service (Designate) has been added to SUSE OpenStack Cloud.

In the Networking section of the Operations Console, you can see alarms for all of the DNS Services (Designate), such as designate-zone-manager, designate-api, designate-pool-manager, designate-mdns, and designate-central after running designate-stop.yml.

You can run designate-start.yml to start the DNS Services back up and the alarms will change from a red status to green and be removed from the New Alarms panel of the Operations Console.

An example of the generated alarms from the Operations Console is provided below after running designate-stop.yml:

ALARM:  STATE:  ALARM ID:  LAST CHECK:  DIMENSION:
Process Check
0f221056-1b0e-4507-9a28-2e42561fac3e 2016-10-03T10:06:32.106Z hostname=ardana-cp1-c1-m1-mgmt,
service=dns,
cluster=cluster1,
process_name=designate-zone-manager,
component=designate-zone-manager,
control_plane=control-plane-1,
cloud_name=entry-scale-kvm

Process Check
50dc4c7b-6fae-416c-9388-6194d2cfc837 2016-10-03T10:04:32.086Z hostname=ardana-cp1-c1-m1-mgmt,
service=dns,
cluster=cluster1,
process_name=designate-api,
component=designate-api,
control_plane=control-plane-1,
cloud_name=entry-scale-kvm

Process Check
55cf49cd-1189-4d07-aaf4-09ed08463044 2016-10-03T10:05:32.109Z hostname=ardana-cp1-c1-m1-mgmt,
service=dns,
cluster=cluster1,
process_name=designate-pool-manager,
component=designate-pool-manager,
control_plane=control-plane-1,
cloud_name=entry-scale-kvm

Process Check
c4ab7a2e-19d7-4eb2-a9e9-26d3b14465ea 2016-10-03T10:06:32.105Z hostname=ardana-cp1-c1-m1-mgmt,
service=dns,
cluster=cluster1,
process_name=designate-mdns,
component=designate-mdns,
control_plane=control-plane-1,
cloud_name=entry-scale-kvm
HTTP Status
c6349bbf-4fd1-461a-9932-434169b86ce5 2016-10-03T10:05:01.731Z service=dns,
cluster=cluster1,
url=http://100.60.90.3:9001/,
hostname=ardana-cp1-c1-m3-mgmt,
component=designate-api,
control_plane=control-plane-1,
api_endpoint=internal,
cloud_name=entry-scale-kvm,
monitored_host_type=instance

Process Check
ec2c32c8-3b91-4656-be70-27ff0c271c89 2016-10-03T10:04:32.082Z hostname=ardana-cp1-c1-m1-mgmt,
service=dns,
cluster=cluster1,
process_name=designate-central,
component=designate-central,
control_plane=control-plane-1,
cloud_name=entry-scale-kvm

9.3 Networking Service Overview Edit source

SUSE OpenStack Cloud Networking is a virtual networking service that leverages the OpenStack Neutron service to provide network connectivity and addressing to SUSE OpenStack Cloud Compute service devices.

The Networking service also provides an API to configure and manage a variety of network services.

You can use the Networking service to connect guest servers or you can define and configure your own virtual network topology.

9.3.1 Installing the Networking service Edit source

SUSE OpenStack Cloud Network Administrators are responsible for planning for the Neutron networking service, and once installed, to configure the service to meet the needs of their cloud network users.

9.3.2 Working with the Networking service Edit source

To perform tasks using the Networking service, you can use the dashboard, API or CLI.

9.3.3 Reconfiguring the Networking service Edit source

If you change any of the network configuration after installation, it is recommended that you reconfigure the Networking service by running the neutron-reconfigure playbook.

On the Cloud Lifecycle Manager:

ardana > cd ~/openstack/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml

9.3.4 For more information Edit source

For information on how to operate your cloud we suggest you read the OpenStack Operations Guide. The Architecture section contains useful information about how an OpenStack Cloud is put together. However, SUSE OpenStack Cloud takes care of these details for you. The Operations section contains information on how to manage the system.

9.3.5 Neutron External Networks Edit source

9.3.5.1 External networks overview Edit source

This topic explains how to create a Neutron external network.

External networks provide access to the internet.

The typical use is to provide an IP address that can be used to reach a VM from an external network which can be a public network like the internet or a network that is private to an organization.

9.3.5.2 Using the Ansible Playbook Edit source

This playbook will query the Networking service for an existing external network, and then create a new one if you do not already have one. The resulting external network will have the name ext-net with a subnet matching the CIDR you specify in the command below.

If you need to specify more granularity, for example specifying an allocation pool for the subnet, use the Section 9.3.5.3, “Using the NeutronClient CLI”.

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts neutron-cloud-configure.yml -e EXT_NET_CIDR=<CIDR>

The table below shows the optional switch that you can use as part of this playbook to specify environment-specific information:

SwitchDescription

-e EXT_NET_CIDR=<CIDR>

Optional. You can use this switch to specify the external network CIDR. If you choose not to use this switch, or use a wrong value, the VMs will not be accessible over the network.

This CIDR will be from the EXTERNAL VM network.

9.3.5.3 Using the NeutronClient CLI Edit source

For more granularity you can utilize the Neutron command line tool to create your external network.

  1. Log in to the Cloud Lifecycle Manager.

  2. Source the Admin creds:

    ardana > source ~/service.osrc
  3. Create the external network and then the subnet using these commands below.

    Creating the network:

    ardana > neutron net-create --router:external <external-network-name>

    Creating the subnet:

    ardana > neutron subnet-create EXTERNAL-NETWORK-NAME CIDR --gateway GATEWAY --allocation-pool start=IP_START,end=IP_END [--disable-dhcp]

    Where:

    ValueDescription
    external-network-name

    This is the name given to your external network. This is a unique value that you will choose. The value ext-net is usually used.

    CIDR

    You can use this switch to specify the external network CIDR. If you choose not to use this switch, or use a wrong value, the VMs will not be accessible over the network.

    This CIDR will be from the EXTERNAL VM network.

    --gateway

    Optional switch to specify the gateway IP for your subnet. If this is not included then it will choose the first available IP.

    --allocation-pool start end

    Optional switch to specify a start and end IP address to use as the allocation pool for this subnet.

    --disable-dhcp

    Optional switch if you want to disable DHCP on this subnet. If this is not specified then DHCP will be enabled.

9.3.5.4 Multiple External Networks Edit source

SUSE OpenStack Cloud provides the ability to have multiple external networks, by using the Network Service (Neutron) provider networks for external networks. You can configure SUSE OpenStack Cloud to allow the use of provider VLANs as external networks by following these steps.

  1. Do NOT include the neutron.l3_agent.external_network_bridge tag in the network_groups definition for your cloud. This results in the l3_agent.ini external_network_bridge being set to an empty value (rather than the traditional br-ex).

  2. Configure your cloud to use provider VLANs, by specifying the provider_physical_network tag on one of the network_groups defined for your cloud.

    For example, to run provider VLANS over the EXAMPLE network group: (some attributes omitted for brevity)

    network-groups:
    
      - name: EXAMPLE
        tags:
          - neutron.networks.vlan:
              provider-physical-network: physnet1
  3. After the cloud has been deployed, you can create external networks using provider VLANs.

    For example, using the Network Service CLI:

    1. Create external network 1 on vlan101

      neutron net-create --provider:network_type vlan --provider:physical_network physnet1 --provider:segmentation_id 101 ext-net1 --router:external true
    2. Create external network 2 on vlan102

      neutron net-create --provider:network_type vlan --provider:physical_network physnet1 --provider:segmentation_id 102 ext-net2 --router:external true

9.3.6 Neutron Provider Networks Edit source

This topic explains how to create a Neutron provider network.

A provider network is a virtual network created in the SUSE OpenStack Cloud cloud that is consumed by SUSE OpenStack Cloud services. The distinctive element of a provider network is that it does not create a virtual router; rather, it depends on L3 routing that is provided by the infrastructure.

A provider network is created by adding the specification to the SUSE OpenStack Cloud input model. It consists of at least one network and one or more subnets.

9.3.6.1 SUSE OpenStack Cloud input model Edit source

The input model is the primary mechanism a cloud admin uses in defining a SUSE OpenStack Cloud installation. It exists as a directory with a data subdirectory that contains YAML files. By convention, any service that creates a Neutron provider network will create a subdirectory under the data directory and the name of the subdirectory shall be the project name. For example, the Octavia project will use Neutron provider networks so it will have a subdirectory named 'octavia' and the config file that specifies the neutron network will exist in that subdirectory.

├── cloudConfig.yml
    ├── data
    │   ├── control_plane.yml
    │   ├── disks_compute.yml
    │   ├── disks_controller_1TB.yml
    │   ├── disks_controller.yml
    │   ├── firewall_rules.yml
    │   ├── net_interfaces.yml
    │   ├── network_groups.yml
    │   ├── networks.yml
    │   ├── neutron
    │   │   └── neutron_config.yml
    │   ├── nic_mappings.yml
    │   ├── server_groups.yml
    │   ├── server_roles.yml
    │   ├── servers.yml
    │   ├── swift
    │   │   └── rings.yml
    │   └── octavia
    │       └── octavia_config.yml
    ├── README.html
    └── README.md

9.3.6.2 Network/Subnet specification Edit source

The elements required in the input model for you to define a network are:

  • name

  • network_type

  • physical_network

Elements that are optional when defining a network are:

  • segmentation_id

  • shared

Required elements for the subnet definition are:

  • cidr

Optional elements for the subnet definition are:

  • allocation_pools which will require start and end addresses

  • host_routes which will require a destination and nexthop

  • gateway_ip

  • no_gateway

  • enable-dhcp

NOTE: Only IPv4 is supported at the present time.

9.3.6.3 Network details Edit source

The following table outlines the network values to be set, and what they represent.

AttributeRequired/optionalAllowed ValuesUsage
nameRequired  
network_typeRequiredflat, vlan, vxlanThe type of desired network
physical_networkRequiredValidName of physical network that is overlayed with the virtual network
segmentation_idOptionalvlan or vxlan rangesVLAN id for vlan or tunnel id for vxlan
sharedOptionalTrueShared by all projects or private to a single project

9.3.6.4 Subnet details Edit source

The following table outlines the subnet values to be set, and what they represent.

AttributeReq/OptAllowed ValuesUsage
cidrRequiredValid CIDR rangefor example, 172.30.0.0/24
allocation_poolsOptionalSee allocation_pools table below 
host_routesOptionalSee host_routes table below 
gateway_ipOptionalValid IP addrSubnet gateway to other nets
no_gatewayOptionalTrueNo distribution of gateway
enable-dhcpOptionalTrueEnable dhcp for this subnet

9.3.6.5 ALLOCATION_POOLS details Edit source

The following table explains allocation pool settings.

AttributeReq/OptAllowed ValuesUsage
startRequiredValid IP addrFirst ip address in pool
endRequiredValid IP addrLast ip address in pool

9.3.6.6 HOST_ROUTES details Edit source

The following table explains host route settings.

AttributeReq/OptAllowed ValuesUsage
destinationRequiredValid CIDRDestination subnet
nexthopRequiredValid IP addrHop to take to destination subnet
Note
Note

Multiple destination/nexthop values can be used.

9.3.6.7 Examples Edit source

The following examples show the configuration file settings for Neutron and Octavia.

Octavia configuration

This file defines the mapping. It does not need to be edited unless you want to change the name of your VLAN.

Path: ~/openstack/my_cloud/definition/data/octavia/octavia_config.yml

---
  product:
    version: 2

  configuration-data:
    - name: OCTAVIA-CONFIG-CP1
      services:
        - octavia
      data:
        amp_network_name: OCTAVIA-MGMT-NET

Neutron configuration

Input your network configuration information for your provider VLANs in neutron_config.yml found here:

~/openstack/my_cloud/definition/data/neutron/.

---
  product:
    version: 2

  configuration-data:
    - name:  NEUTRON-CONFIG-CP1
      services:
        - neutron
      data:
        neutron_provider_networks:
        - name: OCTAVIA-MGMT-NET
          provider:
            - network_type: vlan
              physical_network: physnet1
              segmentation_id: 2754
          cidr: 10.13.189.0/24
          no_gateway:  True
          enable_dhcp: True
          allocation_pools:
            - start: 10.13.189.4
              end: 10.13.189.252
          host_routes:
            # route to MANAGEMENT-NET
            - destination: 10.13.111.128/26
              nexthop:  10.13.189.5

9.3.6.8 Implementing your changes Edit source

  1. Commit the changes to git:

    ardana > git add -A
    ardana > git commit -a -m "configuring provider network"
  2. Run the configuration processor:

    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
  3. Update your deployment directory:

    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
  4. Then continue with your clean cloud installation.

  5. If you are only adding a Neutron Provider network to an existing model, then run the neutron-deploy.yml playbook:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts neutron-deploy.yml

9.3.6.9 Multiple Provider Networks Edit source

The physical network infrastructure must be configured to convey the provider VLAN traffic as tagged VLANs to the cloud compute nodes and network service network nodes. Configuration of the physical network infrastructure is outside the scope of the SUSE OpenStack Cloud 8 software.

SUSE OpenStack Cloud 8 automates the server networking configuration and the Network Service configuration based on information in the cloud definition. To configure the system for provider VLANs, specify the neutron.networks.vlan tag with a provider-physical-network attribute on one or more network groups. For example (some attributes omitted for brevity):

network-groups:

        - name: NET_GROUP_A
        tags:
        - neutron.networks.vlan:
        provider-physical-network: physnet1

        - name: NET_GROUP_B
        tags:
        - neutron.networks.vlan:
        provider-physical-network: physnet2

A network group is associated with a server network interface via an interface model. For example (some attributes omitted for brevity):

interface-models:
        - name: INTERFACE_SET_X
        network-interfaces:
        - device:
        name: bond0
        network-groups:
        - NET_GROUP_A
        - device:
        name: eth3
        network-groups:
        - NET_GROUP_B

A network group used for provider VLANs may contain only a single SUSE OpenStack Cloud network, because that VLAN must span all compute nodes and any Network Service network nodes/controllers (that is, it is a single L2 segment). The SUSE OpenStack Cloud network must be defined with tagged-vlan false, otherwise a Linux VLAN network interface will be created. For example:

networks:

        - name: NET_A
        tagged-vlan: false
        network-group: NET_GROUP_A

        - name: NET_B
        tagged-vlan: false
        network-group: NET_GROUP_B

When the cloud is deployed, SUSE OpenStack Cloud 8 will create the appropriate bridges on the servers, and set the appropriate attributes in the Neutron configuration files (for example, bridge_mappings).

After the cloud has been deployed, create Network Service network objects for each provider VLAN. For example, using the Network Service CLI:

ardana > neutron net-create --provider:network_type vlan --provider:physical_network physnet1 --provider:segmentation_id 101 mynet101
ardana > neutron net-create --provider:network_type vlan --provider:physical_network physnet2 --provider:segmentation_id 234 mynet234

9.3.6.10 More Information Edit source

For more information on the Network Service command-line interface (CLI), see the OpenStack networking command-line client reference: http://docs.openstack.org/cli-reference/content/neutronclient_commands.html

9.3.7 Using IPAM Drivers in the Networking Service Edit source

This topic describes how to choose and implement an IPAM driver.

9.3.7.1 Selecting and implementing an IPAM driver Edit source

Beginning with the Liberty release, OpenStack networking includes a pluggable interface for the IP Address Management (IPAM) function. This interface creates a driver framework for the allocation and de-allocation of subnets and IP addresses, enabling the integration of alternate IPAM implementations or third-party IP Address Management systems.

There are three possible IPAM driver options:

  • Non-pluggable driver. This option is the default when the ipam_driver parameter is not specified in neutron.conf.

  • Pluggable reference IPAM driver. The pluggable IPAM driver interface was introduced in SUSE OpenStack Cloud 8 (OpenStack Liberty). It is a refactoring of the Kilo non-pluggable driver to use the new pluggable interface. The setting in neutron.conf to specify this driver is ipam_driver = internal.

  • Pluggable Infoblox IPAM driver. The pluggable Infoblox IPAM driver is a third-party implementation of the pluggable IPAM interface. the corresponding setting in neutron.conf to specify this driver is ipam_driver = networking_infoblox.ipam.driver.InfobloxPool.

    Note
    Note

    You can use either the non-pluggable IPAM driver or a pluggable one. However, you cannot use both.

9.3.7.2 Using the Pluggable reference IPAM driver Edit source

To indicate that you want to use the Pluggable reference IPAM driver, the only parameter needed is "ipam_driver." You can set it by looking for the following commented line in the neutron.conf.j2 template (ipam_driver = internal) uncommenting it, and committing the file. After following the standard steps to deploy Neutron, Neutron will be configured to run using the Pluggable reference IPAM driver.

As stated, the file you must edit is neutron.conf.j2 on the Cloud Lifecycle Manager in the directory ~/openstack/my_cloud/config/neutron. Here is the relevant section where you can see the ipam_driver parameter commented out:

[DEFAULT]
  ...
  l3_ha_net_cidr = 169.254.192.0/18

  # Uncomment the line below if the Reference Pluggable IPAM driver is to be used
  # ipam_driver = internal
  ...

After uncommenting the line ipam_driver = internal, commit the file using git commit from the openstack/my_cloud directory:

ardana > git commit -a -m 'My config for enabling the internal IPAM Driver'

Then follow the steps to deploy SUSE OpenStack Cloud in the Book “Installing with Cloud Lifecycle Manager”, Preface “Installation Overview” appropriate to your cloud configuration.

Note
Note

Currently there is no migration path from the non-pluggable driver to a pluggable IPAM driver because changes are needed to database tables and Neutron currently cannot make those changes.

9.3.7.3 Using the Infoblox IPAM driver Edit source

As suggested above, using the Infoblox IPAM driver requires changes to existing parameters in nova.conf and neutron.conf. If you want to use the infoblox appliance, you will need to add the "infoblox service-component" to the service-role containing the neutron API server. To use the infoblox appliance for IPAM, both the agent and the Infoblox IPAM driver are required. The infoblox-ipam-agent should be deployed on the same node where the neutron-server component is running. Usually this is a Controller node.

  1. Have the Infoblox appliance running on the management network (the Infoblox appliance admin or the datacenter administrator should know how to perform this step).

  2. Change the control plane definition to add infoblox-ipam-agent as a service in the controller node cluster (see change in bold). Make the changes in control_plane.yml found here: ~/openstack/my_cloud/definition/data/control_plane.yml

    ---
      product:
        version: 2
    
      control-planes:
        - name: ccp
          control-plane-prefix: ccp
     ...
          clusters:
            - name: cluster0
              cluster-prefix: c0
              server-role: ARDANA-ROLE
              member-count: 1
              allocation-policy: strict
              service-components:
                - lifecycle-manager
            - name: cluster1
              cluster-prefix: c1
              server-role: CONTROLLER-ROLE
              member-count: 3
              allocation-policy: strict
              service-components:
                - ntp-server
    ...
                - neutron-server
                - infoblox-ipam-agent
    ...
                - designate-client
                - powerdns
          resources:
            - name: compute
              resource-prefix: comp
              server-role: COMPUTE-ROLE
              allocation-policy: any
  3. Modify the ~/openstack/my_cloud/config/neutron/neutron.conf.j2 file on the controller node to comment and uncomment the lines noted below to enable use with the Infoblox appliance:

    [DEFAULT]
                ...
                l3_ha_net_cidr = 169.254.192.0/18
    
    
                # Uncomment the line below if the Reference Pluggable IPAM driver is to be used
                # ipam_driver = internal
    
    
                # Comment out the line below if the Infoblox IPAM Driver is to be used
                # notification_driver = messaging
    
                # Uncomment the lines below if the Infoblox IPAM driver is to be used
                ipam_driver = networking_infoblox.ipam.driver.InfobloxPool
                notification_driver = messagingv2
    
    
                # Modify the infoblox sections below to suit your cloud environment
    
                [infoblox]
                cloud_data_center_id = 1
                # This name of this section is formed by "infoblox-dc:<infoblox.cloud_data_center_id>"
                # If cloud_data_center_id is 1, then the section name is "infoblox-dc:1"
    
                [infoblox-dc:0]
                http_request_timeout = 120
                http_pool_maxsize = 100
                http_pool_connections = 100
                ssl_verify = False
                wapi_version = 2.2
                admin_user_name = admin
                admin_password = infoblox
                grid_master_name = infoblox.localdomain
                grid_master_host = 1.2.3.4
    
    
                [QUOTAS]
                ...
  4. Change nova.conf.j2 to replace the notification driver "messaging" to "messagingv2"

     ...
    
     # Oslo messaging
     notification_driver = log
    
     #  Note:
     #  If the infoblox-ipam-agent is to be deployed in the cloud, change the
     #  notification_driver setting from "messaging" to "messagingv2".
     notification_driver = messagingv2
     notification_topics = notifications
    
     # Policy
     ...
  5. Commit the changes:

    ardana > cd ~/openstack/my_cloud
    ardana > git commit –a –m 'My config for enabling the Infoblox IPAM driver'
  6. Deploy the cloud with the changes. Due to changes to the control_plane.yml, you will need to rerun the config-processor-run.yml playbook if you have run it already during the install process.

    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts site.yml

9.3.7.4 Configuration parameters for using the Infoblox IPAM driver Edit source

Changes required in the notification parameters in nova.conf:

Parameter NameSection in nova.confDefault ValueCurrent Value Description
notify_on_state_changeDEFAULTNonevm_and_task_state

Send compute.instance.update notifications on instance state changes.

Vm_and_task_state means notify on vm and task state changes.

Infoblox requires the value to be vm_state (notify on vm state change).

Thus NO CHANGE is needed for infoblox

notification_topicsDEFAULTempty listnotifications

NO CHANGE is needed for infoblox.

The infoblox installation guide requires the notifications to be "notifications"

notification_driverDEFAULTNonemessaging

Change needed.

The infoblox installation guide requires the notification driver to be "messagingv2".

Changes to existing parameters in neutron.conf

Parameter NameSection in neutron.confDefault ValueCurrent Value Description
ipam_driverDEFAULTNone

None

(param is undeclared in neutron.conf)

Pluggable IPAM driver to be used by Neutron API server.

For infoblox, the value is "networking_infoblox.ipam.driver.InfobloxPool"

notification_driverDEFAULTempty listmessaging

The driver used to send notifications from the Neutron API server to the Neutron agents.

The installation guide for networking-infoblox calls for the notification_driver to be "messagingv2"

notification_topicsDEFAULTNonenotifications

No change needed.

The row is here show the changes in the Neutron parameters described in the installation guide for networking-infoblox

Parameters specific to the Networking Infoblox Driver. All the parameters for the Infoblox IPAM driver must be defined in neutron.conf.

Parameter NameSection in neutron.confDefault ValueDescription
cloud_data_center_idinfoblox0ID for selecting a particular grid from one or more grids to serve networks in the Infoblox back end
ipam_agent_workersinfoblox1Number of Infoblox IPAM agent works to run
grid_master_hostinfoblox-dc.<cloud_data_center_id>empty stringIP address of the grid master. WAPI requests are sent to the grid_master_host
ssl_verifyinfoblox-dc.<cloud_data_center_id>FalseEnsure whether WAPI requests sent over HTTPS require SSL verification
WAPI Versioninfoblox-dc.<cloud_data_center_id>1.4The WAPI version. Value should be 2.2.
admin_user_nameinfoblox-dc.<cloud_data_center_id>empty stringAdmin user name to access the grid master or cloud platform appliance
admin_passwordinfoblox-dc.<cloud_data_center_id>empty stringAdmin user password
http_pool_connectionsinfoblox-dc.<cloud_data_center_id>100 
http_pool_maxsizeinfoblox-dc.<cloud_data_center_id>100 
http_request_timeoutinfoblox-dc.<cloud_data_center_id>120 

The diagram below shows Nova compute sending notification to the infoblox-ipam-agent

9.3.7.5 Limitations Edit source

  • There is no IPAM migration path from non-pluggable to pluggable IPAM driver (https://bugs.launchpad.net/neutron/+bug/1516156). This means there is no way to reconfigure the Neutron database if you wanted to change Neutron to use a pluggable IPAM driver. Unless you change the default of non-pluggable IPAM configuration to a pluggable driver at install time, you will have no other opportunity to make that change because reconfiguration of SUSE OpenStack Cloud 8from using the default non-pluggable IPAM configuration to SUSE OpenStack Cloud 8 using a pluggable IPAM driver is not supported.

  • Upgrade from previous versions of SUSE OpenStack Cloud to SUSE OpenStack Cloud 8 to use a pluggable IPAM driver is not supported.

  • The Infoblox appliance does not allow for overlapping IPs. For example, only one tenant can have a CIDR of 10.0.0.0/24.

  • The infoblox IPAM driver fails the creation of a subnet when a there is no gateway-ip supplied. For example, the command "neutron subnet-create ... --no-gateway ..." will fail.

9.3.8 Configuring Load Balancing as a Service (LBaaS) Edit source

SUSE OpenStack Cloud 8 LBaaS Configuration

Load Balancing as a Service (LBaaS) is an advanced networking service that allows load balancing of multi-node environments. It provides the ability to spread requests across multiple servers thereby reducing the load on any single server. This document describes the installation steps for LBaaS v1 (see prerequisites) and the configuration for LBaaS v1 and v2.

SUSE OpenStack Cloud 8 can support either LBaaS v1 or LBaaS v2 to allow for wide ranging customer requirements. If the decision is made to utilize LBaaS v1 it is highly unlikely that you will be able to perform an on-line upgrade of the service to v2 after the fact as the internal data structures are significantly different. Should you wish to attempt an upgrade, support will be needed from Sales Engineering and your chosen load balancer partner.

Warning
Warning

The LBaaS architecture is based on a driver model to support different load balancers. LBaaS-compatible drivers are provided by load balancer vendors including F5 and Citrix. A new software load balancer driver was introduced in the OpenStack Liberty release called "Octavia". The Octavia driver deploys a software load balancer called HAProxy. Octavia is the default load balancing provider in SUSE OpenStack Cloud 8 for LBaaS V2. Until Octavia is configured the creation of load balancers will fail with an error. Please refer to Book “Installing with Cloud Lifecycle Manager”, Chapter 31 “Configuring Load Balancer as a Service” document for information on installing Octavia.

Warning
Warning

Before upgrading to SUSE OpenStack Cloud 8, contact F5 and SUSE to determine which F5 drivers have been certified for use with SUSE OpenStack Cloud. Loading drivers not certified by SUSE may result in failure of your cloud deployment.

LBaaS V2 offers with Book “Installing with Cloud Lifecycle Manager”, Chapter 31 “Configuring Load Balancer as a Service” a software load balancing solution that supports both a highly available control plane and data plane. However, should an external hardware load balancer be selected the cloud operation can achieve additional performance and availability.

LBaaS v1

Reasons to select this version.

  1. You must be able to configure LBaaS via Horizon.

  2. Your hardware load balancer vendor does not currently support LBaaS v2.

Reasons not to select this version.

  1. No active development is being performed on this API in the OpenStack community. (Security fixes are still being worked upon).

  2. It does not allow for multiple ports on the same VIP (for example, to support both port 80 and 443 on a single VIP).

  3. It will never be able to support TLS termination/re-encryption at the load balancer.

  4. It will never be able to support L7 rules for load balancing.

  5. LBaaS v1 will likely become officially deprecated by the OpenStack community at the Tokyo (October 2015) summit.

LBaaS v2

Reasons to select this version.

  1. Your vendor already has a driver that supports LBaaS v2. Many hardware load balancer vendors already support LBaaS v2 and this list is growing all the time.

  2. You intend to script your load balancer creation and management so a UI is not important right now (Horizon support will be added in a future release).

  3. You intend to support TLS termination at the load balancer.

  4. You intend to use the Octavia software load balancer (adding HA and scalability).

  5. You do not want to take your load balancers offline to perform subsequent LBaaS upgrades.

  6. You intend in future releases to need L7 load balancing.

Reasons not to select this version.

  1. Your LBaaS vendor does not have a v2 driver.

  2. You must be able to manage your load balancers from Horizon.

  3. You have legacy software which utilizes the LBaaS v1 API.

LBaaS v1 requires configuration changes prior to installation and is not recommended. LBaaS v2 is installed by default with SUSE OpenStack Cloud and requires minimal configuration to start the service.

Note
Note

Only LBaaS V2 API currently supports load balancer failover with Octavia. However, in LBaaS V1 and if Octavia is not deployed when a load balancer is deleted it will need to be manually recreated. LBaaS v2 API includes automatic failover of a deployed load balancer with Octavia. More information about this driver can be found in Book “Installing with Cloud Lifecycle Manager”, Chapter 31 “Configuring Load Balancer as a Service”.

9.3.8.1 Prerequisites Edit source

SUSE OpenStack Cloud LBaaS v1

Installing LBaaS v1

Important
Important

It is not recommended that LBaaS v1 is used in a production environment. It is recommended you use LBaaS v2. If you do deploy LBaaS v1, the upgrade to LBaaS v2 is non-trivial and may require the use of professional services.

Note
Note

If you need to run LBaaS v1 instead of the default LBaaS v2, you should make appropriate installation preparations during SUSE OpenStack Cloud installation since LBaaS v2 is the default. If you have selected to install and use LBaaS v1 you will replace the control_plane.yml directories and neutron.conf.j2 file to use version 1.

Before you modify the control_plane.yml file, it is recommended that you back up the original version of this file. Once you have backed them up, modify the control_plane.yml file.

  1. Edit ~/openstack/my_cloud/definition/data/control_plane.yml - depending on your installation the control_plane.yml file might be in a different location.

  2. In the section specifying the compute nodes (resources/compute) replace neutron-lbaasv2-agent with neutron-lbaas-agent - there will only be one occurrence in that file.

  3. Save the modified file.

  4. Follow the steps in Book “Installing with Cloud Lifecycle Manager”, Chapter 10 “Using Git for Configuration Management” to commit and apply the changes.

  5. To test the installation follow the steps outlined in Book “Installing with Cloud Lifecycle Manager”, Chapter 31 “Configuring Load Balancer as a Service” after you have created a suitable subnet, see: Book “Installing with Cloud Lifecycle Manager”, Chapter 27 “UI Verification”, Section 27.4 “Creating an External Network”.

SUSE OpenStack Cloud LBaaS v2

  1. SUSE OpenStack Cloud must be installed for LBaaS v2.

  2. Follow the instructions to install Book “Installing with Cloud Lifecycle Manager”, Chapter 31 “Configuring Load Balancer as a Service”

9.3.9 Load Balancer: Octavia Driver Administration Edit source

This document provides the instructions on how to enable and manage various components of the Load Balancer Octavia driver if that driver is enabled.

9.3.9.1 Monasca Alerts Edit source

The Monasca-agent has the following Octavia-related plugins:

  • Process checks – checks if octavia processes are running. When it starts, it detects which processes are running and then monitors them.

  • http_connect check – checks if it can connect to octavia api servers.

Alerts are displayed in the Operations Console. For more information see Book “User Guide”, Chapter 1 “Using the Operations Console”, Section 1.1 “Operations Console Overview”.

9.3.9.2 Tuning Octavia Installation Edit source

Homogeneous Compute Configuration

Octavia works only with homogeneous compute node configurations. Currently, Octavia does not support multiple nova flavors. If Octavia needs to be supported on multiple compute nodes, then all the compute nodes should carry same set of physnets (which will be used for Octavia).

Octavia and Floating IPs

Due to a Neutron limitation Octavia will only work with CVR routers. Another option is to use VLAN provider networks which do not require a router.

You cannot currently assign a floating IP address as the VIP (user facing) address for a load balancer created by the Octavia driver if the underlying Neutron network is configured to support Distributed Virtual Router (DVR). The Octavia driver uses a Neutron function known as allowed address pairs to support load balancer fail over.

There is currently a Neutron bug that does not support this function in a DVR configuration

Octavia Configuration Files

The system comes pre-tuned and should not need any adjustments for most customers. If in rare instances manual tuning is needed, follow these steps:

Warning
Warning

Changes might be lost during SUSE OpenStack Cloud upgrades.

Edit the Octavia configuration files in my_cloud/config/octavia. It is recommended that any changes be made in all of the Octavia configuration files.

  • octavia-api.conf.j2

  • octavia-health-manager.conf.j2

  • octavia-housekeeping.conf.j2

  • octavia-worker.conf.j2

After the changes are made to the configuration files, redeploy the service.

  1. Commit changes to git.

    ardana > cd ~/openstack
    ardana > git add -A
    ardana > git commit -m "My Octavia Config"
  2. Run the configuration processor and ready deployment.

    ardana > cd ~/openstack/ardana/ansible/
    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
  3. Run the Octavia reconfigure.

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts octavia-reconfigure.yml

Spare Pools

The Octavia driver provides support for creating spare pools of the HAProxy software installed in VMs. This means instead of creating a new load balancer when loads increase, create new load balancer calls will pull a load balancer from the spare pool. The spare pools feature consumes resources, therefore the load balancers in the spares pool has been set to 0, which is the default and also disables the feature.

Reasons to enable a load balancing spare pool in SUSE OpenStack Cloud

  1. You expect a large number of load balancers to be provisioned all at once (puppet scripts, or ansible scripts) and you want them to come up quickly.

  2. You want to reduce the wait time a customer has while requesting a new load balancer.

To increase the number of load balancers in your spares pool, edit the Octavia configuration files by uncommenting the spare_amphora_pool_size and adding the number of load balancers you would like to include in your spares pool.

# Pool size for the spare pool
# spare_amphora_pool_size = 0
Important
Important

In SUSE OpenStack Cloud the spare pool cannot be used to speed up fail overs. If a load balancer fails in SUSE OpenStack Cloud, Octavia will always provision a new VM to replace that failed load balancer.

9.3.9.3 Managing Amphora Edit source

Octavia starts a separate VM for each load balancing function. These VMs are called amphora.

Updating the Cryptographic Certificates

Octavia uses two-way SSL encryption for communication between amphora and the control plane. Octavia keeps track of the certificates on the amphora and will automatically recycle them. The certificates on the control plane are valid for one year after installation of SUSE OpenStack Cloud.

You can check on the status of the certificate by logging into the controller node as root and running:

ardana > cd /opt/stack/service/octavia-SOME UUID/etc/certs/
openssl x509 -in client.pem  -text –noout

This prints the certificate out where you can check on the expiration dates.

To renew the certificates, reconfigure Octavia. Reconfiguring causes Octavia to automatically generate new certificates and deploy them to the controller hosts.

On the Cloud Lifecycle Manager execute octavia-reconfigure:

ardana > cd ~/scratch/ansible/next/ardana/ansible
ardana > ansible-playbook -i hosts/verb_hosts octavia-reconfigure.yml

Accessing VM information in Nova

You can use openstack project list as an administrative user to obtain information about the tenant or project-id of the Octavia project. In the example below, the Octavia project has a project-id of 37fd6e4feac14741b6e75aba14aea833.

ardana > openstack project list
+----------------------------------+------------------+
| ID                               | Name             |
+----------------------------------+------------------+
| 055071d8f25d450ea0b981ca67f7ccee | glance-swift     |
| 37fd6e4feac14741b6e75aba14aea833 | octavia          |
| 4b431ae087ef4bd285bc887da6405b12 | swift-monitor    |
| 8ecf2bb5754646ae97989ba6cba08607 | swift-dispersion |
| b6bd581f8d9a48e18c86008301d40b26 | services         |
| bfcada17189e4bc7b22a9072d663b52d | cinderinternal   |
| c410223059354dd19964063ef7d63eca | monitor          |
| d43bc229f513494189422d88709b7b73 | admin            |
| d5a80541ba324c54aeae58ac3de95f77 | demo             |
| ea6e039d973e4a58bbe42ee08eaf6a7a | backup           |
+----------------------------------+------------------+

You can then use nova list --tenant <project-id> to list the VMs for the Octavia tenant. Take particular note of the IP address on the OCTAVIA-MGMT-NET; in the example below it is 172.30.1.11. For additional nova command-line options see Section 9.3.9.5, “For More Information”.

ardana > nova list --tenant 37fd6e4feac14741b6e75aba14aea833
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
| ID                                   | Name                                         | Tenant ID                        | Status | Task State | Power State | Networks                                       |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
| 1ed8f651-de31-4208-81c5-817363818596 | amphora-1c3a4598-5489-48ea-8b9c-60c821269e4c | 37fd6e4feac14741b6e75aba14aea833 | ACTIVE | -          | Running     | private=10.0.0.4; OCTAVIA-MGMT-NET=172.30.1.11 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
Important
Important

The Amphora VMs do not have SSH or any other access. In the rare case that there is a problem with the underlying load balancer the whole amphora will need to be replaced.

Initiating Failover of an Amphora VM

Under normal operations Octavia will monitor the health of the amphora constantly and automatically fail them over if there are any issues. This helps to minimize any potential downtime for load balancer users. There are, however, a few cases a failover needs to be initiated manually:

  1. The Loadbalancer has become unresponsive and Octavia has not detected an error.

  2. A new image has become available and existing load balancers need to start using the new image.

  3. The cryptographic certificates to control and/or the HMAC password to verify Health information of the amphora have been compromised.

To minimize the impact for end users we will keep the existing load balancer working until shortly before the new one has been provisioned. There will be a short interruption for the load balancing service so keep that in mind when scheduling the failovers. To achieve that follow these steps (assuming the management ip from the previous step):

  1. Assign the IP to a SHELL variable for better readability.

    ardana > export MGM_IP=172.30.1.11
  2. Identify the port of the vm on the management network.

    ardana > neutron port-list | grep $MGM_IP
    | 0b0301b9-4ee8-4fb6-a47c-2690594173f4 |                                                   | fa:16:3e:d7:50:92 |
    {"subnet_id": "3e0de487-e255-4fc3-84b8-60e08564c5b7", "ip_address": "172.30.1.11"} |
  3. Disable the port to initiate a failover. Note the load balancer will still function but cannot be controlled any longer by Octavia.

    Note
    Note

    Changes after disabling the port will result in errors.

    ardana > neutron port-update --admin-state-up False 0b0301b9-4ee8-4fb6-a47c-2690594173f4
    Updated port: 0b0301b9-4ee8-4fb6-a47c-2690594173f4
  4. You can check to see if the amphora failed over with nova list --tenant <project-id>. This may take some time and in some cases may need to be repeated several times. You can tell that the failover has been successful by the changed IP on the management network.

    ardana > nova list --tenant 37fd6e4feac14741b6e75aba14aea833
    +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
    | ID                                   | Name                                         | Tenant ID                        | Status | Task State | Power State | Networks                                       |
    +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
    | 1ed8f651-de31-4208-81c5-817363818596 | amphora-1c3a4598-5489-48ea-8b9c-60c821269e4c | 37fd6e4feac14741b6e75aba14aea833 | ACTIVE | -          | Running     | private=10.0.0.4; OCTAVIA-MGMT-NET=172.30.1.12 |
    +--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+------------------------------------------------+
Warning
Warning

Do not issue too many failovers at once. In a big installation you might be tempted to initiate several failovers in parallel for instance to speed up an update of amphora images. This will put a strain on the nova service and depending on the size of your installation you might need to throttle the failover rate.

9.3.9.4 Removing load balancers Edit source

The following procedures demonstrate how to delete a load balancer that is in the ERROR, PENDING_CREATE, or PENDING_DELETE state.

Procedure 9.1: Manually deleting load balancers created with neutron lbaasv2 (in an upgrade/migration scenario)
  1. Query the Neutron service for the loadbalancer ID:

    ardana > neutron lbaas-loadbalancer-list
    neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
    +--------------------------------------+---------+----------------------------------+--------------+---------------------+----------+
    | id                                   | name    | tenant_id                        | vip_address  | provisioning_status | provider |
    +--------------------------------------+---------+----------------------------------+--------------+---------------------+----------+
    | 7be4e4ab-e9c6-4a57-b767-da9af5ba7405 | test-lb | d62a1510b0f54b5693566fb8afeb5e33 | 192.168.1.10 | ERROR               | haproxy  |
    +--------------------------------------+---------+----------------------------------+--------------+---------------------+----------+
  2. Connect to the neutron database:

    mysql> use ovs_neutron
  3. Get the pools and healthmonitors associated with the loadbalancer:

    mysql> select id, healthmonitor_id, loadbalancer_id from lbaas_pools where loadbalancer_id = '7be4e4ab-e9c6-4a57-b767-da9af5ba7405';
    +--------------------------------------+--------------------------------------+--------------------------------------+
    | id                                   | healthmonitor_id                     | loadbalancer_id                      |
    +--------------------------------------+--------------------------------------+--------------------------------------+
    | 26c0384b-fc76-4943-83e5-9de40dd1c78c | 323a3c4b-8083-41e1-b1d9-04e1fef1a331 | 7be4e4ab-e9c6-4a57-b767-da9af5ba7405 |
    +--------------------------------------+--------------------------------------+--------------------------------------+
  4. Get the members associated with the pool:

    mysql> select id, pool_id from lbaas_members where pool_id = '26c0384b-fc76-4943-83e5-9de40dd1c78c';
    +--------------------------------------+--------------------------------------+
    | id                                   | pool_id                              |
    +--------------------------------------+--------------------------------------+
    | 6730f6c1-634c-4371-9df5-1a880662acc9 | 26c0384b-fc76-4943-83e5-9de40dd1c78c |
    | 06f0cfc9-379a-4e3d-ab31-cdba1580afc2 | 26c0384b-fc76-4943-83e5-9de40dd1c78c |
    +--------------------------------------+--------------------------------------+
  5. Delete the pool members:

    mysql> delete from lbaas_members where id = '6730f6c1-634c-4371-9df5-1a880662acc9';
    mysql> delete from lbaas_members where id = '06f0cfc9-379a-4e3d-ab31-cdba1580afc2';
  6. Find and delete the listener associated with the loadbalancer:

    mysql> select id, loadbalancer_id, default_pool_id from lbaas_listeners where loadbalancer_id = '7be4e4ab-e9c6-4a57-b767-da9af5ba7405';
    +--------------------------------------+--------------------------------------+--------------------------------------+
    | id                                   | loadbalancer_id                      | default_pool_id                      |
    +--------------------------------------+--------------------------------------+--------------------------------------+
    | 3283f589-8464-43b3-96e0-399377642e0a | 7be4e4ab-e9c6-4a57-b767-da9af5ba7405 | 26c0384b-fc76-4943-83e5-9de40dd1c78c |
    +--------------------------------------+--------------------------------------+--------------------------------------+
    mysql> delete from lbaas_listeners where id = '3283f589-8464-43b3-96e0-399377642e0a';
  7. Delete the pool associated with the loadbalancer:

    mysql> delete from lbaas_pools where id = '26c0384b-fc76-4943-83e5-9de40dd1c78c';
  8. Delete the healthmonitor associated with the pool:

    mysql> delete from lbaas_healthmonitors where id = '323a3c4b-8083-41e1-b1d9-04e1fef1a331';
  9. Delete the loadbalancer:

    mysql> delete from lbaas_loadbalancer_statistics where loadbalancer_id = '7be4e4ab-e9c6-4a57-b767-da9af5ba7405';
    mysql> delete from lbaas_loadbalancers where id = '7be4e4ab-e9c6-4a57-b767-da9af5ba7405';
Procedure 9.2: Manually Deleting Load Balancers Created With Octavia
  1. Query the Octavia service for the loadbalancer ID:

    ardana > openstack loadbalancer list --column id --column name --column provisioning_status
    +--------------------------------------+---------+---------------------+
    | id                                   | name    | provisioning_status |
    +--------------------------------------+---------+---------------------+
    | d8ac085d-e077-4af2-b47a-bdec0c162928 | test-lb | ERROR               |
    +--------------------------------------+---------+---------------------+
  2. Query the Octavia service for the amphora IDs (in this example we use ACTIVE/STANDBY topology with 1 spare Amphora):

    ardana > openstack loadbalancer amphora list
    +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------------+
    | id                                   | loadbalancer_id                      | status    | role   | lb_network_ip | ha_ip       |
    +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------------+
    | 6dc66d41-e4b6-4c33-945d-563f8b26e675 | d8ac085d-e077-4af2-b47a-bdec0c162928 | ALLOCATED | BACKUP | 172.30.1.7    | 192.168.1.8 |
    | 1b195602-3b14-4352-b355-5c4a70e200cf | d8ac085d-e077-4af2-b47a-bdec0c162928 | ALLOCATED | MASTER | 172.30.1.6    | 192.168.1.8 |
    | b2ee14df-8ac6-4bb0-a8d3-3f378dbc2509 | None                                 | READY     | None   | 172.30.1.20   | None        |
    +--------------------------------------+--------------------------------------+-----------+--------+---------------+-------------+
  3. Query the Octavia service for the loadbalancer pools:

    ardana > openstack loadbalancer pool list
    +--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+
    | id                                   | name      | project_id                       | provisioning_status | protocol | lb_algorithm | admin_state_up |
    +--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+
    | 39c4c791-6e66-4dd5-9b80-14ea11152bb5 | test-pool | 86fba765e67f430b83437f2f25225b65 | ACTIVE              | TCP      | ROUND_ROBIN  | True           |
    +--------------------------------------+-----------+----------------------------------+---------------------+----------+--------------+----------------+
  4. Connect to the octavia database:

    mysql> use octavia
  5. Delete any listeners, pools, health monitors, and members from the load balancer:

    mysql> delete from listener where load_balancer_id = 'd8ac085d-e077-4af2-b47a-bdec0c162928';
    mysql> delete from health_monitor where pool_id = '39c4c791-6e66-4dd5-9b80-14ea11152bb5';
    mysql> delete from member where pool_id = '39c4c791-6e66-4dd5-9b80-14ea11152bb5';
    mysql> delete from pool where load_balancer_id = 'd8ac085d-e077-4af2-b47a-bdec0c162928';
  6. Delete the amphora entries in the database:

    mysql> delete from amphora_health where amphora_id = '6dc66d41-e4b6-4c33-945d-563f8b26e675';
    mysql> update amphora set status = 'DELETED' where id = '6dc66d41-e4b6-4c33-945d-563f8b26e675';
    mysql> delete from amphora_health where amphora_id = '1b195602-3b14-4352-b355-5c4a70e200cf';
    mysql> update amphora set status = 'DELETED' where id = '1b195602-3b14-4352-b355-5c4a70e200cf';
  7. Delete the load balancer instance:

    mysql> update load_balancer set provisioning_status = 'DELETED' where id = 'd8ac085d-e077-4af2-b47a-bdec0c162928';
  8. The following script automates the above steps:

    #!/bin/bash
    
    if (( $# != 1 )); then
      echo "Please specify a loadbalancer ID"
      exit 1
    fi
    
    LB_ID=$1
    
    set -u -e -x
    
    readarray -t AMPHORAE < <(openstack loadbalancer amphora list \
      --format value \
      --column id \
      --column loadbalancer_id \
      | grep ${LB_ID} \
      | cut -d ' ' -f 1)
    
    readarray -t POOLS < <(openstack loadbalancer show ${LB_ID} \
      --format value \
      --column pools)
    
    mysql octavia --execute "delete from listener where load_balancer_id = '${LB_ID}';"
    for p in "${POOLS[@]}"; do
      mysql octavia --execute "delete from health_monitor where pool_id = '${p}';"
      mysql octavia --execute "delete from member where pool_id = '${p}';"
    done
    mysql octavia --execute "delete from pool where load_balancer_id = '${LB_ID}';"
    for a in "${AMPHORAE[@]}"; do
      mysql octavia --execute "delete from amphora_health where amphora_id = '${a}';"
      mysql octavia --execute "update amphora set status = 'DELETED' where id = '${a}';"
    done
    mysql octavia --execute "update load_balancer set provisioning_status = 'DELETED' where id = '${LB_ID}';"

9.3.9.5 For More Information Edit source

For more information on the Nova command-line client, see the OpenStack Compute command-line client guide.

For more information on Octavia terminology, see the OpenStack Octavia Glossary

9.3.10 Role-based Access Control in Neutron Edit source

This topic explains how to achieve more granular access control for your Neutron networks.

Previously in SUSE OpenStack Cloud, a network object was either private to a project or could be used by all projects. If the network's shared attribute was True, then the network could be used by every project in the cloud. If false, only the members of the owning project could use it. There was no way for the network to be shared by only a subset of the projects.

Neutron Role Based Access Control (RBAC) solves this problem for networks. Now the network owner can create RBAC policies that give network access to target projects. Members of a targeted project can use the network named in the RBAC policy the same way as if the network was owned by the project. Constraints are described in the section Section 9.3.10.10, “Limitations”.

With RBAC you are able to let another tenant use a network that you created, but as the owner of the network, you need to create the subnet and the router for the network.

9.3.10.1 Creating a Network Edit source

ardana > openstack network create demo-net
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-07-25T17:43:59Z                 |
| description               |                                      |
| dns_domain                |                                      |
| id                        | 9c801954-ec7f-4a65-82f8-e313120aabc4 |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1450                                 |
| name                      | demo-net                             |
| port_security_enabled     | False                                |
| project_id                | cb67c79e25a84e328326d186bf703e1b     |
| provider:network_type     | vxlan                                |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 1009                                 |
| qos_policy_id             | None                                 |
| revision_number           | 2                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | 2018-07-25T17:43:59Z                 |
+---------------------------+--------------------------------------+

9.3.10.2 Creating an RBAC Policy Edit source

Here we will create an RBAC policy where a member of the project called 'demo' will share the network with members of project 'demo2'

To create the RBAC policy, run:

ardana > openstack network rbac create  --target-project DEMO2-PROJECT-ID --type network --action access_as_shared demo-net

Here is an example where the DEMO2-PROJECT-ID is 5a582af8b44b422fafcd4545bd2b7eb5

ardana > openstack network rbac create --target-tenant 5a582af8b44b422fafcd4545bd2b7eb5 \
  --type network --action access_as_shared demo-net

9.3.10.3 Listing RBACs Edit source

To list all the RBAC rules/policies, execute:

ardana > openstack network rbac list
+--------------------------------------+-------------+--------------------------------------+
| ID                                   | Object Type | Object ID                            |
+--------------------------------------+-------------+--------------------------------------+
| 0fdec7f0-9b94-42b4-a4cd-b291d04282c1 | network     | 7cd94877-4276-488d-b682-7328fc85d721 |
+--------------------------------------+-------------+--------------------------------------+

9.3.10.4 Listing the Attributes of an RBAC Edit source

To see the attributes of a specific RBAC policy, run

ardana > openstack network rbac show POLICY-ID

For example:

ardana > openstack network rbac show 0fd89dcb-9809-4a5e-adc1-39dd676cb386

Here is the output:

+---------------+--------------------------------------+
| Field         | Value                                |
+---------------+--------------------------------------+
| action        | access_as_shared                     |
| id            | 0fd89dcb-9809-4a5e-adc1-39dd676cb386 |
| object_id     | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b |
| object_type   | network                              |
| target_tenant | 5a582af8b44b422fafcd4545bd2b7eb5     |
| tenant_id     | 75eb5efae5764682bca2fede6f4d8c6f     |
+---------------+--------------------------------------+

9.3.10.5 Deleting an RBAC Policy Edit source

To delete an RBAC policy, run openstack network rbac delete passing the policy id:

ardana > openstack network rbac delete POLICY-ID

For example:

ardana > openstack network rbac delete 0fd89dcb-9809-4a5e-adc1-39dd676cb386

Here is the output:

Deleted rbac_policy: 0fd89dcb-9809-4a5e-adc1-39dd676cb386

9.3.10.6 Sharing a Network with All Tenants Edit source

Either the administrator or the network owner can make a network shareable by all tenants.

The administrator can make a tenant's network shareable by all tenants. To make the network demo-shareall-net accessible by all tenants in the cloud:

To share a network with all tenants:

  1. Get a list of all projects

    ardana > ~/service.osrc
    ardana > openstack project list

    which produces the list:

    +----------------------------------+------------------+
    | ID                               | Name             |
    +----------------------------------+------------------+
    | 1be57778b61645a7a1c07ca0ac488f9e | demo             |
    | 5346676226274cd2b3e3862c2d5ceadd | admin            |
    | 749a557b2b9c482ca047e8f4abf348cd | swift-monitor    |
    | 8284a83df4df429fb04996c59f9a314b | swift-dispersion |
    | c7a74026ed8d4345a48a3860048dcb39 | demo-sharee      |
    | e771266d937440828372090c4f99a995 | glance-swift     |
    | f43fb69f107b4b109d22431766b85f20 | services         |
    +----------------------------------+------------------+
  2. Get a list of networks:

    ardana > openstack network list

    This produces the following list:

    +--------------------------------------+-------------------+----------------------------------------------------+
    | id                                   | name              | subnets                                            |
    +--------------------------------------+-------------------+----------------------------------------------------+
    | f50f9a63-c048-444d-939d-370cb0af1387 | ext-net           | ef3873db-fc7a-4085-8454-5566fb5578ea 172.31.0.0/16 |
    | 9fb676f5-137e-4646-ac6e-db675a885fd3 | demo-net          | 18fb0b77-fc8b-4f8d-9172-ee47869f92cc 10.0.1.0/24   |
    | 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e | demo-shareall-net | 2bbc85a9-3ffe-464c-944b-2476c7804877 10.0.250.0/24 |
    | 73f946ee-bd2b-42e9-87e4-87f19edd0682 | demo-share-subset | c088b0ef-f541-42a7-b4b9-6ef3c9921e44 10.0.2.0/24   |
    +--------------------------------------+-------------------+----------------------------------------------------+
  3. Set the network you want to share to a shared value of True:

    ardana > openstack network set --share 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e

    You should see the following output:

    Updated network: 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e
  4. Check the attributes of that network by running the following command using the ID of the network in question:

    ardana > openstack network show 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e

    The output will look like this:

    +---------------------------+--------------------------------------+
    | Field                     | Value                                |
    +---------------------------+--------------------------------------+
    | admin_state_up            | UP                                   |
    | availability_zone_hints   |                                      |
    | availability_zones        |                                      |
    | created_at                | 2018-07-25T17:43:59Z                 |
    | description               |                                      |
    | dns_domain                |                                      |
    | id                        | 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e |
    | ipv4_address_scope        | None                                 |
    | ipv6_address_scope        | None                                 |
    | is_default                | None                                 |
    | is_vlan_transparent       | None                                 |
    | mtu                       | 1450                                 |
    | name                      | demo-net                             |
    | port_security_enabled     | False                                |
    | project_id                | cb67c79e25a84e328326d186bf703e1b     |
    | provider:network_type     | vxlan                                |
    | provider:physical_network | None                                 |
    | provider:segmentation_id  | 1009                                 |
    | qos_policy_id             | None                                 |
    | revision_number           | 2                                    |
    | router:external           | Internal                             |
    | segments                  | None                                 |
    | shared                    | False                                |
    | status                    | ACTIVE                               |
    | subnets                   |                                      |
    | tags                      |                                      |
    | updated_at                | 2018-07-25T17:43:59Z                 |
    +---------------------------+--------------------------------------+
  5. As the owner of the demo-shareall-net network, view the RBAC attributes for demo-shareall-net (id=8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e) by first getting an RBAC list:

    ardana > echo $OS_USERNAME ; echo $OS_PROJECT_NAME
    demo
    demo
    ardana > openstack network rbac list

    This produces the list:

    +--------------------------------------+--------------------------------------+
    | id                                   | object_id                            |
    +--------------------------------------+--------------------------------------+
    | ...                                                                         |
    | 3e078293-f55d-461c-9a0b-67b5dae321e8 | 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e |
    +--------------------------------------+--------------------------------------+
  6. View the RBAC information:

    ardana > openstack network rbac show 3e078293-f55d-461c-9a0b-67b5dae321e8
    
    +---------------+--------------------------------------+
    | Field         | Value                                |
    +---------------+--------------------------------------+
    | action        | access_as_shared                     |
    | id            | 3e078293-f55d-461c-9a0b-67b5dae321e8 |
    | object_id     | 8eada4f7-83cf-40ba-aa8c-5bf7d87cca8e |
    | object_type   | network                              |
    | target_tenant | *                                    |
    | tenant_id     | 1be57778b61645a7a1c07ca0ac488f9e     |
    +---------------+--------------------------------------+
  7. With network RBAC, the owner of the network can also make the network shareable by all tenants. First create the network:

    ardana > echo $OS_PROJECT_NAME ; echo $OS_USERNAME
    demo
    demo
    ardana > openstack network create test-net

    The network is created:

    +---------------------------+--------------------------------------+
    | Field                     | Value                                |
    +---------------------------+--------------------------------------+
    | admin_state_up            | UP                                   |
    | availability_zone_hints   |                                      |
    | availability_zones        |                                      |
    | created_at                | 2018-07-25T18:04:25Z                 |
    | description               |                                      |
    | dns_domain                |                                      |
    | id                        | a4bd7c3a-818f-4431-8cdb-fedf7ff40f73 |
    | ipv4_address_scope        | None                                 |
    | ipv6_address_scope        | None                                 |
    | is_default                | False                                |
    | is_vlan_transparent       | None                                 |
    | mtu                       | 1450                                 |
    | name                      | test-net                             |
    | port_security_enabled     | False                                |
    | project_id                | cb67c79e25a84e328326d186bf703e1b     |
    | provider:network_type     | vxlan                                |
    | provider:physical_network | None                                 |
    | provider:segmentation_id  | 1073                                 |
    | qos_policy_id             | None                                 |
    | revision_number           | 2                                    |
    | router:external           | Internal                             |
    | segments                  | None                                 |
    | shared                    | False                                |
    | status                    | ACTIVE                               |
    | subnets                   |                                      |
    | tags                      |                                      |
    | updated_at                | 2018-07-25T18:04:25Z                 |
    +---------------------------+--------------------------------------+
  8. Create the RBAC. It is important that the asterisk is surrounded by single-quotes to prevent the shell from expanding it to all files in the current directory.

    ardana > openstack network rbac create --type network \
      --action access_as_shared --target-project '*' test-net

    Here are the resulting RBAC attributes:

    +---------------+--------------------------------------+
    | Field         | Value                                |
    +---------------+--------------------------------------+
    | action        | access_as_shared                     |
    | id            | 0b797cc6-debc-48a1-bf9d-d294b077d0d9 |
    | object_id     | a4bd7c3a-818f-4431-8cdb-fedf7ff40f73 |
    | object_type   | network                              |
    | target_tenant | *                                    |
    | tenant_id     | 1be57778b61645a7a1c07ca0ac488f9e     |
    +---------------+--------------------------------------+

9.3.10.7 Target Project (demo2) View of Networks and Subnets Edit source

Note that the owner of the network and subnet is not the tenant named demo2. Both the network and subnet are owned by tenant demo. Demo2members cannot create subnets of the network. They also cannot modify or delete subnets owned by demo.

As the tenant demo2, you can get a list of neutron networks:

ardana > openstack network list
+--------------------------------------+-----------+--------------------------------------------------+
| id                                   | name      | subnets                                          |
+--------------------------------------+-----------+--------------------------------------------------+
| f60f3896-2854-4f20-b03f-584a0dcce7a6 | ext-net   | 50e39973-b2e3-466b-81c9-31f4d83d990b             |
| c3d55c21-d8c9-4ee5-944b-560b7e0ea33b | demo-net  | d9b765da-45eb-4543-be96-1b69a00a2556 10.0.1.0/24 |
   ...
+--------------------------------------+-----------+--------------------------------------------------+

And get a list of subnets:

ardana > openstack subnet list --network c3d55c21-d8c9-4ee5-944b-560b7e0ea33b
+--------------------------------------+---------+--------------------------------------+---------------+
| ID                                   | Name    | Network                              | Subnet        |
+--------------------------------------+---------+--------------------------------------+---------------+
| a806f28b-ad66-47f1-b280-a1caa9beb832 | ext-net | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b | 10.0.1.0/24   |
+--------------------------------------+---------+--------------------------------------+---------------+

To show details of the subnet:

ardana > openstack subnet show d9b765da-45eb-4543-be96-1b69a00a2556
+-------------------+--------------------------------------------+
| Field             | Value                                      |
+-------------------+--------------------------------------------+
| allocation_pools  | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr              | 10.0.1.0/24                                |
| dns_nameservers   |                                            |
| enable_dhcp       | True                                       |
| gateway_ip        | 10.0.1.1                                   |
| host_routes       |                                            |
| id                | d9b765da-45eb-4543-be96-1b69a00a2556       |
| ip_version        | 4                                          |
| ipv6_address_mode |                                            |
| ipv6_ra_mode      |                                            |
| name              | sb-demo-net                                |
| network_id        | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b       |
| subnetpool_id     |                                            |
| tenant_id         | 75eb5efae5764682bca2fede6f4d8c6f           |
+-------------------+--------------------------------------------+

9.3.10.8 Target Project: Creating a Port Using demo-net Edit source

The owner of the port is demo2. Members of the network owner project (demo) will not see this port.

Running the following command:

ardana > openstack port create c3d55c21-d8c9-4ee5-944b-560b7e0ea33b

Creates a new port:

+-----------------------+-----------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                               |
+-----------------------+-----------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                                |
| allowed_address_pairs |                                                                                                     |
| binding:vnic_type     | normal                                                                                              |
| device_id             |                                                                                                     |
| device_owner          |                                                                                                     |
| dns_assignment        | {"hostname": "host-10-0-1-10", "ip_address": "10.0.1.10", "fqdn": "host-10-0-1-10.openstacklocal."} |
| dns_name              |                                                                                                     |
| fixed_ips             | {"subnet_id": "d9b765da-45eb-4543-be96-1b69a00a2556", "ip_address": "10.0.1.10"}                    |
| id                    | 03ef2dce-20dc-47e5-9160-942320b4e503                                                                |
| mac_address           | fa:16:3e:27:8d:ca                                                                                   |
| name                  |                                                                                                     |
| network_id            | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b                                                                |
| security_groups       | 275802d0-33cb-4796-9e57-03d8ddd29b94                                                                |
| status                | DOWN                                                                                                |
| tenant_id             | 5a582af8b44b422fafcd4545bd2b7eb5                                                                    |
+-----------------------+-----------------------------------------------------------------------------------------------------+

9.3.10.9 Target Project Booting a VM Using Demo-Net Edit source

Here the tenant demo2 boots a VM that uses the demo-net shared network:

ardana > openstack server create --flavor 1 --image $OS_IMAGE --nic net-id=c3d55c21-d8c9-4ee5-944b-560b7e0ea33b demo2-vm-using-demo-net-nic
+--------------------------------------+------------------------------------------------+
| Property                             | Value                                          |
+--------------------------------------+------------------------------------------------+
| OS-EXT-AZ:availability_zone          |                                                |
| OS-EXT-STS:power_state               | 0                                              |
| OS-EXT-STS:task_state                | scheduling                                     |
| OS-EXT-STS:vm_state                  | building                                       |
| OS-SRV-USG:launched_at               | -                                              |
| OS-SRV-USG:terminated_at             | -                                              |
| accessIPv4                           |                                                |
| accessIPv6                           |                                                |
| adminPass                            | sS9uSv9PT79F                                   |
| config_drive                         |                                                |
| created                              | 2016-01-04T19:23:24Z                           |
| flavor                               | m1.tiny (1)                                    |
| hostId                               |                                                |
| id                                   | 3a4dc44a-027b-45e9-acf8-054a7c2dca2a           |
| image                                | cirros-0.3.3-x86_64 (6ae23432-8636-4e...1efc5) |
| key_name                             | -                                              |
| metadata                             | {}                                             |
| name                                 | demo2-vm-using-demo-net-nic                    |
| os-extended-volumes:volumes_attached | []                                             |
| progress                             | 0                                              |
| security_groups                      | default                                        |
| status                               | BUILD                                          |
| tenant_id                            | 5a582af8b44b422fafcd4545bd2b7eb5               |
| updated                              | 2016-01-04T19:23:24Z                           |
| user_id                              | a0e6427b036344fdb47162987cb0cee5               |
+--------------------------------------+------------------------------------------------+

Run openstack server list:

ardana > openstack server list

See the VM running:

+-------------------+-----------------------------+--------+------------+-------------+--------------------+
| ID                | Name                        | Status | Task State | Power State | Networks           |
+-------------------+-----------------------------+--------+------------+-------------+--------------------+
| 3a4dc...a7c2dca2a | demo2-vm-using-demo-net-nic | ACTIVE | -          | Running     | demo-net=10.0.1.11 |
+-------------------+-----------------------------+--------+------------+-------------+--------------------+

Run openstack port list:

ardana > neutron port-list --device-id 3a4dc44a-027b-45e9-acf8-054a7c2dca2a

View the subnet:

+---------------------+------+-------------------+-------------------------------------------------------------------+
| id                  | name | mac_address       | fixed_ips                                                         |
+---------------------+------+-------------------+-------------------------------------------------------------------+
| 7d14ef8b-9...80348f |      | fa:16:3e:75:32:8e | {"subnet_id": "d9b765da-45...00a2556", "ip_address": "10.0.1.11"} |
+---------------------+------+-------------------+-------------------------------------------------------------------+

Run neutron port-show:

ardana > openstack port show 7d14ef8b-9d48-4310-8c02-00c74d80348f
+-----------------------+-----------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                               |
+-----------------------+-----------------------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                                |
| allowed_address_pairs |                                                                                                     |
| binding:vnic_type     | normal                                                                                              |
| device_id             | 3a4dc44a-027b-45e9-acf8-054a7c2dca2a                                                                |
| device_owner          | compute:None                                                                                        |
| dns_assignment        | {"hostname": "host-10-0-1-11", "ip_address": "10.0.1.11", "fqdn": "host-10-0-1-11.openstacklocal."} |
| dns_name              |                                                                                                     |
| extra_dhcp_opts       |                                                                                                     |
| fixed_ips             | {"subnet_id": "d9b765da-45eb-4543-be96-1b69a00a2556", "ip_address": "10.0.1.11"}                    |
| id                    | 7d14ef8b-9d48-4310-8c02-00c74d80348f                                                                |
| mac_address           | fa:16:3e:75:32:8e                                                                                   |
| name                  |                                                                                                     |
| network_id            | c3d55c21-d8c9-4ee5-944b-560b7e0ea33b                                                                |
| security_groups       | 275802d0-33cb-4796-9e57-03d8ddd29b94                                                                |
| status                | ACTIVE                                                                                              |
| tenant_id             | 5a582af8b44b422fafcd4545bd2b7eb5                                                                    |
+-----------------------+-----------------------------------------------------------------------------------------------------+

9.3.10.10 Limitations Edit source

Note the following limitations of RBAC in Neutron.

  • Neutron network is the only supported RBAC Neutron object type.

  • The "access_as_external" action is not supported – even though it is listed as a valid action by python-neutronclient.

  • The neutron-api server will not accept action value of 'access_as_external'. The access_as_external definition is not found in the specs.

  • The target project users cannot create, modify, or delete subnets on networks that have RBAC policies.

  • The subnet of a network that has an RBAC policy cannot be added as an interface of a target tenant's router. For example, the command neutron router-interface-add tgt-tenant-router <sb-demo-net uuid> will error out.

  • The security group rules on the network owner do not apply to other projects that can use the network.

  • A user in target project can boot up VMs using a VNIC using the shared network. The user of the target project can assign a floating IP (FIP) to the VM. The target project must have SG rules that allows SSH and/or ICMP for VM connectivity.

  • Neutron RBAC creation and management are currently not supported in Horizon. For now, the Neutron CLI has to be used to manage RBAC rules.

  • A RBAC rule tells Neutron whether a tenant can access a network (Allow). Currently there is no DENY action.

  • Port creation on a shared network fails if --fixed-ip is specified in the neutron port-create command.

9.3.11 Configuring Maximum Transmission Units in Neutron Edit source

This topic explains how you can configure MTUs, what to look out for, and the results and implications of changing the default MTU settings. It is important to note that every network within a network group will have the same MTU.

Warning
Warning

An MTU change will not affect existing networks that have had VMs created on them. It will only take effect on new networks created after the reconfiguration process.

9.3.11.1 Overview Edit source

A Maximum Transmission Unit, or MTU is the maximum packet size (in bytes) that a network device can or is configured to handle. There are a number of places in your cloud where MTU configuration is relevant: the physical interfaces managed and configured by SUSE OpenStack Cloud, the virtual interfaces created by Neutron and Nova for Neutron networking, and the interfaces inside the VMs.

SUSE OpenStack Cloud-managed physical interfaces

SUSE OpenStack Cloud-managed physical interfaces include the physical interfaces and the bonds, bridges, and VLANs created on top of them. The MTU for these interfaces is configured via the 'mtu' property of a network group. Because multiple network groups can be mapped to one physical interface, there may have to be some resolution of differing MTUs between the untagged and tagged VLANs on the same physical interface. For instance, if one untagged VLAN, vlan101 (with an MTU of 1500) and a tagged VLAN vlan201 (with an MTU of 9000) are both on one interface (eth0), this means that eth0 can handle 1500, but the VLAN interface which is created on top of eth0 (that is, vlan201@eth0) wants 9000. However, vlan201 cannot have a higher MTU than eth0, so vlan201 will be limited to 1500 when it is brought up, and fragmentation will result.

In general, a VLAN interface MTU must be lower than or equal to the base device MTU. If they are different, as in the case above, the MTU of eth0 can be overridden and raised to 9000, but in any case the discrepancy will have to be reconciled.

Neutron/Nova interfaces

Neutron/Nova interfaces include the virtual devices created by Neutron and Nova during the normal process of realizing a Neutron network/router and booting a VM on it (qr-*, qg-*, tap-*, qvo-*, qvb-*, etc.). There is currently no support in Neutron/Nova for per-network MTUs in which every interface along the path for a particular Neutron network has the correct MTU for that network. There is, however, support for globally changing the MTU of devices created by Neutron/Nova (see network_device_mtu below). This means that if you want to enable jumbo frames for any set of VMs, you will have to enable it for all your VMs. You cannot just enable them for a particular Neutron network.

VM interfaces

VMs typically get their MTU via DHCP advertisement, which means that the dnsmasq processes spawned by the neutron-dhcp-agent actually advertise a particular MTU to the VMs. In SUSE OpenStack Cloud 8, the DHCP server advertises to all VMS a 1400 MTU via a forced setting in dnsmasq-neutron.conf. This is suboptimal for every network type (vxlan, flat, vlan, etc) but it does prevent fragmentation of a VM's packets due to encapsulation.

For instance, if you set the new *-mtu configuration options to a default of 1500 and create a VXLAN network, it will be given an MTU of 1450 (with the remaining 50 bytes used by the VXLAN encapsulation header) and will advertise a 1450 MTU to any VM booted on that network. If you create a provider VLAN network, it will have an MTU of 1500 and will advertise 1500 to booted VMs on the network. It should be noted that this default starting point for MTU calculation and advertisement is also global, meaning you cannot have an MTU of 8950 on one VXLAN network and 1450 on another. However, you can have provider physical networks with different MTUs by using the physical_network_mtus config option, but Nova still requires a global MTU option for the interfaces it creates, thus you cannot really take advantage of that configuration option.

9.3.11.2 Network settings in the input model Edit source

MTU can be set as an attribute of a network group in network_groups.yml. Note that this applies only to KVM. That setting means that every network in the network group will be assigned the specified MTU. The MTU value must be set individually for each network group. For example:

network-groups:
        - name: GUEST
        mtu: 9000
        ...

        - name: EXTERNAL-API
        mtu: 9000
        ...

        - name: EXTERNAL-VM
        mtu: 9000
        ...

9.3.11.3 Infrastructure support for jumbo frames Edit source

If you want to use jumbo frames, or frames with an MTU of 9000 or more, the physical switches and routers that make up the infrastructure of the SUSE OpenStack Cloud installation must be configured to support them. To realize the advantages, generally all devices in the same broadcast domain must have the same MTU.

If you want to configure jumbo frames on compute and controller nodes, then all switches joining the compute and controller nodes must have jumbo frames enabled. Similarly, the "infrastructure gateway" through which the external VM network flows, commonly known as the default route for the external VM VLAN, must also have the same MTU configured.

You can also consider anything in the same broadcast domain to be anything in the same VLAN or anything in the same IP subnet.

9.3.11.4 Enabling end-to-end jumbo frames for a VM Edit source

  1. Add an 'mtu' attribute to all the network groups in your model. Note that adding the MTU for the network groups will only affect the configuration for physical network interfaces.

    To add the mtu attribute, find the YAML file that contains your network-groups entry. We will assume it is network_groups.yml, unless you have changed it. Whatever the file is named, it will be found in ~/openstack/my_cloud/definition/data/.

    To edit these files, begin by checking out the site branch on the Cloud Lifecycle Manager node. You may already be on that branch. If so, you will remain there.

    ardana > cd ~/openstack/ardana/ansible
    ardana > git checkout site

    Then begin editing the files. In network_groups.yml, add mtu: 9000

    network-groups:
                - name: GUEST
                hostname-suffix: guest
                mtu: 9000
                tags:
                - neutron.networks.vxlan

    This will set the physical interface managed by SUSE OpenStack Cloud 8 that has the GUEST network group tag assigned to it. This can be found in the interfaces_set.yml file under the interface-models section.

  2. Next, edit the layer 2 agent config file, ml2_conf.ini.j2, found in ~/openstack/my_cloud/config/neutron/ to set the path_mtu to 0, this ensures that global_physnet_mtu is used.

    [ml2]
    ...
    path_mtu = 0
  3. Next, edit neutron.conf.j2 found in ~/openstack/my_cloud/config/neutron/ to set advertise_mtu (to true) and global_physnet_mtu to 9000 under [DEFAULT]:

    [DEFAULT]
    ...
    advertise_mtu = True
    global_physnet_mtu = 9000

    This allows Neutron to advertise the optimal MTU to instances (based upon global_physnet_mtu minus the encapsulation size).

  4. Next, remove the "dhcp-option-force=26,1400" line from ~/openstack/my_cloud/config/neutron/dnsmasq-neutron.conf.j2.

  5. OvS will set br-int to the value of the lowest physical interface. If you are using Jumbo frames on some of your networks, br-int on the controllers may be set to 1500 instead of 9000. Work around this condition by running:

    ovs-vsctl set int br-int mtu_request=9000
  6. Commit your changes

    ardana > git add -A
    ardana > git commit -m "your commit message goes here in quotes"
  7. If SUSE OpenStack Cloud has not been deployed yet, do normal deployment and skip to step 8.

  8. Assuming it has been deployed already, continue here:

    Run the configuration processor:

    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml

    and ready the deployment:

    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml

    Then run the network_interface-reconfigure.yml playbook, changing directories first:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts network_interface-reconfigure.yml

    Then run neutron-reconfigure.yml:

    ardana > ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml

    Then nova-reconfigure.yml:

    ardana > ansible-playbook -i hosts/verb_hosts nova-reconfigure.yml

    Note: adding/changing network-group mtu settings will likely require a network restart during network_interface-reconfigure.yml.

  9. Follow the normal process for creating a Neutron network and booting a VM or two. In this example, if a VXLAN network is created and a VM is booted on it, the VM will have an MTU of 8950, with the remaining 50 bytes used by the VXLAN encapsulation header.

  10. Test and verify that the VM can send and receive jumbo frames without fragmentation. You can use ping. For example, to test an MTU of 9000 using VXLAN:

    ardana > ping –M do –s 8950 YOUR_VM_FLOATING_IP

    Just substitute your actual floating IP address for the YOUR_VM_FLOATING_IP.

9.3.11.5 Enabling Optimal MTU Advertisement Feature Edit source

To enable the optimal MTU feature, follow these steps:

  1. Edit ~/openstack/my_cloud/config/neutron/neutron.conf.j2 to remove advertise_mtu variable under [DEFAULT]

    [DEFAULT]
    ...
    advertise_mtu = False #remove this
  2. Remove the dhcp-option-force=26,1400 line from ~/openstack/my_cloud/config/neutron/dnsmasq-neutron.conf.j2

  3. If SUSE OpenStack Cloud has already been deployed, follow the remaining steps, otherwise follow the normal deployment procedures.

  4. Commit your changes

    ardana > git add -A
    ardana > git commit -m "your commit message goes here in quotes"
  5. Run the configuration processor:

    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
  6. Run ready deployment:

    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
  7. Run the network_interface-reconfigure.yml playbook, changing directories first:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts network_interface-reconfigure.yml
  8. Run neutron-reconfigure.yml:

    ardana > ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml
Important
Important

If you are upgrading an existing deployment, attention must be paid to avoid creating MTU mismatch between network interfaces in preexisting VMs and that of VMs created after upgrade. If you do have an MTU mismatch, then the new VMs (having interface with 1500 minus the underlay protocol overhead) will not be able to have L2 connectivity with preexisting VMs (with 1400 MTU due to dhcp-option-force).

9.3.12 Improve Network Peformance with Isolated Metadata Settings Edit source

In SUSE OpenStack Cloud, Neutron currently sets enable_isolated_metadata = True by default in dhcp_agent.ini because several services require isolated networks (Neutron networks without a router). This has the effect of spawning a neutron-ns-metadata-proxy process on one of the controller nodes for every active Neutron network.

In environments that create many Neutron networks, these extra neutron-ns-metadata-proxy processes can quickly eat up a lot of memory on the controllers, which does not scale well.

For deployments that do not require isolated metadata (that is, they do not require the Platform Services and will always create networks with an attached router), you can set enable_isolated_metadata = False in dhcp_agent.ini to reduce Neutron memory usage on controllers, allowing a greater number of active Neutron networks.

Note that the dhcp_agent.ini.j2 template is found in ~/openstack/my_cloud/config/neutron on the Cloud Lifecycle Manager node. The edit can be made there and the standard deployment can be run if this is install time. In a deployed cloud, run the Neutron reconfiguration procedure outlined here:

  1. First check out the site branch:

    ardana > cd ~/openstack/my_cloud/config/neutron
    ardana > git checkout site
  2. Edit the dhcp_agent.ini.j2 file to change the enable_isolated_metadata = {{ neutron_enable_isolated_metadata }} line in the [DEFAULT] section to read:

    enable_isolated_metadata = False
  3. Commit the file:

    ardana > git add -A
    ardana > git commit -m "your commit message goes here in quotes"
  4. Run the ready-deployment.yml playbook from ~/openstack/ardana/ansible:

    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
  5. Then run the neutron-reconfigure.yml playbook, changing directories first:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml

9.3.13 Moving from DVR deployments to non_DVR Edit source

If you have an older deployment of SUSE OpenStack Cloud which is using DVR as a default and you are attempting to move to non_DVR, make sure you follow these steps:

  1. Remove all your existing DVR routers and their workloads. Make sure to remove interfaces, floating ips and gateways, if applicable.

    neutron router-interface-delete ROUTER-NAME SUBNET-NAME/SUBNET-ID
    neutron floatingip-disassociate FLOATINGIP-ID PRIVATE-PORT-ID
    neutron router-gateway-clear ROUTER-NAME -NET-NAME/EXT-NET-ID
  2. Then delete the router.

    neutron router-delete ROUTER-NAME
  3. Before you create any non_DVR router make sure that l3-agents and metadata-agents are not running in any compute host. You can run the command neutron agent-list to see if there are any neutron-l3-agent running in any compute-host in your deployment.

    You must disable neutron-l3-agent and neutron-metadata-agent on every compute host by running the following commands:

    ardana > neutron agent-list
    +--------------------------------------+----------------------+--------------------------+-------------------+-------+----------------+---------------------------+
    | id                                   | agent_type           | host                     | availability_zone | alive | admin_state_up | binary                    |
    +--------------------------------------+----------------------+--------------------------+-------------------+-------+----------------+---------------------------+
    | 208f6aea-3d45-4b89-bf42-f45a51b05f29 | Loadbalancerv2 agent | ardana-cp1-comp0001-mgmt |                   | :-)   | True           | neutron-lbaasv2-agent     |
    | 810f0ae7-63aa-4ee3-952d-69837b4b2fe4 | L3 agent             | ardana-cp1-comp0001-mgmt | nova              | :-)   | True           | neutron-l3-agent          |
    | 89ac17ba-2f43-428a-98fa-b3698646543d | Metadata agent       | ardana-cp1-comp0001-mgmt |                   | :-)   | True           | neutron-metadata-agent    |
    | f602edce-1d2a-4c8a-ba56-fa41103d4e17 | Open vSwitch agent   | ardana-cp1-comp0001-mgmt |                   | :-)   | True           | neutron-openvswitch-agent |
    ...
    +--------------------------------------+----------------------+--------------------------+-------------------+-------+----------------+---------------------------+
    
    $ neutron agent-update 810f0ae7-63aa-4ee3-952d-69837b4b2fe4 --admin-state-down
    Updated agent: 810f0ae7-63aa-4ee3-952d-69837b4b2fe4
    
    $ neutron agent-update 89ac17ba-2f43-428a-98fa-b3698646543d --admin-state-down
    Updated agent: 89ac17ba-2f43-428a-98fa-b3698646543d
    Note
    Note

    Only L3 and Metadata agents were disabled.

  4. Once L3 and metadata neutron agents are stopped, follow steps 1 through 7 in the document Book “Planning an Installation with Cloud Lifecycle Manager”, Chapter 12 “Alternative Configurations”, Section 12.2 “Configuring SUSE OpenStack Cloud without DVR” and then run the neutron-reconfigure.yml playbook:

    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml

9.3.14 OVS-DPDK Support Edit source

SUSE OpenStack Cloud uses a version of Open vSwitch (OVS) that is built with the Data Plane Development Kit (DPDK) and includes a QEMU hypervisor which supports vhost-user.

The OVS-DPDK package modifes the OVS fast path, which is normally performed in kernel space, and allows it to run in userspace so there is no context switch to the kernel for processing network packets.

The EAL component of DPDK supports mapping the Network Interface Card (NIC) registers directly into userspace. The DPDK provides a Poll Mode Driver (PMD) that can access the NIC hardware from userspace and uses polling instead of interrupts to avoid the user to kernel transition.

The PMD maps the shared address space of the VM that is provided by the vhost-user capability of QEMU. The vhost-user mode causes Neutron to create a Unix domain socket that allows communication between the PMD and QEMU. The PMD uses this in order to acquire the file descriptors to the pre-allocated VM memory. This allows the PMD to directly access the VM memory space and perform a fast zero-copy of network packets directly into and out of the VMs virtio_net vring.

This yields performance improvements in the time it takes to process network packets.

9.3.14.1 Usage considerations Edit source

The target for a DPDK Open vSwitch is VM performance and VMs only run on compute nodes so the following considerations are compute node specific.

  1. You are required to Section 9.3.14.3, “Configuring Hugepages for DPDK in Neutron Networks” in order to use DPDK with VMs. The memory to be used must be allocated at boot time so you must know beforehand how many VMs will be scheduled on a node. Also, for NUMA considerations, you want those hugepages on the same NUMA node as the NIC. A VM maps its entire address space into a hugepage.

  2. For maximum performance you must reserve logical cores for DPDK poll mode driver (PMD) usage and for hypervisor (QEMU) usage. This keeps the Linux kernel from scheduling processes on those cores. The PMD threads will go to 100% cpu utilization since it uses polling of the hardware instead of interrupts. There will be at least 2 cores dedicated to PMD threads. Each VM will have a core dedicated to it although for less performance VMs can share cores.

  3. VMs can use the virtio_net or the virtio_pmd drivers. There is also a PMD for an emulated e1000.

  4. Only VMs that use hugepages can be sucessfully launched on a DPDK enabled NIC. If there is a need to support both DPDK and non-DPDK based VMs an additional port managed by the Linux kernel must exist.

  5. OVS/DPDK does not support jumbo frames. Please review https://github.com/openvswitch/ovs/blob/branch-2.5/INSTALL.DPDK.md#restrictions for restrictions.

  6. The Open vSwitch firewall driver in networking-ovs-dpdk repo is stateless and not a stateful one that would use iptables and conntrack. In the past, Neutron core has declined to pull in stateless type FW. https://bugs.launchpad.net/neutron/+bug/1531205 The native firewall driver is stateful, which is why conntrack was added to Open vSwitch. But this is not supported on DPDK and will not be until OVS 2.6.

9.3.14.2 For more information Edit source

See the following topics for more information:

9.3.14.3 Configuring Hugepages for DPDK in Neutron Networks Edit source

To take advantage of DPDK and its network performance enhancements, enable hugepages first.

With hugepages, physical RAM is reserved at boot time and dedicated to a virtual machine. Only that virtual machine and Open vSwitch can use this specifically allocated RAM. The host OS cannot access it. This memory is contiguous, and because of its larger size, reduces the number of entries in the memory map and number of times it must be read.

The hugepage reservation is made in /etc/default/grub, but this is handled by the Cloud Lifecycle Manager.

In addition to hugepages, to use DPDK, CPU isolation is required. This is achieved with the 'isolcups' command in /etc/default/grub, but this is also managed by the Cloud Lifecycle Manager using a new input model file.

The two new input model files introduced with this release to help you configure the necessary settings and persist them are:

  • memory_models.yml (for hugepages)

  • cpu_models.yml (for CPU isolation)

9.3.14.3.1 memory_models.yml Edit source

In this file you set your huge page size along with the number of such huge-page allocations.

 ---
  product:
    version: 2

  memory-models:
    - name: COMPUTE-MEMORY-NUMA
      default-huge-page-size: 1G
      huge-pages:
        - size: 1G
          count: 24
          numa-node: 0
        - size: 1G
          count: 24
          numa-node: 1
        - size: 1G
          count: 48
9.3.14.3.2 cpu_models.yml Edit source
---
  product:
    version: 2

  cpu-models:

    - name: COMPUTE-CPU
      assignments:
       - components:
           - nova-compute-kvm
         cpu:
           - processor-ids: 3-5,12-17
             role: vm

       - components:
           - openvswitch
         cpu:
           - processor-ids: 0
             role: eal
           - processor-ids: 1-2
             role: pmd
9.3.14.3.3 NUMA memory allocation Edit source

As mentioned above, the memory used for hugepages is locked down at boot time by an entry in /etc/default/grub. As an admin, you can specify in the input model how to arrange this memory on NUMA nodes. It can be spread across NUMA nodes or you can specify where you want it. For example, if you have only one NIC, you would probably want all the hugepages memory to be on the NUMA node closest to that NIC.

If you do not specify the numa-node settings in the memory_models.yml input model file and use only the last entry indicating "size: 1G" and "count: 48" then this memory is spread evenly across all NUMA nodes.

Also note that the hugepage service runs once at boot time and then goes to an inactive state so you should not expect to see it running. If you decide to make changes to the NUMA memory allocation, you will need to reboot the compute node for the changes to take effect.

9.3.14.4 DPDK Setup for Neutron Networking Edit source

9.3.14.4.1 Hardware requirements Edit source
  • Intel-based compute node. DPDK is not available on AMD-based systems.

  • The following BIOS settings must be enabled for DL360 Gen9:

    1. Virtualization Technology

    2. Intel(R) VT-d

    3. PCI-PT (Also see Section 9.3.15.14, “Enabling PCI-PT on HPE DL360 Gen 9 Servers”)

  • Need adequate host memory to allow for hugepages. The examples below use 1G hugepages for the VMs

9.3.14.4.2 Limitations Edit source
  • DPDK is supported on SLES only.

  • Applies to SUSE OpenStack Cloud 8 only.

  • Tenant network can be untagged vlan or untagged vxlan

  • DPDK port names must be of the form 'dpdk<portid>' where port id is sequential and starts at 0

  • No support for converting DPDK ports to non DPDK ports without rebooting compute node.

  • No security group support, need userspace conntrack.

  • No jumbo frame support.

9.3.14.4.3 Setup instructions Edit source

These setup instructions and example model are for a three-host system. There is one controller with Cloud Lifecycle Manager in cloud control plane and two compute hosts.

  1. After initial run of site.yml all compute nodes must be rebooted to pick up changes in grub for hugepages and isolcpus

  2. Changes to non-uniform memory access (NUMA) memory, isolcpu, or network devices must be followed by a reboot of compute nodes

  3. Run sudo reboot to pick up libvirt change and hugepage/isocpus grub changes

    tux > sudo reboot
  4. Use the bash script below to configure nova aggregates, neutron networks, a new flavor, etc. And then it will spin up two VMs.

VM spin-up instructions

Before running the spin up script you need to get a copy of the cirros image to your Cloud Lifecycle Manager node. You can manually scp a copy of the cirros image to the system. You can copy it locallly with wget like so

ardana > wget http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img

Save the following shell script in the home directory and run it. This should spin up two VMs, one on each compute node.

Warning
Warning

Make sure to change all network-specific information in the script to match your environment.

#!/usr/bin/env bash

source service.osrc

######## register glance image
glance image-create --name='cirros' --container-format=bare --disk-format=qcow2 < ~/cirros-0.3.4-x86_64-disk.img

####### create nova aggregate and flavor for dpdk

MI_NAME=dpdk

nova aggregate-create $MI_NAME nova
nova aggregate-add-host $MI_NAME openstack-cp-comp0001-mgmt
nova aggregate-add-host $MI_NAME openstack-cp-comp0002-mgmt
nova aggregate-set-metadata $MI_NAME pinned=true

nova flavor-create $MI_NAME 6 1024 20 1
nova flavor-key $MI_NAME set hw:cpu_policy=dedicated
nova flavor-key $MI_NAME set aggregate_instance_extra_specs:pinned=true
nova flavor-key $MI_NAME set hw:mem_page_size=1048576

######## sec groups NOTE: no sec groups supported on DPDK.  This is in case we do non-DPDK compute hosts.
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

########  nova keys
nova keypair-add mykey >mykey.pem
chmod 400 mykey.pem

######## create neutron external network
neutron net-create ext-net --router:external --os-endpoint-type internalURL
neutron subnet-create ext-net 10.231.0.0/19 --gateway_ip=10.231.0.1  --ip-version=4 --disable-dhcp  --allocation-pool start=10.231.17.0,end=10.231.17.255

########  neutron network
neutron net-create mynet1
neutron subnet-create mynet1 10.1.1.0/24 --name mysubnet1
neutron router-create myrouter1
neutron router-interface-add myrouter1 mysubnet1
neutron router-gateway-set myrouter1 ext-net
export MYNET=$(neutron net-list|grep mynet|awk '{print $2}')

######## spin up 2 VMs, 1 on each compute
nova boot --image cirros --nic net-id=${MYNET} --key-name mykey --flavor dpdk --availability-zone nova:openstack-cp-comp0001-mgmt vm1
nova boot --image cirros --nic net-id=${MYNET} --key-name mykey --flavor dpdk --availability-zone nova:openstack-cp-comp0002-mgmt vm2

######## create floating ip and attach to instance
export MYFIP1=$(nova floating-ip-create|grep ext-net|awk '{print $4}')
nova add-floating-ip vm1 ${MYFIP1}

export MYFIP2=$(nova floating-ip-create|grep ext-net|awk '{print $4}')
nova add-floating-ip vm2 ${MYFIP2}

nova list

9.3.14.5 DPDK Configurations Edit source

9.3.14.5.1 Base configuration Edit source

The following is specific to DL360 Gen9 and BIOS configuration as detailed in Section 9.3.14.4, “DPDK Setup for Neutron Networking”.

  • EAL cores - 1, isolate: False in cpu-models

  • PMD cores - 1 per NIC port

  • Hugepages - 1G per PMD thread

  • Memory channels - 4

  • Global rx queues - based on needs

9.3.14.5.2 Performance considerations common to all NIC types Edit source

Compute host core frequency

Host CPUs should be running at maximum performance. The following is a script to set that. Note that in this case there are 24 cores. This needs to be modified to fit your environment. For a HP DL360 Gen9, the BIOS should be configured to use "OS Control Mode" which can be found on the iLO Power Settings page.

for i in `seq 0 23`; do echo "performance" > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done

IO non-posted prefetch

The DL360 Gen9 should have the IO non-posted prefetch disabled. Experimental evidence shows this yields an additional 6-8% performance boost.

9.3.14.5.3 Multiqueue configuration Edit source

In order to use multiqueue, a property must be applied to the Glance image and a setting inside the resulting VM must be applied. In this example we create a 4 vCPU flavor for DPDK using 1G hugepages.

MI_NAME=dpdk

nova aggregate-create $MI_NAME nova
nova aggregate-add-host $MI_NAME openstack-cp-comp0001-mgmt
nova aggregate-add-host $MI_NAME openstack-cp-comp0002-mgmt
nova aggregate-set-metadata $MI_NAME pinned=true

nova flavor-create $MI_NAME 6 1024 20 4
nova flavor-key $MI_NAME set hw:cpu_policy=dedicated
nova flavor-key $MI_NAME set aggregate_instance_extra_specs:pinned=true
nova flavor-key $MI_NAME set hw:mem_page_size=1048576

And set the hw_vif_multiqueue_enabled property on the Glance image

ardana > openstack image set --property hw_vif_multiqueue_enabled=true IMAGE UUID

Once the VM is booted using the flavor above, inside the VM, choose the number of combined rx and tx queues to be equal to the number of vCPUs

tux > sudo ethtool -L eth0 combined 4

On the hypervisor you can verify that multiqueue has been properly set by looking at the qemu process

-netdev type=vhost-user,id=hostnet0,chardev=charnet0,queues=4 -device virtio-net-pci,mq=on,vectors=10,

Here you can see that 'mq=on' and vectors=10. The formula for vectors is 2*num_queues+2

9.3.14.6 Troubleshooting DPDK Edit source

9.3.14.6.1 Hardware configuration Edit source

Because there are several variations of hardware, it is up to you to verify that the hardware is configured properly.

  • Only Intel based compute nodes are supported. There is no DPDK available for AMD-based CPUs.

  • PCI-PT must be enabled for the NIC that will be used with DPDK.

  • When using Intel Niantic and the igb_uio driver, the VT-d must be enabled in the BIOS.

  • For DL360 Gen9 systems, the BIOS shared-memory Section 9.3.15.14, “Enabling PCI-PT on HPE DL360 Gen 9 Servers”.

  • Adequate memory must be available for Section 9.3.14.3, “Configuring Hugepages for DPDK in Neutron Networks” usage.

  • Hyper-threading can be enabled but is not required for base functionality.

  • Determine the PCI slot that the DPDK NIC(s) are installed in to determine the associated NUMA node.

  • Only the Intel Haswell, Broadwell, and Skylake microarchitectures are supported. Intel Sandy Bridge is not supported.

9.3.14.6.2 System configuration Edit source
  • Only SLES12-SP3 compute nodes are supported.

  • If a NIC port is used with PCI-PT, SRIOV-only, or PCI-PT+SRIOV, then it cannot be used with DPDK. They are mutually exclusive. This is because DPDK depends on an OvS bridge which does not exist if you use any combination of PCI-PT and SRIOV. You can use DPDK, SRIOV-only, and PCI-PT on difference interfaces of the same server.

  • There is an association between the PCI slot for the NIC and a NUMA node. Make sure to use logical CPU cores that are on the NUMA node associated to the NIC. Use the following to determine which CPUs are on which NUMA node.

    ardana > lscpu
    
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                48
    On-line CPU(s) list:   0-47
    Thread(s) per core:    2
    Core(s) per socket:    12
    Socket(s):             2
    NUMA node(s):          2
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 63
    Model name:            Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
    Stepping:              2
    CPU MHz:               1200.000
    CPU max MHz:           1800.0000
    CPU min MHz:           1200.0000
    BogoMIPS:              3597.06
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              30720K
    NUMA node0 CPU(s):     0-11,24-35
    NUMA node1 CPU(s):     12-23,36-47
9.3.14.6.3 Input model configuration Edit source
  • If you do not specify a driver for a DPDK device, the igb_uio will be selected as default.

  • DPDK devices must be named dpdk<port-id> where the port-id starts at 0 and increments sequentially.

  • Tenant networks supported are untagged VXLAN and VLAN.

  • Jumbo Frames MTU does not work with DPDK OvS. There is an upstream patch most likely showing up in OvS 2.6 and it cannot be backported due to changes this patch relies upon.

  • Sample VXLAN model

  • Sample VLAN model

9.3.14.6.4 Reboot requirements Edit source

A reboot of a compute node must be performed when an input model change causes the following:

  1. After the initial site.yml play on a new OpenStack environment

  2. Changes to an existing OpenStack environment that modify the /etc/default/grub file, such as

    • hugepage allocations

    • CPU isolation

    • iommu changes

  3. Changes to a NIC port usage type, such as

    • moving from DPDK to any combination of PCI-PT and SRIOV

    • moving from DPDK to kernel based eth driver

9.3.14.6.5 Software configuration Edit source

The input model is processed by the Configuration Processor which eventually results in changes to the OS. There are several files that should be checked to verify the proper settings were applied. In addition, after the inital site.yml play is run all compute nodes must be rebooted in order to pickup changes to the /etc/default/grub file for hugepage reservation, CPU isolation and iommu settings.

Kernel settings

Check /etc/default/grub for the following

  1. hugepages

  2. CPU isolation

  3. that iommu is in passthru mode if the igb_uio driver is in use

Open vSwitch settings

Check /etc/default/openvswitch-switchf for

  1. using the --dpdk option

  2. core 0 set aside for EAL and kernel to share

  3. cores assigned to PMD drivers, at least two for each DPDK device

  4. verify that memory is reserved with socket-mem option

  5. Once VNETCORE-2509 merges also verify that the umask is 022 and the group is libvirt-qemu

DPDK settings

  1. check /etc/dpdk/interfacesf for the correct DPDK devices

9.3.14.6.6 DPDK runtime Edit source

All non-bonded DPDK devices will be added to individual OvS bridges. The bridges will be named br-dpdk0, br-dpdk1, etc. The name of the OvS bridge for bonded DPDK devices will be br-dpdkbond0, br-dpdkbond1, etc.

  1. Since each PMD thread is in a polling loop, it will use 100% of the CPU. Thus for two PMDs you would expect to see the ovs-vswitchd process running at 200%. This can be verified by running

    ardana > top
    
    top - 16:45:42 up 4 days, 22:24,  1 user,  load average: 2.03, 2.10, 2.14
    Tasks: 384 total,   2 running, 382 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  9.0 us,  0.2 sy,  0.0 ni, 90.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem:  13171580+total, 10356851+used, 28147296 free,   257196 buffers
    KiB Swap:        0 total,        0 used,        0 free.  1085868 cached Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
     1522 root      10 -10 6475196 287780  10192 S 200.4  0.2  14250:20 ovs-vswitchd
  2. Verify that ovs-vswitchd is running with

    --dpdk option. ps -ef | grep ovs-vswitchd
  3. PMD thread(s) are started when a DPDK port is added to an OvS bridge. Verify the port is on the bridge.

    tux > sudo ovs-vsctl show
  4. A DPDK port cannot be added to an OvS bridge unless it is bound to a driver. Verify that the DPDK port is bound.

    tux > sudo dpdk_nic_bind -s
  5. Verify that the proper number of hugepages is on the correct NUMA node

    tux > sudo virsh freepages --all

    or

    tux > sudo grep -R "" /sys/kernel/mm/hugepages/ /proc/sys/vm/*huge*
  6. Verify that the VM and the DPDK PMD threads have both mapped the same hugepage(s)

    # this will yield 2 process ids, use the 2nd one
    tux > sudo ps -ef | grep ovs-vswitchd
    tux > sudo ls -l /proc/PROCESS-ID/fd | grep huge
    
    # if running more than 1 VM you will need to figure out which one to use
    tux > sudo ps -ef | grep qemu
    tux > sudo ls -l /proc/PROCESS-ID/fd | grep huge
9.3.14.6.7 Errors Edit source

VM does not get fixed IP

  1. DPDK Poll Mode drivers (PMD) communicates with the VM by direct access of the VM hugepage. If a VM is not created using hugepages (see Section 9.3.14.3, “Configuring Hugepages for DPDK in Neutron Networks”), there is no way for DPDK to communicate with the VM and the VM will never be connected to the network.

  2. It has been observed that the DPDK communication with VM fails if the shared-memory is not disabled in BIOS for DL360 Gen9.

Vestiges of non-existent DPDK devices

  1. Incorrect input models that do not use the correct DPDK device name or do not use sequential port IDs starting at 0 may leave non-existent devices in the OvS database. While this does not affect proper functionality it may be confusing.

Startup issues

  1. Running the following will help diagnose startup issues with ovs-vswitchd:

    tux > sudo journalctl -u openvswitch.service --all

9.3.15 SR-IOV and PCI Passthrough Support Edit source

SUSE OpenStack Cloud supports both single-root I/O virtualization (SR-IOV) and PCI passthrough (PCIPT). Both technologies provide for better network performance.

This improves network I/O, decreases latency, and reduces processor overhead.

9.3.15.1 SR-IOV Edit source

A PCI-SIG Single Root I/O Virtualization and Sharing (SR-IOV) Ethernet interface is a physical PCI Ethernet NIC that implements hardware-based virtualization mechanisms to expose multiple virtual network interfaces that can be used by one or more virtual machines simultaneously. With SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF).

When compared with a PCI Passthtrough Ethernet interface, an SR-IOV Ethernet interface:

  • Provides benefits similar to those of a PCI Passthtrough Ethernet interface, including lower latency packet processing.

  • Scales up more easily in a virtualized environment by providing multiple VFs that can be attached to multiple virtual machine interfaces.

  • Shares the same limitations, including the lack of support for LAG, QoS, ACL, and live migration.

  • Has the same requirements regarding the VLAN configuration of the access switches.

The process for configuring SR-IOV includes creating a VLAN provider network and subnet, then attaching VMs to that network.

With SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF)

9.3.15.2 PCI passthrough Ethernet interfaces Edit source

A passthrough Ethernet interface is a physical PCI Ethernet NIC on a compute node to which a virtual machine is granted direct access. PCI passthrough allows a VM to have direct access to the hardware without being brokered by the hypervisor. This minimizes packet processing delays but at the same time demands special operational considerations. For all purposes, a PCI passthrough interface behaves as if it were physically attached to the virtual machine. Therefore any potential throughput limitations coming from the virtualized environment, such as the ones introduced by internal copying of data buffers, are eliminated. However, by bypassing the virtualized environment, the use of PCI passthrough Ethernet devices introduces several restrictions that must be taken into consideration. They include:

  • no support for LAG, QoS, ACL, or host interface monitoring

  • no support for live migration

  • no access to the compute node's OVS switch

A passthrough interface bypasses the compute node's OVS switch completely, and is attached instead directly to the provider network's access switch. Therefore, proper routing of traffic to connect the passthrough interface to a particular tenant network depends entirely on the VLAN tagging options configured on both the passthrough interface and the access port on the switch (TOR).

The access switch routes incoming traffic based on a VLAN ID, which ultimately determines the tenant network to which the traffic belongs. The VLAN ID is either explicit, as found in incoming tagged packets, or implicit, as defined by the access port's default VLAN ID when the incoming packets are untagged. In both cases the access switch must be configured to process the proper VLAN ID, which therefore has to be known in advance

9.3.15.3 Leveraging PCI Passthrough Edit source

Two parts are necessary to leverage PCI passthrough on a SUSE OpenStack Cloud 8 Compute Node: preparing the Compute Node, preparing Nova and Glance.

  1. Preparing the Compute Node

    1. There should be no kernel drivers or binaries with direct access to the PCI device. If there are kernel modules, they should be blacklisted.

      For example, it is common to have a nouveau driver from when the node was installed. This driver is a graphics driver for Nvidia-based GPUs. It must be blacklisted as shown in this example.

      ardana > echo 'blacklist nouveau' >> /etc/modprobe.d/nouveau-default.conf

      The file location and its contents are important; the filename is your choice. Other drivers can be blacklisted in the same manner, possibly including Nvidia drivers.

    2. On the host, iommu_groups should be enabled. To check if IOMMU is enabled:

      root # virt-host-validate
      .....
      QEMU: Checking if IOMMU is enabled by kernel
      : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
      .....

      To modify the kernel cmdline as suggested in the warning, edit the file /etc/default/grub and append intel_iommu=on to the GRUB_CMDLINE_LINUX_DEFAULT variable. Then run update-bootloader.

      A reboot will be required for iommu_groups to be enabled.

    3. After the reboot, check that IOMMU is enabled:

      root # virt-host-validate
      .....
      QEMU: Checking if IOMMU is enabled by kernel
      : PASS
      .....
    4. Confirm IOMMU groups are available by finding the group associated with your PCI device (for example Nvidia GPU):

      ardana > lspci -nn | grep -i nvidia
      08:00.0 VGA compatible controller [0300]: NVIDIA Corporation GT218 [NVS 300] [10de:10d8] (rev a2)
      08:00.1 Audio device [0403]: NVIDIA Corporation High Definition Audio Controller [10de:0be3] (rev a1)

      In this example, 08:00.0 and 08:00.1 are addresses of the PCI device. The vendorID is 10de. The productIDs are 10d8 and 0be3.

    5. Confirm that the devices are available for passthrough:

      ardana > ls -ld /sys/kernel/iommu_groups/*/devices/*08:00.?/
      drwxr-xr-x 3 root root 0 Feb 14 13:05 /sys/kernel/iommu_groups/20/devices/0000:08:00.0/
      drwxr-xr-x 3 root root 0 Feb 19 16:09 /sys/kernel/iommu_groups/20/devices/0000:08:00.1/
      Note
      Note

      With PCI passthrough, only an entire IOMMU group can be passed. Parts of the group cannot be passed. In this example, the IOMMU group is 20.

  2. Preparing Nova and Glance for passthrough

    Information about configuring Nova and Glance is available in the documentation at https://docs.openstack.org/nova/pike/admin/pci-passthrough.html. Both nova-compute and nova-scheduler must be configured.

9.3.15.4 Supported Intel 82599 Devices Edit source

Table 9.1: Intel 82599 devices supported with SRIOV and PCIPT
VendorDeviceTitle
Intel Corporation10f882599 10 Gigabit Dual Port Backplane Connection
Intel Corporation10f982599 10 Gigabit Dual Port Network Connection
Intel Corporation10fb82599ES 10-Gigabit SFI/SFP+ Network Connection
Intel Corporation10fc82599 10 Gigabit Dual Port Network Connection

9.3.15.5 SRIOV PCIPT configuration Edit source

If you plan to take advantage of SR-IOV support in SUSE OpenStack Cloud you will need to plan in advance to meet the following requirements:

  1. Use one of the supported NIC cards:

    • HP Ethernet 10Gb 2-port 560FLR-SFP+ Adapter (Intel Niantic). Product part number: 665243-B21 -- Same part number for the following card options:

      • FlexLOM card

      • PCI slot adapter card

  2. Identify the NIC ports to be used for PCI Passthrough devices and SRIOV devices from each compute node

  3. Ensure that:

    • SRIOV is enabled in the BIOS

    • HP Shared memory is disabled in the BIOS on the compute nodes.

    • The Intel boot agent is disabled on the compute (Section 9.3.15.11, “Intel bootutils” can be used to perform this)

    Note
    Note

    Because of Intel driver limitations, you cannot use a NIC port as an SRIOV NIC as well as a physical NIC. Using the physical function to carry the normal tenant traffic through the OVS bridge at the same time as assigning the VFs from the same NIC device as passthrough to the guest VM is not supported.

If the above prerequisites are met, then SR-IOV or PCIPT can be reconfigured at any time. There is no need to do it at install time.

9.3.15.6 Deployment use cases Edit source

The following are typical use cases that should cover your particular needs:

  1. A device on the host needs to be enabled for both PCI-passthrough and PCI-SRIOV during deployment. At run time Nova decides whether to use physical functions or virtual function depending on vnic_type of the port used for booting the VM.

  2. A device on the host needs to be configured only for PCI-passthrough.

  3. A device on the host needs to be configured only for PCI-SRIOV virtual functions.

9.3.15.7 Input model updates Edit source

SUSE OpenStack Cloud 8 provides various options for the user to configure the network for tenant VMs. These options have been enhanced to support SRIOV and PCIPT.

the Cloud Lifecycle Manager input model changes to support SRIOV and PCIPT are as follows. If you were familiar with the configuration settings previously, you will notice these changes.

net_interfaces.yml: This file defines the interface details of the nodes. In it, the following fields have been added under the compute node interface section:

KeyValue
sriov_only:

Indicates that only SR-IOV be enabled on the interface. This should be set to true if you want to dedicate the NIC interface to support only SR-IOV functionality.

pci-pt:

When this value is set to true, it indicates that PCIPT should be enabled on the interface.

vf-count:

Indicates the number of VFs to be configured on a given interface.

In control_plane.yml, under Compute resource neutron-sriov-nic-agent has been added as service components

under resources:

KeyValue
name: Compute
resource-prefix: Comp
server-role:COMPUTE-ROLE
allocation-policy: Any
min-count: 0
service-components:ntp-client
 nova-compute
 nova-compute-kvm
 neutron-l3-agent
 neutron-metadata-agent
 neutron-openvswitch-agent
 neutron-lbaasv2-agent
 - neutron-sriov-nic-agent*

nic_device_data.yml: This is the new file added with this release to support SRIOV and PCIPT configuration details. It contains information about the specifics of a nic, and is found here: ~/openstack/ardana/services/osconfig/nic_device_data.yml. The fields in this file are as follows.

  1. nic-device-types: The nic-device-types section contains the following key-value pairs:

    KeyValue
    name:

    The name of the nic-device-types that will be referenced in nic_mappings.yml

    family:

    The name of the nic-device-families to be used with this nic_device_type

    device_id:

    Device ID as specified by the vendor for the particular NIC

    type:

    The value of this field can be "simple-port" or "multi-port". If a single bus address is assigned to more than one nic it will be multi-port, else if there is a one-to one mapping between bus address and the nic then it will be simple-port.

  2. nic-device-families: The nic-device-families section contains the following key-value pairs:

    KeyValue
    name:

    The name of the device family that can be used for reference in nic-device-types.

    vendor-id:

    Vendor ID of the NIC

    config-script:

    A script file used to create the virtual functions (VF) on the Compute node.

    driver:

    Indicates the NIC driver that needs to be used.

    vf-count-type:

    This value can be either "port" or "driver".

    “port”:

    Indicates that the device supports per-port virtual function (VF) counts.

    “driver:”

    Indicates that all ports using the same driver will be configured with the same number of VFs, whether or not the interface model specifies a vf-count attribute for the port. If two or more ports specify different vf-count values, the config processor errors out.

    Max-vf-count:

    This field indicates the maximum VFs that can be configured on an interface as defined by the vendor.

control_plane.yml: This file provides the information about the services to be run on a particular node. To support SR-IOV on a particular compute node, you must run neutron-sriov-nic-agent on that node.

Mapping the use cases with various fields in input model

 Vf-countSR-IOVPCIPTOVS bridgeCan be NIC bondedUse case
sriov-only: trueMandatoryYesNoNoNoDedicated to SRIOV
pci-pt : trueNot SpecifiedNoYesNoNoDedicated to PCI-PT
pci-pt : trueSpecifiedYesYesNoNoPCI-PT or SRIOV
pci-pt and sriov-only keywords are not specifiedSpecifiedYesNoYesNoSRIOV with PF used by host
pci-pt and sriov-only keywords are not specifiedNot SpecifiedNoNoYesYesTraditional/Usual use case

9.3.15.8 Mappings between nic_mappings.yml and net_interfaces.yml Edit source

The following diagram shows which fields in nic_mappings.yml map to corresponding fields in net_interfaces.yml:

9.3.15.9 Example Use Cases for Intel Edit source

  1. Nic-device-types and nic-device-families with Intel 82559 with ixgbe as the driver.

    nic-device-types:
        - name: ''8086:10fb
          family: INTEL-82599
          device-id: '10fb'
          type: simple-port
    nic-device-families:
        # Niantic
        - name: INTEL-82599
          vendor-id: '8086'
          config-script: intel-82599.sh
          driver: ixgbe
          vf-count-type: port
          max-vf-count: 63
  2. net_interfaces.yml for the SRIOV-only use case:

    - name: COMPUTE-INTERFACES
       - name: hed1
         device:
           name: hed1
           sriov-only: true
           vf-count: 6
         network-groups:
          - GUEST1
  3. net_interfaces.yml for the PCIPT-only use case:

    - name: COMPUTE-INTERFACES
       - name: hed1
         device:
           name: hed1
           pci-pt: true
        network-groups:
         - GUEST1
  4. net_interfaces.yml for the SRIOV and PCIPT use case

     - name: COMPUTE-INTERFACES
        - name: hed1
          device:
            name: hed1
            pci-pt: true
            vf-count: 6
          network-groups:
          - GUEST1
  5. net_interfaces.yml for SRIOV and Normal Virtio use case

    - name: COMPUTE-INTERFACES
       - name: hed1
         device:
            name: hed1
            vf-count: 6
          network-groups:
          - GUEST1
  6. net_interfaces.yml for PCI-PT (hed1 and hed4 refer to the DUAL ports of the PCI-PT NIC)

        - name: COMPUTE-PCI-INTERFACES
          network-interfaces:
          - name: hed3
            device:
              name: hed3
            network-groups:
              - MANAGEMENT
              - EXTERNAL-VM
            forced-network-groups:
              - EXTERNAL-API
          - name: hed1
            device:
              name: hed1
              pci-pt: true
            network-groups:
              - GUEST
          - name: hed4
            device:
              name: hed4
              pci-pt: true
            network-groups:
              - GUEST

9.3.15.10 Launching Virtual Machines Edit source

Provisioning a VM with SR-IOV NIC is a two-step process.

  1. Create a Neutron port with vnic_type = direct.

    ardana > neutron port-create $net_id --name sriov_port --binding:vnic_type direct
  2. Boot a VM with the created port-id.

    ardana > nova boot --flavor m1.large --image ubuntu_14.04 --nic port-id=$port_id test-sriov

Provisioning a VM with PCI-PT NIC is a two-step process.

  1. Create two Neutron ports with vnic_type = direct-physical.

    ardana > neutron port-create net1 --name pci-port1 --vnic_type=direct-physical
    neutron port-create net1 --name pci-port2  --vnic_type=direct-physical
  2. Boot a VM with the created ports.

    ardana > nova boot --flavor 4 --image opensuse --nic port-id pci-port1-port-id \
    --nic port-id pci-port2-port-id vm1-pci-passthrough

If PCI-PT VM gets stuck (hangs) at boot time when using an Intel NIC, the boot agent should be disabled.

9.3.15.11 Intel bootutils Edit source

When Intel cards are used for PCI-PT, a tenant VM can get stuck at boot time. When this happens, you should download Intel bootutils and use it to should disable bootagent.

  1. Download Preebot.tar.gz from https://downloadcenter.intel.com/download/19186/Intel-Ethernet-Connections-Boot-Utility-Preboot-Images-and-EFI-Drivers

  2. Untar the Preboot.tar.gz on the compute node where the PCI-PT VM is to be hosted.

  3. Go to ~/APPS/BootUtil/Linux_x64

    cd ~/APPS/BootUtil/Linux_x64

    and run following command

    ./bootutil64e -BOOTENABLE disable -all
  4. Boot the PCI-PT VM and it should boot without getting stuck.

    Note
    Note

    Here even though VM console shows VM getting stuck at PXE boot, it is not related to BIOS PXE settings.

9.3.15.12 Making input model changes and implementing PCI PT and SR-IOV Edit source

To implent the configuration you require, log into the Cloud Lifecycle Manager node and update the Cloud Lifecycle Manager model files to enable SR-IOV or PCIPT following the relevent use case explained above. You will need to edit

  • net_interfaces.yml

  • nic_device_data.yml

  • control_plane.yml

To make the edits,

  1. Check out the site branch of the local git repository and change to the correct directory:

    ardana > git checkout site
    ardana > cd ~/openstack/my_cloud/definition/data/
  2. Open each file in vim or another editor and make the necessary changes. Save each file, then commit to the local git repository:

    ardana > git add -A
    ardana > git commit -m "your commit message goes here in quotes"
  3. Here you will have the Cloud Lifecycle Manager enable your changes by running the necessary playbooks:

    ardana > cd ~/openstack/ardana/ansible
    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts site.yml
Note
Note

After running the site.yml playbook above, you must reboot the compute nodes that are configured with Intel PCI devices.

Note
Note

When a VM is running on an SRIOV port on a given compute node, reconfiguration is not supported.

You can set the number of virtual functions that must be enabled on a compute node at install time. You can update the number of virtual functions after deployment. If any VMs have been spawned before you change the number of virtual functions, those VMs may lose connectivity. Therefore, it is always recommended that if any virtual function is used by any tenant VM, you should not reconfigure the virtual functions. Instead, you should delete/migrate all the VMs on that NIC before reconfiguring the number of virtual functions.

9.3.15.13 Limitations Edit source

  • Security groups are not applicable for PCI-PT and SRIOV ports.

  • Live migration is not supported for VMs with PCI-PT and SRIOV ports.

  • Rate limiting (QoS) is not applicable on SRIOV and PCI-PT ports.

  • SRIOV/PCIPT is not supported for VxLAN network.

  • DVR is not supported with SRIOV/PCIPT.

  • For Intel cards, the same NIC cannot be used for both SRIOV and normal VM boot.

  • Current upstream OpenStack code does not support this hot plugin of SRIOV/PCIPT interface using the nova attach_interface command. See https://review.openstack.org/#/c/139910/ for more information.

  • Neutron port-update when admin state is down will not work.

  • SLES Compute Nodes with dual-port PCI-PT NICs, both ports should always be passed in the VM. It is not possible to split the dual port and pass through just a single port.

9.3.15.14 Enabling PCI-PT on HPE DL360 Gen 9 Servers Edit source

The HPE DL360 Gen 9 and HPE ProLiant systems with Intel processors use a region of system memory for sideband communication of management information. The BIOS sets up Reserved Memory Region Reporting (RMRR) to report these memory regions and devices to the operating system. There is a conflict between the Linux kernel and RMRR which causes problems with PCI pass-through (PCI-PT). This is needed for IOMMU use by DPDK. Note that this does not affect SR-IOV.

In order to enable PCI-PT on the HPE DL360 Gen 9 you must have a version of firmware that supports setting this and you must change a BIOS setting.

To begin, get the latest firmware and install it on your compute nodes.

Once the firmware has been updated:

  1. Reboot the server and press F9 (system utilities) during POST (power on self test)

  2. Choose System Configuration

  3. Select the NIC for which you want to enable PCI-PT

  4. Choose Device Level Configuration

  5. Disable the shared memory feature in the BIOS.

  6. Save the changes and reboot server

9.3.16 Setting up VLAN-Aware VMs Edit source

Creating a VM with a trunk port will allow a VM to gain connectivity to one or more networks over the same virtual NIC (vNIC) through the use VLAN interfaces in the guest VM. Connectivity to different networks can be added and removed dynamically through the use of subports. The network of the parent port will be presented to the VM as the untagged VLAN, and the networks of the child ports will be presented to the VM as the tagged VLANs (the VIDs of which can be chosen arbitrarily as long as they are unique to that trunk). The VM will send/receive VLAN-tagged traffic over the subports, and Neutron will mux/demux the traffic onto the subport's corresponding network. This is not to be confused with VLAN transparency where a VM can pass VLAN-tagged traffic transparently across the network without interference from Neutron. VLAN transparency is not supported.

9.3.16.1 Terminology Edit source

  • Trunk: a resource that logically represents a trunked vNIC and references a parent port.

  • Parent port: a Neutron port that a Trunk is referenced to. Its network is presented as the untagged VLAN.

  • Subport: a resource that logically represents a tagged VLAN port on a Trunk. A Subport references a child port and consists of the <port>,<segmentation-type>,<segmentation-id> tuple. Currently only the 'vlan' segmentation type is supported.

  • Child port: a Neutron port that a Subport is referenced to. Its network is presented as a tagged VLAN based upon the segmentation-id used when creating/adding a Subport.

  • Legacy VM: a VM that does not use a trunk port.

  • Legacy port: a Neutron port that is not used in a Trunk.

  • VLAN-aware VM: a VM that uses at least one trunk port.

9.3.16.2 Trunk CLI reference Edit source

CommandAction
network trunk create Create a trunk.
network trunk delete Delete a given trunk.
network trunk list List all trunks.
network trunk show Show information of a given trunk.
network trunk set Add subports to a given trunk.
network subport list List all subports for a given trunk.
network trunk unsetRemove subports from a given trunk.
network trunk setUpdate trunk properties.

9.3.16.3 Enabling VLAN-aware VM capability Edit source

  1. Edit ~/openstack/my_cloud/config/neutron/neutron.conf.j2 to add the "trunk" service_plugin:

    service_plugins = {{ neutron_service_plugins }},trunk
  2. Edit ~/openstack/my_cloud/config/neutron/ml2_conf.ini.j2 to enable the noop firewall driver:

    [securitygroup]
    firewall_driver = neutron.agent.firewall.NoopFirewallDriver
    Note
    Note

    This is a manual configuration step because it must be made apparent that this step disables Neutron security groups completely. The default SUSE OpenStack Cloud firewall_driver is neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewall Driver which does not implement security groups for trunk ports. Optionally, the SUSE OpenStack Cloud default firewall_driver may still be used (that is, skip this step), which would provide security groups for legacy VMs but not for VLAN-aware VMs. However, this mixed environment is not recommended. For more information, see Section 9.3.16.6, “Firewall issues”.

  3. Commit the configuration changes:

    ardana > git add -A
    ardana > git commit -m "Enable vlan-aware VMs"
    ardana > cd ~/openstack/ardana/ansible/
  4. If this is an initial deployment, continue the rest of normal deployment process:

    ardana > ansible-playbook -i hosts/localhost config-processor-run.yml
    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts site.yml
  5. If the cloud has already been deployed and this is a reconfiguration:

    ardana > ansible-playbook -i hosts/localhost ready-deployment.yml
    ardana > cd ~/scratch/ansible/next/ardana/ansible
    ardana > ansible-playbook -i hosts/verb_hosts neutron-reconfigure.yml

9.3.16.4 Use Cases Edit source

Creating a trunk port

Assume that a number of Neutron networks/subnets already exist: private, foo-net, and bar-net. This will create a trunk with two subports allocated to it. The parent port will be on the "private" network, while the two child ports will be on "foo-net" and "bar-net", respectively:

  1. Create a port that will function as the trunk's parent port:

    ardana > neutron port-create --name trunkparent private
  2. Create ports that will function as the child ports to be used in subports:

    ardana > neutron port-create --name subport1 foo-net
    ardana > neutron port-create --name subport2 bar-net
  3. Create a trunk port using the openstack network trunk create command, passing the parent port created in step 1 and child ports created in step 2:

    ardana > openstack network trunk create --parent-port trunkparent --subport port=subport1,segmentation-type=vlan,segmentation-id=1 --subport port=subport2,segmentation-type=vlan,segmentation-id=2 mytrunk
    +-----------------+-----------------------------------------------------------------------------------------------+
    | Field           | Value                                                                                         |
    +-----------------+-----------------------------------------------------------------------------------------------+
    | admin_state_up  | UP                                                                                            |
    | created_at      | 2017-06-02T21:49:59Z                                                                          |
    | description     |                                                                                               |
    | id              | bd822ebd-33d5-423e-8731-dfe16dcebac2                                                          |
    | name            | mytrunk                                                                                       |
    | port_id         | 239f8807-be2e-4732-9de6-c64519f46358                                                          |
    | project_id      | f51610e1ac8941a9a0d08940f11ed9b9                                                              |
    | revision_number | 1                                                                                             |
    | status          | DOWN                                                                                          |
    | sub_ports       | port_id='9d25abcf-d8a4-4272-9436-75735d2d39dc', segmentation_id='1', segmentation_type='vlan' |
    |                 | port_id='e3c38cb2-0567-4501-9602-c7a78300461e', segmentation_id='2', segmentation_type='vlan' |
    | tenant_id       | f51610e1ac8941a9a0d08940f11ed9b9                                                              |
    | updated_at      | 2017-06-02T21:49:59Z                                                                          |
    +-----------------+-----------------------------------------------------------------------------------------------+
    
    $ openstack network subport list --trunk mytrunk
    +--------------------------------------+-------------------+-----------------+
    | Port                                 | Segmentation Type | Segmentation ID |
    +--------------------------------------+-------------------+-----------------+
    | 9d25abcf-d8a4-4272-9436-75735d2d39dc | vlan              |               1 |
    | e3c38cb2-0567-4501-9602-c7a78300461e | vlan              |               2 |
    +--------------------------------------+-------------------+-----------------+

    Optionally, a trunk may be created without subports (they can be added later):

    ardana > openstack network trunk create --parent-port trunkparent mytrunk
    +-----------------+--------------------------------------+
    | Field           | Value                                |
    +-----------------+--------------------------------------+
    | admin_state_up  | UP                                   |
    | created_at      | 2017-06-02T21:45:35Z                 |
    | description     |                                      |
    | id              | eb8a3c7d-9f0a-42db-b26a-ca15c2b38e6e |
    | name            | mytrunk                              |
    | port_id         | 239f8807-be2e-4732-9de6-c64519f46358 |
    | project_id      | f51610e1ac8941a9a0d08940f11ed9b9     |
    | revision_number | 1                                    |
    | status          | DOWN                                 |
    | sub_ports       |                                      |
    | tenant_id       | f51610e1ac8941a9a0d08940f11ed9b9     |
    | updated_at      | 2017-06-02T21:45:35Z                 |
    +-----------------+--------------------------------------+

    A port that is already bound (that is, already in use by a VM) cannot be upgraded to a trunk port. The port must be unbound to be eligible for use as a trunk's parent port. When adding subports to a trunk, the child ports must be unbound as well.

Checking a port's trunk details

Once a trunk has been created, its parent port will show the trunk_details attribute, which consists of the trunk_id and list of subport dictionaries:

ardana > neutron port-show -F trunk_details trunkparent
+---------------+-------------------------------------------------------------------------------------+
| Field         | Value                                                                               |
+---------------+-------------------------------------------------------------------------------------+
| trunk_details | {"trunk_id": "bd822ebd-33d5-423e-8731-dfe16dcebac2", "sub_ports":                   |
|               | [{"segmentation_id": 2, "port_id": "e3c38cb2-0567-4501-9602-c7a78300461e",          |
|               | "segmentation_type": "vlan", "mac_address": "fa:16:3e:11:90:d2"},                   |
|               | {"segmentation_id": 1, "port_id": "9d25abcf-d8a4-4272-9436-75735d2d39dc",           |
|               | "segmentation_type": "vlan", "mac_address": "fa:16:3e:ff:de:73"}]}                  |
+---------------+-------------------------------------------------------------------------------------+

Ports that are not trunk parent ports will not have a trunk_details field:

ardana > neutron port-show -F trunk_details subport1
need more than 0 values to unpack

Adding subports to a trunk

Assuming a trunk and new child port have been created already, the trunk-subport-add command will add one or more subports to the trunk.

  1. Run openstack network trunk set

    ardana > openstack network trunk set --subport port=subport3,segmentation-type=vlan,segmentation-id=3 mytrunk
  2. Run openstack network subport list

    ardana > openstack network subport list --trunk mytrunk
    +--------------------------------------+-------------------+-----------------+
    | Port                                 | Segmentation Type | Segmentation ID |
    +--------------------------------------+-------------------+-----------------+
    | 9d25abcf-d8a4-4272-9436-75735d2d39dc | vlan              |               1 |
    | e3c38cb2-0567-4501-9602-c7a78300461e | vlan              |               2 |
    | bf958742-dbf9-467f-b889-9f8f2d6414ad | vlan              |               3 |
    +--------------------------------------+-------------------+-----------------+
Note
Note

The --subport option may be repeated multiple times in order to add multiple subports at a time.

Removing subports from a trunk

To remove a subport from a trunk, use openstack network trunk unset command:

ardana > openstack network trunk unset --subport subport3 mytrunk

Deleting a trunk port

To delete a trunk port, use the openstack network trunk delete command:

ardana > openstack network trunk delete mytrunk

Once a trunk has been created successfully, its parent port may be passed to the nova boot command, which will make the VM VLAN-aware:

ardana > nova boot --image ubuntu-server --flavor 1 --nic port-id=239f8807-be2e-4732-9de6-c64519f46358 vlan-aware-vm
Note
Note

A trunk cannot be deleted until its parent port is unbound. Mainly, this means you must delete the VM using the trunk port before you are allowed to delete the trunk.

9.3.16.5 VLAN-aware VM network configuration Edit source

This section illustrates how to configure the VLAN interfaces inside a VLAN-aware VM based upon the subports allocated to the trunk port being used.

  1. Run openstack network trunk subport list to see the VLAN IDs in use on the trunk port:

    ardana > openstack network subport list --trunk mytrunk
    +--------------------------------------+-------------------+-----------------+
    | Port                                 | Segmentation Type | Segmentation ID |
    +--------------------------------------+-------------------+-----------------+
    | e3c38cb2-0567-4501-9602-c7a78300461e | vlan              |               2 |
    +--------------------------------------+-------------------+-----------------+
  2. Run neutron port-show on the child port to get its mac_address:

    ardana > neutron port-show -F mac_address 08848e38-50e6-4d22-900c-b21b07886fb7
    +-------------+-------------------+
    | Field       | Value             |
    +-------------+-------------------+
    | mac_address | fa:16:3e:08:24:61 |
    +-------------+-------------------+
  3. Log into the VLAN-aware VM and run the following commands to set up the VLAN interface:

    tux > sudo ip link add link ens3 ens3.2 address fa:16:3e:11:90:d2 broadcast ff:ff:ff:ff:ff:ff type vlan id 2
    $ sudo ip link set dev ens3.2 up
  4. Note the usage of the mac_address from step 2 and VLAN ID from step 1 in configuring the VLAN interface:

    tux > sudo ip link add link ens3 ens3.2 address fa:16:3e:11:90:d2 broadcast ff:ff:ff:ff:ff:ff type vlan id 2
  5. Trigger a DHCP request for the new vlan interface to verify connectivity and retrieve its IP address. On an Ubuntu VM, this might be:

    tux > sudo dhclient ens3.2
    tux > sudo ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast state UP group default qlen 1000
        link/ether fa:16:3e:8d:77:39 brd ff:ff:ff:ff:ff:ff
        inet 10.10.10.5/24 brd 10.10.10.255 scope global ens3
           valid_lft forever preferred_lft forever
        inet6 fe80::f816:3eff:fe8d:7739/64 scope link
           valid_lft forever preferred_lft forever
    3: ens3.2@ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
        link/ether fa:16:3e:11:90:d2 brd ff:ff:ff:ff:ff:ff
        inet 10.10.12.7/24 brd 10.10.12.255 scope global ens3.2
           valid_lft forever preferred_lft forever
        inet6 fe80::f816:3eff:fe11:90d2/64 scope link
           valid_lft forever preferred_lft forever

9.3.16.6 Firewall issues Edit source

The SUSE OpenStack Cloud default firewall_driver is neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver. This default does not implement security groups for VLAN-aware VMs, but it does implement security groups for legacy VMs. For this reason, it is recommended to disable Neutron security groups altogether when using VLAN-aware VMs. To do so, set:

firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Doing this will prevent having a mix of firewalled and non-firewalled VMs in the same environment, but it should be done with caution because all VMs would be non-firewalled.

Print this page