The default networks set up in SUSE OpenStack Cloud are class C networks with 256 IP addresses each. This limits the maximum number of instances that can be started simultaneously. Addresses within the networks are allocated as outlined in the following table. Use the YaST Crowbar module to make customizations (see Chapter 7, Crowbar Setup). The last address in the IP address range of each network is always reserved as the broadcast address. This assignment cannot be changed.

For an overview of the minimum number of IP addresses needed for each of the ranges in the network settings, see Table 2.1, “Minimum Number of IP Addresses for Network Ranges”.

Note: Limitations of the Default Network Proposal

The default network proposal as described below limits the maximum number of Compute Nodes to 80, the maximum number of floating IP addresses to 61 and the maximum number of addresses in the nova_fixed network to 204.

To overcome these limitations you need to reconfigure the network setup by using appropriate address ranges. Do this by either using the YaST Crowbar module as described in Chapter 7, Crowbar Setup, or by manually editing the network template file as described in Section 7.5, “Custom Network Configuration”.

Note: Addresses for Additional Servers

Addresses not used in the ranges mentioned above can be used to add additional servers with static addresses to SUSE OpenStack Cloud. Such servers can be used to provide additional services. A SUSE Manager server inside SUSE OpenStack Cloud, for example, must be configured using one of these addresses.

SUSE OpenStack Cloud supports different network modes defined in Crowbar: single, dual, and team. As of SUSE OpenStack Cloud 7, the networking mode is applied to all nodes and the Administration Server. That means that all machines need to meet the hardware requirements for the chosen mode. The network mode can be configured using the YaST Crowbar module (Chapter 7, Crowbar Setup). The network mode cannot be changed after the cloud is deployed.

Other, more flexible network mode setups can be configured by manually editing the Crowbar network configuration files. See Section 7.5, “Custom Network Configuration” for more information. SUSE or a partner can assist you in creating a custom setup within the scope of a consulting services agreement (see http://www.suse.com/consulting/ for more information on SUSE consulting).

Important: Network Device Bonding is Required for HA

Network device bonding is required for an HA setup of SUSE OpenStack Cloud. If you are planning to move your cloud to an HA setup at a later point in time, make sure to use a network mode in the YaST Crowbar that supports network device bonding.

Otherwise a migration to an HA setup is not supported.

2.1.2.1 Single Network Mode #

In single mode you use one Ethernet card for all the traffic:

2.1.2.2 Dual Network Mode #

Dual mode needs two Ethernet cards (on all nodes but Administration Server) to completely separate traffic between the Admin Network and the public network:

2.1.2.3 Team Network Mode #

Team mode is similar to single mode, except that you combine several Ethernet cards to a “bond” (network device bonding). Team mode needs two or more Ethernet cards.

When using team mode, you must choose a “bonding policy” that defines how to use the combined Ethernet cards. You can either set them up for fault tolerance, performance (load balancing), or a combination of both.

Enabling access to the Administration Server from another network requires an external gateway. This option offers maximum flexibility, but requires additional hardware and may be less secure than you require. Therefore SUSE OpenStack Cloud offers a second option for accessing the Administration Server: the bastion network. You only need a dedicated Ethernet card and a static IP address from the external network to set it up.

The bastion network setup (see Section 7.3.1, “Setting Up a Bastion Network” for setup instructions) enables logging in to the Administration Server via SSH from the company network. A direct login to other nodes in the cloud is not possible. However, the Administration Server can act as a “jump host”: First log in to the Administration Server via SSH, then log in via SSH to other nodes.

The Administration Server acts as a name server for all nodes in the cloud. If the Administration Server has access to the outside, then you can add additional name servers that are automatically used to forward requests. If additional name servers are found on your cloud deployment, the name server on the Administration Server is automatically configured to forward requests for non-local records to these servers.

The Administration Server must have a fully qualified host name. The domain name you specify is used for the DNS zone. It is required to use a sub-domain such as cloud.example.com. The Administration Server must have authority over the domain it is on so that it can create records for discovered nodes. As a result, it will not forward requests for names it cannot resolve in this domain, and thus cannot resolve names for the second-level domain, .e.g. example.com, other than for nodes in the cloud.

This host name must not be changed after SUSE OpenStack Cloud has been deployed. The OpenStack nodes are named after their MAC address by default, but you can provide aliases, which are easier to remember when allocating the nodes. The aliases for the OpenStack nodes can be changed at any time. It is useful to have a list of MAC addresses and the intended use of the corresponding host at hand when deploying the OpenStack nodes.

SUSE OpenStack Cloud offers two different types of services for persistent storage: object and block storage. Object storage lets you upload and download files (similar to an FTP server), whereas a block storage provides mountable devices (similar to a hard disk partition). SUSE OpenStack Cloud provides a repository to store the virtual disk images used to start instances.

Each node in SUSE OpenStack Cloud needs sufficient disk space to store both the operating system and additional data. Requirements and recommendations for the various node types are listed below.

Important: Choose a Hard Disk for the Operating System Installation

The operating system will always be installed on the first hard disk. This is the disk that is listed first in the BIOS, the one from which the machine will boot. Make sure that the hard disk the operating system is installed on will be recognized as the first disk.

2.2.2.3 Compute Nodes #

Unless an instance is started via “Boot from Volume”, it is started with at least one disk, which is a copy of the image from which it has been started. Depending on the flavor you start, the instance may also have a second, so-called “ephemeral” disk. The size of the root disk depends on the image itself. Ephemeral disks are always created as sparse image files that grow up to a defined size as they are “filled”. By default ephemeral disks have a size of 10 GB.

Both disks, root images and ephemeral disk, are directly bound to the instance and are deleted when the instance is terminated. These disks are bound to the Compute Node on which the instance has been started. The disks are created under /var/lib/nova on the Compute Node. Your Compute Nodes should be equipped with enough disk space to store the root images and ephemeral disks.

Note: Ephemeral Disks vs. Block Storage

Do not confuse ephemeral disks with persistent block storage. In addition to an ephemeral disk, which is automatically provided with most instance flavors, you can optionally add a persistent storage device provided by Cinder. Ephemeral disks are deleted when the instance terminates, while persistent storage devices can be reused in another instance.

The maximum disk space required on a compute node depends on the available flavors. A flavor specifies the number of CPUs, RAM, and disk size of an instance. Several flavors ranging from tiny (1 CPU, 512 MB RAM, no ephemeral disk) to xlarge (8 CPUs, 8 GB RAM, 10 GB ephemeral disk) are available by default. Adding custom flavors, and editing and deleting existing flavors is also supported.

To calculate the minimum disk space needed on a compute node, you need to determine the highest disk-space-to-RAM ratio from your flavors. For example:

Flavor small: 2 GB RAM, 100 GB ephemeral disk => 50 GB disk /1 GB RAM

Flavor large: 8 GB RAM, 200 GB ephemeral disk => 25 GB disk /1 GB RAM

So, 50 GB disk /1 GB RAM is the ratio that matters. If you multiply that value by the amount of RAM in GB available on your compute node, you have the minimum disk space required by ephemeral disks. Pad that value with sufficient space for the root disks plus a buffer to leave room for flavors with a higher disk-space-to-RAM ratio in the future.

Warning: Overcommitting Disk Space

The scheduler that decides in which node an instance is started does not check for available disk space. If there is no disk space left on a compute node, this will not only cause data loss on the instances, but the compute node itself will also stop operating. Therefore you must make sure all compute nodes are equipped with enough hard disk space.

The Compute Nodes need to be equipped with a sufficient amount of RAM and CPUs, matching the numbers required by the maximum number of instances running concurrently. An instance started in SUSE OpenStack Cloud cannot share resources from several physical nodes. It uses the resources of the node on which it was started. So if you offer a flavor (see Flavor for a definition) with 8 CPUs and 12 GB RAM, at least one of your nodes should be able to provide these resources. Add 1 GB RAM for every two nodes (including Control Nodes and Storage Nodes) deployed in your cloud.

See Section 2.2.2.3, “Compute Nodes” for storage requirements.

Usually a single CPU and a minimum of 4 GB RAM are sufficient for the Storage Nodes. Memory requirements increase depending on the total number of nodes in SUSE OpenStack Cloud—the higher the number of nodes, the more RAM you need. A deployment with 50 nodes requires a minimum of 20 GB for each Storage Node. If you use Ceph as storage, the storage nodes should be equipped with an additional 2 GB RAM per OSD (Ceph object storage daemon).

For storage requirements, see Section 2.2.2.4, “Storage Nodes (optional)”.

SUSE OpenStack Cloud can be extended by SUSE Enterprise Storage for setting up a Ceph cluster providing block storage services. To store virtual disks for instances, SUSE OpenStack Cloud uses block storage provided by the Cinder module. Cinder itself needs a back-end providing storage. In production environments this usually is a network storage solution. Cinder can use a variety of network storage back-ends, among them solutions from EMC, Fujitsu, or NetApp. In case your organization does not provide a network storage solution that can be used with SUSE OpenStack Cloud, you can set up a Ceph cluster with SUSE Enterprise Storage. SUSE Enterprise Storage provides a reliable and fast distributed storage architecture using commodity hardware platforms.

Deploying SUSE Enterprise Storage (Ceph) within SUSE OpenStack Cloud is fully supported. Ceph nodes can be deployed using the same interface as for all other SUSE OpenStack Cloud components. It requires a SUSE Enterprise Storage subscription. See https://www.suse.com/products/suse-enterprise-storage/ for more information on SUSE Enterprise Storage.

You need seven software repositories to deploy SUSE OpenStack Cloud and to keep a running SUSE OpenStack Cloud up-to-date. This includes the static product repositories, which do not change over the product life cycle, and the update repositories, which constantly change. The following repositories are needed:

As explained in Section 2.6, “High Availability”, Control Nodes in SUSE OpenStack Cloud can optionally be made highly available with the SUSE Linux Enterprise High Availability Extension. SUSE OpenStack Cloud also comes with full support for installing a Ceph storage cluster, which is provided by the SUSE Enterprise Storage extension. (Deploying Ceph is optional.) The following repositories are required to deploy SLES High Availability Extension and SUSE Enterprise Storage nodes:

The product repositories for SUSE Linux Enterprise Server 12 SP2 and SUSE OpenStack Cloud 7 do not change during the life cycle of a product. Thus, they can be copied to the destination directory from the installation media. However, the pool and update repositories must be kept synchronized with their sources on the SUSE Customer Center. SUSE offers two products that synchronize repositories and make them available within your organization: SUSE Manager (http://www.suse.com/products/suse-manager/, and Subscription Management Tool (which ships with SUSE Linux Enterprise Server 12 SP2).

All repositories must be served via HTTP to be available for SUSE OpenStack Cloud deployment. Repositories that are installed on the Administration Server are made available by the Apache Web server running on the Administration Server. If your organization already uses SUSE Manager or SMT, you can use the repositories provided by these servers.

Making the repositories locally available on the Administration Server has the advantage of a simple network setup within SUSE OpenStack Cloud, and it allows you to seal off the SUSE OpenStack Cloud network from other networks in your organization. Hosting the repositories on a remote server has the advantage of using existing resources and services, and it makes setting up the Administration Server much easier. However, this requires a custom network setup for SUSE OpenStack Cloud, since the Administration Server needs access to the remote server.

The Administration Server provides all services needed to manage and deploy all other nodes in the cloud. If the Administration Server is not available, new cloud nodes cannot be allocated, and you cannot add new roles to cloud nodes.

However, only two services on the Administration Server are single points of failure, without which the cloud cannot continue to run properly: DNS and NTP.

2.6.1.1 Administration Server—Avoiding Points of Failure #

To avoid DNS and NTP as potential points of failure, deploy the roles dns-server and ntp-server to multiple nodes.

Note: Access to External Network

If any configured DNS forwarder or NTP external server is not reachable through the admin network from these nodes, allocate an address in the public network for each node that has the dns-server and ntp-server roles:

crowbar network allocate_ip default `hostname -f` public host

Then the nodes can use the public gateway to reach the external servers. The change will only become effective after the next run of chef-client on the affected nodes.

The Control Node(s) usually run a variety of services without which the cloud would not be able to run properly.

2.6.2.1 Control Node(s)—Avoiding Points of Failure #

To prevent the cloud from avoidable downtime if one or more Control Nodes fail, you can make the following roles highly available:

• database-server (database barclamp)

• keystone-server (keystone barclamp)

• rabbitmq-server (rabbitmq barclamp)

• swift-proxy (swift barclamp)

• glance-server (glance barclamp)

• cinder-controller (cinder barclamp)

• neutron-server (neutron barclamp)

• neutron-network (neutron barclamp)

• nova-controller (nova barclamp)

• nova_dashboard-server (nova_dashboard barclamp)

• ceilometer-server (ceilometer barclamp)

• ceilometer-polling (ceilometer barclamp)

• heat-server (heat barclamp)

Instead of assigning these roles to individual cloud nodes, you can assign them to one or several High Availability clusters. SUSE OpenStack Cloud will then use the Pacemaker cluster stack (shipped with the SUSE Linux Enterprise High Availability Extension) to manage the services. If one Control Node fails, the services will fail over to another Control Node. For details on the Pacemaker cluster stack and the SUSE Linux Enterprise High Availability Extension, refer to the Administration Guide, available at http://www.suse.com/documentation/sle-ha-12/. Note that SUSE Linux Enterprise High Availability Extension includes Linux Virtual Server as the load-balancer, and SUSE OpenStack Cloud uses HAProxy for this purpose (http://haproxy.1wt.eu/).

Note: Recommended Setup

Though it is possible to use the same cluster for all of the roles above, the recommended setup is to use three clusters and to deploy the roles as follows:

• data cluster: database-server and rabbitmq-server

• network cluster: neutron-network (as the neutron-network role may result in heavy network load and CPU impact)

• Trove (always needs to be deployed on a dedicated node)

• services cluster: all other roles listed above (as they are related to API/schedulers)

SUSE OpenStack Cloud does not support High Availability for the LBaaS service plug-in. Thus, failover of a neutron load-balancer to another node can only be configured manually by editing the database.

Important: Cluster Requirements and Recommendations

For setting up the clusters, some special requirements and recommendations apply. For details, refer to Section 2.6.5, “Cluster Requirements and Recommendations”.

If a Compute Node fails, all VMs running on that node will go down. While it cannot protect against failures of individual VMs, a High Availability setup for Compute Nodes helps to minimize VM downtime caused by Compute Node failures. If the nova-compute service or libvirtd fail on a Compute Node, Pacemaker will try to automatically recover them. If recovery fails, or the node itself should become unreachable, the node will be fenced and the VMs will be moved to a different Compute Node.

If you decide to use High Availability for Compute Nodes, your Compute Node will be run as Pacemaker remote nodes. With the pacemaker-remote service, High Availability clusters can be extended to control remote nodes without any impact on scalability, and without having to install the full cluster stack (including corosync) on the remote nodes. Instead, each Compute Node only runs the pacemaker-remote service. The service acts as a proxy, allowing the cluster stack on the “normal” cluster nodes to connect to it and to control services remotely. Thus, the node is effectively integrated into the cluster as a remote node. In this way, the services running on the OpenStack compute nodes can be controlled from the core Pacemaker cluster in a lightweight, scalable fashion.

Find more information about the pacemaker_remote service in Pacemaker Remote—Extending High Availability into Virtual Nodes, available at http://www.clusterlabs.org/doc/.

To configure High Availability for Compute Nodes, you need to adjust the following barclamp proposals:

SUSE OpenStack Cloud offers two different types of storage that can be used for the Storage Nodes: object storage (provided by the OpenStack Swift component) and block storage (provided by Ceph).

Both already consider High Availability aspects by design, therefore it does not require much effort to make the storage highly available.

When considering setting up one or more High Availability clusters, refer to the chapter System Requirements in the Administration Guide for SUSE Linux Enterprise High Availability Extension. The guide is available at http://www.suse.com/documentation/sle-ha-12/.

The HA requirements for Control Node also apply to SUSE OpenStack Cloud. Note that by buying SUSE OpenStack Cloud, you automatically get an entitlement for SUSE Linux Enterprise High Availability Extension.

Especially note the following requirements:

For a basic understanding and detailed information on the SUSE Linux Enterprise High Availability Extension (including the Pacemaker cluster stack), read the Administration Guide. It is available at http://www.suse.com/documentation/sle-ha-12/.

In addition to the chapters mentioned in Section 2.6.5, “Cluster Requirements and Recommendations”, the following chapters are especially recommended:

The Administration Guide also provides comprehensive information about the cluster management tools with which you can view and check the cluster status in SUSE OpenStack Cloud. They can also be used to look up details like configuration of cluster resources or global cluster options. Read the following chapters for more information:

Installing a minimal base system is sufficient to set up the Administration Server. The following patterns are the minimum required:

Tip: Installing a Local SMT Server (Optional)

If you do not have a SUSE Manager or SMT server in your organization, or are planning to manually update the repositories required for deployment of the SUSE OpenStack Cloud nodes, you need to set up an SMT server on the Administration Server. Choose the pattern Subscription Management Tool in addition to the patterns listed above to install the SMT server software.

SUSE OpenStack Cloud requires disabling the firewall on the Administration Server. You can disable the firewall during installation in the Firewall and SSH section. If your environment requires a firewall to be active at this stage of the installation, you can disable the firewall during your final network configuration (see Chapter 6, Service Configuration: Administration Server Network Configuration). Optionally, you can also enable SSH access to the Administration Server in this section.

Warning: HTTP_PROXY and NO_PROXY

Setting HTTP_PROXY without properly configuring NO_PROXY for the Administration Server might result in chef-client failing in non-obvious ways.

Mirroring the SUSE Linux Enterprise Server 12 SP2 and SUSE OpenStack Cloud 7 repositories is mandatory. Run the following commands as user root to add them to the list of mirrored repositories:

for REPO in SLES12-SP2-{Pool,Updates} SUSE-OpenStack-Cloud-7-{Pool,Updates}; do
  smt-repos $REPO sle-12-x86_64 -e
done

The following optional repositories provide high availability and storage:

New repositories added to SMT must be updated immediately by running the following command as user root:

smt-mirror -L /var/log/smt/smt-mirror.log

This command will download several GB of patches. This process may last up to several hours. A log file is written to /var/log/smt/smt-mirror.log. After this first manual update the repositories are updated automatically via cron job. A list of all repositories and their location in the file system on the Administration Server can be found at Table 5.2, “SMT Repositories Hosted on the Administration Server”.

When all update and pool repositories are managed by an SMT server installed on the Administration Server (see Chapter 4, Installing and Setting Up an SMT Server on the Administration Server (Optional)), make sure the repository location in YaST Crowbar is set to Local SMT Server (this is the default). For details, see Section 7.4, “Repositories”. No further action is required. The SUSE OpenStack Cloud Crowbar installation automatically detects all available repositories.

To use repositories from a remote SMT server, you first need to make sure all required repositories are mirrored on the server. Refer to Section 4.3, “Setting up Repository Mirroring on the SMT Server” for more information. When all update and pool repositories are managed by a remote SMT server, make sure the repository location in YaST Crowbar is set to Remote SMT Server. For details, see Section 7.4, “Repositories”. No further action is required. The SUSE OpenStack Cloud Crowbar installation automatically detects all available repositories.

Note: Accessing an External SMT Server

When using an external SMT server, it needs to be reachable by all nodes. This means that the SMT server either needs to be part of the admin network or it needs to be accessible via the default route of the nodes. The latter can be either the gateway of the admin network or the gateway of the public network.

To use repositories from SUSE Manager you first need to make sure all required products and extensions are registered, and the corresponding channels are mirrored in SUSE Manager (refer to Table 5.4, “SUSE Manager Repositories (Channels)” for a list of channels).

Important: Accessing a SUSE Manager Server

An external SUSE Manager server needs to be accessible to all nodes in SUSE OpenStack Cloud. The network hosting the SUSE Manager server must be added to the network definitions as described in Section 7.5.8, “Providing Access to External Networks”.

By default SUSE Manager does not expose repositories for direct access. To access them via HTTPS, you need to create a Distribution for auto-installation for the SUSE Linux Enterprise Server 12 SP2 (x86_64) product. Creating this distribution makes the update repositories for this product available, including the repositories for all registered add-on products (like SUSE OpenStack Cloud, SLES High Availability Extension and SUSE Enterprise Storage). Instructions for creating a distribution are in the SUSE Manager documentation in http://www.suse.com/documentation/suse_manager/.

During the distribution setups you need to provide a Label for each the distribution. This label will be part of the URL under which the repositories are available. We recommend choosing a name consisting of characters that do not need to be URL-encoded. In Table 5.4, “SUSE Manager Repositories (Channels)” we assume the following label has been provided: sles12-sp2-x86_64.

When all update and pool repositories are managed by a SUSE Manager server, make sure the repository location in YaST Crowbar is set to SUSE Manager Server. For details, see Section 7.4, “Repositories”. No further action is required. The SUSE OpenStack Cloud Crowbar installation automatically detects all available repositories.

The autoinstallation tree provided by SUSE Manager does not provide the SLES Pool repository. Although this repository is not used for node installation, it needs to be present. To work around this issue, it is sufficient to create an empty Pool repository for SUSE Linux Enterprise Server 12 SP2:

mkdir /srv/tftpboot/suse-12.2/x86_64/repos/SLES12-SP2-Pool/
createrepo /srv/tftpboot/suse-12.2/x86_64/repos/SLES12-SP2-Pool/

If you want to keep your SUSE OpenStack Cloud network as isolated from the company network as possible, or your infrastructure does not allow accessing a SUSE Manager or an SMT server, you can alternatively provide access to the required repositories by one of the following methods:

We strongly recommended making the repositories available at the default locations on the Administration Server as listed in Table 5.5, “Default Repository Locations on the Administration Server”. When choosing these locations, it is sufficient to set the repository location in YaST Crowbar to Custom. You do not need to specify a detailed location for each repository. Refer to Section 7.4, “Repositories” for details. If you prefer to use different locations, you need to announce each location with YaST Crowbar.

If you want to separate the admin and the BMC network, you must change the settings for the networks bmc and bmc_vlan. The bmc_vlan is used to generate a VLAN tagged interface on the Administration Server that can access the bmc network. The bmc_vlan needs to be in the same ranges as bmc, and bmc needs to have VLAN enabled.

Figure 7.3: YaST Crowbar Setup: Network Settings for the BMC Network #

The Network Mode tab of the YaST Crowbar module also lets you set up a Bastion network. As outlined in Section 2.1, “Network”, one way to access the Administration Server from a defined external network is via a Bastion network and a second network card (as opposed to providing an external gateway).

To set up the Bastion network, you need to have a static IP address for the Administration Server from the external network. The example configuration used below assumes that the external network from which to access the admin network has the following addresses. Adjust them according to your needs.

In addition to the values above, you need to enter the Physical Interface Mapping. With this value you specify the Ethernet card that is used for the bastion network. See Section 7.5.5, “Network Conduits” for details on the syntax. The default value ?1g2 matches the second interface (“eth1”) of the system.

Figure 7.4: YaST Crowbar Setup: Network Settings for the Bastion Network #

Warning: No Network Changes After Completing the SUSE OpenStack Cloud Crowbar installation

After you have completed the SUSE OpenStack Cloud Crowbar installation, you cannot change the network setup. If you do need to change it, you must completely set up the Administration Server again.

Important: Accessing Nodes From Outside the Bastion Network

The example configuration from above allows access to the SUSE OpenStack Cloud nodes from within the bastion network. If you want to access nodes from outside the bastion network, make the router for the bastion network the default router for the Administration Server. This is achieved by setting the value for the bastion network's Router preference entry to a lower value than the corresponding entry for the admin network. By default no router preference is set for the Administration Server—in this case, set the preference for the bastion network to 5.

If you use a Linux gateway between the outside and the bastion network, you also need to disable route verification (rp_filter) on the Administration Server. Do so by running the following command on the Administration Server:

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

That command disables route verification for the current session, so the setting will not survive a reboot. Make it permanent by editing /etc/sysctl.conf and setting the value for net.ipv4.conf.all.rp_filter to 0.

The network.json file is located in /etc/crowbar/. The template has the following general structure:

{
   "attributes" : {
      "network" : {
         "mode" : "VALUE",
         "start_up_delay" : VALUE,
         "teaming" : { "mode": VALUE },1
         "enable_tx_offloading" : VALUE,
         "enable_rx_offloading" : VALUE,
         "interface_map"2 : [
            ...
         ],
         "conduit_map"3 : [
            ...
         ],
         "networks"4 : {
            ...
         },
      }
   }
}

Note: Order of Elements

The order in which the entries in the network.json file appear may differ from the one listed above. Use your editor's search function to find certain entries.

The most important options to define in the global attributes section are the default values for the network and bonding modes. The following global attributes exist:

{
   "attributes" : {
      "network" : {
         "mode" : "single",1
         "start_up_delay" : 30,2
         "teaming" : { "mode": 5 },3
         "enable_tx_offloading" : true, 4
         "enable_rx_offloading" : true, 4
         "interface_map" : [
            ...
         ],
         "conduit_map" : [
            ...
         ],
         "networks" : {
            ...
         },
      }
   }
}

1	Network mode. Defines the configuration name (or name space) to be used from the conduit_map (see Section 7.5.5, “Network Conduits”). Your choices are single, dual, or team.
2	Time (in seconds) the Chef-client waits for the network interfaces to come online before timing out.
3	Default bonding mode. For a list of available modes, see Section 7.3, “Network Mode”.
4	Turn on/off TX and RX checksum offloading. If set to false, disable offloading by running ethtool -K and adding the setting to the respective ifcfg configuration file. If set to true, use the defaults of the network driver. If the network driver supports TX and/or RX checksum offloading and enables it by default, it will be used. Checksum offloading is set to true in network.json by default. It is recommended to keep this setting. If you experience problems, such as package losses, try disabling this feature by setting the value to false. Important: Change of the Default Value Starting with SUSE OpenStack Cloud, the default value for TX and RX checksum offloading changed from false to true. To check which defaults a network driver uses, run ethtool -k, for example: tux > sudo ethtool -k eth0 | grep checksumming rx-checksumming: on tx-checksumming: on Note that if the output shows a value marked as [fixed], this value cannot be changed. For more information on TX and RX checksum offloading refer to your hardware vendor's documentation. Detailed technical information can also be obtained from https://www.kernel.org/doc/Documentation/networking/checksum-offloads.txt.

By default, physical network interfaces are used in the order they appear under /sys/class/net/. If you want to apply a different order, you need to create an interface map where you can specify a custom order of the bus IDs. Interface maps are created for specific hardware configurations and are applied to all machines matching this configuration.

{
   "attributes" : {
      "network" : {
         "mode" : "single",
         "start_up_delay" : 30,
         "teaming" : { "mode": 5 },
         "enable_tx_offloading" : true ,
         "enable_rx_offloading" : true ,
         "interface_map" : [
            {
              "pattern" : "PowerEdge R610"1,
              "serial_number" : "0x02159F8E"2,
              "bus_order" : [3
                "0000:00/0000:00:01",
                "0000:00/0000:00:03"
              ]
            }
            ...
         ],
         "conduit_map" : [
            ...
         ],
         "networks" : {
            ...
         },
      }
   }
}

Important: PXE Boot Interface Must be Listed First

The physical interface used to boot the node via PXE must always be listed first.

Note: Interface Map Changes Allowed After Having Completed the SUSE OpenStack Cloud Crowbar Installation

Contrary to all other sections in network.json, you can change interface maps after completing the SUSE OpenStack Cloud Crowbar installation. However, nodes that are already deployed and affected by these changes must be deployed again. Therefore, we do not recommend making changes to the interface map that affect active nodes.

If you change the interface mappings after completing the SUSE OpenStack Cloud Crowbar installation you must not make your changes by editing network.json. You must rather use the Crowbar Web interface and open Barclamps › Crowbar › Network › Edit. Activate your changes by clicking Apply.

Network conduits define mappings for logical interfaces—one or more physical interfaces bonded together. Each conduit can be identified by a unique name, the pattern. This pattern is also called “Network Mode” in this document.

Three network modes are available:

See Section 2.1.2, “Network Modes” for detailed descriptions. Apart from these modes a fallback mode ".*/.*/.*" is also pre-defined—it is applied in case no other mode matches the one specified in the global attributes section. These modes can be adjusted according to your needs. It is also possible to define a custom mode.

The mode name that is specified with mode in the global attributes section is deployed on all nodes in SUSE OpenStack Cloud. It is not possible to use a different mode for a certain node. However, you can define “sub” modes with the same name that only match the following machines:

{
   "attributes" : {
      "network" : {
         "mode" : "single",
         "start_up_delay" : 30,
         "teaming" : { "mode": 5 },
         "enable_tx_offloading" : true,
         "enable_rx_offloading" : true,
         "interface_map" : [
            ...
         ],
         "conduit_map" : [
            {
              "pattern" : "single/.*/.*"1,
              "conduit_list" : {
                "intf2"2 : {
                  "if_list" : ["1g1","1g2"]3,
                  "team_mode" : 54
                },
                "intf1" : {
                  "if_list" : ["1g1","1g2"],
                  "team_mode" : 5
                },
                "intf0" : {
                  "if_list" : ["1g1","1g2"],
                  "team_mode" : 5
                }
              }
            },
         ...
         ],
         "networks" : {
            ...
         },
      }
   }
}

MODE_NAME/NUMBER_OF_NICS/NODE_ROLE

[Quantifier][Speed][Order]

The network definitions contain IP address assignments, the bridge and VLAN setup, and settings for the router preference. Each network is also assigned to a logical interface defined in the network conduit section. In the following the network definition is explained using the example of the admin network definition:

{
   "attributes" : {
      "network" : {
         "mode" : "single",
         "start_up_delay" : 30,
         "teaming" : { "mode": 5 },
         "enable_tx_offloading" : true,
         "enable_rx_offloading" : true,
         "interface_map" : [
            ...
         ],
         "conduit_map" : [
             ...
         ],
         "networks" : {
           "admin" : {
             "conduit" : "intf0"1,
             "add_bridge" : false2,
             "use_vlan" : false3,
             "vlan" : 1004,
             "router_pref" : 105,
             "subnet" : "192.168.124.0"6,
             "netmask" : "255.255.255.0",
             "router" : "192.168.124.1",
             "broadcast" : "192.168.124.255",
             "ranges" : {
               "admin" : {
                 "start" : "192.168.124.10",
                 "end" : "192.168.124.11"
               },
               "switch" : {
                 "start" : "192.168.124.241",
                 "end" : "192.168.124.250"
               },
               "dhcp" : {
                 "start" : "192.168.124.21",
                 "end" : "192.168.124.80"
               },
               "host" : {
                 "start" : "192.168.124.81",
                 "end" : "192.168.124.160"
               }
             }
           },
           "nova_floating": {
             "add_ovs_bridge": false7,
             "bridge_name": "br-public"8,
             ....
           }
         ...
         },
      }
   }
}

Important: VLAN Settings

As of SUSE OpenStack Cloud 7, using a VLAN for the admin network is only supported on a native/untagged VLAN. If you need VLAN support for the admin network, it must be handled at switch level.

When changing the network configuration with YaST or by editing /etc/crowbar/network.json, you can define VLAN settings for each network. For the networks nova-fixed and nova-floating, however, special rules apply:

nova-fixed: The USE VLAN setting will be ignored. However, VLANs will automatically be used if deploying Neutron with VLAN support (using the plugins linuxbridge, openvswitch plus VLAN, or cisco plus VLAN). In this case, you need to specify a correct VLAN ID for this network.

nova-floating: When using a VLAN for nova-floating (which is the default), the USE VLAN and VLAN ID settings for nova-floating and public default to the same.

You have the option of separating public and floating networks with a custom configuration. Configure your own separate floating network (not as a subnet of the public network), and give the floating network its own router. For example, define nova-floating as part of an external network with a custom bridge-name. When you are using different networks and OpenVSwitch is configured, the pre-defined bridge-name won't work.

By default, external networks cannot be reached from nodes in the SUSE OpenStack Cloud. To access external services such as a SUSE Manager server, an SMT server, or a SAN, you need to make the external network(s) known to SUSE OpenStack Cloud. Do so by adding a network definition for each external network to /etc/crowbar/network.json. Refer to Section 7.5, “Custom Network Configuration” for setup instructions.

            "external" : {
               "add_bridge" : false,
               "vlan" : XXX,
               "ranges" : {
                  "host" : {
                     "start" : "192.168.150.1",
                     "end" : "192.168.150.254"
                  }
               },
               "broadcast" : "192.168.150.255",
               "netmask" : "255.255.255.0",
               "conduit" : "intf1",
               "subnet" : "192.168.150.0",
               "use_vlan" : true
            }

Replace the value XXX for the VLAN by a value not used within the SUSE OpenStack Cloud network and not used by Neutron. By default, the following VLANs are already used:

For custom setups, the public and floating networks can be separated. Configure your own separate floating network (not as a subnet of the public network), and give the floating network its own router. For example, define nova-floating as part of an external network with a custom bridge-name. When you are using different networks and OpenVSwitch is configured, the pre-defined bridge-name won't work.

If you need to adjust the Maximum Transmission Unit (MTU) for the Admin and/or Storage Network, adjust /etc/crowbar/network.json as shown below. You can also enable jumbo frames this way by setting the MTU to 9000. The following example enables jumbo frames for both, the storage and the admin network by setting "mtu": 9000.

        "admin": {
          "add_bridge": false,
          "broadcast": "192.168.124.255",
          "conduit": "intf0",
          "mtu": 9000,
          "netmask": "255.255.255.0",
          "ranges": {
            "admin": {
              "end": "192.168.124.11",
              "start": "192.168.124.10"
            },
            "dhcp": {
              "end": "192.168.124.80",
              "start": "192.168.124.21"
            },
            "host": {
              "end": "192.168.124.160",
              "start": "192.168.124.81"
            },
            "switch": {
              "end": "192.168.124.250",
              "start": "192.168.124.241"
            }
          },
          "router": "192.168.124.1",
          "router_pref": 10,
          "subnet": "192.168.124.0",
          "use_vlan": false,
          "vlan": 100
        },
        "storage": {
          "add_bridge": false,
          "broadcast": "192.168.125.255",
          "conduit": "intf1",
          "mtu": 9000,
          "netmask": "255.255.255.0",
          "ranges": {
            "host": {
              "end": "192.168.125.239",
              "start": "192.168.125.10"
            }
          },
          "subnet": "192.168.125.0",
          "use_vlan": true,
          "vlan": 200
        },

Warning: No Network Changes After Completing the SUSE OpenStack Cloud Crowbar installation

After you have completed the SUSE OpenStack Cloud Crowbar installation, you cannot change the network setup, and you cannot change the MTU size.

SUSE OpenStack Cloud includes a new script, network-json-resolve, which matches the physical and logical names of network interfaces, and prints them to stdout. Use this to verify that you are using the correct interface names in network.json. Note that it will only work if OpenStack nodes have been deployed. The following command prints a help menu:

sudo /opt/dell/bin/network-json-resolve -h

network-json-resolve reads your deployed network.json file. To use a different network.json file, specify its full path with the --network-json option. The following example shows how to use a different network.json file, and prints the interface mappings of a single node:

sudo /opt/dell/bin/network-json-resolve --network-json /opt/configs/network.json aliases compute1
 eth0: 0g1, 1g1
 eth1: 0g1, 1g1

You may query the mappings of a specific network interface:

sudo /opt/dell/bin/network-json-resolve aliases compute1 eth0
 eth0: 0g1, 1g1

Print the bus ID order on a node. This returns no bus order defined for node if you did not configure any bus ID mappings:

sudo /opt/dell/bin/network-json-resolve bus_order compute1

Print the defined conduit map for the node:

sudo /opt/dell/bin/network-json-resolve conduit_map compute1
bastion: ?1g1
intf0: ?1g1
intf1: ?1g1
intf2: ?1g1

Resolve conduits to the standard interface names:

sudo /opt/dell/bin/network-json-resolve conduits compute1
bastion:
intf0: eth0
intf1: eth0
intf2: eth0

Resolve the configured networks on a node to the standard interface names:

sudo /opt/dell/bin/network-json-resolve networks compute1
bastion:
bmc_vlan:  eth0
nova_fixed: eth0
nova_floating: eth0
os_sdn: eth0
public: eth0
storage: eth0

Resolve the specified network to the standard interface name(s):

sudo /opt/dell/bin/network-json-resolve networks compute1 public
   public: eth0

Resolve a network.json-style interface to its standard interface name(s):

sudo /opt/dell/bin/network-json-resolve resolve compute1 1g1
 eth0

From this screen you can access HTML and PDF versions of the SUSE OpenStack Cloud manuals that are installed on the Administration Server.

The following procedure shows how to generally edit, create and deploy barclamp proposals. For the description and deployment of the individual barclamps, see Chapter 11, Deploying the OpenStack Services.

A proposal that has not been deployed yet can be deleted in the Edit Proposal view by clicking Delete. To delete a proposal that has already been deployed, see Section 9.3.3, “Deleting a Proposal That Already Has Been Deployed”.

To delete a proposal that has already been deployed, you first need to Deactivate it.

However, deactivating and deleting a barclamp that already had been deployed does not remove packages installed when the barclamp was deployed. Nor does it stop any services that were started during the barclamp deployment. On the affected node, proceed as follows to undo the deployment:

Uninstalling the packages should not be necessary.

When a proposal is applied to one or more nodes that are not yet available for deployment (for example, because they are rebooting or have not been yet fully installed), the proposal will be put in a queue. A message like

Successfully queued the proposal until the following become ready: d52-54-00-6c-25-44

will be shown when having applied the proposal. A new button Dequeue will also become available. Use it to cancel the deployment of the proposal by removing it from the queue.

To keep the operating system and the SUSE OpenStack Cloud software itself up-to-date on the nodes, you can deploy either the Updater barclamp or the SUSE Manager barclamp. The latter requires access to a SUSE Manager server. The Updater barclamp uses Zypper to install updates and patches from repositories made available on the Administration Server.

The easiest way to provide the required repositories on the Administration Server is to set up an SMT server as described in Chapter 4, Installing and Setting Up an SMT Server on the Administration Server (Optional). Alternatives to setting up an SMT server are described in Chapter 5, Software Repository Setup.

The Updater barclamp lets you deploy updates that are available on the update repositories at the moment of deployment. Each time you deploy updates with this barclamp you can choose a different set of nodes to which the updates are deployed. This lets you exactly control where and when updates are deployed.

To deploy the Updater barclamp, proceed as follows. For general instructions on how to edit barclamp proposals refer to Section 9.3, “Deploying Barclamp Proposals”.

zypper keeps track of the packages and patches it installs in /var/log/zypp/history. Review that log file on a node to find out which updates have been installed. A second log file recording debug information on the zypper runs can be found at /var/log/zypper.log on each node.

Warning: Updating Software Packages on Cluster Nodes

Before starting an update for a cluster node, either stop the cluster stack on that node or put the cluster into maintenance mode. If the cluster resource manager on a node is active during the software update, this can lead to unpredictable results like fencing of active nodes. For detailed instructions refer to http://www.suse.com/documentation/sle-ha-12/book_sleha/data/sec_ha_migration_update.html.

To keep the operating system and the SUSE OpenStack Cloud software itself up-to-date on the nodes, you can deploy either SUSE Manager Client barclamp or the Updater barclamp. The latter uses Zypper to install updates and patches from repositories made available on the Administration Server.

To enable the SUSE Manager server to manage the SUSE OpenStack Cloud nodes, you must make the respective SUSE OpenStack Cloud 7 channels, the SUSE Linux Enterprise Server 12 SP2 channels, and the channels for extensions used with your deployment (High Availability Extension, SUSE Enterprise Storage) available via an activation key.

The SUSE Manager Client barclamp requires access to the SUSE Manager server from every node it is deployed to.

To deploy the SUSE Manager Client barclamp, proceed as follows. For general instructions on how to edit barclamp proposals refer to Section 9.3, “Deploying Barclamp Proposals”.

Warning: Updating Software Packages on Cluster Nodes

Before starting an update for a cluster node, either stop the cluster stack on that node or put the cluster into maintenance mode. If the cluster resource manager on a node is active during the software update, this can lead to unpredictable results like fencing of active nodes. For detailed instructions refer to http://www.suse.com/documentation/sle-ha-12/book_sleha/data/sec_ha_migration_update.html.

The NFS barclamp allows you to mount NFS share from a remote host on nodes in the cloud. This feature can, for example, be used to provide an image repository for Glance. Note that all nodes which are to mount an NFS share must be able to reach the NFS server. This requires manually adjusting the network configuration.

To deploy the NFS barclamp, proceed as follows. For general instructions on how to edit barclamp proposals refer to Section 9.3, “Deploying Barclamp Proposals”.

While deploying Ceph from within SUSE OpenStack Cloud is possible, leveraging an external Ceph cluster is also fully supported. Follow the instructions below to use an external Ceph cluster in SUSE OpenStack Cloud.

10.4.4.2 Making Ceph Available on the SUSE OpenStack Cloud Nodes #

Ceph can be used from the KVM Compute Nodes, with Cinder, and with Glance. The following installation steps need to be executed on each node accessing Ceph:

Important: Installation Workflow

The following steps need to be executed before the barclamps get deployed.

Warning: Do Not Deploy the Ceph Barclamp

If using an external Ceph cluster, you must not deploy the SUSE OpenStack Cloud Ceph barclamp. An external and an internal Ceph cluster cannot be used together.

1. Log in as user root to a machine in the Ceph cluster and generate keyring files for Cinder users. Optionally, you can generate keyring files for the Glance users (only needed when using Glance with Ceph/Rados). The keyring file that will be generated for Cinder will also be used on the Compute Nodes. To do so, you need to specify pool names and user names for both services. The default names are:

   	Glance	Cinder
   User	glance	cinder
   Pool	images	volumes

   Make a note of user and pool names in case you do not use the default values. You will need this information later, when deploying Glance and Cinder.

2. 

   Warning: Automatic Changes to the Cluster

   If you decide to use the admin keyring file to connect the external Ceph cluster, be aware that after Crowbar discovers this admin keyring, it will create client keyring files, pools, and capabilities needed to run Glance, Cinder, or Nova integration.

   If you have access to the admin keyring file and agree that automatic changes will be done to the cluster as described above, copy it together with the Ceph configuration file to the Administration Server. If you cannot access this file, create a keyring:

   1. When you can access the admin keyring file ceph.client.admin.keyring, copy it together with ceph.conf (both files are usually located in /etc/ceph) to a temporary location on the Administration Server, for example /root/tmp/.

   2. If you cannot access the admin keyring file create a new keyring file with the following commands. Re-run the commands for Glance, too, if needed. First create a key:

      ceph auth get-or-create-key client.USERNAME mon "allow r" \
      osd 'allow class-read object_prefix rbd_children, allow rwx \
      pool=POOLNAME'

      Replace USERNAME and POOLNAME with the respective values.

      Now use the key to generate the keyring file /etc/ceph/ceph.client.USERNAME.keyring:

      ceph-authtool \
      /etc/ceph/ceph.client.USERNAME.keyring \
      --create-keyring --name=client.USERNAME> \
      --add-key=KEY

      Replace USERNAME with the respective value.

      Copy the Ceph configuration file ceph.conf (usually located in /etc/ceph) and the keyring file(s) generated above to a temporary location on the Administration Server, for example /root/tmp/.

3. Log in to the Crowbar Web interface and check whether the nodes which should have access to the Ceph cluster already have an IP address from the storage network. Do so by going to the Dashboard and clicking the node name. An IP address should be listed for storage. Make a note of the Full name of each node that has no storage network IP address.

4. Log in to the Administration Server as user root and run the following command for all nodes you noted down in the previous step:

   crowbar network allocate_ip "default" NODE "storage" "host"
   chef-client

   NODE needs to be replaced by the node's name.

5. After executing the command in the previous step for all affected nodes, run the command chef-client on the Administration Server.

6. Log in to each affected node as user root. See How can I log in to a node as root? for instructions. On each node, do the following:

   1. Manually install nova, cinder (if using cinder) and/or glance (if using glance) packages with the following commands:

      zypper in openstack-glance
      zypper in openstack-cinder
      zypper in openstack-nova

   2. Copy the ceph.conf file from the Administration Server to /etc/ceph:

      mkdir -p /etc/ceph
      scp root@admin:/root/tmp/ceph.conf /etc/ceph
      chmod 664 /etc/ceph/ceph.conf

   3. Copy the keyring file(s) to /etc/ceph. The exact process depends on whether you have copied the admin keyring file or whether you have created your own keyrings:

      1. If you have copied the admin keyring file, run the following command on the Control Node(s) on which Cinder and Glance will be deployed, and on all KVM Compute Nodes:

         scp root@admin:/root/tmp/ceph.client.admin.keyring /etc/ceph
         chmod 640 /etc/ceph/ceph.client.admin.keyring

      2. If you have created you own keyrings, run the following command on the Control Node on which Cinder will be deployed, and on all KVM Compute Nodes to copy the Cinder keyring:

         scp root@admin:/root/tmp/ceph.client.cinder.keyring /etc/ceph
         chmod 640 /etc/ceph/ceph.client.cinder.keyring

         Now copy the Glance keyring to the Control Node on which Glance will be deployed:

         scp root@admin:/root/tmp/ceph.client.glance.keyring /etc/ceph
         chmod 640 /etc/ceph/ceph.client.glance.keyring

   4. Adjust the ownership of the keyring file as follows:

      Glance: chown root.cinder /etc/ceph/ceph.client.cinder.keyring

      Cinder: chown root.glance /etc/ceph/ceph.client.glance.keyring

      KVM Compute Nodes: chown root.nova /etc/ceph/ceph.volumes.keyring

The nodes can only be accessed via SSH from the Administration Server—it is not possible to connect to them from any other host in the network.

The root account on the nodes has no password assigned, therefore logging in to a node as root@node is only possible via SSH with key authentication. By default, you can only log in with the key of the root of the Administration Server (root@admin) via SSH only.

If you have added additional users to the Administration Server and want to give them permission to log in to the nodes as well, you need to add these users' public SSH keys to root's authorized_keys file on all nodes. Proceed as follows:

To enable SSL to encrypt communication within the cloud (see Section 2.3, “SSL Encryption” for details), all nodes running encrypted services need SSL certificates. An SSL certificate is, at a minimum, required on the Control Node.

Each certificate consists of a pair of files: the certificate file (for example, signing_cert.pem) and the key file (for example, signing_key.pem). If you use your own certificate authority (CA) for signing, you will also need a certificate file for the CA (for example, ca.pem). We recommend copying the files to the /etc directory using the directory structure outlined below. If you use a dedicated certificate for each service, create directories named after the services (for example, /etc/keystone). If you are using shared certificates, use a directory such as /etc/cloud.