Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / SUSE OpenStack Cloud Crowbar 9 Documentation / Deployment Guide using Crowbar / Troubleshooting and Support / Troubleshooting and Support
Applies to SUSE OpenStack Cloud Crowbar 9

18 Troubleshooting and Support

 

Find solutions for the most common pitfalls and technical details on how to create a support request for SUSE OpenStack Cloud Crowbar here.

18.1 FAQ

 

If your problem is not mentioned here, checking the log files on either the Administration Server or the OpenStack nodes may help. A list of log files is available at Section 5.4, “Log Files”.

1. Admin Node Deployment

Q: What to do if the initial SUSE OpenStack Cloud Crowbar installation on the Administration Server fails?

Check the installation routine's log file at /var/log/crowbar/install.log for error messages.

Q: What to do if the initial SUSE OpenStack Cloud Crowbar installation on the Administration Server fails while deploying the IPMI/BMC network?

As of SUSE OpenStack Cloud Crowbar 8, it is assumed that each machine can be accessed directly via IPMI/BMC. However, this is not the case on certain blade hardware, where several nodes are accessed via a common adapter. Such a hardware setup causes an error on deploying the IPMI/BMC network. You need to disable the IPMI deployment running the following command: 

/opt/dell/bin/json-edit -r -a "attributes.ipmi.bmc_enable" \
-v "false" /opt/dell/chef/data_bags/crowbar/bc-template-ipmi.json

Re-run the SUSE OpenStack Cloud Crowbar installation after having disabled the IPMI deployment.

Q: Why am I not able to reach the Administration Server from outside the admin network via the bastion network?

If route -n shows no gateway for the bastion network, check the value of the following entries in /etc/crowbar/network.json: "router_pref": and "router_pref":. Make sure the value for the bastion network's "router_pref": is set to a lower value than "router_pref": for the admin network.

If the router preference is set correctly, route -n shows a gateway for the bastion network. In case the Administration Server is still not accessible via its admin network address (for example, 192.168.124.10), you need to disable route verification (rp_filter). Do so by running the following command on the Administration Server: 

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

If this setting solves the problem, make it permanent by editing /etc/sysctl.conf and setting the value for net.ipv4.conf.all.rp_filter to 0.

Q: Can I change the host name of the Administration Server?

No, after you have run the SUSE OpenStack Cloud Crowbar installation you cannot change the host name. Services like Crowbar, Chef, and the RabbitMQ will fail after changing the host name.

Q: What to do when browsing the Chef Web UI gives a Tampered with cookie error?

You probably have an old cookie in your browser from a previous Chef installation on the same IP address. Remove the cookie named _chef_server_session_id and try again.

Q: How to make custom software repositories from an external server (for example a remote SMT or SUSE Manager server) available for the nodes?

Custom repositories need to be added using the YaST Crowbar module:

  1. Start the YaST Crowbar module and switch to the Repositories tab: YaST › Miscellaneous › Crowbar › Repositories.

  2. Choose Add Repositories

  3. Enter the following data:

    Name

    A unique name to identify the repository.

    URL

    Link or path to the repository.

    Ask On Error

    Access errors to a repository are silently ignored by default. To ensure that you get notified of these errors, set the Ask On Error flag.

    Target Platform/Architecture

    Currently only repositories for SUSE Linux Enterprise Server 12 SP4 on the x86-64 architecture are supported. Make sure to select both options.

  4. Save your settings selecting OK.

2. OpenStack Node Deployment

Q: How can I log in to a node as root?

By default you cannot directly log in to a node as root, because the nodes were set up without a root password. You can only log in via SSH from the Administration Server. You should be able to log in to a node with ssh root@NAME where NAME is the name (alias) of the node.

If name resolution does not work, go to the Crowbar Web interface and open the Node Dashboard. Click the name of the node and look for its admin (eth0) IP Address. Log in to that IP address via SSH as user root.

Q: What to do if a node refuses to boot or boots into a previous installation?

Make sure to change the boot order in the BIOS of the node, so that the first boot option is to boot from the network/boot using PXE.

Q: What to do if a node hangs during hardware discovery after the very first boot using PXE into the “SLEShammer” image?

The root login is enabled at a very early state in discovery mode, so chances are high that you can log in for debugging purposes as described in How can I log in to a node as root? . If logging in as root does not work, you need to set the root password manually. This can either be done by setting the password via the Kernel command line as explained in How to provide Kernel Parameters for the SLEShammer Discovery Image? , or by creating a hook as explained below:

  1. Create a directory on the Administration Server named /updates/discovering-pre 

    mkdir /updates/discovering-pre

  2. Create a hook script setpw.hook in the directory created in the previous step: 

    cat > /updates/discovering-pre/setpw.hook <<EOF
#!/bin/sh
echo "root:linux" | chpasswd
EOF

  3. Make the script executable: 

    chmod a+x  /updates/discovering-pre/setpw.hook

If you are still cannot log in, you very likely hit a bug in the discovery image. Report it at http://bugzilla.suse.com/.

Q: How to provide Kernel Parameters for the SLEShammer Discovery Image?

Kernel Parameters for the SLEShammer Discovery Image can be provided via the Provisioner barclamp. The following example shows how to set a root password:

  1. Open a browser and point it to the Crowbar Web interface available on the Administration Server, for example http://192.168.124.10/. Log in as user crowbar. The password is crowbar by default, if you have not changed it.

  2. Open Barclamps › Crowbar and click Edit in the Provisioner row.

  3. Click Raw in the Attributes section and add the Kernel parameter(s) to the "discovery": { "append": "" } line, for example; 

      "discovery": {
    "append": "DISCOVERY_ROOT_PASSWORD=PASSWORD"
  },

  4. Apply the proposal without changing the assignments in the Deployment section.

Q: What to do when a deployed node fails to boot using PXE with the following error message: “Could not find kernel image: ../suse-12.2/install/boot/x86_64/loader/linux”?

The installation repository on the Administration Server at /srv/tftpboot/suse-12.3/install has not been set up correctly to contain the SUSE Linux Enterprise Server 12 SP4 installation media. Review the instructions at Section 5.1, “Copying the Product Media Repositories”.

Q: Why does my deployed node hang at Unpacking initramfs during boot when using PXE?

The node probably does not have enough RAM. You need at least 4 GB RAM for the deployment process to work.

Q: What to do if a node is reported to be in the state Problem? What to do if a node hangs at “Executing AutoYast script: /usr/sbin/crowbar_join --setup” after the installation is finished?

Be patient—the AutoYaST script may take a while to finish. If it really hangs, log in to the node as root (see How can I log in to a node as root? for details). Check for error messages at the end of /var/log/crowbar/crowbar_join/chef.log. Fix the errors and restart the AutoYaST script by running the following command: 

crowbar_join --start

If successful, the node will be listed in state Ready, when the script has finished.

In cases where the initial --setup wasn't able to complete successfully, you can rerun that once after the previous issue is solved.

If that does not help or if the log does not provide useful information, proceed as follows:

  1. Log in to the Administration Server and run the following command: 

    crowbar crowbar transition $NODE

    NODE needs to be replaced by the alias name you have given to the node when having installed it. Note that this name needs to be prefixed with $.

  2. Log in to the node and run chef-client.

  3. Check the output of the command for failures and error messages and try to fix the cause of these messages.

  4. Reboot the node.

If the node is in a state where login in from the Administration Server is not possible, you need to create a root password for it as described in Direct root Login. Now re-install the node by going to the node on the Crowbar Web interface and clicking Reinstall. After having been re-installed, the node will hang again, but now you can log in and check the log files to find the cause.

Q: Where to find more information when applying a barclamp proposal fails?

Check the Chef client log files on the Administration Server located at /var/log/crowbar/chef-client/d*.log. Further information is available from the Chef client log files located on the node(s) affected by the proposal (/var/log/chef/client.log), and from the log files of the service that failed to be deployed. Additional information may be gained from the Crowbar Web UI log files on the Administration Server. For a list of log file locations refer to Section 5.4, “Log Files”.

Q: How to Prevent the Administration Server from Installing the OpenStack Nodes (Disable PXE and DNS Services)?

By default, the OpenStack nodes are installed by booting a discovery image from the Administration Server using PXE. They are allocated and then boot via PXE into an automatic installation (see Section 11.2, “Node Installation” for details). To install the OpenStack nodes manually or with a custom provisioning tool, you need to disable the PXE boot service and the DNS service on the Administration Server.

As a consequence you also need to provide an external DNS server. Such a server needs to comply with the following requirements:

  • It needs to handle all domain-to-IP requests for SUSE OpenStack Cloud.

  • It needs to handle all IP-to-domain requests for SUSE OpenStack Cloud.

  • It needs to forward unknown requests to other DNS servers.

To disable the PXE and DNS services when setting up the Administration Server, proceed as follows:

Procedure 18.1: Disabling PXE/DNS when Setting Up the Administration Server
 

The following steps need to be performed before starting the SUSE OpenStack Cloud Crowbar installation.

  1. Create the file /etc/crowbar/dns.json with the following content: 

    {
  "attributes": {
    "dns": {
      "nameservers": [ "DNS_SERVER", "DNS_SERVER2" ],
      "auto_assign_server": false
    }
  }
}

    Replace DNS_SERVER and DNS_SERVER2 with the IP address(es) of the external DNS server(s). Specifying more than one server is optional.

  2. Create the file /etc/crowbar/provisioner.json with the following content: 

    {
  "attributes": {
    "provisioner": {
      "enable_pxe": false
    }
  }
}

  3. If these files are present when the SUSE OpenStack Cloud Crowbar installation is started, the Administration Server will be set up using external DNS services and no PXE boot server.

If you already have deployed SUSE OpenStack Cloud, proceed as follows to disable the DNS and PXE services on the Administration Server:

Procedure 18.2: Disabling PXE/DNS on an Administration Server Running Crowbar
 

  1. Open a browser and point it to the Crowbar Web interface available on the Administration Server, for example http://192.168.124.10/. Log in as user crowbar. The password is crowbar by default, if you have not changed it.

  2. Open Barclamps › Crowbar and click Edit in the Provisioner row.

  3. Click Raw in the Attributes section and change the value for enable_pxe to false: 

    "enable_pxe": false,

  4. Apply the proposal without changing the assignments in the Deployment section.

  5. Change to the DNS barclamp via Barclamps › Crowbar and click Edit in the DNS row.

  6. Click Raw in the Attributes section. Change the value for auto_assign_server to false and add the address(es) for the external name server(s): 

    "auto_assign_server": false,
"nameservers": [
  "DNS_SERVER",
  "DNS_SERVER2"
],

    Replace DNS_SERVER and DNS_SERVER2 with the IP address(es) of the external DNS server(s). Specifying more than one server is optional.

  7. Save your changes, but do not apply them, yet!

  8. In the Deployment section of the barclamp remove all nodes from the dns-server role, but do not change the assignments for the dns-client role.

  9. Apply the barclamp.

  10. When the DNS barclamp has been successfully applied, log in to the Administration Server and stop the DNS service: 

    systemctl stop named

Now that the PXE and DNS services are disabled you can install SUSE Linux Enterprise Server 12 SP4 on the OpenStack nodes. When a node is ready, add it to the pool of nodes as described in Section 11.3, “Converting Existing SUSE Linux Enterprise Server 12 SP4 Machines Into SUSE OpenStack Cloud Nodes”.

Q: I have installed a new hard disk on a node that was already deployed. Why is it ignored by Crowbar?

When adding a new hard disk to a node that has already been deployed, it can take up to 15 minutes before the new disk is detected.

Q: How to install additional packages (for example a driver) when nodes are deployed?

SUSE OpenStack Cloud Crowbar offers the possibility to install additional packages that are not part of the default scope of packages installed on the OpenStack nodes. This is for example required if your hardware is only supported by a third party driver. It is also useful if your setup requires to install additional tools that would otherwise need to be installed manually.

Prerequisite for using this feature is that the packages are available in a repository known on the Administration Server. Refer to How to make custom software repositories from an external server (for example a remote SMT or SUSE M..? for details, if the packages you want to install are not part of the repositories already configured.

To add packages for installation on node deployment, proceed as follows:

  1. Open a browser and point it to the Crowbar Web interface on the Administration Server, for example http://192.168.124.10/. Log in as user crowbar. The password is crowbar by default, if you have not changed it during the installation.

  2. Go to Barclamps › Crowbar and click the Edit button for Provisioner.

  3. Next click Raw in the Attributes page to open an editable view of the provisioner configuration.

  4. Add the following JSON code before the last closing curly bracket (replace the PACKAGE placeholders with real package names): 

             "packages": {
    "suse-12.2": ["PACKAGE_1", "PACKAGE_2"],
  }

Note that these packages will get installed on all OpenStack nodes. If the change to the Provisioner barclamp is made after nodes have already been deployed, the packages will be installed on the affected nodes with the next run of Chef or crowbar-register. Package names will be validated against the package naming guidelines to prevent script-injection.

3. Miscellaneous

Q: How to change the keystone credentials after the keystone barclamp has been deployed?

To change the credentials for the keystone administrator (admin) or the regular user (crowbar by default), proceed as follows:

  1. Log in to the Control Node on which keystone is deployed as user root via the Administration Server.

  2. In a shell, source the OpenStack RC file for the project that you want to upload an image to. For details, refer to Set environment variables using the OpenStack RC file in the OpenStack documentation.

  3. Enter the following command to change the PASSWORD for the administrator or the regular user (USER): 

     keystone-manage bootstrap --bootstrap-password PASSWORD \
--bootstrap-username USER

    For a complete list of command line options, run keystone-manage bootstrap --help. Make sure to start the command with a Space to make sure the password does not appear in the command history.

  4. Access the keystone barclamp on the Crowbar Web interface by going to Barclamps › OpenStack and click Edit for the keystone barclamp.

  5. Enter the new password for the same user you specified on the command line before.

  6. Activate the change by clicking Apply. When the proposal has been re-applied, the password has changed and can be used.

Q: How to add or change a configuration value for an OpenStack service?

See Chapter 14, Configuration Files for OpenStack Services.

18.2 Support

 

Before contacting support to help you with a problem on SUSE OpenStack Cloud, it is strongly recommended that you gather as much information about your system and the problem as possible. For this purpose, SUSE OpenStack Cloud Crowbar ships with a tool called supportconfig. It gathers system information such as the current kernel version being used, the hardware, RPM database, partitions, and other items. supportconfig also collects the most important log files, making it easier for the supporters to identify and solve your problem.

It is recommended to always run supportconfig on the Administration Server and on the Control Node(s). If a Compute Node or a Storage Node is part of the problem, run supportconfig on the affected node as well. For details on how to run supportconfig, see https://documentation.suse.com/sles/15-SP1/single-html/SLES-admin/#cha-adm-support.

18.2.1 Applying PTFs (Program Temporary Fixes) Provided by the SUSE L3 Support

 

Under certain circumstances, the SUSE support may provide temporary fixes, the so-called PTFs, to customers with an L3 support contract. These PTFs are provided as RPM packages. To make them available on all nodes in SUSE OpenStack Cloud, proceed as follows.

  1. Download the packages from the location provided by the SUSE L3 Support to a temporary location on the Administration Server.

  2. Move the packages from the temporary download location to the following directories on the Administration Server:

    “noarch” packages (*.noarch.rpm):
    /srv/tftpboot/suse-12.4/x86_64/repos/PTF/rpm/noarch/
    /srv/tftpboot/suse-12.4/s390x/repos/PTF/rpm/noarch/
    “x86_64” packages (*.x86_64.rpm)

    /srv/tftpboot/suse-12.4/x86_64/repos/PTF/rpm/x86_64/

    “s390x” packages (*.s390x.rpm)

    /srv/tftpboot/suse-12.4/s390x/repos/PTF/rpm/s390x/

  3. Create or update the repository metadata: 

    createrepo-cloud-ptf

  4. The repositories are now set up and are available for all nodes in SUSE OpenStack Cloud except for the Administration Server. In case the PTF also contains packages to be installed on the Administration Server, make the repository available on the Administration Server as well: 

    zypper ar -f /srv/tftpboot/suse-12.4/x86_64/repos/PTF PTF

  5. To deploy the updates, proceed as described in Section 11.4.1, “Deploying Node Updates with the Updater Barclamp”. Alternatively, run zypper up manually on each node.