Applies to SUSE Enterprise Storage 7

14 Troubleshooting #

14.1 Debugging Rook #

There are a number of basic actions a user might need to take during debugging. These actions are defined here for reference when they are mentioned in more documentation below.

Important

This document is not devoted to an in-depth explanation of what Kubernetes is, what its features are, how it is used, how to navigate it, or how to debug applications that run on it. This document will use Kubernetes terms, and users are expected to know how to look up Kubernetes information they do not already have. This document will give an outline of how to use Kubernetes tools to get any information needed in the Rook-Ceph context and, when relevant, will briefly explain how Rook uses Kubernetes features.

14.1.1 Setting the operator log level to debug #

In general, the first place to look when encountering a failure is to get logs for the rook-ceph-operator pod. To get the most informative logs possible, set the operator log level to DEBUG.

To do this, modify Helm's values.yaml or modify the operator.yaml manifest. Regardless of the method chosen, the log level can always be set by editing the deployment directly with kubectl. For example:

kubectl@adm > kubectl --namespace rook-ceph set env deployment/rook-ceph-operator ROOK_LOG_LEVEL=DEBUG

After editing the deployment, the operator pod will restart automatically and will start outputting logs with the new log level.

Note

If you are experiencing a particular failure, it may take some time for the Rook operator to reach the failure location again to report debug logs.

14.1.2 Using the toolbox pod #

Use the Rook toolbox pod to interface directly with the Ceph cluster via the CLI. For example:

kubectl@adm > kubectl --namespace rook-ceph exec -it deploy/rook-ceph-tools -- bash

If the rook-ceph-tools deployment does not exist, it should be created using the toolbox.yaml manifest.

Note

To set log levels for Ceph daemons, it is advised to use the Ceph CLI from the toolbox pod.

14.1.3 Using the SES supportutils plugin #

The supportutils plugin for SUSE Enterprise Storage works with Rook clusters. It is installed by the supportutils-plugin-ses package. The plugin collects container logs and more information about a Rook-Ceph cluster, making collection of logs easy. Once the logs are collected, you can browse the collected information and logs without needing to progressively collect more detailed information at each step.

The supportutilsplugin does not alter the Rook log level to DEBUG, and it is advised to set this to DEBUG before running the plugin. The plugin also does not change any Ceph log levels; also consider changing those if the failure merits it before running the plugin.