13 Troubleshooting #
13.1 Debugging Rook #
There are a number of basic actions a user might need to take during debugging. These actions are defined here for reference when they are mentioned in more documentation below.
This document is not devoted to an in-depth explanation of what Kubernetes is, what its features are, how it is used, how to navigate it, or how to debug applications that run on it. This document will use Kubernetes terms, and users are expected to know how to look up Kubernetes information they do not already have. This document will give an outline of how to use Kubernetes tools to get any information needed in the Rook-Ceph context and, when relevant, will briefly explain how Rook uses Kubernetes features.
13.1.1 Setting the operator log level to debug #
In general, the first place to look when encountering a failure is to get
logs for the rook-ceph-operator
pod. To get the most informative logs possible, set the operator log level
to DEBUG.
To do this, modify Helm's values.yaml or modify the
operator.yaml manifest. Regardless of the method
chosen, the log level can always be set by editing the deployment directly
with kubectl. For example:
kubectl@adm > kubectl --namespace rook-ceph set env deployment/rook-ceph-operator ROOK_LOG_LEVEL=DEBUGAfter editing the deployment, the operator pod will restart automatically and will start outputting logs with the new log level.
If you are experiencing a particular failure, it may take some time for the Rook operator to reach the failure location again to report debug logs.
13.1.2 Using the toolbox pod #
Use the Rook toolbox pod to interface directly with the Ceph cluster via the CLI. For example:
kubectl@adm > kubectl --namespace rook-ceph exec -it deploy/rook-ceph-tools -- bash
If the rook-ceph-tools
deployment does not exist, it should be created using the
toolbox.yaml manifest.
To set log levels for Ceph daemons, it is advised to use the Ceph CLI
from the toolbox pod.
13.1.3 Using the SES supportutils plugin #
The supportutils plugin for SUSE Enterprise Storage works with
Rook clusters. It is installed by the
supportutils-plugin-ses package. The plugin collects
container logs and more information about a Rook-Ceph cluster, making
collection of logs easy. Once the logs are collected, you can browse the
collected information and logs without needing to progressively collect
more detailed information at each step.
The supportutilsplugin does not alter the Rook log
level to DEBUG, and it is advised to set this to
DEBUG before running the plugin. The plugin also does
not change any Ceph log levels; also consider changing those if the
failure merits it before running the plugin.