To log in to the Ceph Dashboard, point your browser to its URL including the port number. Run the following command to find the address:

cephuser@adm > ceph mgr services | grep dashboard
"dashboard": "https://host:port/",

The command returns the URL where the Ceph Dashboard is located. If you are having issues with this command, see Book “Troubleshooting Guide”, Chapter 10 “Troubleshooting the Ceph Dashboard”, Section 10.1 “Locating the Ceph Dashboard”.

Figure 2.1: Ceph Dashboard login screen #

  

Log in by using the credentials that you created during cluster deployment (see Book “Deployment Guide”, Chapter 7 “Deploying the bootstrap cluster using ceph-salt”, Section 7.2.9 “Configuring the Ceph Dashboard login credentials”).

Tip: Custom user account

If you do not want to use the default admin account to access the Ceph Dashboard, create a custom user account with administrator privileges. Refer to Chapter 11, Manage users and roles on the command line for more details.

The dashboard user interface is graphically divided into several blocks: the utility menu in the top right-hand side of the screen, the main menu on the left-hand side, and the main content pane.

Figure 2.2: Ceph Dashboard home page #

  

The top right-hand side of the screen contains a utility menu. It includes general tasks related more to the dashboard than to the Ceph cluster. By clicking the options, you can access the following topics:

The dashboard's main menu occupies the left-hand side of the screen. It covers the following topics:

The content pane occupies the main part of the dashboard's screen. The dashboard home page shows plenty of helpful widgets to inform you briefly about the current status of the cluster, capacity, and performance information.

In Ceph Dashboard, you often work with lists—for example, lists of pools, OSD nodes, or RBD devices. All lists will automatically refresh themselves by default every five seconds. The following common widgets help you manage or adjust these list:

Click to trigger a manual refresh of the list.

Click to display or hide individual table columns.

Click and enter (or select) how many rows to display on a single page.

Click inside and filter the rows by typing the string to search for.

Use to change the currently displayed page if the list spans across multiple pages.

Each dashboard widget shows specific status information related to a specific aspect of a running Ceph cluster. Some widgets are active links and after clicking them, they will redirect you to a related detailed page of the topic they represent.

Tip: More details on mouse over

Some graphical widgets show you more detail when you move the mouse over them.

2.6.1 Status widgets #

  

Status widgets give you a brief overview about the cluster's current status.

Figure 2.3: Status widgets #

  

Cluster Status

Presents basic information about the cluster's health.

Hosts

Shows the total number of cluster nodes.

Monitors

Shows the number of running monitors and their quorum.

OSDs

Shows the total number of OSDs, as well as the number of up and in OSDs.

Managers

Shows the number of active and standby Ceph Manager daemons.

Object Gateways

Shows the number of running Object Gateways.

Metadata Servers

Shows the number of Metadata Servers.

iSCSI Gateways

Shows the number of configured iSCSI gateways.

2.6.2 Capacity widgets #

  

Capacity widgets show brief information about the storage capacity.

Figure 2.4: Capacity widgets #

  

Raw Capacity

Shows the ratio of used and available raw storage capacity.

Objects

Shows the number of data objects stored in the cluster.

PG Status

Displays a chart of the placement groups according to their status.

Pools

Shows the number of pools in the cluster.

PGs per OSD

Shows the average number of placement groups per OSD.

2.6.3 Performance widgets #

  

Performance widgets refer to basic performance data of Ceph clients.

Figure 2.5: performance widgets #

  

Client Read/Write

The amount of clients' read and write operations per second.

Client Throughput

The amount of data transferred to and from Ceph clients in bytes per second.

Recovery Throughput

The throughput of data recovered per second.

Scrubbing

Shows the scrubbing (see Section 17.4.9, “Scrubbing a placement group”) status. It is either inactive, enabled, or active.

Click in the utility menu and select User Management.

The list contains each user's user name, full name, e-mail, a list of assigned roles, whether the role is enabled, and the password expiration date.

Figure 3.1: User management #

  

Click Create in the top left of the table heading to add a new user. Enter their user name, password, and optionally a full name and an e-mail.

Figure 3.2: Adding a user #

  

Click the little pen icon to assign predefined roles to the user. Confirm with Create User.

Click a user's table row to highlight the selection Select Edit to edit details about the user. Confirm with Edit User.

Click a user's table row to highlight the selection Select the drop-down button next to Edit and select Delete from the list to delete the user account. Activate the Yes, I am sure check box and confirm with Delete User.

Click in the utility menu and select User Management. Then click the Roles tab.

The list contains each role's name, description, and whether it is a system role.

Figure 3.3: User roles #

  

Click Create in the top left of the table heading to add a new custom role. Enter the Name and Description and next to Permissions, select the appropriate permissions.

Tip: Purging custom roles

If you create custom user roles and intend to remove the Ceph cluster with the ceph-salt purge command later on, you need to purge the custom roles first. Find more details in Section 13.9, “Removing an entire Ceph cluster”.

Figure 3.4: Adding a role #

  

Tip: Multiple activation

By activating the check box that precedes the topic name, you activate all permissions for that topic. By activating the All check box, you activate all permissions for all the topics.

Confirm with Create Role.

Click a user's table row to highlight the selection Select Edit in the top left of the table heading to edit a description and permissions of the custom role. Confirm with Edit Role.

Click a role's table row to highlight the selection Select the drop-down button next to Edit and select Delete from the list to delete the role. Activate the Yes, I am sure check box and confirm with Delete Role.

Click Cluster › Hosts to view a list of cluster nodes.

Figure 4.1: Hosts #

  

Click the drop-down arrow next to a node name in the Hostname column to view the performance details of the node.

The Services column lists all daemons that are running on each related node. Click a daemon name to view its detailed configuration.

Click Cluster › Inventory to view a list of devices. The list includes the device path, type, availability, vendor, model, size, and the OSDs.

Click to select a node name in the Hostname column. When selected, click Identify to identify the device the host is running on. This tells the device to blink its LEDs. Select the duration of this action between 1, 2, 5, 10, or 15 minutes. Click Execute.

Figure 4.2: Services #

  

Click Cluster › Monitors to view a list of cluster nodes with running Ceph monitors. The content pane is split into two views: Status, and In Quorum or Not In Quorum.

The Status table shows general statistics about the running Ceph Monitors, including the following:

The In Quorum and Not In Quorum panes include each monitor's name, rank number, public IP address, and number of open sessions.

Click a node name in the Name column to view the related Ceph Monitor configuration.

Figure 4.3: Ceph Monitors #

  

Click Cluster › Services to view details on each of the available services: crash, Ceph Manager, and Ceph Monitors. The list includes the container image name, container image ID, status of what is running, size, and when it was last refreshed.

Click the drop-down arrow next to a service name in the Service column to view details of the daemon. The detail list includes the host name, daemon type, daemon ID, container ID, container image name, container image ID, version number, status, and when it was last refreshed.

Figure 4.4: Services #

  

Click Cluster › OSDs to view a list of nodes with running OSD daemons. The list includes each node's name, ID, status, device class, number of placement groups, size, usage, reads/writes chart in time, and the rate of read/write operations per second.

Figure 4.5: Ceph OSDs #

  

Select Flags from the Cluster-wide configuration drop-down menu in the table heading to open a pop-up window. This has a list of flags that apply to the whole cluster. You can activate or deactivate individual flags, and confirm with Submit.

Figure 4.6: OSD flags #

  

Select Recovery Priority from the Cluster-wide configuration drop-down menu in the table heading to open a pop-up window. This has a list of OSD recovery priorities that apply to the whole cluster. You can activate the preferred priority profile and fine-tune the individual values below. Confirm with Submit.

Figure 4.7: OSD recovery priority #

  

Click the drop-down arrow next to a node name in the Host column to view an extended table with details about the device settings and performance. Browsing through several tabs, you can see lists of Attributes, Metadata, Device health, Performance counter, a graphical Histogram of reads and writes, and Performance details.

Figure 4.8: OSD details #

  

Tip: Performing specific tasks on OSDs

After you click an OSD node name, the table row is highlighted. This means that you can now perform a task on the node. You can choose to perform any of the following actions: Edit, Create, Scrub, Deep Scrub, Reweight, Mark out, Mark In, Mark Down, Mark Lost, Purge, Destroy, or Delete.

Click the down arrow in the top left of the table heading next to the Create button and select the task you want to perform.

Click Cluster › Configuration to view a complete list of Ceph cluster configuration options. The list contains the name of the option, its short description, and its current and default values, and whether the option is editable.

Figure 4.14: Cluster configuration #

  

Click the drop-down arrow next to a configuration option in the Name column to view an extended table with detailed information about the option, such as its type of value, minimum and maximum permitted values, whether it can be updated at runtime, and many more.

After highlighting a specific option, you can edit its value(s) by clicking the Edit button in the top left of the table heading. Confirm changes with Save.

Click Cluster › CRUSH map to view a CRUSH Map of the cluster. For more general information on CRUSH Maps, refer to Section 17.5, “CRUSH Map manipulation”.

Click the root, nodes, or individual OSDs to view more detailed information, such as crush weight, depth in the map tree, device class of the OSD, and many more.

Figure 4.15: CRUSH Map #

  

Click Cluster › Manager modules to view a list of available Ceph Manager modules. Each line consists of a module name and information on whether it is currently enabled or not.

Figure 4.16: Manager modules #

  

Click the drop-down arrow next to a module in the Name column to view an extended table with detailed settings in the Details table below. Edit them by clicking the Edit button in the top left of the table heading. Confirm changes with Update.

Click the drop-down arrow next to the Edit button in the top left of the table heading to Enable or Disable a module.

Click Cluster › Logs to view a list of cluster's recent log entries. Each line consists of a time stamp, the type of the log entry, and the logged message itself.

Click the Audit Logs tab to view log entries of the auditing subsystem. Refer to Section 11.5, “Auditing API requests” for commands to enable or disable auditing.

Figure 4.17: Logs #

  

Click Cluster › Monitoring to manage and view details on Prometheus alerts.

If you have Prometheus active, in this content pane you can view detailed information on Active Alerts, All Alerts, or Silences.

Note

If you do not have Prometheus deployed, an information banner will appear and link to relevant documentation.

To add a new pool, click Create in the top left of the pools table. In the pool form you can enter the pool's name, type, its applications, compression mode, and quotas including maximum byes and maximum objects. The pool form itself pre-calculates the number of placement groups that best suited to this specific pool. The calculation is based on the amount of OSDs in the cluster and the selected pool type with its specific settings. As soon as a placement groups number is set manually, it will be replaced by a calculated number. Confirm with Create Pool.

Figure 5.2: Adding a new pool #

  

To delete a pool, select the pool in the table row. Click the drop-down arrow next to the Create button and click Delete.

To edit a pool's options, select the pool in the table row and click Edit in the top left of the pools table.

You can change the name of the pool, increase the number of placement groups, change the list of the pool's applications and compression settings. Confirm with Edit Pool.

To view more detailed information about a device, click its row in the table:

Figure 6.2: RBD details #

  

To view detailed configuration of a device, click its row in the table and then the Configuration tab in the lower table:

Figure 6.3: RBD configuration #

  

To add a new device, click Create in the top left of the table heading and do the following on the Create RBD screen:

Figure 6.4: Adding a new RBD #

  

To delete a device, select the device in the table row. Click the drop-down arrow next to the Create button and click Delete. Confirm the deletion with Delete RBD.

Tip: Moving RBDs to trash

Deleting an RBD is an irreversible action. If you Move to Trash instead, you can restore the device later on by selecting it on the Trash tab of the main table and clicking Restore in the top left of the table heading.

To create a RADOS Block Device snapshot, select the device in the table row and the detailed configuration content pane appears. Select the Snapshots tab and click Create in the top left of the table heading. Enter the snapshot's name and confirm with Create RBD Snapshot.

After selecting a snapshot, you can perform additional actions on the device, such as rename, protect, clone, copy, or delete. Rollback restores the device's state from the current snapshot.

Figure 6.5: RBD snapshots #

  

RADOS Block Device images can be asynchronously mirrored between two Ceph clusters. You can use the Ceph Dashboard to configure replication of RBD images between two or more clusters. This capability is available in two modes:

Mirroring is configured on a per-pool basis within peer clusters and can be configured on a specific subset of images within the pool or configured to automatically mirror all images within a pool when using journal-based mirroring only.

Mirroring is configured using the rbd command, which is installed by default in SUSE Enterprise Storage 7. The rbd-mirror daemon is responsible for pulling image updates from the remote, peer cluster and applying them to the image within the local cluster. See Section 6.6.2, “Enabling the rbd-mirror daemon” for more information on enabling the rbd-mirror daemon.

Depending on the need for replication, RADOS Block Device mirroring can be configured for either one- or two-way replication:

Important

Each instance of the rbd-mirror daemon must be able to connect to both the local and remote Ceph clusters simultaneously, for example all monitor and OSD hosts. Additionally, the network must have sufficient bandwidth between the two data centers to handle mirroring workload.

Tip: General information

For general information and the command line approach to RADOS Block Device mirroring, refer to Section 20.4, “RBD image mirrors”.

6.6.1 Configuring primary and secondary clusters #

  

A primary cluster is where the original pool with images is created. A secondary cluster is where the pool or images are replicated from the primary cluster.

Note: Relative naming

The primary and secondary terms can be relative in the context of replication because they relate more to individual pools than to clusters. For example, in two-way replication, one pool can be mirrored from the primary cluster to the secondary one, while another pool can be mirrored from the secondary cluster to the primary one.

cephuser@adm > rbd mirror pool disable POOL_NAME

cephuser@adm > rbd mirror pool peer bootstrap create [--site-name local-site-name] pool-name

cephuser@adm > rbd --cluster primary mirror pool peer bootstrap create --site-name primary
  image-pool eyJmc2lkIjoiOWY1MjgyZGItYjg5OS00NTk2LTgwOTgtMzIwYzFmYzM5NmYzIiwiY2xpZW50X2lkIjoicmJkL \
  W1pcnJvci1wZWVyIiwia2V5IjoiQVFBUnczOWQwdkhvQmhBQVlMM1I4RmR5dHNJQU50bkFTZ0lOTVE9PSIsIm1vbl9ob3N0I \
  joiW3YyOjE5Mi4xNjguMS4zOjY4MjAsdjE6MTkyLjE2OC4xLjM6NjgyMV0ifQ==

cephuser@adm > rbd mirror pool peer bootstrap import [--site-name local-site-name] \
[--direction rx-only or rx-tx] pool-name token-path

cephuser@adm > cat >>EOF < token
eyJmc2lkIjoiOWY1MjgyZGItYjg5OS00NTk2LTgwOTgtMzIwYzFmYzM5NmYzIiwiY2xpZW50X2lkIjoicmJkLW1pcn \
Jvci1wZWVyIiwia2V5IjoiQVFBUnczOWQwdkhvQmhBQVlMM1I4RmR5dHNJQU50bkFTZ0lOTVE9PSIsIm1vbl9ob3N0I \
joiW3YyOjE5Mi4xNjguMS4zOjY4MjAsdjE6MTkyLjE2OC4xLjM6NjgyMV0ifQ==
EOF
cephuser@adm > rbd --cluster secondary mirror pool peer bootstrap import --site-name secondary image-pool token

cephuser@adm > rbd mirror pool peer remove pool-name peer-uuid

Tip: More information on iSCSI Gateways

For more general information about iSCSI Gateways, refer to Chapter 22, Ceph iSCSI gateway.

To list all available gateways and mapped images, click Block › iSCSI from the main menu. An Overview tab opens, listing currently configured iSCSI Gateways and mapped RBD images.

The Gateways table lists each gateway's state, number of iSCSI targets, and number of sessions. The Images table lists each mapped image's name, related pool name backstore type, and other statistical details.

The Targets tab lists currently configured iSCSI targets.

Figure 6.14: List of iSCSI targets #

  

To view more detailed information about a target, click the drop-down arrow on the target table row. A tree-structured schema opens, listing disks, portals, initiators, and groups. Click an item to expand it and view its detailed contents, optionally with a related configuration in the table on the right.

Figure 6.15: iSCSI target details #

  

6.7.1 Adding iSCSI targets #

  

To add a new iSCSI target, click Create in the top left of the Targets table and enter the required information.

Figure 6.16: Adding a new target #

  

1. Enter the target address of the new gateway.

2. Click Add portal and select one or multiple iSCSI portals from the list.

3. Click Add image and select one or multiple RBD images for the gateway.

4. If you need to use authentication to access the gateway, activate the ACL Authentication check box and enter the credentials. You can find more advanced authentication options after activating Mutual authentication and Discovery authentication.

5. Confirm with Create Target.

Tip: For more information

For more general information and a description of RBD QoS configuration options, refer to Section 20.6, “QoS settings”.

The QoS options can be configured at different levels.

The global configuration is at the top of the list and will be used for all newly created RBD images and for those images that do not override these values on the pool or RBD image layer. An option value specified globally can be overridden on a per-pool or per-image basis. Options specified on a pool will be applied to all RBD images of that pool unless overridden by a configuration option set on an image. Options specified on an image will override options specified on a pool and will override options specified globally.

This way it is possible to define defaults globally, adapt them for all RBD images of a specific pool, and override the pool configuration for individual RBD images.

6.8.2 Configuring options on a new pool #

  

To create a new pool and configure RBD configuration options on it, click Pools › Create. Select replicated as pool type. You will then need to add the rbd application tag to the pool to be able to configure the RBD QoS options.

Note

It is not possible to configure RBD QoS configuration options on an erasure coded pool. To configure the RBD QoS options for erasure coded pools, you need to edit the replicated metadata pool of an RBD image. The configuration will then be applied to the erasure coded data pool of that image.

6.8.3 Configuring options on an existing pool #

  

To configure RBD QoS options on an existing pool, click Pools, then click the pool's table row and select Edit at the top left of the table.

You should see the RBD Configuration section in the dialog, followed by a Quality of Service section.

Note

If you see neither the RBD Configuration nor the Quality of Service section, you are likely either editing an erasure coded pool, which cannot be used to set RBD configuration options, or the pool is not configured to be used by RBD images. In the latter case, assign the rbd application tag to the pool and the corresponding configuration sections will show up.

To add a new NFS export, click Create in the top left of the exports table and enter the required information.

Figure 7.3: Adding a new NFS export #

  

To delete an export, select and highlight the export in the table row. Click the drop-down arrow next to the Edit button and select Delete. Activate the Yes, I am sure check box and confirm with Delete NFS export.

To edit an existing export, select and highlight the export in the table row and click Edit in the top left of the exports table.

You can then adjust all the details of the NFS export.

Figure 7.4: Editing an NFS export #

  

Click Filesystems from the main menu to view the overview of configured file systems. The main table shows each file system's name, date of creation, and whether it is enabled or not.

By clicking a file system's table row, you reveal details about its rank and pools added to the file system.

Figure 8.1: CephFS details #

  

At the bottom of the screen, you can see statistics counting the number of related MDS inodes and client requests, collected in real time.

Figure 8.2: CephFS details #

  

To view a list of configured Object Gateways, click Object Gateway › Daemons. The list includes the ID of the gateway, host name of the cluster node where the gateway daemon is running, and the gateway's version number.

Click the drop-down arrow next to the gateway's name to view detailed information about the gateway. The Performance Counters tab shows details about read/write operations and cache statistics.

Figure 9.1: Gateway's details #

  

Click Object Gateway › Users to view a list of existing Object Gateway users.

Click the drop-down arrow next to the user name to view details about the user account, such as status information or the user and bucket quota details.

Figure 9.2: Gateway users #

  

9.2.1 Adding a new gateway user #

  

To add a new gateway user, click Create in the top left of the table heading. Fill in their credentials, details about the S3 key and user and bucket quotas, then confirm with Create User.

Figure 9.3: Adding a new gateway user #

  

Object Gateway (OGW) buckets implement the functionality of OpenStack Swift containers. Object Gateway buckets serve as containers for storing data objects.

Click Object Gateway › Buckets to view a list of Object Gateway buckets.

9.3.1 Adding a new bucket #

  

To add a new Object Gateway bucket, click Create in the top left of the table heading. Enter the bucket's name, select the owner, and set the placement target. Confirm with Create Bucket.

Note

At this stage you can also enable locking by selecting Enabled; however, this is configurable after creation. See Section 9.3.3, “Editing the bucket” for more information.

9.3.2 Viewing bucket details #

  

To view detailed information about an Object Gateway bucket, click the drop-down arrow next to the bucket name.

Figure 9.4: Gateway bucket details #

  

Tip: Bucket quota

Below the Details table, you can find details about the bucket quota and locking settings.

9.3.3 Editing the bucket #

  

Select and highlight a bucket, then click Edit in the top left of the table heading.

You can update the owner of the bucket or enable versioning, multi-factor authentication or locking. Confirm any changes with Edit Bucket.

Figure 9.5: Editing the bucket details #

  

All HTTP connections to the dashboard are secured with TLS/SSL by default. A secure connection requires an SSL certificate. You can either use a self-signed certificate, or generate a certificate and have a well known certificate authority (CA) sign it.

Tip: Disabling SSL

You may want to disable the SSL support for a specific reason. For example, if the dashboard is running behind a proxy that does not support SSL.

Use caution when disabling SSL as user names and passwords will be sent to the dashboard unencrypted.

To disable SSL, run:

cephuser@adm > ceph config set mgr mgr/dashboard/ssl false

Tip: Restarting the Ceph Manager processes

You need to restart the Ceph Manager processes manually after changing the SSL certificate and key. You can do so by either running

cephuser@adm > ceph mgr fail ACTIVE-MANAGER-NAME

or by disabling and re-enabling the dashboard module, which also triggers the manager to respawn itself:

cephuser@adm > ceph mgr module disable dashboard
cephuser@adm > ceph mgr module enable dashboard

10.1.1 Creating self-signed certificates #

  

Creating a self-signed certificate for secure communication is simple. This way you can get the dashboard running quickly.

Note: Web browsers complaint

Most Web browsers will complain about a self-signed certificate and require explicit confirmation before establishing a secure connection to the dashboard.

To generate and install a self-signed certificate, use the following built-in command:

cephuser@adm > ceph dashboard create-self-signed-cert

10.1.2 Using certificates signed by CA #

  

To properly secure the connection to the dashboard and to eliminate Web browser complaints about a self-signed certificate, we recommend using a certificate that is signed by a CA.

You can generate a certificate key pair with a command similar to the following:

# openssl req -new -nodes -x509 \
  -subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650 \
  -keyout dashboard.key -out dashboard.crt -extensions v3_ca

The above command outputs dashboard.key and dashboard.crt files. After you get the dashboard.crt file signed by a CA, enable it for all Ceph Manager instances by running the following commands:

cephuser@adm > ceph dashboard set-ssl-certificate -i dashboard.crt
cephuser@adm > ceph dashboard set-ssl-certificate-key -i dashboard.key

Tip: Different certificates for each manager instance

If you require different certificates for each Ceph Manager instance, modify the commands and include the name of the instance as follows. Replace NAME with the name of the Ceph Manager instance (usually the related host name):

cephuser@adm > ceph dashboard set-ssl-certificate NAME -i dashboard.crt
cephuser@adm > ceph dashboard set-ssl-certificate-key NAME -i dashboard.key

The Ceph Dashboard binds to a specific TCP/IP address and TCP port. By default, the currently active Ceph Manager that hosts the dashboard binds to TCP port 8443 (or 8080 when SSL is disabled).

Note

If a firewall is enabled on the hosts running Ceph Manager (and thus the Ceph Dashboard), you may need to change the configuration to enable access to these ports. For more information on firewall settings for Ceph, see Book “Troubleshooting Guide”, Chapter 13 “Hints and tips”, Section 13.7 “Firewall settings for Ceph”.

The Ceph Dashboard binds to "::" by default, which corresponds to all available IPv4 and IPv6 addresses. You can change the IP address and port number of the Web application so that they apply to all Ceph Manager instances by using the following commands:

cephuser@adm > ceph config set mgr mgr/dashboard/server_addr IP_ADDRESS
cephuser@adm > ceph config set mgr mgr/dashboard/server_port PORT_NUMBER

Tip: Configuring Ceph Manager instances separately

Since each ceph-mgr daemon hosts its own instance of the dashboard, you may need to configure them separately. Change the IP address and port number for a specific manager instance by using the following commands (replace NAME with the ID of the ceph-mgr instance):

cephuser@adm > ceph config set mgr mgr/dashboard/NAME/server_addr IP_ADDRESS
cephuser@adm > ceph config set mgr mgr/dashboard/NAME/server_port PORT_NUMBER

Tip: Listing configured endpoints

The ceph mgr services command displays all endpoints that are currently configured. Look for the dashboard key to obtain the URL for accessing the dashboard.

If you do not want to use the default administrator account, create a different user account and associate it with at least one role. We provide a set of predefined system roles that you can use. For more details refer to Chapter 11, Manage users and roles on the command line.

To create a user with administrator privileges, use the following command:

cephuser@adm > ceph dashboard ac-user-create USER_NAME PASSWORD administrator

To use the Object Gateway management functionality of the dashboard, you need to provide the login credentials of a user with the system flag enabled:

Note

By default the firewall is enabled in SUSE Linux Enterprise Server 15 SP2. For more information on firewall configuration, see Book “Troubleshooting Guide”, Chapter 13 “Hints and tips”, Section 13.7 “Firewall settings for Ceph”.

There are several points to consider:

The Ceph Dashboard manages iSCSI targets using the REST API provided by the rbd-target-api service of the Ceph iSCSI gateway. Ensure it is installed and enabled on iSCSI gateways.

Note

The iSCSI management functionality of the Ceph Dashboard depends on the latest version 3 of the ceph-iscsi project. Ensure that your operating system provides the correct version, otherwise the Ceph Dashboard will not enable the management features.

If the ceph-iscsi REST API is configured in HTTPS mode and it is using a self-signed certificate, configure the dashboard to avoid SSL certificate verification when accessing ceph-iscsi API.

Disable API SSL verification:

cephuser@adm > ceph dashboard set-iscsi-api-ssl-verification false

Define the available iSCSI gateways:

cephuser@adm > ceph dashboard iscsi-gateway-list
cephuser@adm > ceph dashboard iscsi-gateway-add scheme://username:password@host[:port]
cephuser@adm > ceph dashboard iscsi-gateway-rm gateway_name

Single Sign-On (SSO) is an access control method that enables users to log in with a single ID and password to multiple applications simultaneously.

The Ceph Dashboard supports external authentication of users via the SAML 2.0 protocol. Because authorization is still performed by the dashboard, you first need to create user accounts and associate them with the desired roles. However, the authentication process can be performed by an existing Identity Provider (IdP).

To configure Single Sign-On, use the following command:

cephuser@adm > ceph dashboard sso setup saml2 CEPH_DASHBOARD_BASE_URL \
 IDP_METADATA IDP_USERNAME_ATTRIBUTE \
 IDP_ENTITY_ID SP_X_509_CERT \
 SP_PRIVATE_KEY

Parameters:

Note: SAML requests

The issuer value of SAML requests will follow this pattern:

CEPH_DASHBOARD_BASE_URL/auth/saml2/metadata

To display the current SAML 2.0 configuration, run:

cephuser@adm > ceph dashboard sso show saml2

To disable Single Sign-On, run:

cephuser@adm > ceph dashboard sso disable

To check if SSO is enabled, run:

cephuser@adm > ceph dashboard sso status

To enable SSO, run:

cephuser@adm > ceph dashboard sso enable saml2

By default the password policy feature is enabled including the following checks:

The password policy feature can be switched on or off completely:

cephuser@adm > ceph dashboard set-pwd-policy-enabled true|false

The following individual checks can be switched on or off:

cephuser@adm > ceph dashboard set-pwd-policy-check-length-enabled true|false
cephuser@adm > ceph dashboard set-pwd-policy-check-oldpwd-enabled true|false
cephuser@adm > ceph dashboard set-pwd-policy-check-username-enabled true|false
cephuser@adm > ceph dashboard set-pwd-policy-check-exclusion-list-enabled true|false
cephuser@adm > ceph dashboard set-pwd-policy-check-complexity-enabled true|false
cephuser@adm > ceph dashboard set-pwd-policy-check-sequential-chars-enabled true|false
cephuser@adm > ceph dashboard set-pwd-policy-check-repetitive-chars-enabled true|false

In addition, the following options are available to configure the password policy behaviour.

The Ceph Dashboard supports managing multiple user accounts. Each user account consists of a user name, a password (stored in encrypted form using bcrypt), an optional name, and an optional e-mail address.

User accounts are stored in Ceph Monitor's configuration database and are shared globally across all Ceph Manager instances.

Use the following commands to manage user accounts:

This section describes what security scopes you can assign to a user role, how to manage user roles and assign them to user accounts.

11.3.2 Specifying user roles #

  

A role specifies a set of mappings between a security scope and a set of permissions. There are four types of permissions: 'read', 'create', 'update', and 'delete'.

The following example specifies a role where a user has 'read' and 'create' permissions for features related to pool management, and has full permissions for features related to RBD image management:

{
  'role': 'my_new_role',
  'description': 'My new role',
  'scopes_permissions': {
    'pool': ['read', 'create'],
    'rbd-image': ['read', 'create', 'update', 'delete']
  }
}

The dashboard already provides a set of predefined roles that we call system roles. You can instantly use them after a fresh Ceph Dashboard installation:

administrator

Provides full permissions for all security scopes.

read-only

Provides read permission for all security scopes except the dashboard settings.

block-manager

Provides full permissions for 'rbd-image', 'rbd-mirroring', and 'iscsi' scopes.

rgw-manager

Provides full permissions for the 'rgw' scope.

cluster-manager

Provides full permissions for the 'hosts', 'osd', 'monitor', 'manager', and 'config-opt' scopes.

pool-manager

Provides full permissions for the 'pool' scope.

cephfs-manager

Provides full permissions for the 'cephfs' scope.

11.3.2.1 Managing custom roles #

  

You can create new user roles by using the following commands:

Create a new role:

cephuser@adm > ceph dashboard ac-role-create ROLENAME [DESCRIPTION]

Delete a role:

cephuser@adm > ceph dashboard ac-role-delete ROLENAME

Add scope permissions to a role:

cephuser@adm > ceph dashboard ac-role-add-scope-perms ROLENAME SCOPENAME PERMISSION [PERMISSION...]

Delete scope permissions from a role:

cephuser@adm > ceph dashboard ac-role-del-perms ROLENAME SCOPENAME

11.3.2.2 Assigning roles to user accounts #

  

Use the following commands to assign roles to users:

Set user roles:

cephuser@adm > ceph dashboard ac-user-set-roles USERNAME ROLENAME [ROLENAME ...]

Add additional roles to a user:

cephuser@adm > ceph dashboard ac-user-add-roles USERNAME ROLENAME [ROLENAME ...]

Delete roles from a user:

cephuser@adm > ceph dashboard ac-user-del-roles USERNAME ROLENAME [ROLENAME ...]

Tip: Purging custom roles

If you create custom user roles and intend to remove the Ceph cluster with the ceph.purge runner later on, you need to purge the custom roles first. Find more details in Section 13.9, “Removing an entire Ceph cluster”.

11.3.2.3 Example: Creating a user and a custom role #

  

This section illustrates a procedure for creating a user account capable of managing RBD images, viewing and creating Ceph pools, and having read-only access to any other scopes.

1. Create a new user named tux:

    cephuser@adm > ceph dashboard ac-user-create tux PASSWORD

2. Create a role and specify scope permissions:

   cephuser@adm > ceph dashboard ac-role-create rbd/pool-manager
   cephuser@adm > ceph dashboard ac-role-add-scope-perms rbd/pool-manager \
    rbd-image read create update delete
   cephuser@adm > ceph dashboard ac-role-add-scope-perms rbd/pool-manager pool read create

3. Associate the roles with the tux user:

   cephuser@adm > ceph dashboard ac-user-set-roles tux rbd/pool-manager read-only

If you want to establish a fixed URL to reach the Ceph Dashboard or if you do not want to allow direct connections to the manager nodes, you can set up a proxy that automatically forwards incoming requests to the currently active ceph-mgr instance.

  cephuser@adm > ceph config set mgr mgr/dashboard/url_prefix URL_PREFIX

cephuser@adm > ceph config set mgr mgr/dashboard/standby_behaviour "error"

cephuser@adm > ceph config set mgr mgr/dashboard/standby_behaviour "redirect"

cephuser@adm > ceph config set mgr mgr/dashboard/standby_error_status_code 503

11.4.4 HAProxy example configuration #

  

The following example configuration is for TLS/SSL pass through using HAProxy.

Note

The configuration works under the following conditions: If the dashboard fails over, the front-end client might receive an HTTP redirect (303) response and will be redirected to an unresolvable host.

This happens when the failover occurs during two HAProxy health checks. In this situation the previously active dashboard node will now respond with a 303 which points to the new active node. To prevent that situation you should consider disabling the redirection behaviour on standby nodes.

  defaults
    log global
    option log-health-checks
    timeout connect 5s
    timeout client 50s
    timeout server 450s

  frontend dashboard_front
    mode http
    bind *:80
    option httplog
    redirect scheme https code 301 if !{ ssl_fc }

  frontend dashboard_front_ssl
    mode tcp
    bind *:443
    option tcplog
    default_backend dashboard_back_ssl

  backend dashboard_back_ssl
    mode tcp
    option httpchk GET /
    http-check expect status 200
    server x HOST:PORT ssl check verify none
    server y HOST:PORT ssl check verify none
    server z HOST:PORT ssl check verify none

The Ceph Dashboard's REST API can log PUT, POST, and DELETE requests to the Ceph audit log. Logging is disabled by default, but you can enable it with the following command:

cephuser@adm > ceph dashboard set-audit-api-enabled true

If enabled, the following parameters are logged per each request:

An example log entry looks like this:

2019-02-06 10:33:01.302514 mgr.x [INF] [DASHBOARD] \
 from='https://[::ffff:127.0.0.1]:37022' path='/api/rgw/user/exu' method='PUT' \
 user='admin' params='{"max_buckets": "1000", "display_name": "Example User", "uid": "exu", "suspended": "0", "email": "user@example.com"}'

Tip: Disable logging of request payload

The logging of the request payload (the list of arguments and their values) is enabled by default. You can disable it as follows:

cephuser@adm > ceph dashboard set-audit-api-log-payload false

Ceph Dashboard can manage NFS Ganesha exports that use CephFS or Object Gateway as their backstore. The dashboard manages NFS Ganesha configuration files stored in RADOS objects on the CephFS cluster. NFS Ganesha must store part of their configuration in the Ceph cluster.

Run the following command to configure the NFS Ganesha configuration object's location:

cephuser@adm > ceph dashboard set-ganesha-clusters-rados-pool-namespace pool_name[/namespace]

You can now manage NFS Ganesha exports using the Ceph Dashboard.

cephuser@adm > ceph dashboard set-ganesha-clusters-rados-pool-namespace cluster_id:pool_name[/namespace](,cluster_id:pool_name[/namespace])*

Ceph Dashboard plugins extend the functionality of the dashboard. The debug plugin allows the customization of the behaviour of the dashboard according to the debug mode. It can be enabled, disabled, or checked with the following command:

cephuser@adm > ceph dashboard debug status
Debug: 'disabled'
cephuser@adm > ceph dashboard debug enable
Debug: 'enabled'
cephuser@adm > dashboard debug disable
Debug: 'disabled'

By default, this is disabled. This is the recommended setting for production deployments. If required, debug mode can be enabled without need of restarting.

You can find the immediate state of the cluster using ceph status or ceph -s:

cephuser@adm > ceph -s
cluster:
    id:     b4b30c6e-9681-11ea-ac39-525400d7702d
    health: HEALTH_OK

  services:
    mon: 5 daemons, quorum ses-min1,ses-master,ses-min2,ses-min4,ses-min3 (age 2m)
    mgr: ses-min1.gpijpm(active, since 3d), standbys: ses-min2.oopvyh
    mds: my_cephfs:1 {0=my_cephfs.ses-min1.oterul=up:active}
    osd: 3 osds: 3 up (since 3d), 3 in (since 11d)
    rgw: 2 daemons active (myrealm.myzone.ses-min1.kwwazo, myrealm.myzone.ses-min2.jngabw)

  task status:
    scrub status:
        mds.my_cephfs.ses-min1.oterul: idle

  data:
    pools:   7 pools, 169 pgs
    objects: 250 objects, 10 KiB
    usage:   3.1 GiB used, 27 GiB / 30 GiB avail
    pgs:     169 active+clean

The output provides the following information:

Tip: How Ceph calculates data usage

The used value reflects the actual amount of raw storage used. The xxx GB / xxx GB value means the amount available (the lesser number) of the overall storage capacity of the cluster. The notional number reflects the size of the stored data before it is replicated, cloned or snapshot. Therefore, the amount of data actually stored typically exceeds the notional amount stored, because Ceph creates replicas of the data and may also use storage capacity for cloning and snapshotting.

Other commands that display immediate status information are:

To get the information updated in real time, put any of these commands (including ceph -s) as an argument of the watch command:

# watch -n 10 'ceph -s'

Press Ctrl–C when you are tired of watching.

After you start your cluster and before you start reading and/or writing data, check your cluster's health:

cephuser@adm > ceph health
HEALTH_WARN 10 pgs degraded; 100 pgs stuck unclean; 1 mons down, quorum 0,2 \
node-1,node-2,node-3

Tip

If you specified non-default locations for your configuration or keyring, you may specify their locations:

cephuser@adm > ceph -c /path/to/conf -k /path/to/keyring health

The Ceph cluster returns one of the following health codes:

Tip

If you specified non-default locations for your configuration or keyring, you may specify their locations:

# ceph -c /path/to/conf -k /path/to/keyring health

To check a cluster’s data usage and distribution among pools, use the ceph df command. To get more details, use ceph df detail.

cephuser@adm > ceph df
--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED     RAW USED  %RAW USED
hdd    30 GiB  27 GiB  121 MiB   3.1 GiB      10.40
TOTAL  30 GiB  27 GiB  121 MiB   3.1 GiB      10.40

--- POOLS ---
POOL                   ID  STORED   OBJECTS  USED     %USED  MAX AVAIL
device_health_metrics   1      0 B        0      0 B      0    8.5 GiB
cephfs.my_cephfs.meta   2  1.0 MiB       22  4.5 MiB   0.02    8.5 GiB
cephfs.my_cephfs.data   3      0 B        0      0 B      0    8.5 GiB
.rgw.root               4  1.9 KiB       13  2.2 MiB      0    8.5 GiB
myzone.rgw.log          5  3.4 KiB      207    6 MiB   0.02    8.5 GiB
myzone.rgw.control      6      0 B        8      0 B      0    8.5 GiB
myzone.rgw.meta         7      0 B        0      0 B      0    8.5 GiB

The RAW STORAGE section of the output provides an overview of the amount of storage your cluster uses for your data.

The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section does not reflect replicas, clones or snapshots. For example, if you store an object with 1MB of data, the notional usage will be 1MB, but the actual usage may be 2MB or more depending on the number of replicas, clones and snapshots.

Note

The numbers in the POOLS section are notional. They are not inclusive of the number of replicas, snapshots or clones. As a result, the sum of the USEDand %USED amounts will not add up to the RAW USED and %RAW USED amounts in the RAW STORAGE section of the output.

You can check OSDs to ensure they are up and on by executing:

cephuser@adm > ceph osd stat

or

cephuser@adm > ceph osd dump

You can also view OSDs according to their position in the CRUSH map.

ceph osd tree will print a CRUSH tree with a host, its OSDs, whether they are up, and their weight:

   cephuser@adm > ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
-1      3  0.02939  root default
-3      3  0.00980    rack mainrack
-2      3  0.00980            host osd-host
0       1  0.00980                    osd.0   up   1.00000   1.00000
1       1  0.00980                    osd.1   up   1.00000   1.00000
2       1  0.00980                    osd.2   up   1.00000   1.00000

Ceph prevents you from writing to a full OSD so that you do not lose data. In an operational cluster, you should receive a warning when your cluster is getting near its full ratio. The mon osd full ratio defaults to 0.95, or 95% of capacity before it stops clients from writing data. The mon osd nearfull ratio defaults to 0.85, or 85% of capacity, when it generates a health warning.

Full OSD nodes will be reported by ceph health:

cephuser@adm > ceph health
  HEALTH_WARN 1 nearfull osds
  osd.2 is near full at 85%

or

cephuser@adm > ceph health
  HEALTH_ERR 1 nearfull osds, 1 full osds
  osd.2 is near full at 85%
  osd.3 is full at 97%

The best way to deal with a full cluster is to add new OSD hosts/disks allowing the cluster to redistribute data to the newly available storage.

Tip: Preventing full OSDs

After an OSD becomes full—it uses 100% of its disk space—it will normally crash quickly without warning. Following are a few tips to remember when administering OSD nodes.

• Each OSD's disk space (usually mounted under /var/lib/ceph/osd/osd-{1,2..}) needs to be placed on a dedicated underlying disk or partition.

• Check the Ceph configuration files and make sure that Ceph does not store its log file to the disks/partitions dedicated for use by OSDs.

• Make sure that no other process writes to the disks/partitions dedicated for use by OSDs.

After you start the cluster and before first reading and/or writing data, check the Ceph Monitors' quorum status. When the cluster is already serving requests, check the Ceph Monitors' status periodically to ensure that they are running.

To display the monitor map, execute the following:

cephuser@adm > ceph mon stat

or

cephuser@adm > ceph mon dump

To check the quorum status for the monitor cluster, execute the following:

cephuser@adm > ceph quorum_status

Ceph will return the quorum status. For example, a Ceph cluster consisting of three monitors may return the following:

{ "election_epoch": 10,
  "quorum": [
        0,
        1,
        2],
  "monmap": { "epoch": 1,
      "fsid": "444b489c-4f16-4b75-83f0-cb8097468898",
      "modified": "2011-12-12 13:28:27.505520",
      "created": "2011-12-12 13:28:27.505520",
      "mons": [
            { "rank": 0,
              "name": "a",
              "addr": "192.168.1.10:6789\/0"},
            { "rank": 1,
              "name": "b",
              "addr": "192.168.1.11:6789\/0"},
            { "rank": 2,
              "name": "c",
              "addr": "192.168.1.12:6789\/0"}
           ]
    }
}

Placement groups map objects to OSDs. When you monitor your placement groups, you will want them to be active and clean. For a detailed discussion, refer to Section 12.9, “Monitoring OSDs and placement groups”.

When a Ceph storage cluster gets close to its maximum capacity, Ceph prevents you from writing to or reading from Ceph OSDs as a safety measure to prevent data loss. Therefore, letting a production cluster approach its full ratio is not a good practice, because it sacrifices high availability. The default full ratio is set to .95, meaning 95% of capacity. This a very aggressive setting for a test cluster with a small number of OSDs.

Tip: Increase Storage Capacity

When monitoring your cluster, be alert to warnings related to the nearfull ratio. It means that a failure of some OSDs could result in a temporary service disruption if one or more OSDs fails. Consider adding more OSDs to increase storage capacity.

A common scenario for test clusters involves a system administrator removing a Ceph OSD from the Ceph storage cluster to watch the cluster rebalance. Then removing another Ceph OSD, and so on until the cluster eventually reaches the full ratio and locks up. We recommend a bit of capacity planning even with a test cluster. Planning enables you to estimate how much spare capacity you will need in order to maintain high availability. Ideally, you want to plan for a series of Ceph OSD failures where the cluster can recover to an active + clean state without replacing those Ceph OSDs immediately. You can run a cluster in an active + degraded state, but this is not ideal for normal operating conditions.

The following diagram depicts a simplistic Ceph storage cluster containing 33 Ceph nodes with one Ceph OSD per host, each of them reading from and writing to a 3 TB drive. This exemplary cluster has a maximum actual capacity of 99 TB. The mon osd full ratio option is set to 0.95. If the cluster falls to 5 TB of the remaining capacity, it will not allow the clients to read and write data. Therefore the storage cluster’s operating capacity is 95 TB, not 99 TB.

Figure 12.1: Ceph cluster #

  

It is normal in such a cluster for one or two OSDs to fail. A less frequent but reasonable scenario involves a rack’s router or power supply failing, which brings down multiple OSDs simultaneously (for example, OSDs 7-12). In such a scenario, you should still strive for a cluster that can remain operational and achieve an active + clean state—even if that means adding a few hosts with additional OSDs in short order. If your capacity usage is too high, you may not lose data. But you could still sacrifice data availability while resolving an outage within a failure domain if capacity usage of the cluster exceeds the full ratio. For this reason, we recommend at least some rough capacity planning.

Identify two numbers for your cluster:

If you divide the total capacity of your cluster by the number of OSDs in your cluster, you will find the mean average capacity of an OSD within your cluster. Consider multiplying that number by the number of OSDs you expect will fail simultaneously during normal operations (a relatively small number). Finally, multiply the capacity of the cluster by the full ratio to arrive at a maximum operating capacity. Then, subtract the number of the amount of data from the OSDs you expect to fail to arrive at a reasonable full ratio. Repeat the foregoing process with a higher number of OSD failures (a rack of OSDs) to arrive at a reasonable number for a near full ratio.

The following settings only apply on cluster creation and are then stored in the OSD map:

[global]
 mon osd full ratio = .80
 mon osd backfillfull ratio = .75
 mon osd nearfull ratio = .70

Tip

These settings only apply during cluster creation. Afterward they need to be changed in the OSD Map using the ceph osd set-nearfull-ratio and ceph osd set-full-ratio commands.

Tip: Check OSD weight

If some OSDs are nearfull, but others have plenty of capacity, you may have a problem with the CRUSH weight for the nearfull OSDs.

High availability and high reliability require a fault-tolerant approach to managing hardware and software issues. Ceph has no single point-of-failure, and can service requests for data in a 'degraded' mode. Ceph’s data placement introduces a layer of indirection to ensure that data does not bind directly to particular OSD addresses. This means that tracking down system faults requires finding the placement group and the underlying OSDs at root of the problem.

Tip: Access in case of failure

A fault in one part of the cluster may prevent you from accessing a particular object. That does not mean that you cannot access other objects. When you run into a fault, follow the steps for monitoring your OSDs and placement groups. Then begin troubleshooting.

Ceph is generally self-repairing. However, when problems persist, monitoring OSDs and placement groups will help you identify the problem.

12.9.1 Monitoring OSDs #

  

An OSD’s status is either in the cluster ('in') or out of the cluster ('out'). At the same time, it is either up and running ('up') or it is down and not running ('down'). If an OSD is 'up', it may be either in the cluster (you can read and write data) or out of the cluster. If it was in the cluster and recently moved out of the cluster, Ceph will migrate placement groups to other OSDs. If an OSD is out of the cluster, CRUSH will not assign placement groups to it. If an OSD is 'down', it should also be 'out'.

Note: Unhealthy state

If an OSD is 'down' and 'in', there is a problem and the cluster will not be in a healthy state.

If you execute a command such as ceph health, ceph -s or ceph -w, you may notice that the cluster does not always echo back HEALTH OK. With regard to OSDs, you should expect that the cluster will not echo HEALTH OK under the following circumstances:

• You have not started the cluster yet (it will not respond).

• You have started or restarted the cluster and it is not ready yet, because the placement groups are being created and the OSDs are in the process of peering.

• You have added or removed an OSD.

• You have modified your cluster map.

An important aspect of monitoring OSDs is to ensure that when the cluster is up and running, all the OSDs in the cluster are up and running, too. To see if all the OSDs are running, execute:

# ceph osd stat
x osds: y up, z in; epoch: eNNNN

The result should tell you the total number of OSDs (x), how many are 'up' (y), how many are 'in' (z), and the map epoch (eNNNN). If the number of OSDs that are 'in' the cluster is more than the number of OSDs that are 'up', execute the following command to identify the ceph-osd daemons that are not running:

# ceph osd tree
#ID CLASS WEIGHT  TYPE NAME             STATUS REWEIGHT PRI-AFF
-1       2.00000 pool openstack
-3       2.00000 rack dell-2950-rack-A
-2       2.00000 host dell-2950-A1
0   ssd 1.00000      osd.0                up  1.00000 1.00000
1   ssd 1.00000      osd.1              down  1.00000 1.00000

For example, if an OSD with ID 1 is down, start it:

cephuser@osd > sudo systemctl start ceph-CLUSTER_ID@osd.0.service

See Book “Troubleshooting Guide”, Chapter 4 “Troubleshooting OSDs”, Section 4.3 “OSDs not running” for problems associated with OSDs that have stopped or that will not restart.

12.9.2 Assigning placement group sets #

  

When CRUSH assigns placement groups to OSDs, it looks at the number of replicas for the pool and assigns the placement group to OSDs such that each replica of the placement group gets assigned to a different OSD. For example, if the pool requires three replicas of a placement group, CRUSH may assign them to osd.1, osd.2 and osd.3 respectively. CRUSH actually seeks a pseudo-random placement that will take into account failure domains you set in your CRUSH Map, so you will rarely see placement groups assigned to nearest neighbor OSDs in a large cluster. We refer to the set of OSDs that should contain the replicas of a particular placement group as the acting set. In some cases, an OSD in the acting set is down or otherwise not able to service requests for objects in the placement group. When these situations arise, it may match one of the following scenarios:

• You added or removed an OSD. Then, CRUSH reassigned the placement group to other OSDs and therefore changed the composition of the acting set, causing the migration of data with a 'backfill' process.

• An OSD was 'down', was restarted, and is now recovering.

• An OSD in the acting set is 'down' or unable to service requests, and another OSD has temporarily assumed its duties.

  Ceph processes a client request using the up set, which is the set of OSDs that will actually handle the requests. In most cases, the up set and the acting set are virtually identical. When they are not, it may indicate that Ceph is migrating data, an OSD is recovering, or that there is a problem (for example, Ceph usually echoes a HEALTH WARN state with a 'stuck stale' message in such scenarios).

To retrieve a list of placement groups, run:

cephuser@adm > ceph pg dump

To view which OSDs are within the acting set or the up set for a given placement group, run:

cephuser@adm > ceph pg map PG_NUM
osdmap eNNN pg RAW_PG_NUM (PG_NUM) -> up [0,1,2] acting [0,1,2]

The result should tell you the osdmap epoch (eNNN), the placement group number (PG_NUM), the OSDs in the up set ('up'), and the OSDs in the acting set ('acting'):

Tip: Cluster problem indicator

If the up set and acting set do not match, this may be an indicator either of the cluster rebalancing itself, or of a potential problem with the cluster.

12.9.3 Peering #

  

Before you can write data to a placement group, it must be in an active state, and it should be in a clean state. For Ceph to determine the current state of a placement group, the primary OSD of the placement group (the first OSD in the acting set), peers with the secondary and tertiary OSDs to establish agreement on the current state of the placement group (assuming a pool with three replicas of the PG).

Figure 12.2: Peering schema #

  

12.9.4 Monitoring placement group states #

  

If you execute a command such as ceph health, ceph -s or ceph -w, you may notice that the cluster does not always echo back the HEALTH OK message. After you check to see if the OSDs are running, you should also check placement group states.

Expect that the cluster will not echo HEALTH OK in a number of placement group peering-related circumstances:

• You have created a pool and placement groups have not peered yet.

• The placement groups are recovering.

• You have added an OSD to or removed an OSD from the cluster.

• You have modified your CRUSH Map and your placement groups are migrating.

• There is inconsistent data in different replicas of a placement group.

• Ceph is scrubbing a placement group’s replicas.

• Ceph does not have enough storage capacity to complete backfilling operations.

If one of the above mentioned circumstances causes Ceph to echo HEALTH WARN, do not panic. In many cases, the cluster will recover on its own. In some cases, you may need to take action. An important aspect of monitoring placement groups is to ensure that when the cluster is up and running, all placement groups are 'active' and preferably in the 'clean state'. To see the status of all placement groups, run:

cephuser@adm > ceph pg stat
x pgs: y active+clean; z bytes data, aa MB used, bb GB / cc GB avail

The result should tell you the total number of placement groups (x), how many placement groups are in a particular state such as 'active+clean' (y) and the amount of data stored (z).

In addition to the placement group states, Ceph will also echo back the amount of storage capacity used (aa), the amount of storage capacity remaining (bb), and the total storage capacity for the placement group. These numbers can be important in a few cases:

• You are reaching your near full ratio or full ratio.

• Your data is not getting distributed across the cluster because of an error in your CRUSH configuration.

Tip: Placement group IDs

Placement group IDs consist of the pool number (not pool name) followed by a period (.) and the placement group ID—a hexadecimal number. You can view pool numbers and their names from the output of ceph osd lspools. For example, the default pool rbd corresponds to pool number 0. A fully qualified placement group ID has the following form:

POOL_NUM.PG_ID

And it typically looks like this:

0.1f

To retrieve a list of placement groups, run the following:

cephuser@adm > ceph pg dump

You can also format the output in JSON format and save it to a file:

cephuser@adm > ceph pg dump -o FILE_NAME --format=json

To query a particular placement group, run the following:

cephuser@adm > ceph pg POOL_NUM.PG_ID query

The following list describes the common placement group states in detail.

CREATING

When you create a pool, it will create the number of placement groups you specified. Ceph will echo 'creating' when it is creating one or more placement groups. When they are created, the OSDs that are part of the placement group’s acting set will peer. When peering is complete, the placement group status should be 'active+clean', which means that a Ceph client can begin writing to the placement group.

Figure 12.3: Placement groups status #

  

PEERING

When Ceph is peering a placement group, it is bringing the OSDs that store the replicas of the placement group into agreement about the state of the objects and metadata in the placement group. When Ceph completes peering, this means that the OSDs that store the placement group agree about the current state of the placement group. However, completion of the peering process does not mean that each replica has the latest contents.

Note: Authoritative history

Ceph will not acknowledge a write operation to a client until all OSDs of the acting set persist the write operation. This practice ensures that at least one member of the acting set will have a record of every acknowledged write operation since the last successful peering operation.

With an accurate record of each acknowledged write operation, Ceph can construct and enlarge a new authoritative history of the placement group—a complete and fully ordered set of operations that, if performed, would bring an OSD’s copy of a placement group up to date.

ACTIVE

When Ceph completes the peering process, a placement group may become active. The active state means that the data in the placement group is generally available in the primary placement group and the replicas for read and write operations.

CLEAN

When a placement group is in the clean state, the primary OSD and the replica OSDs have successfully peered and there are no stray replicas for the placement group. Ceph replicated all objects in the placement group the correct number of times.

DEGRADED

When a client writes an object to the primary OSD, the primary OSD is responsible for writing the replicas to the replica OSDs. After the primary OSD writes the object to storage, the placement group will remain in a 'degraded' state until the primary OSD has received an acknowledgement from the replica OSDs that Ceph created the replica objects successfully.

The reason a placement group can be 'active+degraded' is that an OSD may be 'active' even though it does not hold all of the objects yet. If an OSD goes down, Ceph marks each placement group assigned to the OSD as 'degraded'. The OSDs must peer again when the OSD comes back up. However, a client can still write a new object to a degraded placement group if it is 'active'.

If an OSD is 'down' and the 'degraded' condition persists, Ceph may mark the down OSD as 'out' of the cluster and remap the data from the 'down' OSD to another OSD. The time between being marked 'down' and being marked 'out' is controlled by the mon osd down out interval option, which is set to 600 seconds by default.

A placement group can also be 'degraded' because Ceph cannot find one or more objects that should be in the placement group. While you cannot read or write to unfound objects, you can still access all of the other objects in the 'degraded' placement group.

RECOVERING

Ceph was designed for fault-tolerance at a scale where hardware and software problems are ongoing. When an OSD goes 'down', its contents may fall behind the current state of other replicas in the placement groups. When the OSD is back 'up', the contents of the placement groups must be updated to reflect the current state. During that time period, the OSD may reflect a 'recovering' state.

Recovery is not always trivial, because a hardware failure may cause a cascading failure of multiple OSDs. For example, a network switch for a rack or cabinet may fail, which can cause the OSDs of a number of host machines to fall behind the current state of the cluster. Each of the OSDs must recover when the fault is resolved.

Ceph provides a number of settings to balance the resource contention between new service requests and the need to recover data objects and restore the placement groups to the current state. The osd recovery delay start setting allows an OSD to restart, re-peer and even process some replay requests before starting the recovery process. The osd recovery thread timeout sets a thread timeout, because multiple OSDs may fail, restart and re-peer at staggered rates. The osd recovery max active setting limits the number of recovery requests an OSD will process simultaneously to prevent the OSD from failing to serve. The osd recovery max chunk setting limits the size of the recovered data chunks to prevent network congestion.

BACK FILLING

When a new OSD joins the cluster, CRUSH will reassign placement groups from OSDs in the cluster to the newly added OSD. Forcing the new OSD to accept the reassigned placement groups immediately can put excessive load on the new OSD. Backfilling the OSD with the placement groups allows this process to begin in the background. When backfilling is complete, the new OSD will begin serving requests when it is ready.

During the backfill operations, you may see one of several states: 'backfill_wait' indicates that a backfill operation is pending, but is not yet in progress; 'backfill' indicates that a backfill operation is in progress; 'backfill_too_full' indicates that a backfill operation was requested, but could not be completed because of insufficient storage capacity. When a placement group cannot be backfilled, it may be considered 'incomplete'.

Ceph provides a number of settings to manage the load associated with reassigning placement groups to an OSD (especially a new OSD). By default, osd max backfills sets the maximum number of concurrent backfills to or from an OSD to 10. The backfill full ratio enables an OSD to refuse a backfill request if the OSD is approaching its full ratio (90%, by default) and change with ceph osd set-backfillfull-ratio command. If an OSD refuses a backfill request, the osd backfill retry interval enables an OSD to retry the request (after 10 seconds, by default). OSDs can also set osd backfill scan min and osd backfill scan max to manage scan intervals (64 and 512, by default).

REMAPPED

When the acting set that services a placement group changes, the data migrates from the old acting set to the new acting set. It may take some time for a new primary OSD to service requests. So it may ask the old primary to continue to service requests until the placement group migration is complete. When data migration completes, the mapping uses the primary OSD of the new acting set.

STALE

While Ceph uses heartbeats to ensure that hosts and daemons are running, the ceph-osd daemons may also get into a 'stuck' state where they are not reporting statistics in a timely manner (for example, a temporary network fault). By default, OSD daemons report their placement group, boot and failure statistics every half second (0.5), which is more frequent than the heartbeat thresholds. If the primary OSD of a placement group’s acting set fails to report to the monitor or if other OSDs have reported the primary OSD as 'down', the monitors will mark the placement group as 'stale'.

When you start your cluster, it is common to see the 'stale' state until the peering process completes. After your cluster has been running for a while, seeing placement groups in the 'stale' state indicates that the primary OSD for those placement groups is down or not reporting placement group statistics to the monitor.

cephuser@adm > ceph osd map POOL_NAME OBJECT_NAME [NAMESPACE]

cephuser@adm > rados put test-object-1 testfile.txt --pool=data

cephuser@adm > rados -p data ls

cephuser@adm > ceph osd map data test-object-1
osdmap e537 pool 'data' (0) object 'test-object-1' -> pg 0.d1743484 \
(0.4) -> up ([1,0], p0) acting ([1,0], p0)

cephuser@adm > rados rm test-object-1 --pool=data

To modify the configuration of an existing Ceph cluster, follow these steps:

To add a new node to a Ceph cluster, follow these steps:

Tip: Remove OSDs

If the node that you are going to remove runs OSDs, remove the OSDs from it first and check that no OSDs are running on that node. Refer to Section 13.4.4, “Removing OSDs” for more details on removing OSDs.

To remove a node from a cluster, do the following:

Important

In the event of a failure and the minion you are trying to remove is in a permanently powered-off state, you will need to remove the node from the Salt Master:

root@master # salt-key -d minion_id

Then, manually remove the node from pillar_root/ceph-salt.sls. This is typically located in /srv/pillar/ceph-salt.sls.

This section describes how to add, erase, or remove OSDs in a Ceph cluster.

cephuser@adm > ceph orch device ls
HOST       PATH      TYPE SIZE  DEVICE  AVAIL REJECT REASONS
ses-master /dev/vda  hdd  42.0G         False locked
ses-min1   /dev/vda  hdd  42.0G         False locked
ses-min1   /dev/vdb  hdd  8192M  387836 False locked, LVM detected, Insufficient space (<5GB) on vgs
ses-min2   /dev/vdc  hdd  8192M  450575 True

13.4.2 Erasing disk devices #

  

To re-use a disk device, you need to erase (or zap) it first:

ceph orch device zap HOST_NAME DISK_DEVICE

For example:

cephuser@adm > ceph orch device zap ses-min2 /dev/vdc

Note

If you previously deployed OSDs by using DriveGroups or the --all-available-devices option while the unmanaged flag was not set, cephadm will deploy these OSDs automatically after you erase them.

13.4.3 Adding OSDs using DriveGroups specification #

  

DriveGroups specify the layouts of OSDs in the Ceph cluster. They are defined in a single YAML file. In this section, we will use drive_groups.yml as an example.

An administrator should manually specify a group of OSDs that are interrelated (hybrid OSDs that are deployed on a mixture of HDDs and SDDs) or share identical deployment options (for example, the same object store, same encryption option, stand-alone OSDs). To avoid explicitly listing devices, DriveGroups use a list of filter items that correspond to a few selected fields of ceph-volume's inventory reports. cephadm will provide code that translates these DriveGroups into actual device lists for inspection by the user.

The command to apply the OSD specification to the cluster is:

cephuser@adm > ceph orch apply osd -i drive_groups.yml

To see a preview of actions and test your application, you can use the --dry-run option together with the ceph orch apply osd command. For example:

cephuser@adm > ceph orch apply osd -i drive_groups.yml --dry-run
...
+---------+------+------+----------+----+-----+
|SERVICE  |NAME  |HOST  |DATA      |DB  |WAL  |
+---------+------+------+----------+----+-----+
|osd      |test  |mgr0  |/dev/sda  |-   |-    |
|osd      |test  |mgr0  |/dev/sdb  |-   |-    |
+---------+------+------+----------+----+-----+

If the --dry-run output matches your expectations, then simply re-run the command without the --dry-run option.

13.4.3.1 Unmanaged OSDs #

  

All available clean disk devices that match the DriveGroups specification will be used as OSDs automatically after you add them to the cluster. This behavior is called a managed mode.

To disable the managed mode, add the unmanaged: true line to the relevant specifications, for example:

service_type: osd
service_id: example_drvgrp_name
placement:
 hosts:
 - ses-min2
 - ses-min3
encrypted: true
unmanaged: true

Tip

To change already deployed OSDs from the managed to unmanaged mode, add the unmanaged: true lines where applicable during the procedure described in Section 13.1, “Modifying the cluster configuration”.

13.4.3.2 DriveGroups specification #

  

Following is an example DriveGroups specification file:

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  drive_spec: DEVICE_SPECIFICATION
db_devices:
  drive_spec: DEVICE_SPECIFICATION
wal_devices:
  drive_spec: DEVICE_SPECIFICATION
block_wal_size: '5G'  # (optional, unit suffixes permitted)
block_db_size: '5G'   # (optional, unit suffixes permitted)
encrypted: true       # 'True' or 'False' (defaults to 'False')

Note

The option previously called "encryption" in DeepSea has been renamed to "encrypted". When applying DriveGroups in SUSE Enterprise Storage 7, ensure you use this new terminology in your service specification, otherwise the ceph orch apply operation will fail.

13.4.3.3 Matching disk devices #

  

You can describe the specification using the following filters:

• By a disk model:

  model: DISK_MODEL_STRING

• By a disk vendor:

  vendor: DISK_VENDOR_STRING

  

  Tip

  Always enter the DISK_VENDOR_STRING in lowercase.

  To obtain details about disk model and vendor, examine the output of the following command:

  cephuser@adm > ceph orch device ls
  HOST     PATH     TYPE  SIZE DEVICE_ID                  MODEL            VENDOR
  ses-min1 /dev/sdb ssd  29.8G SATA_SSD_AF34075704240015  SATA SSD         ATA
  ses-min2 /dev/sda ssd   223G Micron_5200_MTFDDAK240TDN  Micron_5200_MTFD ATA
  [...]

• Whether a disk is rotational or not. SSDs and NVMe drives are not rotational.

  rotational: 0

• Deploy a node using all available drives for OSDs:

  data_devices:
    all: true

• Additionally, by limiting the number of matching disks:

  limit: 10

13.4.3.4 Filtering devices by size #

  

You can filter disk devices by their size—either by an exact size, or a size range. The size: parameter accepts arguments in the following form:

• '10G' - Includes disks of an exact size.

• '10G:40G' - Includes disks whose size is within the range.

• ':10G' - Includes disks less than or equal to 10 GB in size.

• '40G:' - Includes disks equal to or greater than 40 GB in size.

Example 13.1: Matching by disk size #

  

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  size: '40TB:'
db_devices:
  size: ':2TB'

Note: Quotes required

When using the ':' delimiter, you need to enclose the size in quotes, otherwise the ':' sign will be interpreted as a new configuration hash.

Tip: Unit shortcuts

Instead of Gigabytes (G), you can specify the sizes in Megabytes (M) or Terabytes (T).

13.4.3.5 DriveGroups examples #

  

This section includes examples of different OSD setups.

Example 13.2: Simple setup #

  

This example describes two nodes with the same setup:

• 20 HDDs

  • Vendor: Intel

  • Model: SSD-123-foo

  • Size: 4 TB

• 2 SSDs

  • Vendor: Micron

  • Model: MC-55-44-ZX

  • Size: 512 GB

The corresponding drive_groups.yml file will be as follows:

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  model: SSD-123-foo
db_devices:
  model: MC-55-44-XZ

Such a configuration is simple and valid. The problem is that an administrator may add disks from different vendors in the future, and these will not be included. You can improve it by reducing the filters on core properties of the drives:

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  rotational: 1
db_devices:
  rotational: 0

In the previous example, we are enforcing all rotating devices to be declared as 'data devices' and all non-rotating devices will be used as 'shared devices' (wal, db).

If you know that drives with more than 2 TB will always be the slower data devices, you can filter by size:

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  size: '2TB:'
db_devices:
  size: ':2TB'

Example 13.3: Advanced setup #

  

This example describes two distinct setups: 20 HDDs should share 2 SSDs, while 10 SSDs should share 2 NVMes.

• 20 HDDs

  • Vendor: Intel

  • Model: SSD-123-foo

  • Size: 4 TB

• 12 SSDs

  • Vendor: Micron

  • Model: MC-55-44-ZX

  • Size: 512 GB

• 2 NVMes

  • Vendor: Samsung

  • Model: NVME-QQQQ-987

  • Size: 256 GB

Such a setup can be defined with two layouts as follows:

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  rotational: 0
db_devices:
  model: MC-55-44-XZ

service_type: osd
service_id: example_drvgrp_name2
placement:
  host_pattern: '*'
data_devices:
  model: MC-55-44-XZ
db_devices:
  vendor: samsung
  size: 256GB

Example 13.4: Advanced setup with non-uniform nodes #

  

The previous examples assumed that all nodes have the same drives. However, that is not always the case:

Nodes 1-5:

• 20 HDDs

  • Vendor: Intel

  • Model: SSD-123-foo

  • Size: 4 TB

• 2 SSDs

  • Vendor: Micron

  • Model: MC-55-44-ZX

  • Size: 512 GB

Nodes 6-10:

• 5 NVMes

  • Vendor: Intel

  • Model: SSD-123-foo

  • Size: 4 TB

• 20 SSDs

  • Vendor: Micron

  • Model: MC-55-44-ZX

  • Size: 512 GB

You can use the 'target' key in the layout to target specific nodes. Salt target notation helps to keep things simple:

service_type: osd
service_id: example_drvgrp_one2five
placement:
  host_pattern: 'node[1-5]'
data_devices:
  rotational: 1
db_devices:
  rotational: 0

followed by

service_type: osd
service_id: example_drvgrp_rest
placement:
  host_pattern: 'node[6-10]'
data_devices:
  model: MC-55-44-XZ
db_devices:
  model: SSD-123-foo

Example 13.5: Expert setup #

  

All previous cases assumed that the WALs and DBs use the same device. It is however possible to deploy the WAL on a dedicated device as well:

• 20 HDDs

  • Vendor: Intel

  • Model: SSD-123-foo

  • Size: 4 TB

• 2 SSDs

  • Vendor: Micron

  • Model: MC-55-44-ZX

  • Size: 512 GB

• 2 NVMes

  • Vendor: Samsung

  • Model: NVME-QQQQ-987

  • Size: 256 GB

service_type: osd
service_id: example_drvgrp_name
placement:
  host_pattern: '*'
data_devices:
  model: MC-55-44-XZ
db_devices:
  model: SSD-123-foo
wal_devices:
  model: NVME-QQQQ-987

Example 13.6: Complex (and unlikely) setup #

  

In the following setup, we are trying to define:

• 20 HDDs backed by 1 NVMe

• 2 HDDs backed by 1 SSD(db) and 1 NVMe (wal)

• 8 SSDs backed by 1 NVMe

• 2 SSDs stand-alone (encrypted)

• 1 HDD is spare and should not be deployed

The summary of used drives is as follows:

• 23 HDDs

  • Vendor: Intel

  • Model: SSD-123-foo

  • Size: 4 TB

• 10 SSDs

  • Vendor: Micron

  • Model: MC-55-44-ZX

  • Size: 512 GB

• 1 NVMe

  • Vendor: Samsung

  • Model: NVME-QQQQ-987

  • Size: 256 GB

The DriveGroups definition will be the following:

service_type: osd
service_id: example_drvgrp_hdd_nvme
placement:
  host_pattern: '*'
data_devices:
  rotational: 0
db_devices:
  model: NVME-QQQQ-987

service_type: osd
service_id: example_drvgrp_hdd_ssd_nvme
placement:
  host_pattern: '*'
data_devices:
  rotational: 0
db_devices:
  model: MC-55-44-XZ
wal_devices:
  model: NVME-QQQQ-987

service_type: osd
service_id: example_drvgrp_ssd_nvme
placement:
  host_pattern: '*'
data_devices:
  model: SSD-123-foo
db_devices:
  model: NVME-QQQQ-987

service_type: osd
service_id: example_drvgrp_standalone_encrypted
placement:
  host_pattern: '*'
data_devices:
  model: SSD-123-foo
encrypted: True

One HDD will remain as the file is being parsed from top to bottom.