Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
SUSE Enterprise Storage 6

Administration Guide

Authors: Tomáš Bažant, Alexandra Settle, Liam Proven, and Sven Seeberg
Publication Date: 07/04/2021
About This Guide
Available Documentation
Documentation Conventions
About the Making of This Manual
Ceph Contributors
I Cluster Management
1 User Privileges and Command Prompts
1.1 Salt/DeepSea Related Commands
1.2 Ceph Related Commands
1.3 General Linux Commands
1.4 Additional Information
2 Salt Cluster Administration
2.1 Adding New Cluster Nodes
2.2 Adding New Roles to Nodes
2.3 Removing and Reinstalling Cluster Nodes
2.4 Redeploying Monitor Nodes
2.5 Verify an Encrypted OSD
2.6 Adding an OSD Disk to a Node
2.7 Removing an OSD
2.8 Replacing an OSD Disk
2.9 Recovering a Reinstalled OSD Node
2.10 Moving the Admin Node to a New Server
2.11 Automated Installation via Salt
2.12 Updating the Cluster Nodes
2.13 Halting or Rebooting Cluster
2.14 Adjusting ceph.conf with Custom Settings
2.15 Enabling AppArmor Profiles
2.16 Deactivating Tuned Profiles
2.17 Removing an Entire Ceph Cluster
3 Backing Up Cluster Configuration and Data
3.1 Back Up Ceph Configuration
3.2 Back Up Salt Configuration
3.3 Back Up DeepSea Configuration
3.4 Back Up Custom Configurations
II Ceph Dashboard
4 About Ceph Dashboard
5 Dashboard's Web User Interface
5.1 Log In
5.2 Utility Menu
5.3 Main Menu
5.4 The Content Pane
5.5 Common Web UI Features
5.6 Dashboard Widgets
6 Managing Dashboard Users and Roles
6.1 Listing Users
6.2 Adding New Users
6.3 Editing Users
6.4 Deleting Users
6.5 Listing User Roles
6.6 Adding Custom Roles
6.7 Editing Custom Roles
6.8 Deleting Custom Roles
7 Viewing Cluster Internals
7.1 Cluster Nodes
7.2 Ceph Monitors
7.3 Ceph OSDs
7.4 Cluster Configuration
7.5 CRUSH Map
7.6 Manager Modules
7.7 Logs
8 Managing Pools
8.1 Adding a New Pool
8.2 Deleting Pools
8.3 Editing a Pool's Options
9 Managing RADOS Block Devices
9.1 Viewing Details about RBDs
9.2 Viewing RBD's Configuration
9.3 Creating RBDs
9.4 Deleting RBDs
9.5 RADOS Block Device Snapshots
9.6 Managing iSCSI Gateways
9.7 RBD Quality of Service (QoS)
9.8 RBD Mirroring
10 Managing NFS Ganesha
10.1 Adding NFS Exports
10.2 Deleting NFS Exports
10.3 Editing NFS Exports
11 Managing Ceph File Systems
11.1 Viewing CephFS Overview
12 Managing Object Gateways
12.1 Viewing Object Gateways
12.2 Managing Object Gateway Users
12.3 Managing the Object Gateway Buckets
13 Manual Configuration
13.1 TLS/SSL Support
13.2 Host Name and Port Number
13.3 User Name and Password
13.4 Enabling the Object Gateway Management Front-end
13.5 Enable Single Sign-On
14 Managing Users and Roles on the Command Line
14.1 User Accounts
14.2 User Roles and Permissions
14.3 Reverse Proxies
14.4 Auditing
III Operating a Cluster
15 Introduction
16 Operating Ceph Services
16.1 Operating Ceph Cluster Related Services Using systemd
16.2 Restarting Ceph Services Using DeepSea
16.3 Shutdown and Start of the Whole Ceph Cluster
17 Determining Cluster State
17.1 Checking a Cluster's Status
17.2 Checking Cluster Health
17.3 Watching a Cluster
17.4 Checking a Cluster's Usage Stats
17.5 Checking OSD Status
17.6 Checking for Full OSDs
17.7 Checking Monitor Status
17.8 Checking Placement Group States
17.9 Using the Admin Socket
17.10 Storage Capacity
17.11 Monitoring OSDs and Placement Groups
17.12 OSD Is Not Running
18 Monitoring and Alerting
18.1 Pillar Variables
18.2 Grafana
18.3 Prometheus
18.4 Alertmanager
18.5 Troubleshooting Alerts
19 Authentication with cephx
19.1 Authentication Architecture
19.2 Key Management
20 Stored Data Management
20.1 Devices
20.2 Buckets
20.3 Rule Sets
20.4 Placement Groups
20.5 CRUSH Map Manipulation
20.6 Scrubbing
21 Ceph Manager Modules
21.1 Balancer
21.2 Telemetry Module
22 Managing Storage Pools
22.1 Associate Pools with an Application
22.2 Operating Pools
22.3 Pool Migration
22.4 Pool Snapshots
22.5 Data Compression
23 RADOS Block Device
23.1 Block Device Commands
23.2 Mounting and Unmounting
23.3 Snapshots
23.4 Mirroring
23.5 Cache Settings
23.6 QoS Settings
23.7 Read-ahead Settings
23.8 Advanced Features
23.9 Mapping RBD Using Old Kernel Clients
24 Erasure Coded Pools
24.1 Prerequisite for Erasure Coded Pools
24.2 Creating a Sample Erasure Coded Pool
24.3 Erasure Code Profiles
24.4 Erasure Coded Pools with RADOS Block Device
25 Ceph Cluster Configuration
25.1 Runtime Configuration
25.2 Ceph OSD and BlueStore
25.3 Ceph Object Gateway
IV Accessing Cluster Data
26 Ceph Object Gateway
26.1 Object Gateway Restrictions and Naming Limitations
26.2 Deploying the Object Gateway
26.3 Operating the Object Gateway Service
26.4 Configuration Options
26.5 Managing Object Gateway Access
26.6 HTTP Front-ends
26.7 Enabling HTTPS/SSL for Object Gateways
26.8 Synchronization Modules
26.9 LDAP Authentication
26.10 Bucket Index Sharding
26.11 Integrating OpenStack Keystone
26.12 Pool Placement and Storage Classes
26.13 Multisite Object Gateways
26.14 Load Balancing the Object Gateway Servers with HAProxy
27 Ceph iSCSI Gateway
27.1 Connecting to ceph-iscsi Managed Targets
27.2 Conclusion
28 Clustered File System
28.1 Mounting CephFS
28.2 Unmounting CephFS
28.3 CephFS in /etc/fstab
28.4 Multiple Active MDS Daemons (Active-Active MDS)
28.5 Managing Failover
28.6 Setting CephFS Quotas
28.7 Managing CephFS Snapshots
29 Exporting Ceph Data via Samba
29.1 Export CephFS via Samba Share
29.2 Samba Gateway Joining Active Directory
30 NFS Ganesha: Export Ceph Data via NFS
30.1 Installation
30.2 Configuration
30.3 Custom NFS Ganesha Roles
30.4 Starting or Restarting NFS Ganesha
30.5 Setting the Log Level
30.6 Verifying the Exported NFS Share
30.7 Mounting the Exported NFS Share
V Integration with Virtualization Tools
31 Using libvirt with Ceph
31.1 Configuring Ceph
31.2 Preparing the VM Manager
31.3 Creating a VM
31.4 Configuring the VM
31.5 Summary
32 Ceph as a Back-end for QEMU KVM Instance
32.1 Installation
32.2 Usage
32.3 Creating Images with QEMU
32.4 Resizing Images with QEMU
32.5 Retrieving Image Info with QEMU
32.6 Running QEMU with RBD
32.7 Enabling Discard/TRIM
32.8 QEMU Cache Options
VI FAQs, Tips and Troubleshooting
33 Hints and Tips
33.1 Identifying Orphaned Partitions
33.2 Adjusting Scrubbing
33.3 Stopping OSDs without Rebalancing
33.4 Time Synchronization of Nodes
33.5 Checking for Unbalanced Data Writing
33.6 Btrfs Subvolume for /var/lib/ceph on Ceph Monitor Nodes
33.7 Increasing File Descriptors
33.8 Integration with Virtualization Software
33.9 Firewall Settings for Ceph
33.10 Testing Network Performance
33.11 How to Locate Physical Disks Using LED Lights
34 Frequently Asked Questions
34.1 How Does the Number of Placement Groups Affect the Cluster Performance?
34.2 Can I Use SSDs and Hard Disks on the Same Cluster?
34.3 What are the Trade-offs of Using a Journal on SSD?
34.4 What Happens When a Disk Fails?
34.5 What Happens When a Journal Disk Fails?
35 Troubleshooting
35.1 Reporting Software Problems
35.2 Sending Large Objects with rados Fails with Full OSD
35.3 Corrupted XFS File system
35.4 'Too Many PGs per OSD' Status Message
35.5 'nn pg stuck inactive' Status Message
35.6 OSD Weight is 0
35.7 OSD is Down
35.8 Finding Slow OSDs
35.9 Fixing Clock Skew Warnings
35.10 Poor Cluster Performance Caused by Network Problems
35.11 /var Running Out of Space
35.12 OSD Panic Occurs when Media Error Happens during FileStore Directory Split
A DeepSea Stage 1 Custom Example
B Ceph Maintenance Updates Based on Upstream 'Nautilus' Point Releases
C Documentation Updates
C.1 Maintenance update of SUSE Enterprise Storage 6 documentation
C.2 June 2019 (Release of SUSE Enterprise Storage 6)
List of Figures
5.1 Ceph Dashboard Login Screen
5.2 Ceph Dashboard Home Page
5.3 Status Widgets
5.4 performance Widgets
5.5 Capacity Widgets
6.1 User Management
6.2 Adding a User
6.3 User Roles
6.4 Adding a Role
7.1 Hosts
7.2 Ceph Monitors
7.3 Ceph OSDs
7.4 OSD Flags
7.5 OSD Recovery Priority
7.6 OSD Details
7.7 Cluster Configuration
7.8 CRUSH Map
7.9 Manager Modules
7.10 Logs
8.1 List of Pools
8.2 Adding a New Pool
9.1 List of RBD Images
9.2 RBD Details
9.3 RBD Configuration
9.4 Adding a New RBD
9.5 RBD Snapshots
9.6 List of iSCSI Targets
9.7 iSCSI Target Details
9.8 Adding a New Target
9.9 Running rbd-mirror Daemons
9.10 Creating a Pool with RBD Application
9.11 Configuring the Replication Mode
9.12 Adding Peer Credentials
9.13 List of Replicated Pools
9.14 New RBD Image
9.15 New RBD Image Synchronized
9.16 RBD Images' Replication Status
10.1 List of NFS Exports
10.2 NFS Export Details
10.3 Adding a New NFS Export
10.4 Editing an NFS Export
11.1 CephFS Details
12.1 Gateway's Details
12.2 Gateway Users
12.3 Adding a New Gateway User
12.4 Gateway Bucket Details
17.1 Ceph Cluster
17.2 Peering Schema
17.3 Placement Groups Status
19.1 Basic cephx Authentication
19.2 cephx Authentication
19.3 cephx Authentication - MDS and OSD
20.1 OSDs with Mixed Device Classes
20.2 Example Tree
20.3 Node Replacement Methods
20.4 Placement Groups in a Pool
20.5 Placement Groups and OSDs
22.1 Pools before Migration
22.2 Cache Tier Setup
22.3 Data Flushing
22.4 Setting Overlay
22.5 Migration Complete
23.1 RADOS Protocol
27.1 iSCSI Initiator Properties
27.2 Discover Target Portal
27.3 Target Portals
27.4 Targets
27.5 iSCSI Target Properties
27.6 Device Details
27.7 New Volume Wizard
27.8 Offline Disk Prompt
27.9 Confirm Volume Selections
27.10 iSCSI Initiator Properties
27.11 Add Target Server
27.12 Manage Multipath Devices
27.13 Paths Listing for Multipath
27.14 Add Storage Dialog
27.15 Custom Space Setting
27.16 iSCSI Datastore Overview

Copyright © 2021 SUSE LLC

Copyright © 2016, RedHat, Inc, and contributors.

The text of and illustrations in this document are licensed under a Creative Commons Attribution-Share Alike 4.0 International ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/4.0/legalcode. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux® is the registered trademark of Linus Torvalds in the United States and other countries. Java® is a registered trademark of Oracle and/or its affiliates. XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. All other trademarks are the property of their respective owners.

For SUSE trademarks, see http://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors nor the translators shall be held liable for possible errors or the consequences thereof.

Print this page