Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
SUSE Enterprise Storage 5.5 (SES 5 & SES 5.5)

Administration Guide

Authors: Tomáš Bažant, Jana Haláčková, and Sven Seeberg
Publication Date: 07/04/2021
About This Guide
Available Documentation
Documentation Conventions
About the Making of This Manual
Ceph Contributors
I Cluster Management
1 Salt Cluster Administration
1.1 Adding New Cluster Nodes
1.2 Adding New Roles to Nodes
1.3 Removing and Reinstalling Cluster Nodes
1.4 Redeploying Monitor Nodes
1.5 Adding an OSD Disk to a Node
1.6 Removing an OSD
1.7 Replacing an OSD Disk
1.8 Recovering a Reinstalled OSD Node
1.9 Automated Installation via Salt
1.10 Updating the Cluster Nodes
1.11 Halting or Rebooting Cluster
1.12 Adjusting ceph.conf with Custom Settings
1.13 Enabling AppArmor Profiles
II Operating a Cluster
2 Introduction
3 Operating Ceph Services
3.1 Operating Ceph Cluster Related Services using systemd
3.2 Restarting Ceph Services using DeepSea
3.3 Shutdown and Restart of the Whole Ceph Cluster
4 Determining Cluster State
4.1 Checking a Cluster's Status
4.2 Checking Cluster Health
4.3 Watching a Cluster
4.4 Checking a Cluster's Usage Stats
4.5 Checking OSD Status
4.6 Checking for Full OSDs
4.7 Checking Monitor Status
4.8 Checking Placement Group States
4.9 Using the Admin Socket
5 Monitoring and Alerting
5.1 Configuration File
5.2 Custom Alerts
6 Authentication with cephx
6.1 Authentication Architecture
6.2 Key Management
7 Stored Data Management
7.1 Devices
7.2 Buckets
7.3 Rule Sets
7.4 CRUSH Map Manipulation
7.5 Scrubbing
8 Managing Storage Pools
8.1 Associate Pools with an Application
8.2 Operating Pools
8.3 Pool Migration
8.4 Pool Snapshots
8.5 Data Compression
9 RADOS Block Device
9.1 Block Device Commands
9.2 Mounting and Unmounting
9.3 Snapshots
9.4 Mirroring
9.5 Advanced Features
10 Erasure Coded Pools
10.1 Prerequisite for Erasure Coded Pools
10.2 Creating a Sample Erasure Coded Pool
10.3 Erasure Code Profiles
10.4 Erasure Coded Pools with RADOS Block Device
11 Cache Tiering
11.1 Tiered Storage Terminology
11.2 Points to Consider
11.3 When to Use Cache Tiering
11.4 Cache Modes
11.5 Erasure Coded Pool and Cache Tiering
11.6 Setting Up an Example Tiered Storage
11.7 Configuring a Cache Tier
12 Ceph Cluster Configuration
12.1 Runtime Configuration
12.2 Ceph OSD and BlueStore
III Accessing Cluster Data
13 Ceph Object Gateway
13.1 Object Gateway Restrictions and Naming Limitations
13.2 Deploying the Object Gateway
13.3 Operating the Object Gateway Service
13.4 Configuration Parameters
13.5 Managing Object Gateway Access
13.6 Enabling HTTPS/SSL for Object Gateways
13.7 Sync Modules
13.8 LDAP Authentication
13.9 Bucket Index Sharding
13.10 Integrating OpenStack Keystone
13.11 Multisite Object Gateways
13.12 Load Balancing the Object Gateway Servers with HAProxy
14 Ceph iSCSI Gateway
14.1 Connecting to lrbd-managed Targets
14.2 Conclusion
15 Clustered File System
15.1 Mounting CephFS
15.2 Unmounting CephFS
15.3 CephFS in /etc/fstab
15.4 Multiple Active MDS Daemons (Active-Active MDS)
15.5 Managing Failover
16 NFS Ganesha: Export Ceph Data via NFS
16.1 Installation
16.2 Configuration
16.3 Custom NFS Ganesha Roles
16.4 Starting or Restarting NFS Ganesha
16.5 Setting the Log Level
16.6 Verifying the Exported NFS Share
16.7 Mounting the Exported NFS Share
16.8 Additional Resources
IV Managing Cluster with GUI Tools
17 openATTIC
17.1 openATTIC Deployment and Configuration
17.2 openATTIC Web User Interface
17.3 Dashboard
17.4 Ceph Related Tasks
V Integration with Virtualization Tools
18 Using libvirt with Ceph
18.1 Configuring Ceph
18.2 Preparing the VM Manager
18.3 Creating a VM
18.4 Configuring the VM
18.5 Summary
19 Ceph as a Back-end for QEMU KVM Instance
19.1 Installation
19.2 Usage
19.3 Creating Images with QEMU
19.4 Resizing Images with QEMU
19.5 Retrieving Image Info with QEMU
19.6 Running QEMU with RBD
19.7 Enabling Discard/TRIM
19.8 QEMU Cache Options
VI FAQs, Tips and Troubleshooting
20 Hints and Tips
20.1 Identifying Orphaned Partitions
20.2 Adjusting Scrubbing
20.3 Stopping OSDs without Rebalancing
20.4 Time Synchronization of Nodes
20.5 Checking for Unbalanced Data Writing
20.6 Btrfs Sub-volume for /var/lib/ceph
20.7 Increasing File Descriptors
20.8 How to Use Existing Partitions for OSDs Including OSD Journals
20.9 Integration with Virtualization Software
20.10 Firewall Settings for Ceph
20.11 Testing Network Performance
20.12 Replacing Storage Disk
21 Frequently Asked Questions
21.1 How Does the Number of Placement Groups Affect the Cluster Performance?
21.2 Can I Use SSDs and Hard Disks on the Same Cluster?
21.3 What are the Trade-offs of Using a Journal on SSD?
21.4 What Happens When a Disk Fails?
21.5 What Happens When a Journal Disk Fails?
22 Troubleshooting
22.1 Reporting Software Problems
22.2 Sending Large Objects with rados Fails with Full OSD
22.3 Corrupted XFS File system
22.4 'Too Many PGs per OSD' Status Message
22.5 'nn pg stuck inactive' Status Message
22.6 OSD Weight is 0
22.7 OSD is Down
22.8 Finding Slow OSDs
22.9 Fixing Clock Skew Warnings
22.10 Poor Cluster Performance Caused by Network Problems
22.11 /var Running Out of Space
22.12 Too Many PGs Per OSD
A DeepSea Stage 1 Custom Example
B Default Alerts for SUSE Enterprise Storage
C Example Procedure of Manual Ceph Installation
D Documentation Updates
D.1 The Latest Documentation Update
D.2 October, 2018 (Documentation Maintenance Update)
D.3 November 2017 (Documentation Maintenance Update)
D.4 October, 2017 (Release of SUSE Enterprise Storage 5.5)
List of Figures
6.1 Basic cephx Authentication
6.2 cephx Authentication
6.3 cephx Authentication - MDS and OSD
7.1 OSDs with Mixed Device Classes
7.2 Example Tree
7.3 Node Replacement Methods
8.1 Pools before Migration
8.2 Cache Tier Setup
8.3 Data Flushing
8.4 Setting Overlay
8.5 Migration Complete
9.1 RADOS Protocol
11.1 Bloom Filter with 3 Stored Objects
14.1 iSCSI Initiator Properties
14.2 Discover Target Portal
14.3 Target Portals
14.4 Targets
14.5 iSCSI Target Properties
14.6 Device Details
14.7 New Volume Wizard
14.8 Offline Disk Prompt
14.9 Confirm Volume Selections
14.10 iSCSI Initiator Properties
14.11 Add Target Server
14.12 Manage Multipath Devices
14.13 Paths Listing for Multipath
14.14 Add Storage Dialog
14.15 Custom Space Setting
14.16 iSCSI Datastore Overview
17.1 openATTIC Login Screen
17.2 openATTIC Dashboard
17.3 Basic Widgets
17.4 Capacity Widgets
17.5 Latency Widgets
17.6 Throughput
17.7 List of OSD nodes
17.8 List of RBDs
17.9 RBD Details
17.10 RBD Snapshots
17.11 Deleting RBD
17.12 Adding a New RBD
17.13 List of Pools
17.14 Pool Details
17.15 Deleting Pools
17.16 Adding a New Pool
17.17 List of Nodes
17.18 List of NFS Exports
17.19 NFS Export Details
17.20 Adding a New NFS Export
17.21 Editing an NFS Export
17.22 List of iSCSI Gateways
17.23 Gateway Details
17.24 Adding a New iSCSI Gateway
17.25 CRUSH Map
17.26 Replication rules
17.27 List of Object Gateway Users
17.28 Adding a New Object Gateway User
17.29 User quota
17.30 Bucket Quota
17.31 Adding a Subuser
17.32 View S3 keys
17.33 Capabilities
17.34 Object Gateway Buckets
17.35 Adding a New Bucket
17.36 Bucket Details
17.37 Editing an Object Gateway Bucket
17.38 Deleting Buckets

Copyright © 2021 SUSE LLC

Copyright © 2016, RedHat, Inc, and contributors.

The text of and illustrations in this document are licensed under a Creative Commons Attribution-Share Alike 4.0 International ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/4.0/legalcode. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux® is the registered trademark of Linus Torvalds in the United States and other countries. Java® is a registered trademark of Oracle and/or its affiliates. XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. All other trademarks are the property of their respective owners.

For SUSE trademarks, see http://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors nor the translators shall be held liable for possible errors or the consequences thereof.

Print this page