SUSE Enterprise Storage 7 uses the Salt-based ceph-salt
tool to prepare the operating system on each participating cluster node for deployment via cephadm. cephadm deploys and manages a Ceph cluster by connecting to hosts from the Ceph Manager daemon via SSH. cephadm manages the full life-cycle of a Ceph cluster.
It starts by bootstrapping a tiny cluster on a single node (one MON and MGR service) and then uses the orchestration interface to expand the cluster to include all hosts and to provision all Ceph services.
You can perform this via the Ceph command line interface (CLI) or partially via Ceph Dashboard (GUI).
Note that the Ceph community documentation uses the cephadm bootstrap
command during initial deployment.
The ceph-salt
calls the cephadm bootstrap
command and should not be run directly.
Any Ceph cluster deployment manually using the cephadm bootstrap
will be unsupported.
To deploy a Ceph cluster by using cephadm, you need to complete the following tasks:
Install and do basic configuration of the underlying operating system—SUSE Linux Enterprise Server 15 SP2—on all cluster nodes.
Deploy the Salt infrastructure on all cluster nodes for performing the initial deployment preparations via ceph-salt
.
Configure the basic properties of the cluster via ceph-salt
and deploy it.
Add new nodes and roles to the cluster and deploy services to them using cephadm.
Install and register SUSE Linux Enterprise Server 15 SP2 on each cluster node. During installation of SUSE Enterprise Storage, access to the update repositories is required, therefore registration is mandatory. Include at least the following modules:
Basesystem Module
Server Applications Module
Find more details on how to install SUSE Linux Enterprise Server in https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-install.html.
Install the SUSE Enterprise Storage 7 extension on each cluster node.
You can either install the SUSE Enterprise Storage 7 extension separately after you have installed SUSE Linux Enterprise Server 15 SP2, or you can add it during the SUSE Linux Enterprise Server 15 SP2 installation procedure.
Find more details on how to install extensions in https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-register-sle.html.
Configure network settings including proper DNS name resolution on each node. For more information on configuring a network, see https://documentation.suse.com/sles/15-SP2/single-html/SLES-admin/#sec-network-yast For more information on configuring a DNS server, see https://documentation.suse.com/sles/15-SP2/single-html/SLES-admin/#cha-dns.
SUSE Enterprise Storage uses Salt and ceph-salt
for the initial cluster preparation. Salt helps you configure and run commands on multiple cluster nodes simultaneously from one dedicated host called the Salt Master.
Before deploying Salt, consider the following important points:
Salt Minions are the nodes controlled by a dedicated node called Salt Master.
If the Salt Master host should be part of the Ceph cluster, it needs to run its own Salt Minion, but this is not a requirement.
You will get the best performance from your Ceph cluster when each role is deployed on a separate node. But real deployments sometimes require sharing one node for multiple roles. To avoid trouble with performance and the upgrade procedure, do not deploy the Ceph OSD, Metadata Server, or Ceph Monitor role to the Admin Node.
Salt Minions need to correctly resolve the Salt Master's host name over the network.
By default, they look for the salt
host name, but you can specify any other network-reachable host name in the /etc/salt/minion
file.
Install the salt-master
on the Salt Master node:
root@master #
zypper in salt-master
Check that the salt-master
service is enabled and started, and enable and start it if needed:
root@master #
systemctl enable salt-master.serviceroot@master #
systemctl start salt-master.service
If you intend to use the firewall, verify that the Salt Master node has ports 4505 and 4506 open to all Salt Minion nodes.
If the ports are closed, you can open them using the yast2 firewall
command by allowing the service for the appropriate zone.
For example, public
.
Install the package salt-minion
on all minion nodes.
root@minion >
zypper in salt-minion
Edit /etc/salt/minion
and uncomment the following line:
#log_level_logfile: warning
Change the warning
log level to info
.
log_level_logfile
and log_level
While log_level
controls which log messages will be displayed on the screen, log_level_logfile
controls which log messages will be written to /var/log/salt/minion
.
Ensure you change the log level on all cluster (minion) nodes.
Make sure that the fully qualified domain name of each node can be resolved to an IP address on the public cluster network by all the other nodes.
Configure all minions to connect to the master.
If your Salt Master is not reachable by the host name salt
, edit the file /etc/salt/minion
or create a new file /etc/salt/minion.d/master.conf
with the following content:
master: host_name_of_salt_master
If you performed any changes to the configuration files mentioned above, restart the Salt service on all related Salt Minions:
root@minion >
systemctl restart salt-minion.service
Check that the salt-minion
service is enabled and started on all nodes.
Enable and start it if needed:
root #
systemctl enable salt-minion.serviceroot #
systemctl start salt-minion.service
Verify each Salt Minion's fingerprint and accept all salt keys on the Salt Master if the fingerprints match.
If the Salt Minion fingerprint comes back empty, make sure the Salt Minion has a Salt Master configuration and that it can communicate with the Salt Master.
View each minion's fingerprint:
root@minion >
salt-call --local key.finger
local:
3f:a3:2f:3f:b4:d3:d9:24:49:ca:6b:2c:e1:6c:3f:c3:83:37:f0:aa:87:42:e8:ff...
After gathering fingerprints of all the Salt Minions, list fingerprints of all unaccepted minion keys on the Salt Master:
root@master #
salt-key -F
[...]
Unaccepted Keys:
minion1:
3f:a3:2f:3f:b4:d3:d9:24:49:ca:6b:2c:e1:6c:3f:c3:83:37:f0:aa:87:42:e8:ff...
If the minions' fingerprints match, accept them:
root@master #
salt-key --accept-all
Verify that the keys have been accepted:
root@master #
salt-key --list-all
Test whether all Salt Minions respond:
root@master #
salt-run manage.status
This section guides you through the process of deploying a basic Ceph cluster. Read the following subsections carefully and execute the included commands in the given order.
ceph-salt
#Edit source
ceph-salt
provides tools for deploying Ceph clusters managed by cephadm. ceph-salt
uses the Salt infrastructure to perform OS management—for example, software updates or time synchronization—and defining roles for Salt Minions.
On the Salt Master, install the ceph-salt package:
root@master #
zypper install ceph-salt
The above command installed ceph-salt-formula as a dependency which modified the Salt Master configuration by inserting additional files in the /etc/salt/master.d
directory.
To apply the changes, restart salt-master.service
and synchronize Salt modules:
root@master #
systemctl restart salt-master.serviceroot@master #
salt \* saltutil.sync_all
Use the ceph-salt config
command to configure the basic properties of the cluster.
The /etc/ceph/ceph.conf
file is managed by cephadm and users should not edit it. Ceph configuration parameters should be set using the new ceph config
command.
See Section 28.2, “Configuration database” for more information.
ceph-salt
shell #Edit source
If you run ceph-salt config
without any path or subcommand, you will enter an interactive ceph-salt
shell.
The shell is convenient if you need to configure multiple properties in one batch and do not want type the full command syntax.
root@master #
ceph-salt config/>
ls o- / ............................................................... [...] o- ceph_cluster .................................................. [...] | o- minions .............................................. [no minions] | o- roles ....................................................... [...] | o- admin .............................................. [no minions] | o- bootstrap ........................................... [no minion] | o- cephadm ............................................ [no minions] | o- tuned ..................................................... [...] | o- latency .......................................... [no minions] | o- throughput ....................................... [no minions] o- cephadm_bootstrap ............................................. [...] | o- advanced .................................................... [...] | o- ceph_conf ................................................... [...] | o- ceph_image_path .................................. [ no image path] | o- dashboard ................................................... [...] | | o- force_password_update ................................. [enabled] | | o- password ................................................ [admin] | | o- ssl_certificate ....................................... [not set] | | o- ssl_certificate_key ................................... [not set] | | o- username ................................................ [admin] | o- mon_ip .................................................. [not set] o- containers .................................................... [...] | o- registries_conf ......................................... [enabled] | | o- registries .............................................. [empty] | o- registry_auth ............................................... [...] | o- password .............................................. [not set] | o- registry .............................................. [not set] | o- username .............................................. [not set] o- ssh ............................................... [no key pair set] | o- private_key .................................. [no private key set] | o- public_key .................................... [no public key set] o- time_server ........................... [enabled, no server host set] o- external_servers .......................................... [empty] o- servers ................................................... [empty] o- subnet .................................................. [not set]
As you can see from the output of ceph-salt
's ls
command, the cluster configuration is organized in a tree structure.
To configure a specific property of the cluster in the ceph-salt
shell, you have two options:
Run the command from the current position and enter the absolute path to the property as the first argument:
/>
/cephadm_bootstrap/dashboard ls o- dashboard ....................................................... [...] o- force_password_update ..................................... [enabled] o- password .................................................... [admin] o- ssl_certificate ........................................... [not set] o- ssl_certificate_key ....................................... [not set] o- username .................................................... [admin]/> /cephadm_bootstrap/dashboard/username set ceph-admin
Value set.
Change to the path whose property you need to configure and run the command:
/>
cd /cephadm_bootstrap/dashboard//ceph_cluster/minions>
ls o- dashboard ....................................................... [...] o- force_password_update ..................................... [enabled] o- password .................................................... [admin] o- ssl_certificate ........................................... [not set] o- ssl_certificate_key ....................................... [not set] o- username ................................................[ceph-admin]
While in a ceph-salt
shell, you can use the autocompletion feature similar to a normal Linux shell (Bash) autocompletion.
It completes configuration paths, subcommands, or Salt Minion names.
When autocompleting a configuration path, you have two options:
To let the shell finish a path relative to your current position,press the TAB key →| twice.
To let the shell finish an absolute path, enter / and press the TAB key →| twice.
If you enter cd
from the ceph-salt
shell without any path, the command will print a tree structure of the cluster configuration with the line of the current path active.
You can use the up and down cursor keys to navigate through individual lines.
After you confirm with Enter, the configuration path will change to the last active one.
To keep the documentation consistent, we will use a single command syntax without entering the ceph-salt
shell.
For example, you can list the cluster configuration tree by using the following command:
root@master #
ceph-salt config ls
Include all or a subset of Salt Minions that we deployed and accepted in Section 5.2, “Deploying Salt” to the Ceph cluster configuration.
You can either specify the Salt Minions by their full names, or use a glob expressions '*' and '?' to include multiple Salt Minions at once.
Use the add
subcommand under the /ceph_cluster/minions
path.
The following command includes all accepted Salt Minions:
root@master #
ceph-salt config /ceph_cluster/minions add '*'
Verify that the specified Salt Minions were added:
root@master #
ceph-salt config /ceph_cluster/minions ls
o- minions ................................................. [Minions: 5]
o- ses-master.example.com .................................. [no roles]
o- ses-min1.example.com .................................... [no roles]
o- ses-min2.example.com .................................... [no roles]
o- ses-min3.example.com .................................... [no roles]
o- ses-min4.example.com .................................... [no roles]
Specify which nodes will belong to the Ceph cluster and will be managed by cephadm. Include all nodes that will run Ceph services as well as the Admin Node:
root@master #
ceph-salt config /ceph_cluster/roles/cephadm add '*'
The Admin Node is the node where the ceph.conf
configuration file and the Ceph admin keyring is installed.
You usually run Ceph related commands on the Admin Node.
In a homogeneous environment where all or most hosts belong to SUSE Enterprise Storage, we recommend having the Admin Node on the same host as the Salt Master.
In a heterogeneous environment where one Salt infrastructure hosts more than one cluster, for example, SUSE Enterprise Storage together with SUSE Manager, do not place the Admin Node on the same host as Salt Master.
To specify the Admin Node, run the following command:
root@master #
ceph-salt config /ceph_cluster/roles/admin add ses-master.example.com 1 minion added.root@master #
ceph-salt config /ceph_cluster/roles/admin ls o- admin ................................................... [Minions: 1] o- ses-master.example.com ...................... [Other roles: cephadm]
ceph.conf
and the admin keyring on multiple nodesYou can install the Ceph configuration file and admin keyring on multiple nodes if your deployment requires it. For security reasons, avoid installing them on all the cluster's nodes.
You need to specify which of the cluster's Salt Minions will bootstrap the cluster. This minion will become the first one running Ceph Monitor and Ceph Manager services.
root@master #
ceph-salt config /ceph_cluster/roles/bootstrap set ses-min1.example.com Value set.root@master #
ceph-salt config /ceph_cluster/roles/bootstrap ls o- bootstrap ..................................... [ses-min1.example.com]
Additionally, you need to specify the bootstrap MON's IP address on the public network to ensure that the public_network
parameter is set correctly, for example:
root@master #
ceph-salt config /cephadm_bootstrap/mon_ip set 192.168.10.20
You need to specify which of the cluster's minions have actively tuned profiles. To do so, add these roles explicitly with the following commands:
One minion cannot have both the latency
and throughput
roles.
root@master #
ceph-salt config /ceph_cluster/roles/tuned/latency add ses-min1.example.com Adding ses-min1.example.com... 1 minion added.root@master #
ceph-salt config /ceph_cluster/roles/tuned/throughput add ses-min2.example.com Adding ses-min2.example.com... 1 minion added.
cephadm uses the SSH protocol to communicate with cluster nodes.
A user account named cephadm
is automatically created and used for SSH communication.
You need to generate the private and public part of the SSH key pair:
root@master #
ceph-salt config /ssh generate Key pair generated.root@master #
ceph-salt config /ssh ls o- ssh .................................................. [Key Pair set] o- private_key ..... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83] o- public_key ...... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
All cluster nodes need to have their time synchronized with a reliable time source. There are several scenarios to approach time synchronization:
If all cluster nodes are already configured to synchronize their time using an NTP service of choice, disable time server handling completely:
root@master #
ceph-salt config /time_server disable
If your site already has a single source of time, specify the host name of the time source:
root@master #
ceph-salt config /time_server/servers add time-server.example.com
Alternatively, ceph-salt
has the ability to configure one of the Salt Minion to serve as the time server for the rest of the cluster.
This is sometimes referred to as an "internal time server".
In this scenario, ceph-salt
will configure the internal time server (which should be one of the Salt Minion) to synchronize its time with an external time server, such as pool.ntp.org
, and configure all the other minions to get their time from the internal time server.
This can be achieved as follows:
root@master #
ceph-salt config /time_server/servers add ses-master.example.comroot@master #
ceph-salt config /time_server/external_servers add pool.ntp.org
The /time_server/subnet
option specifies the subnet from which NTP clients are allowed to access the NTP server.
It is automatically set when you specify /time_server/servers
.
If you need to change it or specify it manually, run:
root@master #
ceph-salt config /time_server/subnet set 10.20.6.0/24
Check the time server settings:
root@master #
ceph-salt config /time_server ls
o- time_server ................................................ [enabled]
o- external_servers ............................................... [1]
| o- pool.ntp.org ............................................... [...]
o- servers ........................................................ [1]
| o- ses-master.example.com ..................................... [...]
o- subnet .............................................. [10.20.6.0/24]
Find more information on setting up time synchronization in https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-ntp.html#sec-ntp-yast.
Ceph Dashboard will be available after the basic cluster is deployed. To access it, you need to set a valid user name and password, for example:
root@master #
ceph-salt config /cephadm_bootstrap/dashboard/username set adminroot@master #
ceph-salt config /cephadm_bootstrap/dashboard/password set PWD
By default, the first dashboard user will be forced to change their password on first login to the dashboard. To disable this feature, run the following command:
root@master #
ceph-salt config /cephadm_bootstrap/dashboard/force_password_update disable
cephadm needs to know a valid URI path to container images that will be used during the deployment step. Verify whether the default path is set:
root@master #
ceph-salt config /cephadm_bootstrap/ceph_image_path ls
If there is no default path set or your deployment requires a specific path, add it as follows:
root@master #
ceph-salt config /cephadm_bootstrap/ceph_image_path set registry.suse.com/ses/7/ceph/ceph
For the monitoring stack, there are more container images needed. For an air-gapped deployment as well as deploying from a local registry, you may want to get those images at this point to prepare your local registry accordingly.
Note that those container images will not be used by ceph-salt
for the deployment.
It is a preparation for a later step where cephadm will be used for the deployment or migration of monitoring components.
For more information about the images used by the monitoring stack and how to customize them, visit the Section 16.1, “Configuring custom or local images” page.
Optionally, you can set a local container registry.
This will serve as a mirror of the registry.suse.com
registry.
Remember you need to re-sync the local registry whenever there are new updated containers available from registry.suse.com
.
Creating a local registry is useful in the following scenarios:
You have a lot of cluster nodes and want to save download time and bandwidth by creating a local mirror of container images.
Your cluster has no access to the online registry (an air-gapped deployment) and you need a local mirror to pull the container images from.
If configuration or network issues prevent your cluster from accessing remote registries across a secure link, so you need a local, unencrypted registry instead.
In order to deploy Program Temporary Fixes (PTFs) on a supported system, you need to deploy a local container registry.
To configure a local registry URL together with access credentials, do the following:
Configure the URL of the local registry:
cephuser@adm >
ceph-salt config /containers/registry_auth/registry set REGISTRY_URL
Configure the user name and password to access the local registry:
cephuser@adm >
ceph-salt config /containers/registry_auth/username set REGISTRY_USERNAME
cephuser@adm >
ceph-salt config /containers/registry_auth/password set REGISTRY_PASSWORD
Run ceph-salt apply
to update the Salt pillar on all minions.
To avoid re-syncing the local registry when new updated containers appear, you can configure a registry cache.
Cloud Native Application Development and Delivery methods require a registry and a CI/CD (Continuous Integration/Delivery) instance for the development and production of container images. You can use a private registry in this instance.
The Messenger v2 protocol (MSGR2) is Ceph's on-wire protocol. It provides a security mode that encrypts all data passing over the network, encapsulation of authentication payloads, and the enabling of future integration of new authentication modes (such as Kerberos).
msgr2 is not currently supported by Linux kernel Ceph clients, such as CephFS and RADOS Block Device.
Ceph daemons can bind to multiple ports, allowing both legacy Ceph clients and new v2-capable clients to connect to the same cluster. By default, MONs now bind to the new IANA-assigned port 3300 (CE4h or 0xCE4) for the new v2 protocol, while also binding to the old default port 6789 for the legacy v1 protocol.
The v2 protocol (MSGR2) supports two connection modes:
A strong initial authentication when the connection is established and a CRC32C integrity check.
A strong initial authentication when the connection is established and full encryption of all post-authentication traffic, including a cryptographic integrity check.
For most connections, there are options that control which modes are used:
The connection mode (or permitted modes) used for intra-cluster communication between Ceph daemons. If multiple modes are listed, the modes listed first are preferred.
A list of permitted modes for clients to use when connecting to the cluster.
A list of connection modes, in order of preference, for clients to use (or allow) when talking to a Ceph cluster.
There are a parallel set of options that apply specifically to monitors, allowing administrators to set different (usually more secure) requirements on communication with the monitors.
The connection mode (or permitted modes) to use between monitors.
A list of permitted modes for clients or other Ceph daemons to use when connecting to monitors.
A list of connection modes, in order of preference, for clients or non-monitor daemons to use when connecting to monitors.
In order to enable MSGR2 encryption mode during the deployment, you need to add some configuration options to the ceph-salt
configuration before running ceph-salt apply
.
To use secure
mode, run the following commands.
Add the global section to ceph_conf
in the ceph-salt
configuration tool:
root@master #
ceph-salt config /cephadm_bootstrap/ceph_conf add global
Set the following options:
root@master #
ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_cluster_mode "secure crc"root@master #
ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_service_mode "secure crc"root@master #
ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_client_mode "secure crc"
Ensure secure
precedes crc
.
To force secure
mode, run the following commands:
root@master #
ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_cluster_mode secureroot@master #
ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_service_mode secureroot@master #
ceph-salt config /cephadm_bootstrap/ceph_conf/global set ms_client_mode secure
If you want to change any of the above settings, set the configuration changes in the monitor configuration store.
This is achieved using the ceph config set
command.
root@master #
ceph config set global CONNECTION_OPTION CONNECTION_MODE [--force]
For example:
root@master #
ceph config set global ms_cluster_mode "secure crc"
If you want to check the current value, including default value, run the following command:
root@master #
ceph config get CEPH_COMPONENT CONNECTION_OPTION
For example, to get the ms_cluster_mode
for OSD's, run:
root@master #
ceph config get osd ms_cluster_mode
Optionally, if you are running a separate cluster network, you may need to set the cluster network IP address followed by the subnet mask part after the slash sign, for example 192.168.10.22/24
.
Run the following commands to enable cluster_network
:
root@master #
ceph-salt config /cephadm_bootstrap/ceph_conf add globalroot@master #
ceph-salt config /cephadm_bootstrap/ceph_conf/global set cluster_network NETWORK_ADDR
The minimal cluster configuration is finished. Inspect it for obvious errors:
root@master #
ceph-salt config ls
o- / ............................................................... [...]
o- ceph_cluster .................................................. [...]
| o- minions .............................................. [Minions: 5]
| | o- ses-master.example.com .................................. [admin]
| | o- ses-min1.example.com ......................... [bootstrap, admin]
| | o- ses-min2.example.com ................................. [no roles]
| | o- ses-min3.example.com ................................. [no roles]
| | o- ses-min4.example.com ................................. [no roles]
| o- roles ....................................................... [...]
| o- admin .............................................. [Minions: 2]
| | o- ses-master.example.com ....................... [no other roles]
| | o- ses-min1.example.com ................. [other roles: bootstrap]
| o- bootstrap ................................ [ses-min1.example.com]
| o- cephadm ............................................ [Minions: 5]
| o- tuned ..................................................... [...]
| o- latency .......................................... [no minions]
| o- throughput ....................................... [no minions]
o- cephadm_bootstrap ............................................. [...]
| o- advanced .................................................... [...]
| o- ceph_conf ................................................... [...]
| o- ceph_image_path ............... [registry.suse.com/ses/7/ceph/ceph]
| o- dashboard ................................................... [...]
| o- force_password_update ................................. [enabled]
| o- password ................................... [randomly generated]
| o- username ................................................ [admin]
| o- mon_ip ............................................ [192.168.10.20]
o- containers .................................................... [...]
| o- registries_conf ......................................... [enabled]
| | o- registries .............................................. [empty]
| o- registry_auth ............................................... [...]
| o- password .............................................. [not set]
| o- registry .............................................. [not set]
| o- username .............................................. [not set]
o- ssh .................................................. [Key Pair set]
| o- private_key ..... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
| o- public_key ...... [53:b1:eb:65:d2:3a:ff:51:6c:e2:1b:ca:84:8e:0e:83]
o- time_server ............................................... [enabled]
o- external_servers .............................................. [1]
| o- 0.pt.pool.ntp.org ......................................... [...]
o- servers ....................................................... [1]
| o- ses-master.example.com .................................... [...]
o- subnet ............................................. [10.20.6.0/24]
You can check if the configuration of the cluster is valid by running the following command:
root@master #
ceph-salt status
cluster: 5 minions, 0 hosts managed by cephadm
config: OK
After you have configured the basic cluster and its configuration is valid, it is a good idea to export its configuration to a file:
root@master #
ceph-salt export > cluster.json
The output of the ceph-salt export
includes the SSH private key.
If you are concerned about the security implications, do not execute this command without taking appropriate precautions.
In case you break the cluster configuration and need to revert to a backup state, run:
root@master #
ceph-salt import cluster.json
Before you deploy the cluster, update all software packages on all nodes:
root@master #
ceph-salt update
If a node reports Reboot is needed
during the update, important OS packages—such as the kernel—were updated to a newer version and you need to reboot the node to apply the changes.
To reboot all nodes that require rebooting, either append the --reboot
option
root@master #
ceph-salt update --reboot
Or, reboot them in a separate step:
root@master #
ceph-salt reboot
The Salt Master is never rebooted by ceph-salt update --reboot
or ceph-salt reboot
commands.
If the Salt Master needs rebooting, you need to reboot it manually.
After the nodes are updated, bootstrap the minimal cluster:
root@master #
ceph-salt apply
When bootstrapping is complete, the cluster will have one Ceph Monitor and one Ceph Manager.
The above command will open an interactive user interface that shows the current progress of each minion.
If you need to apply the configuration from a script, there is also a non-interactive mode of deployment. This is also useful when deploying the cluster from a remote machine because constant updating of the progress information on the screen over the network may become distracting:
root@master #
ceph-salt apply --non-interactive
After the ceph-salt apply
command has completed, you should have one Ceph Monitor and one Ceph Manager.
You should be able to run the ceph status
command successfully on any of the minions that were given the admin
role as root
or the cephadm
user using sudo
.
The next steps involve using the cephadm to deploy additional Ceph Monitor, Ceph Manager, OSDs, the Monitoring Stack, and Gateways.
Before you continue, review your new cluster's network settings.
At this point, the public_network
setting has been populated based on what was entered for /cephadm_bootstrap/mon_ip
in the ceph-salt
configuration.
However, this setting was only applied to Ceph Monitor.
You can review this setting with the following command:
root@master #
ceph config get mon public_network
This is the minimum that Ceph requires to work, but we recommend making this public_network
setting global
, which means it will apply to all types of Ceph daemons, and not only to MONs:
root@master #
ceph config set global public_network "$(ceph config get mon public_network)"
This step is not required. However, if you do not use this setting, the Ceph OSDs and other daemons (except Ceph Monitor) will listen on all addresses.
If you want your OSDs to communicate amongst themselves using a completely separate network, run the following command:
root@master #
ceph config set global cluster_network "cluster_network_in_cidr_notation"
Executing this command will ensure that the OSDs created in your deployment will use the intended cluster network from the start.
If your cluster is set to have dense nodes (greater than 62 OSDs per host), make sure to assign sufficient ports for Ceph OSDs.
The default range (6800-7300) currently allows for no more than 62 OSDs per host.
For a cluster with dense nodes, adjust the setting ms_bind_port_max
to a suitable value.
Each
OSD will consume eight additional ports.
For example, given a host that is set to run 96 OSDs, 768 ports will be needed. ms_bind_port_max
should be set at least to 7568 by running the following command:
root@master #
ceph config set osd.* ms_bind_port_max 7568
You will need to adjust your firewall settings accordingly for this to work. See Section 13.7, “Firewall settings for Ceph” for more information.
After deploying the basic Ceph cluster, deploy core services to more cluster nodes. To make the cluster data accessible to clients, deploy additional services as well.
Currently, we support deployment of Ceph services on the command line by using the Ceph orchestrator (ceph orch
subcommands).
ceph orch
command #Edit source
The Ceph orchestrator command ceph orch
—which is an interface to the cephadm module—will take care of listing cluster components and deploying Ceph services on new cluster nodes.
The following command shows the current mode and status of the Ceph orchestrator.
cephuser@adm >
ceph orch status
To list all disk devices, run the following:
cephuser@adm >
ceph orch device ls
Hostname Path Type Serial Size Health Ident Fault Available
ses-master /dev/vdb hdd 0d8a... 10.7G Unknown N/A N/A No
ses-min1 /dev/vdc hdd 8304... 10.7G Unknown N/A N/A No
ses-min1 /dev/vdd hdd 7b81... 10.7G Unknown N/A N/A No
[...]
Service is a general term for a Ceph service of a specific type, for example Ceph Manager.
Daemon is a specific instance of a service, for example a process mgr.ses-min1.gdlcik
running on a node called ses-min1
.
To list all services known to cephadm, run:
cephuser@adm >
ceph orch ls
NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID
mgr 1/0 5m ago - <no spec> registry.example.com/[...] 5bf12403d0bd
mon 1/0 5m ago - <no spec> registry.example.com/[...] 5bf12403d0bd
You can limit the list to services on a particular node with the optional -–host
parameter and services of a particular type with the optional --service-type
parameter (acceptable types are mon
, osd
, mgr
, mds
, and rgw
).
To list all running daemons deployed by cephadm, run:
cephuser@adm >
ceph orch ps
NAME HOST STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID
mgr.ses-min1.gd ses-min1 running) 8m ago 12d 15.2.0.108 5bf12403d0bd b8104e09814c
mon.ses-min1 ses-min1 running) 8m ago 12d 15.2.0.108 5bf12403d0bd a719e0087369
To query the status of a particular daemon, use --daemon_type
and --daemon_id
.
For
OSDs, the ID is the numeric OSD ID.
For
MDS, the ID is the file system name:
cephuser@adm >
ceph orch ps --daemon_type osd --daemon_id 0cephuser@adm >
ceph orch ps --daemon_type mds --daemon_id my_cephfs
The recommended way to specify the deployment of Ceph services is to create a YAML-formatted file with the specification of the services that you intend to deploy.
You can create a separate specification file for each type of service, for example:
root@master #
cat nfs.yml
service_type: nfs
service_id: EXAMPLE_NFS
placement:
hosts:
- ses-min1
- ses-min2
spec:
pool: EXAMPLE_POOL
namespace: EXAMPLE_NAMESPACE
Alternatively, you can specify multiple (or all) service types in one file—for example, cluster.yml
—that describes which nodes will run specific services.
Remember to separate individual service types with three dashes (---
):
cephuser@adm >
cat cluster.yml
service_type: nfs
service_id: EXAMPLE_NFS
placement:
hosts:
- ses-min1
- ses-min2
spec:
pool: EXAMPLE_POOL
namespace: EXAMPLE_NAMESPACE
---
service_type: rgw
service_id: REALM_NAME.ZONE_NAME
placement:
hosts:
- ses-min1
- ses-min2
- ses-min3
---
[...]
The aforementioned properties have the following meaning:
service_type
The type of the service.
It can be either a Ceph service (mon
, mgr
, mds
, crash
, osd
, or rbd-mirror
), a gateway (nfs
or rgw
), or part of the monitoring stack (alertmanager
, grafana
, node-exporter
, or prometheus
).
service_id
The name of the service.
Specifications of type mon
, mgr
, alertmanager
, grafana
, node-exporter
, and prometheus
do not require the service_id
property.
placement
Specifies which nodes will be running the service. Refer to Section 5.4.2.2, “Creating placement specification” for more details.
spec
Additional specification relevant for the service type.
Ceph cluster services have usually a number of properties specific to them. For examples and details of individual services' specification, refer to Section 5.4.3, “Deploy Ceph services”.
To deploy Ceph services, cephadm needs to know on which nodes to deploy them.
Use the placement
property and list the short host names of the nodes that the service applies to:
cephuser@adm >
cat cluster.yml
[...]
placement:
hosts:
- host1
- host2
- host3
[...]
After you have created a full cluster.yml
file with specifications of all services and their placement, you can apply the cluster by running the following command:
cephuser@adm >
ceph orch apply -i cluster.yml
To view the status of the cluster, run the ceph orch status
command.
For more details, see Section 5.4.1.1, “Displaying the orchestrator status”.
Although you deployed services to the Ceph cluster by using the specification files as described in Section 5.4.2, “Service and placement specification”, the configuration of the cluster may diverge from the original specification during its operation. Also, you may have removed the specification files accidentally.
To retrieve a complete specification of a running cluster, run:
cephuser@adm >
ceph orch ls --export
placement:
hosts:
- hostname: ses-min1
name: ''
network: ''
service_id: my_cephfs
service_name: mds.my_cephfs
service_type: mds
---
placement:
count: 2
service_name: mgr
service_type: mgr
---
[...]
You can append the --format
option to change the default yaml
output format.
You can select from json
, json-pretty
, or yaml
.
For example:
ceph orch ls --export --format json
After the basic cluster is running, you can deploy Ceph services to additional nodes.
Ceph cluster has three or five MONs deployed across different nodes. If there are five or more nodes in the cluster, we recommend deploying five MONs. A good practice is to have MGRs deployed on the same nodes as MONs.
When deploying MONs and MGRs, remember to include the first MON that you added when configuring the basic cluster in Section 5.3.2.5, “Specifying first MON/MGR node”.
To deploy MONs, apply the following specification:
service_type: mon placement: hosts: - ses-min1 - ses-min2 - ses-min3
If you need to add another node, append the host name to the same YAML list. For example:
service_type: mon placement: hosts: - ses-min1 - ses-min2 - ses-min3 - ses-min4
Similarly, to deploy MGRs, apply the following specification:
Ensure your deployment has at least three Ceph Managers in each deployment.
service_type: mgr placement: hosts: - ses-min1 - ses-min2 - ses-min3
If MONs or MGRs are not on the same subnet, you need to append the subnet addresses. For example:
service_type: mon placement: hosts: - ses-min1:10.1.2.0/24 - ses-min2:10.1.5.0/24 - ses-min3:10.1.10.0/24
A storage device is considered available if all of the following conditions are met:
The device has no partitions.
The device does not have any LVM state.
The device is not be mounted.
The device does not contain a file system.
The device does not contain a BlueStore OSD.
The device is larger than 5 GB.
If the above conditions are not met, Ceph refuses to provision such OSDs.
There are two ways you can deploy OSDs:
Tell Ceph to consume all available and unused storage devices:
cephuser@adm >
ceph orch apply osd --all-available-devices
Use DriveGroups (see Section 13.4.3, “Adding OSDs using DriveGroups specification”) to create OSD specification describing devices that will be deployed based on their properties, such as device type (SSD or HDD), device model names, size, or the nodes on which the devices exist. Then apply the specification by running the following command:
cephuser@adm >
ceph orch apply osd -i drive_groups.yml
CephFS requires one or more Metadata Server (MDS) services. To create a CephFS, first create MDS servers by applying the following specification:
Ensure you have at least two pools, one for CephFS data and one for CephFS metadata, created before applying the following specification.
service_type: mds service_id: CEPHFS_NAME placement: hosts: - ses-min1 - ses-min2 - ses-min3
After MDSs are functional, create the CephFS:
ceph fs new CEPHFS_NAME metadata_pool data_pool
cephadm deploys an Object Gateway as a collection of daemons that manage a particular realm and zone.
You can either relate an Object Gateway service to already existing realm and zone, (refer to Section 21.13, “Multisite Object Gateways” for more details), or you can specify a non-existing REALM_NAME and ZONE_NAME and they will be created automatically after you apply the following configuration:
service_type: rgw service_id: REALM_NAME.ZONE_NAME placement: hosts: - ses-min1 - ses-min2 - ses-min3 spec: rgw_realm: RGW_REALM rgw_zone: RGW_ZONE
To use a secure SSL connection to the Object Gateway, you need a pair of valid SSL certificate and key files (see Section 21.7, “Enable HTTPS/SSL for Object Gateways” for more details). You need to enable SSL, specify a port number for SSL connections, and the SSL certificate and key files.
To enable SSL and specify the port number, include the following in your specification:
spec: ssl: true rgw_frontend_port: 443
To specify the SSL certificate and key, you can paste their contents directly into the YAML specification file.
The pipe sign (|
) at the end of line tells the parser to expect a multi-line string as a value.
For example:
spec: ssl: true rgw_frontend_port: 443 rgw_frontend_ssl_certificate: | -----BEGIN CERTIFICATE----- MIIFmjCCA4KgAwIBAgIJAIZ2n35bmwXTMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV BAYTAkFVMQwwCgYDVQQIDANOU1cxHTAbBgNVBAoMFEV4YW1wbGUgUkdXIFNTTCBp [...] -----END CERTIFICATE----- rgw_frontend_ssl_key: | -----BEGIN PRIVATE KEY----- MIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQDLtFwg6LLl2j4Z BDV+iL4AO7VZ9KbmWIt37Ml2W6y2YeKX3Qwf+3eBz7TVHR1dm6iPpCpqpQjXUsT9 [...] -----END PRIVATE KEY-----
Instead of pasting the content of SSL certificate and key files, you can omit the rgw_frontend_ssl_certificate:
and rgw_frontend_ssl_key:
keywords and upload them to the configuration database:
cephuser@adm >
ceph config-key set rgw/cert/REALM_NAME/ZONE_NAME.crt \ -i SSL_CERT_FILEcephuser@adm >
ceph config-key set rgw/cert/REALM_NAME/ZONE_NAME.key \ -i SSL_KEY_FILE
Subclusters help you organize the nodes in your clusters to isolate workloads and make elastic scaling easier. If you are deploying with a subcluster, apply the following configuration:
service_type: rgw service_id: REALM_NAME.ZONE_NAME.SUBCLUSTER placement: hosts: - ses-min1 - ses-min2 - ses-min3 spec: rgw_realm: RGW_REALM rgw_zone: RGW_ZONE subcluster: SUBCLUSTER
cephadm deploys an iSCSI Gateway which is a storage area network (SAN) protocol that allows clients (called initiators) to send SCSI commands to SCSI storage devices (targets) on remote servers.
Apply the following configuration to deploy.
Ensure trusted_ip_list
contains the IP addresses of all iSCSI Gateway and Ceph Manager nodes (see the example output below).
Ensure the pool is created before applying the following specification.
service_type: iscsi service_id: EXAMPLE_ISCSI placement: hosts: - ses-min1 - ses-min2 - ses-min3 spec: pool: EXAMPLE_POOL api_user: EXAMPLE_USER api_password: EXAMPLE_PASSWORD trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2"
Ensure the IPs listed for trusted_ip_list
do not have a space after the comma separation.
To use a secure SSL connection between the Ceph Dashboard and the iSCSI target API, you need a pair of valid SSL certificate and key files.
These can be either CA-issued or self-signed (see Section 10.1.1, “Creating self-signed certificates”).
To enable SSL, include the api_secure: true
setting in your specification file:
spec: api_secure: true
To specify the SSL certificate and key, you can paste the content directly into the YAML specification file.
The pipe sign (|
) at the end of line tells the parser to expect a multi-line string as a value.
For example:
spec: pool: EXAMPLE_POOL api_user: EXAMPLE_USER api_password: EXAMPLE_PASSWORD trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2" api_secure: true ssl_cert: | -----BEGIN CERTIFICATE----- MIIDtTCCAp2gAwIBAgIYMC4xNzc1NDQxNjEzMzc2MjMyXzxvQ7EcMA0GCSqGSIb3 DQEBCwUAMG0xCzAJBgNVBAYTAlVTMQ0wCwYDVQQIDARVdGFoMRcwFQYDVQQHDA5T [...] -----END CERTIFICATE----- ssl_key: | -----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC5jdYbjtNTAKW4 /CwQr/7wOiLGzVxChn3mmCIF3DwbL/qvTFTX2d8bDf6LjGwLYloXHscRfxszX/4h [...] -----END PRIVATE KEY-----
NFS Ganesha supports NFS version 4.1 and newer. It does not support NFS version 3.
cephadm deploys NFS Ganesha using a pre-defined RADOS pool and an optional name-space. To deploy NFS Ganesha, apply the following specification:
You need to have a pre-defined RADOS pool otherwise the ceph orch apply
operation will fail.
For more information on creating a pool, see Section 18.1, “Creating a pool”.
service_type: nfs service_id: EXAMPLE_NFS placement: hosts: - ses-min1 - ses-min2 spec: pool: EXAMPLE_POOL namespace: EXAMPLE_NAMESPACE
EXAMPLE_NFS with an arbitrary string that identifies the NFS export.
EXAMPLE_POOL with the name of the pool where the NFS Ganesha RADOS configuration object will be stored.
EXAMPLE_NAMESPACE (optional) with the desired Object Gateway NFS namespace (for example, ganesha
).
rbd-mirror
#Edit source
The rbd-mirror
service takes care of synchronizing RADOS Block Device images between two Ceph clusters (for more details, see Section 20.4, “RBD image mirrors”).
To deploy rbd-mirror
, use the following specification:
service_type: rbd-mirror service_id: EXAMPLE_RBD_MIRROR placement: hosts: - ses-min3
The monitoring stack consists of Prometheus, Prometheus exporters, Prometheus Alertmanager, and Grafana. Ceph Dashboard makes use of these components to store and visualize detailed metrics on cluster usage and performance.
If your deployment requires custom or locally served container images of the monitoring stack services, refer to Section 16.1, “Configuring custom or local images”.
To deploy the monitoring stack, follow these steps:
Enable the prometheus
module in the Ceph Manager daemon.
This exposes the internal Ceph metrics so that Prometheus can read them:
cephuser@adm >
ceph mgr module enable prometheus
Ensure this command is run before Prometheus is deployed. If the command was not run before the deployment, you must redeploy Prometheus to update Prometheus' configuration:
cephuser@adm >
ceph orch redeploy prometheus
Create a specification file (for example monitoring.yaml
) with a content similar to the following:
service_type: prometheus placement: hosts: - ses-min2 --- service_type: node-exporter --- service_type: alertmanager placement: hosts: - ses-min4 --- service_type: grafana placement: hosts: - ses-min3
Apply monitoring services by running:
cephuser@adm >
ceph orch apply -i monitoring.yaml
It may take a minute or two for the monitoring services to be deployed.
Prometheus, Grafana, and the Ceph Dashboard are all automatically configured to talk to each other, resulting in a fully functional Grafana integration in the Ceph Dashboard when deployed as described above.
The only exception to this rule is monitoring with RBD images. See Section 16.5.4, “Enabling RBD-image monitoring” for more information.