To clean up all the artifacts created by the file system demo:

kubectl@adm > kubectl delete -f kube-registry.yaml

To delete the file system components and backing data, delete the Filesystem CRD.

Note

WARNING: Data will be deleted if preservePoolsOnDelete=false.

kubectl@adm > kubectl -n rook-ceph delete cephfilesystem myfs

Note: If the “preservePoolsOnDelete” file system attribute is set to true, the above command won’t delete the pools. Creating the file system again with the same CRD will reuse the previous pools.

If these settings are changed in the CRD, the operator will update the number of MONs during a periodic check of the MON health, which by default is every 45 seconds.

To change the defaults that the operator uses to determine the MON health and whether to failover a MON, refer to the Section 6.1.3.15, “Health settings”. The intervals should be small enough that you have confidence the MONs will maintain quorum, while also being long enough to ignore network blips where MONs are failed over too often.

You can use the cluster CR to enable or disable any manager module. For example, this can be configured:

mgr:
  modules:
  - name: <name of the module>
    enabled: true

Some modules will have special configuration to ensure the module is fully functional after being enabled. Specifically, the pg_autoscaler—Rook will configure all new pools with PG autoscaling by setting: osd_pool_default_pg_autoscale_mode = on

If not specified, the default SDN will be used. Configure the network that will be enabled for the cluster and services.

Note

Changing networking configuration after a Ceph cluster has been deployed is not supported and will result in a non-functioning cluster.

To use host networking, set provider: host.

In addition to the cluster level settings specified above, each individual node can also specify configuration to override the cluster level settings and defaults. If a node does not specify any configuration, then it will inherit the cluster level settings.

When useAllNodes is set to true, Rook attempts to make Ceph cluster management as hands-off as possible while still maintaining reasonable data safety. If a usable node comes online, Rook will begin to use it automatically. To maintain a balance between hands-off usability and data safety, nodes are removed from Ceph as OSD hosts only (1) if the node is deleted from Kubernetes itself or (2) if the node has its taints or affinities modified in such a way that the node is no longer usable by Rook. Any changes to taints or affinities, intentional or unintentional, may affect the data reliability of the Ceph cluster. In order to help protect against this somewhat, deletion of nodes by taint or affinity modifications must be confirmed by deleting the Rook-Ceph operator pod and allowing the operator deployment to restart the pod.

For production clusters, we recommend that useAllNodes is set to false to prevent the Ceph cluster from suffering reduced data reliability unintentionally due to a user mistake. When useAllNodes is set to false, Rook relies on the user to be explicit about when nodes are added to or removed from the Ceph cluster. Nodes are only added to the Ceph cluster if the node is added to the Ceph cluster resource. Similarly, nodes are only removed if the node is removed from the Ceph cluster resource.

kubectl -n rook-ceph edit cephcluster rook-ceph

Below are the settings available, both at the cluster and individual node level, for selecting which storage resources will be included in the cluster.

The following are the settings for Storage Class Device Sets which can be configured to create OSDs that are backed by block mode PVs.

Ceph DriveGroups allow for specifying highly advanced OSD layouts. Refer to Book “Administration and Operations Guide”, Chapter 13 “Operational tasks”, Section 13.4.3 “Adding OSDs using DriveGroups specification” for both general information and detailed specification of DriveGroups with useful examples.

Important

When managing a Rook/Ceph cluster’s OSD layouts with DriveGroups, the storage configuration is mostly ignored. storageClassDeviceSets can still be used to create OSDs on PVC, but Rook will no longer use storage configurations for creating OSDs on a node's devices. To avoid confusion, we recommend using the storage configuration or DriveGroups, but never both. Because storage and DriveGroups should not be used simultaneously, Rook only supports provisioning OSDs with DriveGroups on new Rook-Ceph clusters.

DriveGroups are defined by a name, a Ceph DriveGroups spec, and a Rook placement.

Annotations and Labels can be specified so that the Rook components will have those annotations or labels added to them.

You can set annotations and labels for Rook components for the list of key value pairs:

When other keys are set, all will be merged together with the specific component.

Placement configuration for the cluster services. It includes the following keys: mgr, mon, osd, cleanup, and all. Each service will have its placement configuration generated by merging the generic configuration under all with the most specific one (which will override any attributes).

Note

Placement of OSD pods is controlled using the Section 6.1.3.8, “Storage class device sets”, not the general placement configuration.

A placement configuration is specified (according to the Kubernetes PodSpec) as:

If you use labelSelector for OSD pods, you must write two rules both for rook-ceph-osd and rook-ceph-osd-prepare.

The Rook Ceph operator creates a job called rook-ceph-detect-version to detect the full Ceph version used by the given cephVersion.image. The placement from the MON section is used for the job except for the PodAntiAffinity field.

Resources should be specified so that the Rook components are handled after Kubernetes Pod Quality of Service classes. This allows to keep Rook components running when for example a node runs out of memory and the Rook components are not killed depending on their Quality of Service class.

You can set resource requests/limits for Rook components through the Section 6.1.3.13, “Resource requirements and limits” structure in the following keys:

In order to provide the best possible experience running Ceph in containers, Rook internally recommends minimum memory limits if resource limits are passed. If a user configures a limit or request value that is too low, Rook will still run the pod(s) and print a warning to the operator log.

Priority class names can be specified so that the Rook components will have those priority class names added to them.

You can set priority class names for Rook components for the list of key value pairs:

The specific component keys will act as overrides to all.

Rook-Ceph will monitor the state of the CephCluster on various components by default. The following CRD settings are available:

Currently three health checks are implemented:

The liveness probe of each daemon can also be controlled via livenessProbe. The setting is valid for mon, mgr and osd. Here is a complete example for both daemonHealth and livenessProbe:

healthCheck:
  daemonHealth:
    mon:
      disabled: false
      interval: 45s
      timeout: 600s
    osd:
      disabled: false
      interval: 60s
    status:
      disabled: false
  livenessProbe:
    mon:
      disabled: false
    mgr:
      disabled: false
    osd:
      disabled: false

You can change the mgr probe by applying the following:

healthCheck:
  livenessProbe:
    mgr:
      disabled: false
      probe:
        httpGet:
          path: /
          port: 9283
        initialDelaySeconds: 3
        periodSeconds: 3

Changing the liveness probe is an advanced operation and should rarely be necessary. If you want to change these settings, start with the probe specification that Rook generates by default and then modify the desired settings.

Individual nodes and their configurations can be specified so that only the named nodes below will be used as storage resources. Each node’s “name” field should match their “kubernetes.io/hostname” label.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v15.2.4
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: true
  dashboard:
    enabled: true
  # cluster level storage configuration and selection
  storage:
    useAllNodes: false
    useAllDevices: false
    deviceFilter:
    config:
      metadataDevice:
      databaseSizeMB: "1024" # this value can be removed for environments with normal sized disks (100 GB or larger)
    nodes:
    - name: "172.17.4.201"
      devices:             # specific devices to use for storage can be specified for each node
      - name: "sdb" # Whole storage device
      - name: "sdc1" # One specific partition. Should not have a file system on it.
      - name: "/dev/disk/by-id/ata-ST4000DM004-XXXX" # both device name and explicit udev links are supported
      config:         # configuration can be specified at the node level which overrides the cluster level config
        storeType: bluestore
    - name: "172.17.4.301"
      deviceFilter: "^sd."

To control where various services will be scheduled by Kubernetes, use the placement configuration sections below. The example under “all” would have all services scheduled on Kubernetes nodes labeled with “role=storage-node” and tolerate taints with a key of “storage-node”.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v15.2.4
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: true
  # enable the Ceph dashboard for viewing cluster status
  dashboard:
    enabled: true
  placement:
    all:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: role
              operator: In
              values:
              - storage-node
      tolerations:
      - key: storage-node
        operator: Exists
    mgr:
      nodeAffinity:
      tolerations:
    mon:
      nodeAffinity:
      tolerations:
    osd:
      nodeAffinity:
      tolerations:

To control how many resources the Rook components can request/use, you can set requests and limits in Kubernetes for them. You can override these requests and limits for OSDs per node when using useAllNodes: false in the node item in the nodes list.

Warning

Before setting resource requests/limits, review the Ceph documentation for hardware recommendations for each component.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v15.2.4
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: true
  # enable the Ceph dashboard for viewing cluster status
  dashboard:
    enabled: true
  # cluster level resource requests/limits configuration
  resources:
  storage:
    useAllNodes: false
    nodes:
    - name: "172.17.4.201"
      resources:
        limits:
          cpu: "2"
          memory: "4096Mi"
        requests:
          cpu: "2"
          memory: "4096Mi"

The topology of the cluster is important in production environments where you want your data spread across failure domains. The topology can be controlled by adding labels to the nodes. When the labels are found on a node at first OSD deployment, Rook will add them to the desired level in the CRUSH map.

The complete list of labels in hierarchy order from highest to lowest is:

topology.kubernetes.io/region
topology.kubernetes.io/zone
topology.rook.io/datacenter
topology.rook.io/room
topology.rook.io/pod
topology.rook.io/pdu
topology.rook.io/row
topology.rook.io/rack
topology.rook.io/chassis

For example, if the following labels were added to a node:

kubectl label node mynode topology.kubernetes.io/zone=zone1
kubectl label node mynode topology.rook.io/rack=rack1

Note

For versions previous to K8s 1.17, use the topology key: failure-domain.beta.kubernetes.io/zone or region.

These labels would result in the following hierarchy for OSDs on that node (this command can be run in the Rook toolbox):

[root@mynode /]# ceph osd tree
ID CLASS WEIGHT  TYPE NAME                 STATUS REWEIGHT PRI-AFF
-1       0.01358 root default
-5       0.01358     zone zone1
-4       0.01358         rack rack1
-3       0.01358             host mynode
 0   hdd 0.00679                 osd.0         up  1.00000 1.00000
 1   hdd 0.00679                 osd.1         up  1.00000 1.00000

Ceph requires unique names at every level in the hierarchy (CRUSH map). For example, you cannot have two racks with the same name that are in different zones. Racks in different zones must be named uniquely.

Note that the host is added automatically to the hierarchy by Rook. The host cannot be specified with a topology label. All topology labels are optional.

Tip

When setting the node labels prior to CephCluster creation, these settings take immediate effect. However, applying this to an already deployed CephCluster requires removing each node from the cluster first and then re-adding it with new configuration to take effect. Do this node by node to keep your data safe! Check the result with ceph osd tree from the Chapter 8, Toolboxes. The OSD tree should display the hierarchy for the nodes that already have been re-added.

To utilize the failureDomain based on the node labels, specify the corresponding option in the CephBlockPool.

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: rack  # this matches the topology labels on nodes
  replicated:
    size: 3

This configuration will split the replication of volumes across unique racks in the data center setup.

In the CRD specification below three monitors are created each using a 10Gi PVC created by Rook using the local-storage storage class.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v15.2.4
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
    volumeClaimTemplate:
      spec:
        storageClassName: local-storage
        resources:
          requests:
            storage: 10Gi
  dashboard:
    enabled: true
  storage:
    useAllNodes: true
    useAllDevices: true
    deviceFilter:
    config:
      metadataDevice:
      databaseSizeMB: "1024" # this value can be removed for environments with normal sized disks (100 GB or larger)
      journalSizeMB: "1024"  # this value can be removed for environments with normal sized disks (20 GB or larger)
      osdsPerDevice: "1"

In the CRD specification below, three OSDs (having specific placement and resource values) and three MONs with each using a 10Gi PVC, are created by Rook using the local-storage storage class.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
    volumeClaimTemplate:
      spec:
        storageClassName: local-storage
        resources:
          requests:
            storage: 10Gi
  cephVersion:
    image: ceph/ceph:v15.2.4
    allowUnsupported: false
  dashboard:
    enabled: true
  network:
    hostNetwork: false
  storage:
    storageClassDeviceSets:
    - name: set1
      count: 3
      portable: false
      tuneDeviceClass: false
      resources:
        limits:
          cpu: "500m"
          memory: "4Gi"
        requests:
          cpu: "500m"
          memory: "4Gi"
      placement:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: "rook.io/cluster"
                  operator: In
                  values:
                    - cluster1
                topologyKey: "topology.kubernetes.io/zone"
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          resources:
            requests:
              storage: 10Gi
          storageClassName: local-storage
          volumeMode: Block
          accessModes:
            - ReadWriteOnce

In the simplest case, Ceph OSD BlueStore consumes a single (primary) storage device. BlueStore is the engine used by the OSD to store data.

The storage device is normally used as a whole, occupying the full device that is managed directly by BlueStore. It is also possible to deploy BlueStore across additional devices such as a DB device. This device can be used for storing BlueStore’s internal metadata. BlueStore (or rather, the embedded RocksDB) will put as much metadata as it can on the DB device to improve performance. If the DB device fills up, metadata will spill back onto the primary device (where it would have been otherwise). Again, it is only helpful to provision a DB device if it is faster than the primary device.

You can have multiple volumeClaimTemplates where each might either represent a device or a metadata device. So just taking the storage section this will give something like:

  storage:
   storageClassDeviceSets:
    - name: set1
      count: 3
      portable: false
      tuneDeviceClass: false
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          resources:
            requests:
              storage: 10Gi
          # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
          storageClassName: gp2
          volumeMode: Block
          accessModes:
            - ReadWriteOnce
      - metadata:
          name: metadata
        spec:
          resources:
            requests:
              # Find the right size https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing
              storage: 5Gi
          # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, io1)
          storageClassName: io1
          volumeMode: Block
          accessModes:
            - ReadWriteOnce

Note

Rook only supports three naming conventions for a given template:

• data: represents the main OSD block device, where your data is being stored.

• metadata: represents the metadata (including block.db and block.wal) device used to store the Ceph Bluestore database for an OSD.

• “wal”: represents the block.wal device used to store the Ceph BlueStore database for an OSD. If this device is set, “metadata” device will refer specifically to the block.db device. It is recommended to use a faster storage class for the metadata or wal device, with a slower device for the data. Otherwise, having a separate metadata device will not improve the performance.

The BlueStore partition has the following reference combinations supported by the ceph-volume utility:

With the present configuration, each OSD will have its main block allocated a 10 GB device as well a 5 GB device to act as a BlueStore database.

The minimum supported Ceph version for the External Cluster is Luminous 12.2.x.

The features available from the external cluster will vary depending on the version of Ceph. The following table shows the minimum version of Ceph for some of the features:

6.1.4.9.1 Prerequisites #

  

In order to configure an external Ceph cluster with Rook, we need to inject some information in order to connect to that cluster. You can use the cluster/examples/kubernetes/ceph/import-external-cluster.sh script to achieve that. The script will look for the following populated environment variables:

• NAMESPACE: the namespace where the configmap and secrets should be injected

• ROOK_EXTERNAL_FSID: the FSID of the external Ceph cluster. This can be retrieved via the ceph fsid command.

• ROOK_EXTERNAL_CEPH_MON_DATA: this is a comma- separated list of running monitors' IP addresses along with their ports. For example, a=172.17.0.4:6789,b=172.17.0.5:6789,c=172.17.0.6:6789. You do not need to specify all the monitors; you can simply pass one, and the operator will discover the rest. The name of the monitor is the name that appears in the ceph status output.

Now, we need to give Rook a key to connect to the cluster in order to perform various operations, such as cluster health checks, CSI keys management, etc. We recommend generating keys with minimal access, so the admin key does not need to be used by the external cluster. In this case, the admin key is only needed to generate the keys that will be used by the external cluster. If the admin key is to be used by the external cluster, however, set the following variable:

• ROOK_EXTERNAL_ADMIN_SECRET: OPTIONAL: the external Ceph cluster admin secret key. This can be retrieved via the ceph auth get-key client.admin command.

Note

WARNING: If you plan to create CRs (pool, rgw, mds, nfs) in the external cluster, you MUST inject the client.admin keyring as well as injecting cluster-external-management.yaml

Example:

export NAMESPACE=rook-ceph-external
export ROOK_EXTERNAL_FSID=3240b4aa-ddbc-42ee-98ba-4ea7b2a61514
export ROOK_EXTERNAL_CEPH_MON_DATA=a=172.17.0.4:6789
export ROOK_EXTERNAL_ADMIN_SECRET=AQC6Ylxdja+NDBAAB7qy9MEAr4VLLq4dCIvxtg==

If the Ceph admin key is not provided, the following script needs to be executed on a machine that can connect to the Ceph cluster using the Ceph admin key. On that machine, run cluster/examples/kubernetes/ceph/create-external-cluster-resources.sh. The script will automatically create users and keys with the lowest possible privileges and populate the necessary environment variables for cluster/examples/kubernetes/ceph/import-external-cluster.sh to work correctly.

Finally, execute the script like this from a machine that has access to your Kubernetes cluster:

bash cluster/examples/kubernetes/ceph/import-external-cluster.sh

6.1.4.9.2 CephCluster example (consumer) #

  

Assuming the above section has successfully completed, here is a CR example:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph-external
  namespace: rook-ceph-external
spec:
  external:
    enable: true
  crashCollector:
    disable: true
  # optionally, the ceph-mgr IP address can be pass to gather metric from the prometheus exporter
  #monitoring:
    #enabled: true
    #rulesNamespace: rook-ceph
    #externalMgrEndpoints:
      #- ip: 192.168.39.182

Choose the namespace carefully; if you have an existing cluster managed by Rook, you have likely already injected common.yaml. Additionally, you need to inject common-external.yaml too.

You can now create it like this:

kubectl create -f cluster/examples/kubernetes/ceph/cluster-external.yaml

If the previous section has not been completed, the Rook Operator will still acknowledge the CR creation but will wait forever to receive connection information.

Warning

If no cluster is managed by the current Rook Operator, you need to inject common.yaml, then modify cluster-external.yaml and specify rook-ceph as namespace.

If this is successful, you will see the CephCluster status as connected.

kubectl get CephCluster -n rook-ceph-external
NAME                 DATADIRHOSTPATH   MONCOUNT   AGE    STATE       HEALTH
rook-ceph-external   /var/lib/rook                162m   Connected   HEALTH_OK

Before you create a StorageClass with this cluster you will need to create a pool in your external Ceph Cluster.

rados df
POOL_NAME     USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS  RD WR_OPS  WR USED COMPR UNDER COMPR
replicated_2g  0 B       0      0      0                  0       0        0      0 0 B      0 0 B        0 B         0 B

cat << EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block-ext
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
    # clusterID is the namespace where the rook cluster is running
    clusterID: rook-ceph-external
    # Ceph pool into which the RBD image shall be created
    pool: replicated_2g

    # RBD image format. Defaults to "2".
    imageFormat: "2"

    # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.
    imageFeatures: layering

    # The secrets contain Ceph admin credentials.
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph-external
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph-external
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph-external

    # Specify the filesystem type of the volume. If not specified, csi-provisioner
    # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock
    # in hyperconverged settings where the volume is mounted on the same node as the osds.
    csi.storage.k8s.io/fstype: ext4

# Delete the rbd volume when a PVC is deleted
reclaimPolicy: Delete
allowVolumeExpansion: true
EOF

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph-external
  namespace: rook-ceph-external
spec:
  external:
    enable: true
  dataDirHostPath: /var/lib/rook
  cephVersion:
    image: ceph/ceph:v15.2.4 # Should match external cluster version

Rook has the ability to cleanup resources and data that were deployed when a delete cephcluster command is issued. The policy represents the confirmation that cluster data should be forcibly deleted. The cleanupPolicy should only be added to the cluster when the cluster is about to be deleted. After the confirmation field of the cleanup policy is set, Rook will stop configuring the cluster as if the cluster is about to be destroyed in order to prevent these settings from being deployed unintentionally. The cleanupPolicy CR settings has different fields:

To automate activation of the cleanup, you can use the following command:

Warning

Data will be permanently deleted.

kubectl -n rook-ceph patch cephcluster rook-ceph --type merge \
 -p '{"spec":{"cleanupPolicy":{"confirmation":"yes-really-destroy-data"}}}'

Nothing will happen until the deletion of the CR is requested, so this can still be reverted. However, all new configuration by the operator will be blocked with this cleanup policy enabled.

Rook waits for the deletion of PVs provisioned using the CephCluster before proceeding to delete the CephCluster. To force deletion of the CephCluster without waiting for the PVs to be deleted, you can set the allowUninstallWithVolumes to true under spec.CleanupPolicy.

For optimal performance, while also adding redundancy, this sample will configure Ceph to make three full copies of the data on multiple nodes.

Note

This sample requires at least one OSD per node, with each OSD located on three different nodes.

Each OSD must be located on a different node, because the failureDomain is set to host and the replicated.size is set to three.

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
  deviceClass: hdd

This sample will lower the overall storage capacity requirement, while also adding redundancy by using Section 6.2.2.4, “Erasure coding”.

Note

This sample requires at least three BlueStore OSDs.

The OSDs can be located on a single Ceph node or spread across multiple nodes, because the failureDomain is set to osd and the erasureCoded chunk settings require at least three different OSDs (two dataChunks + one codingChunks).

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: ecpool
  namespace: rook-ceph
spec:
  failureDomain: osd
  erasureCoded:
    dataChunks: 2
    codingChunks: 1
  deviceClass: hdd

High performance applications typically will not use erasure coding due to the performance overhead of creating and distributing the chunks in the cluster.

When creating an erasure-coded pool, we recommend creating the pool when you have BlueStore OSDs in your cluster.

With poolProperties you can set any pool property:

spec:
  parameters:
    <name of the parameter>: <parameter value>

For example:

spec:
  parameters:
    min_size: 1

Erasure coding allows you to keep your data safe while reducing the storage overhead. Instead of creating multiple replicas of the data, erasure coding divides the original data into chunks of equal size, then generates extra chunks of that same size for redundancy.

For example, if you have an object of size 2 MB, the simplest erasure coding with two data chunks would divide the object into two chunks of size 1 MB each (data chunks). One more chunk (coding chunk) of size 1 MB will be generated. In total, 3 MB will be stored in the cluster. The object will be able to suffer the loss of any one of the chunks and still be able to reconstruct the original object.

The number of data and coding chunks you choose will depend on your resiliency to loss and how much storage overhead is acceptable in your storage cluster. Here are some examples to illustrate how the number of chunks affects the storage and loss toleration.

The failureDomain must be also be taken into account when determining the number of chunks. The failure domain determines the level in the Ceph CRUSH hierarchy where the chunks must be uniquely distributed. This decision will impact whether node losses or disk losses are tolerated. There could also be performance differences of placing the data across nodes or OSDs.

If you do not have a sufficient number of hosts or OSDs for unique placement the pool can be created, writing to the pool will hang.

Rook currently only configures two levels in the CRUSH map. It is also possible to configure other levels such as rack with by adding topology labels to the nodes.

Note

This sample requires at least one OSD per node, with each OSD located on three different nodes.

Each OSD must be located on a different node, because both of the defined pools set the failureDomain to host and the replicated.size to three.

The failureDomain can also be set to another location type (for example, rack), if it has been added as a location in the Storage Selection Settings.

  apiVersion: ceph.rook.io/v1
  kind: CephFilesystem
  metadata:
    name: myfs
    namespace: rook-ceph
  spec:
    metadataPool:
      failureDomain: host
      replicated:
        size: 3
    dataPools:
      - failureDomain: host
        replicated:
          size: 3
    preservePoolsOnDelete: true
    metadataServer:
      activeCount: 1
      activeStandby: true
      # A key/value list of annotations
      annotations:
      #  key: value
      placement:
      #  nodeAffinity:
      #    requiredDuringSchedulingIgnoredDuringExecution:
      #      nodeSelectorTerms:
      #      - matchExpressions:
      #        - key: role
      #          operator: In
      #          values:
      #          - mds-node
      #  tolerations:
      #  - key: mds-node
      #    operator: Exists
      #  podAffinity:
      #  podAntiAffinity:
      #  topologySpreadConstraints:
      resources:
      #  limits:
      #    cpu: "500m"
      #    memory: "1024Mi"
      #  requests:
      #    cpu: "500m"
      #    memory: "1024Mi"

These definitions can be found in the filesystem.yaml file.

Erasure coded pools require the OSDs to use BlueStore for the configured storeType. Additionally, erasure coded pools can only be used with dataPools. The metadataPool must use a replicated pool.

Note

This sample requires at least three BlueStore OSDs, with each OSD located on a different node.

The OSDs must be located on different nodes, because the failureDomain will be set to host by default, and the erasureCoded chunk settings require at least three different OSDs (two dataChunks + one codingChunks).

  apiVersion: ceph.rook.io/v1
  kind: CephFilesystem
  metadata:
    name: myfs-ec
    namespace: rook-ceph
  spec:
    metadataPool:
      replicated:
        size: 3
    dataPools:
      - erasureCoded:
          dataChunks: 2
          codingChunks: 1
    metadataServer:
      activeCount: 1
      activeStandby: true

These definitions can also be found in the filesystem-ec.yaml file.

The pools allow all of the settings defined in the Pool CRD spec. In the example above, there must be at least three hosts (size three) and at least eight devices (six data + two coding chunks) in the cluster.

The number of PGs and PGPs can be configured on a per-pool basis, but it is highly advised to set default values that are appropriate for your Ceph cluster. Appropriate values depend on the number of OSDs the user expects to have backing each pool.

Pools created prior to v1.1 will have a default PG count of 100. Pools created after v1.1 will have Ceph's default PG count.

An easier option exists for Rook-Ceph clusters running Ceph Nautilus (v14.2.x) or newer. Nautilus introduced the PG auto-scaler mgr module capable of automatically managing PG and PGP values for pools.

In Nautilus, this module is not enabled by default, but can be enabled by the following setting in the CephCluster CR:

  spec:
    mgr:
      modules:
      - name: pg_autoscaler
        enabled: true

In Octopus (v15.2.x), this module is enabled by default without the aforementioned setting.

With that setting, the autoscaler will be enabled for all new pools. If you do not desire to have the autoscaler enabled for all new pools, you will need to use the Rook toolbox to enable the module and enable the autoscaling on individual pools.

The autoscaler is not enabled for the existing pools after enabling the module. So if you want to enable the autoscaling for these existing pools, they must be configured from the toolbox.

The most recommended way of configuring Ceph is to set Ceph's configuration directly. The first method for doing so is to use Ceph's CLI from the Rook-Ceph toolbox pod. From the toolbox, the user can change Ceph configurations, enable manager modules, create users and pools, and much more.

The Ceph Dashboard is another way of setting some of Ceph’s configuration directly. Configuration by the Ceph Dashboard is recommended with the same priority as configuration via the Ceph CLI (above).

Setting configuration options via Ceph’s CLI requires that at least one MON be available for the configuration options to be set, and setting configuration options via dashboard requires at least one mgr to be available. Ceph may also have a small number of very advanced settings that are not able to be modified easily via CLI or dashboard. The least recommended method for configuring Ceph is intended as a last-resort fallback in situations like these.

The operator can automatically remove OSD deployments that are considered “safe-to-destroy” by Ceph. After the steps above, the OSD will be considered safe to remove since the data has all been moved to other OSDs. But this will only be done automatically by the operator if you have this setting in the cluster CR:

removeOSDsIfOutAndSafeToRemove: true

Otherwise, you will need to delete the deployment directly:

kubectl@adm > kubectl delete deployment -n rook-ceph rook-ceph-osd-ID

Ceph can provide raw block device volumes to pods. Each example below sets up a storage class which can then be used to provision a block device in Kubernetes pods.

The storage classes are found in different sub-directories depending on the driver:

CephFS (CephFS) allows the user to mount a shared POSIX-compliant folder into one or more hosts (pods in the container world). This storage is similar to NFS shared storage or CIFS shared folders.

File storage contains multiple pools that can be configured for different scenarios:

Dynamic provisioning is possible with the CSI driver. The storage class for shared file systems is found in the csi/cephfs directory.

Ceph supports storing blobs of data called objects that support HTTP[S]-type get/put/post and delete semantics.

Object Storage contains multiple pools that can be configured for different scenarios:

The Ceph operator also runs an object store bucket provisioner which can grant access to existing buckets or dynamically provision new buckets.

Note

When you are using the Helm chart to install the Rook Ceph operator, and have set mountSecurityMode to—for example— Restricted, then the below ClusterRole has already been created for you.

This ClusterRole is needed no matter whether you want to use one RoleBinding per namespace or a ClusterRoleBinding.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rook-ceph-agent-mount
  labels:
    operator: rook
    storage-backend: ceph
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get

Note

You either need a RoleBinding in each namespace in which a mount secret resides in, or create a ClusterRoleBinding with which the Rook Ceph agent has access to Kubernetes secrets in all namespaces.

Create the RoleBinding shown here in each namespace for which the Rook Ceph agent should read secrets for mounting. The RoleBinding subjects' namespace must be the one the Rook Ceph agent runs in (default rook-ceph for version 1.0 and newer; for previous versions, the default namespace was rook-ceph-system).

Replace namespace: name-of-namespace-with-mountsecret according to the name of all namespaces a mountSecret can be in.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-agent-mount
  namespace: name-of-namespace-with-mountsecret
  labels:
    operator: rook
    storage-backend: ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-agent-mount
subjects:
- kind: ServiceAccount
  name: rook-ceph-system
  namespace: rook-ceph

This ClusterRoleBinding only needs to be created once, as it covers the whole cluster.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-agent-mount
  labels:
    operator: rook
    storage-backend: ceph
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-agent-mount
subjects:
- kind: ServiceAccount
  name: rook-ceph-system
  namespace: rook-ceph

# Get OSD Pods
# This uses the example/default cluster name "rook"
OSD_PODS=$(kubectl get pods --all-namespaces -l \
app=rook-ceph-osd,rook_cluster=rook-ceph -o jsonpath='{.items[*].metadata.name}')

# Find node and drive associations from OSD pods
for pod in $(echo ${OSD_PODS})
do
  echo "Pod:  ${pod}"
  echo "Node: $(kubectl -n rook-ceph get pod ${pod} -o jsonpath='{.spec.nodeName}')"
  kubectl -n rook-ceph exec ${pod} -- sh -c '\
  for i in /var/lib/ceph/osd/ceph-*; do
    [ -f ${i}/ready ] || continue
    echo -ne "-$(basename ${i}) "
    echo $(lsblk -n -o NAME,SIZE ${i}/block 2> /dev/null || \
    findmnt -n -v -o SOURCE,SIZE -T ${i}) $(cat ${i}/type)
  done | sort -V
  echo'
done

The output should look as follows:

Pod:  osd-m2fz2
Node: node1.zbrbdl
-osd0  sda3  557.3G  bluestore
-osd1  sdf3  110.2G  bluestore
-osd2  sdd3  277.8G  bluestore
-osd3  sdb3  557.3G  bluestore
-osd4  sde3  464.2G  bluestore
-osd5  sdc3  557.3G  bluestore

Pod:  osd-nxxnq
Node: node3.zbrbdl
-osd6   sda3  110.7G  bluestore
-osd17  sdd3  1.8T    bluestore
-osd18  sdb3  231.8G  bluestore
-osd19  sdc3  231.8G  bluestore

Pod:  osd-tww1h
Node: node2.zbrbdl
-osd7   sdc3  464.2G  bluestore
-osd8   sdj3  557.3G  bluestore
-osd9   sdf3  66.7G   bluestore
-osd10  sdd3  464.2G  bluestore
-osd11  sdb3  147.4G  bluestore
-osd12  sdi3  557.3G  bluestore
-osd13  sdk3  557.3G  bluestore
-osd14  sde3  66.7G   bluestore
-osd15  sda3  110.2G  bluestore
-osd16  sdh3  135.1G  bluestore

Note

Since Ceph Nautilus (v14.x), you can use the Ceph Manager pg_autoscaler module to auto-scale the PGs as needed. If you want to enable this feature, please refer to Section 7.1.1.1, “Default PG and PGP counts”.

The general rules for deciding how many PGs your pool(s) should contain is:

If you have more than 50 OSDs, you need to know how to calculate the pg_num value by yourself. For calculating pg_num yourself, please make use of the pgcalc tool at http://ceph.com/pgcalc/.

If you are already using a pool, it is generally safe to set pg_count on the fly (see Section 11.1.6.2, “Setting PG count”). Decreasing the PG count is not recommended on a pool that is in use. The safest way to decrease the PG count is to back up the data, delete the pool, and recreate it.

Be sure to read the Section 11.1.6.1, “Sizing placement groups” section before changing the number of PGs.

# Set the number of PGs in the rbd pool to 512
cephuser@adm > ceph osd pool set rbd pg_num 512

In this example we will set the default pool size to two, and tell OSD daemons not to change the weight of OSDs on startup.

Warning

Modify Ceph settings carefully. You are leaving the sandbox tested by Rook. Changing the settings could result in unhealthy daemons or even data loss if used incorrectly.

When the Rook Operator creates a cluster, a placeholder ConfigMap is created that will allow you to override Ceph configuration settings. When the daemon pods are started, the settings specified in this ConfigMap will be merged with the default settings generated by Rook.

The default override settings are blank. Cutting out the extraneous properties, we would see the following defaults after creating a cluster:

kubectl@adm > kubectl -n rook-ceph get ConfigMap rook-config-override -o yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: rook-config-override
  namespace: rook-ceph
data:
  config: ""

To apply your desired configuration, you will need to update this ConfigMap. The next time the daemon pod(s) start, they will use the updated configs.

kubectl@adm > kubectl -n rook-ceph edit configmap rook-config-override

Modify the settings and save. Each line you add should be indented from the config property as such:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rook-config-override
  namespace: rook-ceph
data:
  config: |
    [global]
    osd crush update on start = false
    osd pool default size = 2

The CRUSH weight controls the ratio of data that should be distributed to each OSD. This also means a higher or lower amount of disk I/O operations for an OSD with higher or lower weight, respectively.

By default, OSDs get a weight relative to their storage capacity, which maximizes overall cluster capacity by filling all drives at the same rate, even if drive sizes vary. This should work for most use-cases, but the following situations could warrant weight changes:

This example sets the weight of osd.0 which is 600 GiB.

cephuser@adm > ceph osd crush reweight osd.0 .600

When pools are set with a size setting greater than one, data is replicated between nodes and OSDs. For every chunk of data a Primary OSD is selected to be used for reading that data to be sent to clients. You can control how likely it is for an OSD to become a Primary using the Primary Affinity setting. This is similar to the OSD weight setting, except it only affects reads on the storage device, not capacity or writes.

In this example, we will make sure osd.0 is only selected as Primary if all other OSDs holding replica data are unavailable:

cephuser@adm >  osd primary-affinity osd.0 0

The below sample will create a CephObjectStore that starts the Object Gateway service in the cluster with an S3 API.

Note

This sample requires at least three BlueStore OSDs, with each OSD located on a different node.

The OSDs must be located on different nodes, because the failureDomain is set to host and the erasureCoded chunk settings require at least three different OSDs (two dataChunks + one codingChunks).

  apiVersion: ceph.rook.io/v1
  kind: CephObjectStore
  metadata:
    name: my-store
    namespace: rook-ceph
  spec:
    metadataPool:
      failureDomain: host
      replicated:
        size: 3
    dataPool:
      failureDomain: host
      erasureCoded:
        dataChunks: 2
        codingChunks: 1
    preservePoolsOnDelete: true
    gateway:
      type: s3
      sslCertificateRef:
      port: 80
      securePort:
      instances: 1
    healthCheck:
      bucket:
        disabled: false
        interval: 60s

After the CephObjectStore is created, the Rook operator will then create all the pools and other resources necessary to start the service. This may take a minute to complete.

Create the object store:

kubectl@adm > kubectl create -f object.yaml

To confirm the object store is configured, wait for the rgw pod to start:

kubectl@adm > kubectl -n rook-ceph get pod -l app=rook-ceph-rgw

Rook can connect to existing Object Gateway gateways to work in conjunction with the external mode of the CephCluster CRD. If you have an external CephCluster CR, you can instruct Rook to consume external gateways with the following:

  apiVersion: ceph.rook.io/v1
  kind: CephObjectStore
  metadata:
    name: external-store
    namespace: rook-ceph
  spec:
    gateway:
      port: 8080
      externalRgwEndpoints:
        - ip: 192.168.39.182
    healthCheck:
      bucket:
        enabled: true
        interval: 60s

You can use the existing object-external.yaml file. When ready the ceph-object-controller will output a message in the Operator log similar to this one:

ceph-object-controller: ceph object store gateway service running at 10.100.28.138:8080

You can now get and access the store via:

kubectl@adm > kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
rook-ceph-rgw-my-store   ClusterIP   10.100.28.138     none        8080/TCP   6h59m

Any pod from your cluster can now access this endpoint:

kubectl@adm > curl 10.100.28.138:8080

It is also possible to use the internally registered DNS name:

kubectl@adm > curl rook-ceph-rgw-my-store.rook-ceph:8080

The DNS name is created with the following schema: rook-ceph-rgw-$STORE_NAME.$NAMESPACE.

The following commands extract key pieces of information from the secret and configmap:

#config-map, secret, OBC will part of default if no specific name space mentioned
export AWS_HOST=$(kubectl -n default get cm ceph-bucket -o yaml | grep BUCKET_HOST | awk '{print $2}')
export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-bucket -o yaml | grep AWS_ACCESS_KEY_ID | awk '{print $2}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-bucket -o yaml | grep AWS_SECRET_ACCESS_KEY | awk '{print $2}' | base64 --decode)

To simplify the S3 client commands, you will want to set the four environment variables for use by your client (for example, inside the toolbox). See above for retrieving the variables for a bucket created by an ObjectBucketClaim.

export AWS_HOST=HOST
export AWS_ENDPOINT=ENDPOINT
export AWS_ACCESS_KEY_ID=ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=SECRET_KEY

The variables for the user generated in this example might be:

export AWS_HOST=rook-ceph-rgw-my-store.rook-ceph
export AWS_ENDPOINT=10.104.35.31:80
export AWS_ACCESS_KEY_ID=XEZDB3UJ6X7HVBE7X7MA
export AWS_SECRET_ACCESS_KEY=7yGIZON7EhFORz0I40BFniML36D2rl8CQQ5kXU6l

The access key and secret key can be retrieved as described in the section above on Section 12.1.2.1, “Client connections” or below in the section Section 12.1.5, “Creating a user” if you are not creating the buckets with an ObjectBucketClaim.

To test the CephObjectStore we will install the s3cmd tool into the toolbox pod.

zypper --assumeyes install s3cmd

Upload a file to the newly created bucket:

echo "Hello Rook" > /tmp/rookObj
s3cmd put /tmp/rookObj --no-ssl --host=${AWS_HOST} --host-bucket=  s3://rookbucket

Download and verify the file from the bucket:

s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=${AWS_HOST} --host-bucket=
cat /tmp/rookObj-download

Erasure-coded pools require the OSDs to use bluestore for the configured storeType. Additionally, erasure- coded pools can only be used with dataPools. The metadataPool must use a replicated pool.

Note

This sample requires at least three BlueStore OSDs, with each OSD located on a different node.

The OSDs must be located on different nodes, because the failureDomain is set to host and the erasureCoded chunk settings require at least three different OSDs (two dataChunks + one codingChunks).

  apiVersion: ceph.rook.io/v1
  kind: CephObjectStore
  metadata:
    name: my-store
    namespace: rook-ceph
  spec:
    metadataPool:
      failureDomain: host
      replicated:
        size: 3
    dataPool:
      failureDomain: host
      erasureCoded:
        dataChunks: 2
        codingChunks: 1
    preservePoolsOnDelete: true
    gateway:
      type: s3
      sslCertificateRef:
      port: 80
      securePort:
      instances: 1
      # A key/value list of annotations
      annotations:
      #  key: value
      placement:
      #  nodeAffinity:
      #    requiredDuringSchedulingIgnoredDuringExecution:
      #      nodeSelectorTerms:
      #      - matchExpressions:
      #        - key: role
      #          operator: In
      #          values:
      #          - rgw-node
      #  tolerations:
      #  - key: rgw-node
      #    operator: Exists
      #  podAffinity:
      #  podAntiAffinity:
      #  topologySpreadConstraints:
      resources:
      #  limits:
      #    cpu: "500m"
      #    memory: "1024Mi"
      #  requests:
      #    cpu: "500m"
      #    memory: "1024Mi"
    #zone:
      #name: zone-a

The pools allow all of the settings defined in the Pool CRD spec. In the example above, there must be at least three hosts (size 3) and at least three devices (two data + one coding chunks) in the cluster.

When the zone section is set, pools with the object store's name will not be created, since the object-store will the using the pools created by the ceph-object-zone.

Rook provides a default mime.types file for each Ceph Object Storage. This file is stored in a Kubernetes ConfigMap with the name rook-ceph-rgw-<STORE-NAME>-mime-types. For most users, the default file should suffice, however, the option is available to users to edit the mime.types file in the ConfigMap as they desire. Users may have their own special file types, and particularly security conscious users may wish to pare down the file to reduce the possibility of a file type execution attack.

Rook will not overwrite an existing mime.types ConfigMap so that user modifications will not be destroyed. If the object store is destroyed and recreated, the ConfigMap will also be destroyed and created anew.

After you connect to the dashboard, you will need to login for secure access. Rook creates a default user named admin and generates a secret called rook-ceph-dashboard-admin-password in the namespace where the Rook Ceph cluster is running. To retrieve the generated password, you can run the following:

kubectl@adm > kubectl -n rook-ceph get secret rook-ceph-dashboard-password \
 -o jsonpath="{['data']['password']}" | base64 --decode && echo

The simplest way to expose the service in minikube or similar environments is using the NodePort to open a port on the VM that can be accessed by the host. To create a service with the NodePort, save this YAML file as dashboard-external-https.yaml.

  apiVersion: v1
  kind: Service
  metadata:
    name: rook-ceph-mgr-dashboard-external-https
    namespace: rook-ceph
    labels:
      app: rook-ceph-mgr
      rook_cluster: rook-ceph
  spec:
    ports:
    - name: dashboard
      port: 8443
      protocol: TCP
      targetPort: 8443
    selector:
      app: rook-ceph-mgr
      rook_cluster: rook-ceph
    sessionAffinity: None
    type: NodePort

Now create the service:

kubectl@adm > kubectl create -f dashboard-external-https.yaml

You will see the new service rook-ceph-mgr-dashboard-external-https created:

kubectl@adm > kubectl -n rook-ceph get service
NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
rook-ceph-mgr                           ClusterIP   10.108.111.192   <none>        9283/TCP         4h
rook-ceph-mgr-dashboard                 ClusterIP   10.110.113.240   <none>        8443/TCP         4h
rook-ceph-mgr-dashboard-external-https  NodePort    10.101.209.6     <none>        8443:31176/TCP   4h

In this example, port 31176 will be opened to expose port 8443 from the ceph-mgr pod. Find the IP address of the VM. If using minikube, you can run minikube ip to find the IP address. Now you can enter the URL in your browser such as https://192.168.99.110:31176 and the dashboard will appear.

If you have a cluster on a cloud provider that supports load balancers, you can create a service that is provisioned with a public hostname. The yaml is the same as dashboard-external-https.yaml except for the following property:

  spec:
  [...]
    type: LoadBalancer

Now create the service:

kubectl@adm > kubectl create -f dashboard-loadbalancer.yaml

You will see the new service rook-ceph-mgr-dashboard-loadbalancer created:

kubectl@adm > kubectl -n rook-ceph get service
NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)             AGE
rook-ceph-mgr                            ClusterIP      172.30.11.40     <none>                                                                    9283/TCP      4h
rook-ceph-mgr-dashboard                  ClusterIP      172.30.203.185   <none>                                                                    8443/TCP      4h
rook-ceph-mgr-dashboard-loadbalancer     LoadBalancer   172.30.27.242    a7f23e8e2839511e9b7a5122b08f2038-1251669398.us-east-1.elb.amazonaws.com   8443:32747/TCP      4h

Now you can enter the URL in your browser such as https://a7f23e8e2839511e9b7a5122b08f2038-1251669398.us-east-1.elb.amazonaws.com:8443 and the dashboard will appear.

If you have a cluster with an Nginx Ingress Controller and a Certificate Manager, then you can create an Ingress like the one below. This example achieves four things:

  apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: rook-ceph-mgr-dashboard
    namespace: rook-ceph
    annotations:
      kubernetes.io/ingress.class: "nginx"
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      nginx.ingress.kubernetes.io/server-snippet: |
        proxy_ssl_verify off;
  spec:
    tls:
     - hosts:
       - rook-ceph.example.com
       secretName: rook-ceph.example.com
    rules:
    - host: rook-ceph.example.com
      http:
        paths:
        - path: /
          backend:
            serviceName: rook-ceph-mgr-dashboard
            servicePort: https-dashboard

Customise the Ingress resource to match your cluster. Replace the example domain name rook-ceph.example.com with a domain name that will resolve to your Ingress Controller (creating the DNS entry if required).

Now create the Ingress:

kubectl@adm > kubectl create -f dashboard-ingress-https.yaml

You will see the new Ingress rook-ceph-mgr-dashboard created:

kubectl@adm > kubectl -n rook-ceph get ingress
NAME                      HOSTS                      ADDRESS   PORTS     AGE
rook-ceph-mgr-dashboard   rook-ceph.example.com      80, 443   5m

And the new Secret for the TLS certificate:

kubectl@adm > kubectl -n rook-ceph get secret rook-ceph.example.com
NAME                       TYPE                DATA      AGE
rook-ceph.example.com      kubernetes.io/tls   2         4m

You can now browse to https://rook-ceph.example.com/ to log into the dashboard.

After you verify the basic health of the running pods, next you will want to run Ceph tools for status of the storage components. There are two ways to run the Ceph tools, either in the Rook toolbox or inside other Rook pods that are already running.

kubectl@adm > kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

Create a rook-ceph-tools pod to investigate the current state of Ceph. Here is an example of what one might see. In this case, the ceph status command would just hang, so a CTRL-C needed to be sent.

kubectl@adm > kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
cephuser@adm > ceph status
^CCluster connection interrupted or timed out

Another indication is when one or more of the MON pods restart frequently. Note the “mon107” that has only been up for 16 minutes in the following output.

kubectl@adm > kubectl -n rook-ceph get all -o wide --show-all
  NAME                                 READY     STATUS    RESTARTS   AGE       IP               NODE
  po/rook-ceph-mgr0-2487684371-gzlbq   1/1       Running   0          17h       192.168.224.46   k8-host-0402
  po/rook-ceph-mon107-p74rj            1/1       Running   0          16m       192.168.224.28   k8-host-0402
  rook-ceph-mon1-56fgm                 1/1       Running   0          2d        192.168.91.135   k8-host-0404
  rook-ceph-mon2-rlxcd                 1/1       Running   0          2d        192.168.123.33   k8-host-0403
  rook-ceph-osd-bg2vj                  1/1       Running   0          2d        192.168.91.177   k8-host-0404
  rook-ceph-osd-mwxdm                  1/1       Running   0          2d        192.168.123.31   k8-host-0403

What is happening here is that the MON pods are restarting and one or more of the Ceph daemons are not getting configured with the proper cluster information. This is commonly the result of not specifying a value for dataDirHostPath in your Cluster CRD.

The dataDirHostPath setting specifies a path on the local host for the Ceph daemons to store configuration and data. Setting this to a path like /var/lib/rook, reapplying your cluster CRD and restarting all the Ceph daemons (MON, MGR, OSD, RGW) should solve this problem. After the Object Gateway daemons have been restarted, it is advisable to restart the rook-tools pod.

When the operator is starting a cluster, the operator will start one MON at a time and check that they are healthy before continuing to bring up all three MONs. If the first MON is not detected healthy, the operator will continue to check until it is healthy. If the first MON fails to start, a second and then a third MON may attempt to start. However, they will never form a quorum, and orchestration will be blocked from proceeding.

The likely causes for the MON health not being detected:

kubectl@adm > kubectl -n rook-ceph logs -l app=rook-ceph-operator

  2018-01-21 21:47:32.375833 I | exec: Running command: ceph mon_status --cluster=rook --conf=/var/lib/rook/rook-ceph/rook.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/442263890
  2018-01-21 21:52:35.370533 I | exec: 2018-01-21 21:52:35.071462 7f96a3b82700  0 monclient(hunting): authenticate timed out after 300
  2018-01-21 21:52:35.071462 7f96a3b82700  0 monclient(hunting): authenticate timed out after 300
  2018-01-21 21:52:35.071524 7f96a3b82700  0 librados: client.admin authentication error (110) Connection timed out
  2018-01-21 21:52:35.071524 7f96a3b82700  0 librados: client.admin authentication error (110) Connection timed out
  [errno 110] error connecting to the cluster

kubectl@adm > kubectl -n rook-ceph get pod -l app=rook-ceph-mon
NAME                                READY     STATUS               RESTARTS   AGE
rook-ceph-mon-a-69fb9c78cd-58szd    1/1       CrashLoopBackOff     2          47s

kubectl@adm > kubectl -n rook-ceph describe pod -l mon=rook-ceph-mon0
[...]
Last State:    Terminated
Reason:    Error
Message:    The keyring does not match the existing keyring in /var/lib/rook/rook-ceph-mon0/data/keyring.
You may need to delete the contents of dataDirHostPath on the host from a previous deployment.
[...]

NAME                               READY   STATUS    RESTARTS   AGE
rook-ceph-mon-a-7d9fd97d9b-cdq7g   1/1     Running   0          10m
rook-ceph-mon-d-77df8454bd-r5jwr   1/1     Running   0          9m2s
rook-ceph-mon-f-58b4f8d9c7-89lgs   1/1     Running   0          7m38s

15.1.3.2.4 Identifying the solution #

  

This is a common problem reinitializing the Rook cluster when the local directory used for persistence has not been purged. This directory is the dataDirHostPath setting in the cluster CRD, and is typically set to /var/lib/rook. To fix the issue, you will need to delete all components of Rook and then delete the contents of /var/lib/rook (or the directory specified by dataDirHostPath) on each of the hosts in the cluster. Then, when the cluster CRD is applied to start a new cluster, the rook-operator should start all the pods as expected.

Important

Deleting the dataDirHostPath folder is destructive to the storage. Only delete the folder if you are trying to permanently purge the Rook cluster.

When you create a PVC based on a Rook storage class, it stays pending indefinitely.

For the Wordpress example, you might see two PVCs in the pending state.

kubectl@adm > kubectl get pvc
NAME             STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim   Pending                                      rook-ceph-block   8s
wp-pv-claim      Pending                                      rook-ceph-block   16s

There are two common causes for the PVCs staying in the pending state:

cephuser@adm > ceph status
  cluster:
  id:     a0452c76-30d9-4c1a-a948-5d8405f19a7c
  health: HEALTH_OK

  services:
  mon: 3 daemons, quorum a,b,c (age 11m)
  mgr: a(active, since 10m)
  osd: 1 osds: 1 up (since 46s), 1 in (since 109m)

kubectl@adm > kubectl -n rook-ceph get pod -l app=rook-ceph-osd-prepare
NAME                                 ...  READY   STATUS      RESTARTS   AGE
rook-ceph-osd-prepare-minikube-9twvk   0/2     Completed   0          30m

kubectl@adm > kubectl -n rook-ceph get pod -l app=csi-rbdplugin-provisioner

kubectl@adm > kubectl -n rook-ceph logs csi-cephfsplugin-provisioner-d77bb49c6-q9hwq csi-provisioner

If the OSD prepare logs did not give you enough clues about why the OSDs were not being created, review your cluster.yaml configuration. The common mistakes include:

When an OSD starts, the device or directory will be configured for consumption. If there is an error with the configuration, the pod will crash and you will see the CrashLoopBackoff status for the pod. Look in the OSD pod logs for an indication of the failure.

kubectl@adm > kubectl -n rook-ceph logs rook-ceph-osd-fl8fs

One common case for failure is that you have re-deployed a test cluster and some state may remain from a previous deployment. If your cluster is larger than a few nodes, you may get lucky enough that the monitors were able to start and form a quorum. However, now the OSDs pods may fail to start due to the old state. Looking at the OSD pod logs, you will see an error about the file already existing.

kubectl -n rook-ceph logs rook-ceph-osd-fl8fs
[...]
2017-10-31 20:13:11.187106 I | mkfs-osd0: 2017-10-31 20:13:11.186992 7f0059d62e00 -1 bluestore(/var/lib/rook/osd0) _read_fsid unparsable uuid
2017-10-31 20:13:11.187208 I | mkfs-osd0: 2017-10-31 20:13:11.187026 7f0059d62e00 -1 bluestore(/var/lib/rook/osd0) _setup_block_symlink_or_file failed to create block symlink to /dev/disk/by-partuuid/651153ba-2dfc-4231-ba06-94759e5ba273: (17) File exists
2017-10-31 20:13:11.187233 I | mkfs-osd0: 2017-10-31 20:13:11.187038 7f0059d62e00 -1 bluestore(/var/lib/rook/osd0) mkfs failed, (17) File exists
2017-10-31 20:13:11.187254 I | mkfs-osd0: 2017-10-31 20:13:11.187042 7f0059d62e00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (17) File exists
2017-10-31 20:13:11.187275 I | mkfs-osd0: 2017-10-31 20:13:11.187121 7f0059d62e00 -1  ** ERROR: error creating empty object store in /var/lib/rook/osd0: (17) File exists

If the error is from the file that already exists, this is a common problem reinitializing the Rook cluster when the local directory used for persistence has not been purged. This directory is the dataDirHostPath setting in the cluster CRD and is typically set to /var/lib/rook. To fix the issue you will need to delete all components of Rook and then delete the contents of /var/lib/rook (or the directory specified by dataDirHostPath) on each of the hosts in the cluster. Then when the cluster CRD is applied to start a new cluster, the rook-operator should start all the pods as expected.

First, ensure that you have specified the devices correctly in the CRD. The cluster CRD has several ways to specify the devices that are to be consumed by the Rook storage:

Second, if Rook determines that a device is not available (has existing partitions or a formatted file system), Rook will skip consuming the devices. If Rook is not starting OSDs on the devices you expect, Rook may have skipped it for this reason. To see if a device was skipped, view the OSD preparation log on the node where the device was skipped. Note that it is completely normal and expected for OSD prepare pod to be in the completed state. After the job is complete, Rook leaves the pod around in case the logs need to be investigated.

Get the prepare pods in the cluster:

kubectl@adm > kubectl -n rook-ceph get pod -l app=rook-ceph-osd-prepare
NAME                                   READY     STATUS      RESTARTS   AGE
rook-ceph-osd-prepare-node1-fvmrp      0/1       Completed   0          18m
rook-ceph-osd-prepare-node2-w9xv9      0/1       Completed   0          22m
rook-ceph-osd-prepare-node3-7rgnv      0/1       Completed   0          22m

View the logs for the node of interest in the "provision" container:

kubectl@adm > kubectl -n rook-ceph logs rook-ceph-osd-prepare-node1-fvmrp provision

Here are some key lines to look for in the log. A device will be skipped if Rook sees it has partitions or a file system:

2019-05-30 19:02:57.353171 W | cephosd: skipping device sda that is in use
2019-05-30 19:02:57.452168 W | skipping device "sdb5": ["Used by ceph-disk"]

Other messages about a disk being unusable by Ceph include:

Insufficient space (<5GB) on vgs
Insufficient space (<5GB)
LVM detected
Has BlueStore device label
locked
read-only

A device is going to be configured:

2019-05-30 19:02:57.535598 I | cephosd: device sdc to be configured by ceph-volume

For each device configured, you will see a report printed to the log:

2019-05-30 19:02:59.844642 I |   Type            Path                                                    LV Size         % of device
2019-05-30 19:02:59.844651 I | ----------------------------------------------------------------------------------------------------
2019-05-30 19:02:59.844677 I |   [data]          /dev/sdc                                                7.00 GB         100%

Either update the CR with the correct settings, or clean the partitions or file system from your devices.

After the settings are updated or the devices are cleaned, trigger the operator to analyze the devices again by restarting the operator. Each time the operator starts, it will ensure all the desired devices are configured. The operator does automatically deploy OSDs in most scenarios, but an operator restart will cover any scenarios that the operator does not detect automatically.

Restart the operator to ensure devices are configured. A new pod will automatically be started when the current operator pod is deleted.

kubectl@adm > kubectl -n rook-ceph delete pod -l app=rook-ceph-operator

If it is feasible to upgrade your kernel, you should upgrade to 4.x, even better is 4.7 or above, due to a feature for CephFS added to the kernel.

If you are unable to upgrade the kernel, you need to go to each host that will consume storage and run:

modprobe rbd

This command inserts the rbd module into the kernel.

To persist this fix, you need to add the rbd kernel module to either /etc/modprobe.d/ or /etc/modules-load.d/. For both paths create a file called rbd.conf with the following content:

rbd

Now when a host is restarted, the module should be loaded automatically.

The only solution to this problem is to upgrade your kernel to 4.7 or higher. This is due to a mount flag added in kernel version 4.7, which allows choosing the file system by name.

This happens when the following conditions are satisfied.

In addition, when this problem happens, you can see the following messages in dmesg.

dmesg
...
[51717.039319] INFO: task kworker/2:1:5938 blocked for more than 120 seconds.
[51717.039361]       Not tainted 4.15.0-72-generic #81-Ubuntu
[51717.039388] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
...

This is the so-called hung_task problem and means that there is a deadlock in the kernel.

You can bypass this problem by using ext4 or any other file systems rather than XFS. The file system type can be specified with csi.storage.k8s.io/fstype in StorageClass resource.

See Book “Troubleshooting Guide”, Chapter 5 “Troubleshooting placement groups (PGs)” for more information.

There is a critical flaw in OSD on LV-backed PVC. LVM metadata can be corrupted if both the host and OSD container modify it simultaneously. For example, the administrator might modify it on the host, while the OSD initialization process in a container could modify it too. In addition, if lvmetad is running, the possibility of occurrence gets higher. In this case, the change of LVM metadata in OSD container is not reflected to LVM metadata cache in host for a while.

If you still decide to configure an OSD on LVM, keep the following in mind to reduce the probability of this issue.

You can know whether the above-mentioned tag exists tag by running

#  lvs -o lv_name,lv_tags

If the lv_tag field is empty in an LV corresponding to the OSD lv_tags, this OSD encountered the problem. In this case, retire this OSD or replace with other new OSD before restarting.