5 Block Storage #
Block Storage allows a single pod to mount storage. This guide shows how to create a simple, multi-tier web application on Kubernetes using persistent volumes enabled by Rook.
5.1 Provisioning Block Storage #
Before Rook can provision storage, a StorageClass
and a
CephBlockPool
need to be created. This will allow Kubernetes
to interoperate with Rook when provisioning persistent volumes.
This sample requires at least one OSD per node, with each OSD located on three different nodes.
Each OSD must be located on a different node, because the
failureDomain
is set to host
and the replicated.size
is set to 3
.
This example uses the CSI driver, which is the preferred driver going forward for Kubernetes 1.13 and newer. Examples are found in the CSI RBD directory.
Save this StorageClass
definition as
storageclass.yaml
:
apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: replicapool namespace: rook-ceph spec: failureDomain: host replicated: size: 3 --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: rook-ceph-block # Change "rook-ceph" provisioner prefix to match the operator namespace if needed provisioner: rook-ceph.rbd.csi.ceph.com parameters: # clusterID is the namespace where the rook cluster is running clusterID: rook-ceph # Ceph pool into which the RBD image shall be created pool: replicapool # RBD image format. Defaults to "2". imageFormat: "2" # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature. imageFeatures: layering # The secrets contain Ceph admin credentials. csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # Specify the filesystem type of the volume. If not specified, csi-provisioner # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock # in hyperconverged settings where the volume is mounted on the same node as the osds. csi.storage.k8s.io/fstype: ext4 # Delete the rbd volume when a PVC is deleted reclaimPolicy: Delete
If you have deployed the Rook operator in a namespace other than “rook-ceph”, change the prefix in the provisioner to match the namespace you used. For example, if the Rook operator is running in the namespace “my-namespace” the provisioner value should be “my-namespace.rbd.csi.ceph.com”.
Create the storage class.
kubectl@adm >
kubectl create -f cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml
As
specified
by Kubernetes, when using the Retain
reclaim policy,
any Ceph RBD image that is backed by a
PersistentVolume
will continue to exist even after the
PersistentVolume
has been deleted. These Ceph RBD
images will need to be cleaned up manually using rbd rm
.
5.2 Consuming storage: WordPress sample #
In this example, we will create a sample application to consume the block storage provisioned by Rook with the classic WordPress and MySQL apps. Both of these applications will make use of block volumes provisioned by Rook.
Start MySQL and WordPress from the
cluster/examples/kubernetes
folder:
kubectl@adm >
kubectl create -f mysql.yaml
kubectl create -f wordpress.yaml
Both of these applications create a block volume, and mount it to their respective pod. You can see the Kubernetes volume claims by running the following:
kubectl@adm >
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
mysql-pv-claim Bound pvc-95402dbc-efc0-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m
wp-pv-claim Bound pvc-39e43169-efc1-11e6-bc9a-0cc47a3459ee 20Gi RWO 1m
Once the WordPress and MySQL pods are in the Running
state, get the cluster IP of the WordPress app and enter it in your browser:
kubectl@adm >
kubectl get svc wordpress
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
wordpress 10.3.0.155 <pending> 80:30841/TCP 2m
You should see the WordPress application running.
If you are using Minikube, the WordPress URL can be retrieved with this one-line command:
kubectl@adm >
echo http://$(minikube ip):$(kubectl get service wordpress -o jsonpath='{.spec.ports[0].nodePort}')
When running in a Vagrant environment, there will be no external IP address
to reach WordPress with. You will only be able to reach WordPress via the
CLUSTER-IP
from inside the Kubernetes cluster.
5.3 Consuming the storage: Toolbox #
With the pool that was created above, we can also create a block image and mount it directly in a pod.
5.4 Teardown #
To clean up all the artifacts created by the block-storage demonstration:
kubectl@adm >
kubectl delete -f wordpress.yamlkubectl@adm >
kubectl delete -f mysql.yamlkubectl@adm >
kubectl delete -n rook-ceph cephblockpools.ceph.rook.io replicapoolkubectl@adm >
kubectl delete storageclass rook-ceph-block
5.5 Advanced Example: Erasure-Coded Block Storage #
If you want to use erasure-coded pools with RBD, your OSDs must use
bluestore
as their storeType
.
Additionally, the nodes that will mount the erasure-coded RBD block storage
must have Linux kernel 4.11
or above.
This example requires at least three bluestore OSDs, with each OSD located on a different node.
The OSDs must be located on different nodes, because the
failureDomain
is set to host
and the
erasureCoded
chunk settings require at least three
different OSDs (two dataChunks
plus one
codingChunk
).
To be able to use an erasure-coded pool, you need to create two pools (as seen below in the definitions): one erasure-coded, and one replicated.
5.5.1 Erasure coded CSI driver #
The erasure-coded pool must be set as the dataPool
parameter in
storageclass-ec.yaml
It is used for the data of the RBD images.