Using an External Ceph Driver
These instructions are about using the external Ceph driver in an RKE2 cluster. If you are using RKE, additional steps are required. For details, refer to this section.
Using the Ceph Driver with RKE
The resources below are fully compatible with RKE based clusters, but there is a need to do an additional kubelet configuration for RKE.
On RKE clusters, the kubelet component is running in a Docker container and doesn’t have access to the host’s kernel modules as rbd and libceph by default.
To solve this limitation, you can either run modprobe rbd
on worker nodes, or configure the kubelet containers to automatically mount the /lib/modules
directory from the host into the container.
For the kubelet configuration, put the following lines into the cluster.yml
file prior to RKE cluster provisioning. You can also modify the cluster.yml
later in the Rancher UI by clicking on and restarting the worker nodes.
services:
kubelet:
extra_binds:
- '/lib/modules:/lib/modules:ro'
For more information about the extra_binds
directive, refer to this section.
Installing the ceph-csi driver on an SUSE® Rancher Prime RKE2 cluster
These steps are needed for dynamic RBD provisioning only. |
For more information about the ceph-csi-rbd
chart, refer to this page.
To get details about your SES cluster, run:
ceph mon dump
Read its output:
dumped monmap epoch 3 epoch 3 fsid 79179d9d-98d8-4976-ab2e-58635caa7235 last_changed 2021-02-11T10:56:42.110184+0000 created 2021-02-11T10:56:22.913321+0000 min_mon_release 15 (octopus) 0: [v2:10.85.8.118:3300/0,v1:10.85.8.118:6789/0] mon.a 1: [v2:10.85.8.123:3300/0,v1:10.85.8.123:6789/0] mon.b 2: [v2:10.85.8.124:3300/0,v1:10.85.8.124:6789/0] mon.c
Later you’ll need the fsid and mon addresses values.
Install the ceph-csi Driver Using Helm
Run these commands:
helm repo add ceph-csi https://ceph.github.io/csi-charts helm repo update helm search repo ceph-csi -l helm inspect values ceph-csi/ceph-csi-rbd > ceph-csi-rbd-values.yaml
Modify the ceph-csi-rbd-values.yaml
file and keep there only the required changes:
# ceph-csi-rbd-values.yaml
csiConfig:
- clusterID: "79179d9d-98d8-4976-ab2e-58635caa7235"
monitors:
- "10.85.8.118:6789"
- "10.85.8.123:6789"
- "10.85.8.124:6789"
provisioner:
name: provisioner
replicaCount: 2
Make sure the ceph monitors are reachable from the RKE2 cluster, for example, by ping.
kubectl create namespace ceph-csi-rbd helm install --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yaml kubectl rollout status deployment ceph-csi-rbd-provisioner -n ceph-csi-rbd helm status ceph-csi-rbd -n ceph-csi-rbd
in case you’d like to modify the configuration directly via Helm, you may adapt the ceph-csi-rbd-values.yaml
file and call:
helm upgrade \ --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yaml
Creating RBD Ceph Resources
# Create a ceph pool: ceph osd pool create myPool 64 64 # Create a block device pool: rbd pool init myPool # Create a block device image: rbd create -s 2G myPool/image # Create a block device user and record the key: ceph auth get-or-create-key client.myPoolUser mon "allow r" osd "allow class-read object_prefix rbd_children, allow rwx pool=myPool" | tr -d '\n' | base64 QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ== # Encode the ceph user myPoolUser into a bash64 hash: echo "myPoolUser" | tr -d '\n' | base64 bXlQb29sVXNlcg== # Create a block device admin user and record the key: ceph auth get-or-create-key client.myPoolAdmin mds 'allow *' mgr 'allow *' mon 'allow *' osd 'allow * pool=myPool' | tr -d '\n' | base64 QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ== # Encode the ceph user myPoolAdmin into a bash64 hash: echo "myPoolAdmin" | tr -d '\n' | base64 bXlQb29sQWRtaW4=
Configure RBD Ceph Access Secrets
User Account
For static RBD provisioning (the image within the ceph pool must exist), run these commands:
cat > ceph-user-secret.yaml << EOF apiVersion: v1 kind: Secret metadata: name: ceph-user namespace: default type: kubernetes.io/rbd data: userID: bXlQb29sVXNlcg== userKey: QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ== EOF kubectl apply -f ceph-user-secret.yaml
Admin Account
For dynamic RBD provisioning (used for automatic image creation within a given ceph pool), run these commands:
cat > ceph-admin-secret.yaml << EOF apiVersion: v1 kind: Secret metadata: name: ceph-admin namespace: default type: kubernetes.io/rbd data: userID: bXlQb29sQWRtaW4= userKey: QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ== EOF kubectl apply -f ceph-admin-secret.yaml
Create RBD Testing Resources
Using RBD in Pods
# pod
cat > ceph-rbd-pod-inline.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
name: ceph-rbd-pod-inline
spec:
containers:
- name: ceph-rbd-pod-inline
image: busybox
command: ["sleep", "infinity"]
volumeMounts:
- mountPath: /mnt/ceph_rbd
name: volume
volumes:
- name: volume
rbd:
monitors:
- 10.85.8.118:6789
- 10.85.8.123:6789
- 10.85.8.124:6789
pool: myPool
image: image
user: myPoolUser
secretRef:
name: ceph-user
fsType: ext4
readOnly: false
EOF
kubectl apply -f ceph-rbd-pod-inline.yaml
kubectl get pod
kubectl exec pod/ceph-rbd-pod-inline -- df -k | grep rbd
Using RBD in Persistent Volumes
# pod-pvc-pv
cat > ceph-rbd-pod-pvc-pv-allinone.yaml << EOF
apiVersion: v1
kind: PersistentVolume
metadata:
name: ceph-rbd-pv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
rbd:
monitors:
- 10.85.8.118:6789
- 10.85.8.123:6789
- 10.85.8.124:6789
pool: myPool
image: image
user: myPoolUser
secretRef:
name: ceph-user
fsType: ext4
readOnly: false
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ceph-rbd-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
name: ceph-rbd-pod-pvc-pv
spec:
containers:
- name: ceph-rbd-pod-pvc-pv
image: busybox
command: ["sleep", "infinity"]
volumeMounts:
- mountPath: /mnt/ceph_rbd
name: volume
volumes:
- name: volume
persistentVolumeClaim:
claimName: ceph-rbd-pvc
EOF
kubectl apply -f ceph-rbd-pod-pvc-pv-allinone.yaml
kubectl get pv,pvc,pod
kubectl exec pod/ceph-rbd-pod-pvc-pv -- df -k | grep rbd
Using RBD in Storage Classes
This example is for dynamic provisioning. The ceph-csi driver is needed.
# pod-pvc-sc
cat > ceph-rbd-pod-pvc-sc-allinone.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph-rbd-sc
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: rbd.csi.ceph.com
parameters:
clusterID: 79179d9d-98d8-4976-ab2e-58635caa7235
pool: myPool
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: ceph-admin
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/controller-expand-secret-name: ceph-admin
csi.storage.k8s.io/controller-expand-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: ceph-admin
csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ceph-rbd-sc-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
storageClassName: ceph-rbd-sc
---
apiVersion: v1
kind: Pod
metadata:
name: ceph-rbd-pod-pvc-sc
spec:
containers:
- name: ceph-rbd-pod-pvc-sc
image: busybox
command: ["sleep", "infinity"]
volumeMounts:
- mountPath: /mnt/ceph_rbd
name: volume
volumes:
- name: volume
persistentVolumeClaim:
claimName: ceph-rbd-sc-pvc
EOF
kubectl apply -f ceph-rbd-pod-pvc-sc-allinone.yaml
kubectl get pv,pvc,sc,pod
kubectl exec pod/ceph-rbd-pod-pvc-sc -- df -k | grep rbd
SUSE® Rancher Prime RKE2 Server/Master Provisioning
sudo su curl -sfL https://get.rke2.io | sh - systemctl enable --now rke2-server cat > /root/.bashrc << EOF export PATH=$PATH:/var/lib/rancher/rke2/bin/ export KUBECONFIG=/etc/rancher/rke2/rke2.yaml EOF cat /var/lib/rancher/rke2/server/node-token token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED
SUSE® Rancher Prime RKE2 Agent/Worker provisioning
mkdir -p /etc/rancher/rke2/ cat > /etc/rancher/rke2/config.yaml << EOF server: https://10.100.103.23:9345 token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED EOF curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh - systemctl enable --now rke2-agent.service
To import the cluster into Rancher, click ☰ > Cluster Management. Then on the Clusters page, click Import Existing. Then run the provided kubectl command on the server/master node.
Tested Versions
OS for running RKE2 nodes: JeOS SLE15-SP2 with installed kernel-default-5.3.18-24.49
kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-22T12:00:00Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7+rke2r1", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-20T01:50:52Z", GoVersion:"go1.15.5b5", Compiler:"gc", Platform:"linux/amd64"} helm version version.BuildInfo{Version:"3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.12"}
Kubernetes version on RKE2 cluster: v1.19.7+rke2r1
Troubleshooting
In case you are using SUSE’s ceph-rook based on SES7, it might be useful to expose the monitors on hostNetwork by editing rook-1.4.5/ceph/cluster.yaml
and setting spec.network.hostNetwork=true
.
Also for operating the ceph-rook cluster, it is useful to deploy a toolbox on the Kubernetes cluster where ceph-rook is provisioned by kubectl apply -f rook-1.4.5/ceph/toolbox.yaml
Then all the ceph related commands can be executed in the toolbox pod, for example, by running kubectl exec -it -n rook-ceph rook-ceph-tools-686d8b8bfb-2nvqp -- bash
Operating with the ceph - basic commands:
ceph osd pool stats ceph osd pool delete myPool myPool --yes-i-really-really-mean-it rbd list -p myPool > csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38 > image
Delete the image: rbd rm csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38 -p myPool
CephFS commands in rook toolbox:
ceph -s ceph fs ls ceph fs fail cephfs ceph fs rm cephfs --yes-i-really-mean-it ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it
To prepare a cephfs filesystem, you can run this command on a rook cluster:
kubectl apply -f rook-1.4.5/ceph/filesystem.yaml