Applies to SUSE Enterprise Storage 7.1

13 Object Storage #

13.1 Object Storage #

Object Storage exposes an S3 API to the storage cluster for applications to put and get data.

13.1.1 Configuring the Object Storage #

Rook has the ability to either deploy an Object Storage in Kubernetes or to connect to an external Object Gateway service. Most commonly, the Object Storage will be configured locally by Rook.

13.1.1.1 Creating a local Object Storage #

The below sample will create a CephObjectStore that starts the Object Gateway service in the cluster with an S3 API.

Note

This sample requires at least three BlueStore OSDs, with each OSD located on a different node.

The OSDs must be located on different nodes, because the failureDomain is set to host and the erasureCoded chunk settings require at least three different OSDs (two dataChunks + one codingChunks).

  apiVersion: ceph.rook.io/v1
  kind: CephObjectStore
  metadata:
    name: my-store
    namespace: rook-ceph
  spec:
    metadataPool:
      failureDomain: host
      replicated:
        size: 3
    dataPool:
      failureDomain: host
      erasureCoded:
        dataChunks: 2
        codingChunks: 1
    preservePoolsOnDelete: true
    gateway:
      type: s3
      sslCertificateRef:
      port: 80
      securePort:
      instances: 1
    healthCheck:
      bucket:
        disabled: false
        interval: 60s

After the CephObjectStore is created, the Rook Operator will then create all the pools and other resources necessary to start the service. This may take a minute to complete.

Create the object store:

kubectl@adm > kubectl create -f object.yaml

To confirm the object store is configured, wait for the rgw pod to start:

kubectl@adm > kubectl -n rook-ceph get pod -l app=rook-ceph-rgw

13.1.1.2 Connecting to an external Object Storage #

Rook can connect to existing Object Gateway gateways to work in conjunction with the external mode of the CephCluster CRD. If you have an external CephCluster CR, you can instruct Rook to consume external gateways with the following:

  apiVersion: ceph.rook.io/v1
  kind: CephObjectStore
  metadata:
    name: external-store
    namespace: rook-ceph
  spec:
    gateway:
      port: 8080
      externalRgwEndpoints:
        - ip: 192.168.39.182
    healthCheck:
      bucket:
        enabled: true
        interval: 60s

You can use the existing object-external.yaml file. When ready the ceph-object-controller will output a message in the Operator log similar to this one:

ceph-object-controller: ceph object store gateway service running at 10.100.28.138:8080

You can now get and access the store via:

kubectl@adm > kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
rook-ceph-rgw-my-store   ClusterIP   10.100.28.138     none        8080/TCP   6h59m

Any pod from your cluster can now access this endpoint:

kubectl@adm > curl 10.100.28.138:8080

It is also possible to use the internally registered DNS name:

kubectl@adm > curl rook-ceph-rgw-my-store.rook-ceph:8080

The DNS name is created with the following schema: rook-ceph-rgw-$STORE_NAME.$NAMESPACE.

13.1.2 Creating a bucket #

Now that the object store is configured, next we need to create a bucket where a client can read and write objects. A bucket can be created by defining a storage class, similar to the pattern used by block and file storage. First, define the storage class that will allow object clients to create a bucket. The storage class defines the object storage system, the bucket retention policy, and other properties required by the administrator. Save the following as storageclass-bucket-delete.yaml (the example is named as such due to the Delete reclaim policy).

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
     name: rook-ceph-bucket
  provisioner: rook-ceph.ceph.rook.io/bucket
  reclaimPolicy: Delete
  parameters:
    objectStoreName: my-store
    objectStoreNamespace: rook-ceph
    region: us-east-1

kubectl@adm > kubectl create -f storageclass-bucket-delete.yaml

Based on this storage class, an object client can now request a bucket by creating an Object Bucket Claim (OBC). When the OBC is created, the Rook-Ceph bucket provisioner will create a new bucket. Notice that the OBC references the storage class that was created above. Save the following as object-bucket-claim-delete.yaml (the example is named as such due to the Delete reclaim policy):

  apiVersion: objectbucket.io/v1alpha1
  kind: ObjectBucketClaim
  metadata:
    name: ceph-bucket
  spec:
    generateBucketName: ceph-bkt
    storageClassName: rook-ceph-bucket

kubectl@adm > kubectl create -f object-bucket-claim-delete.yaml

Now that the claim is created, the operator will create the bucket as well as generate other artifacts to enable access to the bucket. A secret and ConfigMap are created with the same name as the OBC and in the same namespace. The secret contains credentials used by the application pod to access the bucket. The ConfigMap contains bucket endpoint information and is also consumed by the pod.

13.1.2.1 Client connections #

The following commands extract key pieces of information from the secret and configmap:

#config-map, secret, OBC will part of default if no specific name space mentioned
export AWS_HOST=$(kubectl -n default get cm ceph-bucket -o yaml | grep BUCKET_HOST | awk '{print $2}')
export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-bucket -o yaml | grep AWS_ACCESS_KEY_ID | awk '{print $2}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-bucket -o yaml | grep AWS_SECRET_ACCESS_KEY | awk '{print $2}' | base64 --decode)

13.1.3 Consuming the Object Storage #

Now that you have the Object Storage configured and a bucket created, you can consume the object storage from an S3 client.

This section will guide you through testing the connection to the CephObjectStore and uploading and downloading from it. Run the following commands after you have connected to the Rook toolbox.

13.1.3.1 Setting environment variables #

To simplify the S3 client commands, you will want to set the four environment variables for use by your client (for example, inside the toolbox). See above for retrieving the variables for a bucket created by an ObjectBucketClaim.

export AWS_HOST=HOST
export AWS_ENDPOINT=ENDPOINT
export AWS_ACCESS_KEY_ID=ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=SECRET_KEY

Host: The DNS host name where the Object Gateway service is found in the cluster. Assuming you are using the default rook-ceph cluster, it will be rook-ceph-rgw-my-store.rook-ceph.
Endpoint: The endpoint where the Object Gateway service is listening. Run the following command and then combine the clusterIP and the port.
```
kubectl@adm > kubectl -n rook-ceph get svc rook-ceph-rgw-my-store
```
Access key: The user’s access_key as printed above
Secret key: The user’s secret_key as printed above

The variables for the user generated in this example might be:

export AWS_HOST=rook-ceph-rgw-my-store.rook-ceph
export AWS_ENDPOINT=10.104.35.31:80
export AWS_ACCESS_KEY_ID=XEZDB3UJ6X7HVBE7X7MA
export AWS_SECRET_ACCESS_KEY=7yGIZON7EhFORz0I40BFniML36D2rl8CQQ5kXU6l

The access key and secret key can be retrieved as described in the section above on Section 13.1.2.1, “Client connections” or below in the section Section 13.1.5, “Creating a user” if you are not creating the buckets with an ObjectBucketClaim.

13.1.3.2 Installing the s3cmd package #

To test the CephObjectStore we will install the s3cmd tool into the toolbox pod.

zypper --assumeyes install s3cmd

13.1.3.3 PUT or GET an object #

Upload a file to the newly created bucket:

echo "Hello Rook" > /tmp/rookObj
s3cmd put /tmp/rookObj --no-ssl --host=${AWS_HOST} --host-bucket=  s3://rookbucket

Download and verify the file from the bucket:

s3cmd get s3://rookbucket/rookObj /tmp/rookObj-download --no-ssl --host=${AWS_HOST} --host-bucket=
cat /tmp/rookObj-download

13.1.4 Setting up external access to the cluster #

Rook sets up the object storage so pods will have access internal to the cluster. If your applications are running outside the cluster, you will need to setup an external service through a NodePort.

First, note the service that exposes RGW internal to the cluster. We will leave this service intact and create a new service for external access.

kubectl@adm > kubectl -n rook-ceph get service rook-ceph-rgw-my-store
NAME                     CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
rook-ceph-rgw-my-store   10.3.0.177   none        80/TCP      2m

Save the external service as rgw-external.yaml:

  apiVersion: v1
  kind: Service
  metadata:
    name: rook-ceph-rgw-my-store-external
    namespace: rook-ceph
    labels:
      app: rook-ceph-rgw
      rook_cluster: rook-ceph
      rook_object_store: my-store
  spec:
    ports:
    - name: rgw
      port: 80
      protocol: TCP
      targetPort: 80
    selector:
      app: rook-ceph-rgw
      rook_cluster: rook-ceph
      rook_object_store: my-store
    sessionAffinity: None
    type: NodePort

Now, create the external service:

kubectl@adm > kubectl create -f rgw-external.yaml

See both Object Gateway services running and notice what port the external service is running on:

kubectl@adm > kubectl -n rook-ceph get service rook-ceph-rgw-my-store rook-ceph-rgw-my-store-external
NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
rook-ceph-rgw-my-store            ClusterIP   10.104.82.228    none          80/TCP         4m
rook-ceph-rgw-my-store-external   NodePort    10.111.113.237   none          80:31536/TCP   39s

Internally the Object Gateway service is running on port 80. The external port in this case is 31536.

13.1.5 Creating a user #

If you need to create an independent set of user credentials to access the S3 endpoint, create a CephObjectStoreUser. The user will be used to connect to the Object Gateway service in the cluster using the S3 API. The user will be independent of any object bucket claims that you might have created in the earlier instructions in this document.

  apiVersion: ceph.rook.io/v1
  kind: CephObjectStoreUser
  metadata:
    name: my-user
    namespace: rook-ceph
  spec:
    store: my-store
    displayName: "my display name"

When the CephObjectStoreUser is created, the Rook operator will then create the RGW user on the specified CephObjectStore and store the Access Key and Secret Key in a kubernetes secret in the same namespace as the CephObjectStoreUser.

Create the object store user:

kubectl@adm > kubectl create -f object-user.yaml

To confirm the object store user is configured, describe the secret:

kubectl@adm > kubectl -n rook-ceph describe secret rook-ceph-object-user-my-store-my-user
  Name:		rook-ceph-object-user-my-store-my-user
  Namespace:	rook-ceph
  Labels:			app=rook-ceph-rgw
  			      rook_cluster=rook-ceph
  			      rook_object_store=my-store
  Annotations:	none

  Type:	kubernetes.io/rook

  Data
  ====
  AccessKey:	20 bytes
  SecretKey:	40 bytes

The AccessKey and SecretKey data fields can be mounted in a pod as an environment variable.

To directly retrieve the secrets:

kubectl@adm > kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml \
 | grep AccessKey | awk '{print $2}' | base64 --decode
kubectl@adm > kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o yaml \
 | grep SecretKey | awk '{print $2}' | base64 --decode

13.2 Ceph Object Storage CRD #

Rook allows creation and customization of object stores through the custom resource definitions (CRDs). The following settings are available for Ceph Object Storage.

13.2.1 Sample #

13.2.1.1 Erasure code #

Erasure coded pools require the OSDs to use bluestore for the configured storeType. Additionally, erasure coded pools can only be used with dataPools. The metadataPool must use a replicated pool.

Note

This sample requires at least three BlueStore OSDs, with each OSD located on a different node.

  apiVersion: ceph.rook.io/v1
  kind: CephObjectStore
  metadata:
    name: my-store
    namespace: rook-ceph
  spec:
    metadataPool:
      failureDomain: host
      replicated:
        size: 3
    dataPool:
      failureDomain: host
      erasureCoded:
        dataChunks: 2
        codingChunks: 1
    preservePoolsOnDelete: true
    gateway:
      type: s3
      sslCertificateRef:
      port: 80
      securePort:
      instances: 1
      # A key/value list of annotations
      annotations:
      #  key: value
      placement:
      #  nodeAffinity:
      #    requiredDuringSchedulingIgnoredDuringExecution:
      #      nodeSelectorTerms:
      #      - matchExpressions:
      #        - key: role
      #          operator: In
      #          values:
      #          - rgw-node
      #  tolerations:
      #  - key: rgw-node
      #    operator: Exists
      #  podAffinity:
      #  podAntiAffinity:
      #  topologySpreadConstraints:
      resources:
      #  limits:
      #    cpu: "500m"
      #    memory: "1024Mi"
      #  requests:
      #    cpu: "500m"
      #    memory: "1024Mi"
    #zone:
      #name: zone-a

13.2.2 Object store settings #

13.2.2.1 Metadata #

name: The name of the object store to create, which will be reflected in the pool and other resource names.
namespace: The namespace of the Rook cluster where the object store is created.

13.2.2.2 Pools #

The pools allow all of the settings defined in the pool CRD specification. In the example above, there must be at least three hosts (size 3) and at least three devices (two data + one coding chunks) in the cluster.

When the zone section is set, pools with the object store's name will not be created, since the object-store will the using the pools created by the ceph-object-zone.

metadataPool: The settings used to create all of the object store metadata pools. Must use replication.
dataPool: The settings to create the object store data pool. Can use replication or erasure coding.
preservePoolsOnDelete: If it is set to “true”, the pools used to support the object store will remain when the object store will be deleted. This is a security measure to avoid accidental loss of data. It is set to “false” by default. If it is not specified, this is also deemed as “false”.

13.2.3 Creating gateway settings #

The gateway settings correspond to the Object Gateway daemon settings.

type: S3 is supported
sslCertificateRef: If the certificate is not specified, SSL will not be configured. If specified, this is the name of the Kubernetes secret that contains the SSL certificate to be used for secure connections to the object store. Rook will look in the secret provided at the cert key name. The value of the cert key must be in the format expected by the Object Gateway service: “The server key, server certificate, and any other CA or intermediate certificates be supplied in one file. Each of these items must be in pem form.”
port: The port on which the Object service will be reachable. If host networking is enabled, the Object Gateway daemons will also listen on that port. If running on SDN, the Object Gateway daemon listening port will be 8080 internally.
securePort: The secure port on which Object Gateway pods will be listening. An SSL certificate must be specified.
instances: The number of pods that will be started to load-balance this object store.
externalRgwEndpoints: A list of IP addresses to connect to external existing Object Gateways (works with external mode). This setting will be ignored if the CephCluster does not have external spec enabled.
annotations: Key-value pair list of annotations to add.
labels: Key-value pair list of labels to add.
placement: The Kubernetes placement settings to determine where the Object Gateway pods should be started in the cluster.
resources: Set resource requests/limits for the Gateway Pod(s).
priorityClassName: Set priority class name for the Gateway Pod(s).

Example of external Object Gateway endpoints to connect to:

  gateway:
    port: 80
    externalRgwEndpoints:
      - ip: 192.168.39.182

This will create a service with the endpoint 192.168.39.182 on port 80, pointing to the Ceph object external gateway. All the other settings from the gateway section will be ignored, except for securePort.

13.2.4 Zone settings #

The zone settings allow the object store to join custom created ceph-object-zone.

name: the name of the ceph-object-zone the object store will be in.

13.2.5 Runtime settings #

13.2.5.1 MIME types #

Rook provides a default mime.types file for each Ceph Object Storage. This file is stored in a Kubernetes ConfigMap with the name rook-ceph-rgw-<STORE-NAME>-mime-types. For most users, the default file should suffice, however, the option is available to users to edit the mime.types file in the ConfigMap as they desire. Users may have their own special file types, and particularly security conscious users may wish to pare down the file to reduce the possibility of a file type execution attack.

Rook will not overwrite an existing mime.types ConfigMap so that user modifications will not be destroyed. If the object store is destroyed and re-created, the ConfigMap will also be destroyed and re-created.

13.2.6 Health settings #

Rook-Ceph will be default monitor the state of the object store endpoints. The following CRD settings are available:

healthCheck: main object store health monitoring section

For example:

  healthCheck:
    bucket:
      disabled: false
      interval: 60s

The endpoint health check procedure is the following:

Create an S3 user.
Create a bucket with that user.
PUT the file in the object store.
GET the file from the object store.
Verify object consistency.
Update CR health status check.

Rook-Ceph always keeps the bucket and the user for the health check; it just does a PUT and GET of an S3 object, since creating a bucket is an expensive operation.

13.3 Ceph object bucket claim #

Rook supports the creation of new buckets and access to existing buckets via two custom resources:

An Object Bucket Claim (OBC) is custom resource which requests a bucket (new or existing) and is described by a Custom Resource Definition (CRD) shown below.
An Object Bucket (OB) is a custom resource automatically generated when a bucket is provisioned. It is a global resource, typically not visible to non-admin users, and contains information specific to the bucket. It is described by an OB CRD, also shown below.

An OBC references a storage class which is created by an administrator. The storage class defines whether the bucket requested is a new bucket or an existing bucket. It also defines the bucket retention policy. Users request a new or existing bucket by creating an OBC which is shown below. The ceph provisioner detects the OBC and creates a new bucket or grants access to an existing bucket, depending the the storage class referenced in the OBC. It also generates a Secret which provides credentials to access the bucket, and a ConfigMap which contains the bucket’s endpoint. Application pods consume the information in the Secret and ConfigMap to access the bucket. Please note that to make provisioner watch the cluster namespace only you need to set ROOK_OBC_WATCH_OPERATOR_NAMESPACE to true in the operator manifest, otherwise it watches all namespaces.

13.3.1 Sample #

13.3.1.1 OBC custom resource #

  apiVersion: objectbucket.io/v1alpha1
  kind: ObjectBucketClaim
  metadata:
    name: ceph-bucket [1]
    namespace: rook-ceph [2]
  spec:
    bucketName: [3]
    generateBucketName: photo-booth [4]
    storageClassName: rook-ceph-bucket [4]
    additionalConfig: [5]
      maxObjects: "1000"
      maxSize: "2G"

name of the ObjectBucketClaim. This name becomes the name of the Secret and ConfigMap.
namespace(optional) of the ObjectBucketClaim, which is also the namespace of the ConfigMap and Secret.
bucketName name of the bucket. Not recommended for new buckets, since names must be unique within an entire object store.
generateBucketName value becomes the prefix for a randomly-generated name; if supplied, then bucketName must be empty. If both bucketName and generateBucketName are supplied, then BucketName has precedence and GenerateBucketName is ignored. If both bucketName and generateBucketName are blank or omitted, then the storage class is expected to contain the name of an existing bucket. It is an error if all three bucket-related names are blank or omitted.
storageClassName which defines the StorageClass which contains the names of the bucket provisioner, the object store, and specifies the bucket-retention policy.
additionalConfig is an optional list of key-value pairs used to define attributes specific to the bucket being provisioned by this OBC. This information is typically tuned to a particular bucket provisioner, and may limit application portability. Options supported:
- maxObjects: The maximum number of objects in the bucket
- maxSize: The maximum size of the bucket, please note minimum recommended value is 4K.

13.3.1.2 OBC custom resource after bucket provisioning #

  apiVersion: objectbucket.io/v1alpha1
  kind: ObjectBucketClaim
  metadata:
    creationTimestamp: "2019-10-18T09:54:01Z"
    generation: 2
    name: ceph-bucket
    namespace: default [1]
    resourceVersion: "559491"
  spec:
    ObjectBucketName: obc-default-ceph-bucket [2]
    additionalConfig: null
    bucketName: photo-booth-c1178d61-1517-431f-8408-ec4c9fa50bee [3]
    cannedBucketAcl: ""
    ssl: false
    storageClassName: rook-ceph-bucket [4]
    versioned: false
  status:
    Phase: bound [5]

namespace where OBC got created.
ObjectBucketName generated OB name created using name space and OBC name.
The generated (in this case), unique bucket name for the new bucket.
Name of the storage class from OBC got created.
Phases of bucket creation:
- Pending: the operator is processing the request.
- Bound: the operator finished processing the request and linked the OBC and OB
- Released: the OB has been deleted, leaving the OBC unclaimed but unavailable.
- Failed: not currently set.

13.3.1.3 App pod #

  apiVersion: v1
  kind: Pod
  metadata:
    name: app-pod
    namespace: dev-user
  spec:
    containers:
    - name: mycontainer
      image: redis
      envFrom: [1]
      - configMapRef:
          name: ceph-bucket [2]
      - secretRef:
          name: ceph-bucket [3]

Use env: if mapping of the defined key names to the environment-variable names used by the app is needed.
Makes available to the pod as environment variables: BUCKET_HOST, BUCKET_PORT, BUCKET_NAME
makes available to the pod as environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

13.3.1.4 `StorageClass` #

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: rook-ceph-bucket
    labels:
      aws-s3/object [1]
  provisioner: rook-ceph.ceph.rook.io/bucket [2]
  parameters: [3]
    objectStoreName: my-store
    objectStoreNamespace: rook-ceph
    region: us-west-1
    bucketName: ceph-bucket [4]
  reclaimPolicy: Delete [5]

label(optional) here associates this StorageClass to a specific provisioner.
provisioner responsible for handling OBCs referencing this StorageClass.
all parameter required.
bucketName is required for access to existing buckets but is omitted when provisioning new buckets. Unlike greenfield provisioning, the brownfield bucket name appears in the StorageClass, not the OBC.
Rook-Ceph provisioner decides how to treat the reclaimPolicy when an OBC is deleted for the bucket.

Delete = physically delete the bucket.
Retain = do not physically delete the bucket.

13.4 Ceph Object Storage user custom resource definitions (CRD) #

Rook allows creation and customization of object store users through the custom resource definitions (CRDs). The following settings are available for Ceph object store users.

13.4.1 Sample #

  apiVersion: ceph.rook.io/v1
  kind: CephObjectStoreUser
  metadata:
    name: my-user
    namespace: rook-ceph
  spec:
    store: my-store
    displayName: my-display-name

13.4.2 Object Storage user settings #

13.4.2.1 Metadata #

name: The name of the object store user to create, which will be reflected in the secret and other resource names.
namespace: The namespace of the Rook cluster where the object store user is created.

13.4.2.2 Specification #

store: The object store in which the user will be created. This matches the name of the Object Storage CRD.
displayName: The display name which will be passed to the radosgw-admin user create command.

13 Object Storage #

13.1 Object Storage #

13.1.1 Configuring the Object Storage #

13.1.1.1 Creating a local Object Storage #

13.1.1.2 Connecting to an external Object Storage #

13.1.2 Creating a bucket #

13.1.2.1 Client connections #

13.1.3 Consuming the Object Storage #

13.1.3.1 Setting environment variables #

13.1.3.2 Installing the s3cmd package #

13.1.3.3 PUT or GET an object #

13.1.4 Setting up external access to the cluster #

13.1.5 Creating a user #

13.2 Ceph Object Storage CRD #

13.2.1 Sample #

13.2.1.1 Erasure code #

13.2.2 Object store settings #

13.2.2.1 Metadata #

13.2.2.2 Pools #

13.2.3 Creating gateway settings #

13.2.4 Zone settings #

13.2.5 Runtime settings #

13.2.5.1 MIME types #

13.2.6 Health settings #

13.3 Ceph object bucket claim #

13.3.1 Sample #

13.3.1.1 OBC custom resource #

13.3.1.2 OBC custom resource after bucket provisioning #

13.3.1.3 App pod #

13.3.1.4 StorageClass #

13.4 Ceph Object Storage user custom resource definitions (CRD) #

13.4.1 Sample #

13.4.2 Object Storage user settings #

13.4.2.1 Metadata #

13.4.2.2 Specification #

13.3.1.4 `StorageClass` #