Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Cloud Application Platform 1.5.2

10 Deploying SUSE Cloud Application Platform on Amazon Elastic Kubernetes Service (EKS)

Important
Important
README first!

Before you start deploying SUSE Cloud Application Platform, review the following documents:

Read the Release Notes: Release Notes SUSE Cloud Application Platform

Read Chapter 3, Deployment and Administration Notes

This chapter describes how to deploy SUSE Cloud Application Platform on Amazon Elastic Kubernetes Service (EKS), using Amazon's Elastic Load Balancer to provide fault-tolerant access to your cluster.

10.1 Prerequisites

The following are required to deploy and use SUSE Cloud Application Platform on EKS:

Important
Important

The prerequisites and configurations described is this chapter only reflect the requirements for a minimal SUSE Cloud Application Platform deployment. For a more production-ready environment, consider incoporating the following features:

Stratos Web Console

For details, see Chapter 6, Installing the Stratos Web Console.

High Availability

For details, see Chapter 7, SUSE Cloud Application Platform High Availability.

LDAP Integration

For details, see Chapter 8, LDAP Integration.

External Log Server Integration

For details, see Chapter 24, Logging.

Managing Certificates

For details, see Chapter 25, Managing Certificates.

Other Features

Refer to the Administration Guide at Part III, “SUSE Cloud Application Platform Administration” for additional features.

10.2 IAM Requirements for EKS

These IAM policies provide sufficient access to use EKS.

10.2.1 Unscoped Operations

Some of these permissions are very broad. They are difficult to scope effectively, in part because many resources are created and named dynamically when deploying an EKS cluster using the CloudFormation console. It may be helpful to enforce certain naming conventions, such as prefixing cluster names with ${aws:username} for pattern-matching in Conditions. However, this requires special consideration beyond the EKS deployment guide, and should be evaluated in the broader context of organizational IAM policies.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "UnscopedOperations",
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateUploadBucket",
                "cloudformation:EstimateTemplateCost",
                "cloudformation:ListExports",
                "cloudformation:ListStacks",
                "cloudformation:ListImports",
                "cloudformation:DescribeAccountLimits",
                "eks:ListClusters",
                "cloudformation:ValidateTemplate",
                "cloudformation:GetTemplateSummary",
                "eks:CreateCluster"
            ],
            "Resource": "*"
        },
        {
            "Sid": "EffectivelyUnscopedOperations",
            "Effect": "Allow",
            "Action": [
                "iam:CreateInstanceProfile",
                "iam:DeleteInstanceProfile",
                "iam:GetRole",
                "iam:DetachRolePolicy",
                "iam:RemoveRoleFromInstanceProfile",
                "cloudformation:*",
                "iam:CreateRole",
                "iam:DeleteRole",
                "eks:*"
            ],
            "Resource": [
                "arn:aws:eks:*:*:cluster/*",
                "arn:aws:cloudformation:*:*:stack/*/*",
                "arn:aws:cloudformation:*:*:stackset/*:*",
                "arn:aws:iam::*:instance-profile/*",
                "arn:aws:iam::*:role/*"
            ]
        }
    ]
}

10.2.2 Scoped Operations

These policies deal with sensitive access controls, such as passing roles and attaching/detaching policies from roles.

This policy, as written, allows unrestricted use of only customer-managed policies, and not Amazon-managed policies. This prevents potential security holes such as attaching the IAMFullAccess policy to a role. If you are using roles in a way that would be undermined by this, you should strongly consider integrating a Permissions Boundary before using this policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "UseCustomPoliciesWithCustomRoles",
            "Effect": "Allow",
            "Action": [
                "iam:DetachRolePolicy",
                "iam:AttachRolePolicy"
            ],
            "Resource": [
                "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/*",
                "arn:aws:iam::<YOUR_ACCOUNT_ID>:policy/*"
            ],
            "Condition": {
                "ForAllValues:ArnNotLike": {
                    "iam:PolicyARN": "arn:aws:iam::aws:policy/*"
                }
            }
        },
        {
            "Sid": "AllowPassingRoles",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/*"
        },
        {
            "Sid": "AddCustomRolesToInstanceProfiles",
            "Effect": "Allow",
            "Action": "iam:AddRoleToInstanceProfile",
            "Resource": "arn:aws:iam::<YOUR_ACCOUNT_ID>:instance-profile/*"
        },
        {
            "Sid": "AssumeServiceRoleForEKS",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<EKS_SERVICE_ROLE_NAME>"
        },
        {
            "Sid": "DenyUsingAmazonManagedPoliciesUnlessNeededForEKS",
            "Effect": "Deny",
            "Action": "iam:*",
            "Resource": "arn:aws:iam::aws:policy/*",
            "Condition": {
                "ArnNotEquals": {
                    "iam:PolicyARN": [
                        "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
                        "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
                        "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
                    ]
                }
            }
        },
        {
            "Sid": "AllowAttachingSpecificAmazonManagedPoliciesForEKS",
            "Effect": "Allow",
            "Action": [
                "iam:DetachRolePolicy",
                "iam:AttachRolePolicy"
            ],
            "Resource": "*",
            "Condition": {
                "ArnEquals": {
                    "iam:PolicyARN": [
                        "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
                        "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
                        "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
                    ]
                }
            }
        }
    ]
}

10.3 Install Helm Client and Tiller

Helm is a Kubernetes package manager. It consists of a client and server component, both of which are required in order to install and manage Cloud Application Platform.

The Helm client, helm, can be installed on your remote administration computer by referring to the documentation at https://v2.helm.sh/docs/using_helm/#installing-helm. Cloud Application Platform is compatible with both Helm 2 and Helm 3. Examples in this guide are based on Helm 2. To use Helm 3, refer to the Helm documentation at https://helm.sh/docs/.

Tiller, the Helm server component, needs to be installed on your Kubernetes cluster. Follow the instructions at https://v2.helm.sh/docs/using_helm/#installing-tiller to install Tiller with a service account and ensure your installation is appropriately secured according to your requirements as described in https://v2.helm.sh/docs/using_helm/#securing-your-helm-installation.

10.4 Default Storage Class

This example creates a simple storage class for your cluster in storage-class.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
allowVolumeExpansion: true

Then apply the new storage class configuration with this command:

tux > kubectl create --filename storage-class.yaml

10.5 DNS Configuration

Creation of the Elastic Load Balancer is triggered by a setting in the Cloud Application Platform deployment configuration file. First deploy uaa, then create CNAMEs for your domain and uaa subdomains (see the table below). Then deploy scf, and create the appropriate scf CNAMEs. This is described in more detail in the deployment sections.

The following table lists the required domain and sub-domains, using example.com as the example domain:

Warning
Warning: Supported Domains

When selecting a domain, SUSE Cloud Application Platform expects DOMAIN to be either a subdomain or a root domain. Setting DOMAIN to a top-level domain, such suse, is not supported.

DomainsServices
uaa.example.comuaa-uaa-public
*.uaa.example.comuaa-uaa-public
example.comrouter-gorouter-public
*.example.comrouter-gorouter-public
tcp.example.comtcp-router-tcp-router-public
ssh.example.comdiego-ssh-ssh-proxy-public

A SUSE Cloud Application Platform cluster exposes these four services:

Kubernetes service descriptionsKubernetes service names
User Account and Authentication (uaa)uaa-uaa-public
Cloud Foundry (CF) TCP routing servicetcp-router-tcp-router-public
CF application SSH accessdiego-ssh-ssh-proxy-public
CF routerrouter-gorouter-public

uaa-uaa-public is in the uaa namespace, and the rest are in the scf namespace.

10.6 Deployment Configuration

Use this example scf-config-values.yaml as a template for your configuration.

Warning
Warning: Supported Domains

When selecting a domain, SUSE Cloud Application Platform expects DOMAIN to be either a subdomain or a root domain. Setting DOMAIN to a top-level domain, such suse, is not supported.

### example deployment configuration file
### scf-config-values.yaml

env:
  DOMAIN: example.com
  UAA_HOST: uaa.example.com
  UAA_PORT: 2793

  GARDEN_ROOTFS_DRIVER: overlay-xfs
  GARDEN_APPARMOR_PROFILE: ""

services:
  loadbalanced: true

kube:
  storage_class:
    # Change the value to the storage class you use
    persistent: "gp2"
    shared: "gp2"

  # The default registry images are fetched from
  registry:
    hostname: "registry.suse.com"
    username: ""
    password: ""
  organization: "cap"

secrets:
  # Create a very strong password for user 'admin'
  CLUSTER_ADMIN_PASSWORD: password

  # Create a very strong password, and protect it because it
  # provides root access to everything
  UAA_ADMIN_CLIENT_SECRET: password

Take note of the following Helm values when defining your scf-config-values.yaml file.

GARDEN_ROOTFS_DRIVER

For SUSE® CaaS Platform and other Kubernetes deployments where the nodes are based on SUSE Linux Enterprise, the btrfs file system driver must be used. By default, btrfs is selected as the default.

For Microsoft AKS, Amazon EKS, Google GKE, and other Kubernetes deployments where the nodes are based on other operating systems, the overlay-xfs file system driver must be used.

Important
Important: Protect UAA_ADMIN_CLIENT_SECRET

The UAA_ADMIN_CLIENT_SECRET is the master password for access to your Cloud Application Platform cluster. Make this a very strong password, and protect it just as carefully as you would protect any root password.

10.7 Deploying Cloud Application Platform

The following list provides an overview of Helm commands to complete the deployment. Included are links to detailed descriptions.

  1. Download the SUSE Kubernetes charts repository (Section 10.8, “Add the Kubernetes Charts Repository”)

  2. Deploy uaa, then create appropriate CNAMEs (Section 10.9, “Deploy uaa)

  3. Copy the uaa secret and certificate to the scf namespace, deploy scf, create CNAMES (Section 10.10, “Deploy scf)

10.8 Add the Kubernetes Charts Repository

Download the SUSE Kubernetes charts repository with Helm:

tux > helm repo add suse https://kubernetes-charts.suse.com/

You may replace the example suse name with any name. Verify with helm:

tux > helm repo list
NAME            URL
stable          https://kubernetes-charts.storage.googleapis.com
local           http://127.0.0.1:8879/charts
suse            https://kubernetes-charts.suse.com/

List your chart names, as you will need these for some operations:

tux > helm search suse
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
suse/cf                         2.20.3          1.5.2           A Helm chart for SUSE Cloud Foundry
suse/cf-usb-sidecar-mysql       1.0.1                           A Helm chart for SUSE Universal Service Broker Sidecar fo...
suse/cf-usb-sidecar-postgres    1.0.1                           A Helm chart for SUSE Universal Service Broker Sidecar fo...
suse/console                    3.1.0           1.5.2           A Helm chart for deploying Stratos UI Console
suse/log-agent-rsyslog      	1.0.1        	8.39.0     	Log Agent for forwarding logs of K8s control pl...
suse/metrics                    1.1.2           1.5.2           A Helm chart for Stratos Metrics
suse/minibroker                 0.3.1                           A minibroker for your minikube
suse/nginx-ingress              0.28.4          0.15.0          An nginx Ingress controller that uses ConfigMap to store ...
suse/uaa                        2.20.3          1.5.2           A Helm chart for SUSE UAA

10.9 Deploy uaa

Note
Note: Embedded uaa in scf

The User Account and Authentication (uaa) Server is included as an optional feature of the scf Helm chart. This simplifies the Cloud Application Platform deployment process as a separate installation and/or upgrade of uaa is no longer a prerequisite to installing and/or upgrading scf.

It is important to note that:

  • This feature should only be used when uaa is not shared with other projects.

  • You cannot migrate from an existing external uaa to an embedded one. In this situation, enabling this feature during an upgrade will result in a single admin account.

To enable this feature, add the following to your scf-config-values.yaml.

enable:
  uaa: true

When deploying and/or upgrading scf, run helm install and/or helm upgrade and note that:

  • Installing and/or upgrading uaa using helm install suse/uaa ... and/or helm upgrade is no longer required.

  • It is no longer necessary to set the UAA_CA_CERT parameter. Previously, this parameter was passed the CA_CERT variable, which was assigned the CA certificate of uaa.

Use Helm to deploy the uaa (User Account and Authentication) server. You may create your own release --name:

tux > helm install suse/uaa \
--name susecf-uaa \
--namespace uaa \
--values scf-config-values.yaml

Wait until you have a successful uaa deployment before going to the next steps, which you can monitor with the watch command:

tux > watch --color 'kubectl get pods --namespace uaa'

When uaa is successfully deployed, the following is observed:

  • For the secret-generation pod, the STATUS is Completed and the READY column is at 0/1.

  • All other pods have a Running STATUS and a READY value of n/n.

Press CtrlC to exit the watch command.

When uaa is finished deploying, create CNAMEs for the required domains. (See Section 10.5, “DNS Configuration”.) Use kubectl to find the service host names. These host names include the elb sub-domain, so use this to get the correct results:

tux > kubectl get services --namespace uaa | grep elb

Use curl to verify you are able to connect to the uaa OAuth server on the DNS name configured:

tux > curl --insecure https://uaa.example.com:2793/.well-known/openid-configuration

This should return a JSON object, as this abbreviated example shows:

{"issuer":"https://uaa.example.com:2793/oauth/token",
"authorization_endpoint":"https://uaa.example.com:2793
/oauth/authorize","token_endpoint":"https://uaa.example.com:2793/oauth/token"

10.10 Deploy scf

Before deploying scf, ensure the DNS records for the uaa domains have been set up as specified in the previous section. Next, use Helm to deploy scf:

Note
Note: Setting UAA_CA_CERT

Starting with SUSE Cloud Application Platform 1.5.2, you no longer need to set UAA_CA_CERT when using an external UAA with a certificate signed by a well known Certificate Authority (CA). It is only needed when you use an external UAA with either a certificate generated by the secret-generator or a self-signed certificate.

If you need to set UAA_CA_CERT:

  1. Obtain your UAA secret and certificate:

    tux > SECRET=$(kubectl get pods --namespace uaa \
    --output jsonpath='{.items[?(.metadata.name=="uaa-0")].spec.containers[?(.name=="uaa")].env[?(.name=="INTERNAL_CA_CERT")].valueFrom.secretKeyRef.name}')
    
    tux > CA_CERT="$(kubectl get secret $SECRET --namespace uaa \
    --output jsonpath="{.data['internal-ca-cert']}" | base64 --decode -)"
  2. Then pass --set "secrets.UAA_CA_CERT=${CA_CERT}" as part of your helm command.

tux > helm install suse/cf \
--name susecf-scf \
--namespace scf \
--values scf-config-values.yaml

Wait until you have a successful scf deployment before going to the next steps, which you can monitor with the watch command:

tux > watch --color 'kubectl get pods --namespace scf'

When scf is successfully deployed, the following is observed:

  • For the secret-generation and post-deployment-setup pods, the STATUS is Completed and the READY column is at 0/1.

  • All other pods have a Running STATUS and a READY value of n/n.

Press CtrlC to exit the watch command.

Before traffic is routed to EKS load balancers, health checks are performed on them. These health checks require access to ports on the tcp-router service, which are only opened by request via cf map-route and other associated commands.

Use kubectl patch to patch the tcp-router service to expose a port to the EKS load balancer so the health checks can be completed.

tux > kubectl patch service tcp-router-tcp-router-public --namespace scf \
--type strategic \
--patch '{"spec": {"ports": [{"name": "healthcheck", "port": 8080}]}}'

Remove port 8080 from the load balancer's listeners list. This ensures the port is not exposed to external traffic while still allowing it to serve as the port for health checks internally.

tux > aws elb delete-load-balancer-listeners \
--load-balancer-name  a635338ecc2ed11e607790a6b408ef5f \
--load-balancer-ports 8080

Where the --load-balancer-name can be found by examining the EXTERNAL-IP field after running command below. The load balancer name will be the string preceding the first - character.

tux > kubectl get service -n scf tcp-router-tcp-router-public

The health checks should now operate correctly. When the status shows RUNNING for all of the scf pods, create CNAMEs for the required domains. (See Section 10.5, “DNS Configuration”) Use kubectl to find the service host names. These host names include the elb sub-domain, so use this to get the correct results:

tux > kubectl get services --namespace scf | grep elb

10.11 Deploying and Using the AWS Service Broker

The AWS Service Broker provides integration of native AWS services with SUSE Cloud Application Platform.

10.11.1 Prerequisites

Deploying and using the AWS Service Broker requires the following:

10.11.2 Setup

  1. Create the required DynamoDB table where the AWS service broker will store its data. This example creates a table named awssb:

    tux > aws dynamodb create-table \
    		--attribute-definitions \
    			AttributeName=id,AttributeType=S \
    			AttributeName=userid,AttributeType=S \
    			AttributeName=type,AttributeType=S \
    		--key-schema \
    			AttributeName=id,KeyType=HASH \
    			AttributeName=userid,KeyType=RANGE \
    		--global-secondary-indexes \
    			'IndexName=type-userid-index,KeySchema=[{AttributeName=type,KeyType=HASH},{AttributeName=userid,KeyType=RANGE}],Projection={ProjectionType=INCLUDE,NonKeyAttributes=[id,userid,type,locked]},ProvisionedThroughput={ReadCapacityUnits=5,WriteCapacityUnits=5}' \
    		--provisioned-throughput \
    			ReadCapacityUnits=5,WriteCapacityUnits=5 \
    		--region ${AWS_REGION} --table-name awssb
  2. Wait until the table has been created. When it is ready, the TableStatus will change to ACTIVE. Check the status using the describe-table command:

    aws dynamodb describe-table --table-name awssb

    (For more information about the describe-table command, see https://docs.aws.amazon.com/cli/latest/reference/dynamodb/describe-table.html.)

  3. Set a name for the Kubernetes namespace you will install the service broker to. This name will also be used in the service broker URL:

    tux > BROKER_NAMESPACE=aws-sb
  4. Create a server certificate for the service broker:

    1. Create and use a separate directory to avoid conflicts with other CA files:

      tux > mkdir /tmp/aws-service-broker-certificates && cd $_
    2. Get the CA certificate:

      kubectl get secret --namespace scf --output jsonpath='{.items[*].data.internal-ca-cert}' | base64 -di > ca.pem
    3. Get the CA private key:

      kubectl get secret --namespace scf --output jsonpath='{.items[*].data.internal-ca-cert-key}' | base64 -di > ca.key
    4. Create a signing request. Replace BROKER_NAMESPACE with the namespace assigned in Step 3:

      tux > openssl req -newkey rsa:4096 -keyout tls.key.encrypted -out tls.req -days 365 \
        -passout pass:1234 \
        -subj '/CN=aws-servicebroker.'${BROKER_NAMESPACE} -batch \
        -subj '/CN=aws-servicebroker-aws-servicebroker.aws-sb.svc.cluster.local' -batch
        </dev/null
    5. Decrypt the generated broker private key:

      tux > openssl rsa -in tls.key.encrypted -passin pass:1234 -out tls.key
    6. Sign the request with the CA certificate:

      tux > openssl x509 -req -CA ca.pem -CAkey ca.key -CAcreateserial -in tls.req -out tls.pem
  5. Install the AWS service broker as documented at https://github.com/awslabs/aws-servicebroker/blob/master/docs/getting-started-k8s.md. Skip the installation of the Kubernetes Service Catalog. While installing the AWS Service Broker, make sure to update the Helm chart version (the version as of this writing is 1.0.1). For the broker install, pass in a value indicating the Cluster Service Broker should not be installed (for example --set deployClusterServiceBroker=false). Ensure an account and role with adequate IAM rights is chosen (see Section 10.11.1, “Prerequisites”:

    tux > helm install aws-sb/aws-servicebroker \
    	     --name aws-servicebroker \
    	     --namespace $BROKER_NAMESPACE \
    	     --version 1.0.1 \
    	     --set aws.secretkey=$AWS_ACCESS_KEY \
    	     --set aws.accesskeyid=$AWS_KEY_ID \
    	     --set deployClusterServiceBroker=false \
    	     --set tls.cert="$(base64 -w0 tls.pem)" \
    	     --set tls.key="$(base64 -w0 tls.key)" \
    	     --set-string aws.targetaccountid=$AWS_TARGET_ACCOUNT_ID \
    	     --set aws.targetrolename=$AWS_TARGET_ROLE_NAME \
    	     --set aws.tablename=awssb \
    	     --set aws.vpcid=$VPC_ID \
    	     --set aws.region=$AWS_REGION \
    	     --set authenticate=false

    To find the values of aws.targetaccoundid, aws.targetrolename, and vpcId run the following command.

    tux > aws eks describe-cluster --name $CLUSTER_NAME

    For aws.targetaccoundid and aws.targetrolename , examine the cluster.roleArn field. For vpcId, refer to the cluster.resourcesVpcConfig.vpcId field.

  6. Log into your Cloud Application Platform deployment. Select an organization and space to work with, creating them if needed:

    tux > cf api --skip-ssl-validation https://api.example.com
    tux > cf login -u admin -p password
    tux > cf create-org org
    tux > cf create-space space
    tux > cf target -o org -s space
  7. Create a service broker in scf. Note the name of the service broker should be the same as the one specified for the --name flag in the helm install step (for example aws-servicebroker. Note that the username and password parameters are only used as dummy values to pass to the cf command:

    tux > cf create-service-broker aws-servicebroker username password https://aws-servicebroker-aws-servicebroker.aws-sb.svc.cluster.local
  8. Verify the service broker has been registered:

    tux > cf service-brokers
  9. List the available service plans:

    tux > cf service-access
  10. Enable access to a service. This example uses the -p to enable access to a specific service plan. See https://github.com/awslabs/aws-servicebroker/blob/master/templates/rdsmysql/template.yaml for information about all available services and their associated plans:

    tux > cf enable-service-access rdsmysql -p custom
  11. Create a service instance. As an example, a custom MySQL instance can be created as:

    tux > cf create-service rdsmysql custom mysql-instance-name -c '{
      "AccessCidr": "192.0.2.24/32",
      "BackupRetentionPeriod": 0,
      "MasterUsername": "master",
      "DBInstanceClass": "db.t2.micro",
      "EngineVersion": "5.7.17",
      "PubliclyAccessible": "true",
      "region": "$AWS_REGION",
      "StorageEncrypted": "false",
      "VpcId": "$VPC_ID",
      "target_account_id": "$AWS_TARGET_ACCOUNT_ID",
      "target_role_name": "$AWS_TARGET_ROLE_NAME"
    }'

10.11.3 Cleanup

When the AWS Service Broker and its services are no longer required, perform the following steps:

  1. Unbind any applications using any service instances then delete the service instance:

    tux > cf unbind-service my_app mysql-instance-name
    tux > cf delete-service mysql-instance-name
  2. Delete the service broker in scf:

    tux > cf delete-service-broker aws-servicebroker
  3. Delete the deployed Helm chart and the namespace:

    tux > helm delete --purge aws-servicebroker
    tux > kubectl delete namespace ${BROKER_NAMESPACE}
  4. The manually created DynamoDB table will need to be deleted as well:

    tux > aws dynamodb delete-table --table-name awssb --region ${AWS_REGION}

10.12 Resizing Persistent Volumes

Depending on your workloads, the default persistent volume (PV) sizes of your Cloud Application Platform deployment may be insufficient. This section describes the process to resize a persistent volume in your Cloud Application Platform deployment, by modifying the persistent volumes claim (PVC) object.

Note that PVs can only be expanded, but cannot be shrunk. shrunk.

10.12.1 Prerequisites

The following are required in order to use the process below to resize a PV.

10.12.2 Example Procedure

The following describes the process required to resize a PV, using the PV and PVC associated with uaa's mysql as an example.

  1. Find the storage class and PVC associated with the PV being expanded. In This example, the storage class is called persistent and the PVC is called mysql-data-mysql-0.

    tux > kubectl get persistentvolume
  2. Verify whether the storage class has allowVolumeExpansion set to true. If it does not, run the following command to update the storage class.

    tux > kubectl get storageclass persistent --output json

    If it does not, run the below command to update the storage class.

    tux > kubectl patch storageclass persistent \
    --patch '{"allowVolumeExpansion": true}'
  3. Cordon all nodes in your cluster.

    1. tux > export VM_NODES=$(kubectl get nodes -o name)
    2. tux > for i in $VM_NODES
       do
        kubectl cordon `echo "${i//node\/}"`
      done
  4. Increase the storage size of the PVC object associated with the PV being expanded.

    tux > kubectl patch persistentvolumeclaim --namespace uaa mysql-data-mysql-0 \
    --patch '{"spec": {"resources": {"requests": {"storage": "25Gi"}}}}'
  5. List all pods that use the PVC, in any namespace.

    tux > kubectl get pods --all-namespaces --output=json | jq -c '.items[] | {name: .metadata.name, namespace: .metadata.namespace, claimName: .spec |  select( has ("volumes") ).volumes[] | select( has ("persistentVolumeClaim") ).persistentVolumeClaim.claimName }'
  6. Restart all pods that use the PVC.

    tux > kubectl delete pod mysql-0 --namespace uaa
  7. Run kubectl describe persistentvolumeclaim and monitor the status.conditions field.

    tux > watch 'kubectl get persistentvolumeclaim --namespace uaa mysql-data-mysql-0 --output json'

    When the following is observed, press CtrlC to exit the watch command and proceed to the next step.

    • status.conditions.message is

      message: Waiting for user to (re-)start a pod to finish file system resize of volume on node.
    • status.conditions.type is

      type: FileSystemResizePending
  8. Uncordon all nodes in your cluster.

    tux > for i in $VM_NODES
     do
      kubectl uncordon `echo "${i//node\/}"`
    done
  9. Wait for the resize to finish. Verify the storage size values match for status.capacity.storage and spec.resources.requests.storage.

    tux > watch 'kubectl get persistentvolumeclaim --namespace uaa mysql-data-mysql-0 --output json'
  10. Also verify the storage size in the pod itself is updated.

    tux > kubectl --namespace uaa exec mysql-0 -- df --human-readable