Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Cloud Application Platform 2.1.1

21 Backup and Restore

21.1 Backup and Restore Using cf-plugin-backup

cf-plugin-backup backs up and restores your Cloud Controller Database (CCDB), using the Cloud Foundry command line interface (cf CLI). (See Section 26.1, “Using the cf CLI with SUSE Cloud Application Platform”.)

cf-plugin-backup is not a general-purpose backup and restore plugin. It is designed to save the state of a KubeCF instance before making changes to it. If the changes cause problems, use cf-plugin-backup to restore the instance from scratch. Do not use it to restore to a non-pristine KubeCF instance. Some of the limitations for applying the backup to a non-pristine KubeCF instance are:

  • Application configuration is not restored to running applications, as the plugin does not have the ability to determine which applications should be restarted to load the restored configurations.

  • User information is managed by the User Account and Authentication (uaa) Server, not the Cloud Controller (CC). As the plugin talks only to the CC it cannot save full user information, nor restore users. Saving and restoring users must be performed separately, and user restoration must be performed before the backup plugin is invoked.

  • The set of available stacks is part of the KubeCF instance setup, and is not part of the CC configuration. Trying to restore applications using stacks not available on the target KubeCF instance will fail. Setting up the necessary stacks must be performed separately before the backup plugin is invoked.

  • Buildpacks are not saved. Applications using custom buildpacks not available on the target KubeCF instance will not be restored. Custom buildpacks must be managed separately, and relevant buildpacks must be in place before the affected applications are restored.

21.1.1 Installing the cf-plugin-backup

Download the plugin from https://github.com/SUSE/cf-plugin-backup/releases.

Then install it with cf, using the name of the plugin binary that you downloaded:

tux > cf install-plugin cf-plugin-backup-1.0.8.0.g9e8438e.linux-amd64
 Attention: Plugins are binaries written by potentially untrusted authors.
 Install and use plugins at your own risk.
 Do you want to install the plugin
 backup-plugin/cf-plugin-backup-1.0.8.0.g9e8438e.linux-amd64? [yN]: y
 Installing plugin backup...
 OK
 Plugin backup 1.0.8 successfully installed.

Verify installation by listing installed plugins:

tux > cf plugins
 Listing installed plugins...

 plugin   version   command name      command help
 backup   1.0.8     backup-info       Show information about the current snapshot
 backup   1.0.8     backup-restore    Restore the CloudFoundry state from a
  backup created with the snapshot command
 backup   1.0.8     backup-snapshot   Create a new CloudFoundry backup snapshot
  to a local file

 Use 'cf repo-plugins' to list plugins in registered repos available to install.

21.1.2 Using cf-plugin-backup

The plugin has three commands:

  • backup-info

  • backup-snapshot

  • backup-restore

View the online help for any command, like this example:

tux >  cf backup-info --help
 NAME:
   backup-info - Show information about the current snapshot

 USAGE:
   cf backup-info

Create a backup of your SUSE Cloud Application Platform data and applications. The command outputs progress messages until it is completed:

tux > cf backup-snapshot
 2018/08/18 12:48:27 Retrieving resource /v2/quota_definitions
 2018/08/18 12:48:30 org quota definitions done
 2018/08/18 12:48:30 Retrieving resource /v2/space_quota_definitions
 2018/08/18 12:48:32 space quota definitions done
 2018/08/18 12:48:32 Retrieving resource /v2/organizations
 [...]

Your Cloud Application Platform data is saved in the current directory in cf-backup.json, and application data in the app-bits/ directory.

View the current backup:

tux > cf backup-info
 - Org  system

Restore from backup:

tux > cf backup-restore

There are two additional restore options: --include-security-groups and --include-quota-definitions.

21.1.3 Scope of Backup

The following table lists the scope of the cf-plugin-backup backup. Organization and space users are backed up at the SUSE Cloud Application Platform level. The user account in uaa/LDAP, the service instances and their application bindings, and buildpacks are not backed up. The sections following the table goes into more detail.

ScopeRestore
OrgsYes
Org auditorsYes
Org billing-managerYes
Quota definitionsOptional
SpacesYes
Space developersYes
Space auditorsYes
Space managersYes
AppsYes
App binariesYes
RoutesYes
Route mappingsYes
DomainsYes
Private domainsYes
Stacksnot available
Feature flagsYes
Security groupsOptional
Custom buildpacksNo

cf backup-info reads the cf-backup.json snapshot file found in the current working directory, and reports summary statistics on the content.

cf backup-snapshot extracts and saves the following information from the CC into a cf-backup.json snapshot file. Note that it does not save user information, but only the references needed for the roles. The full user information is handled by the uaa server, and the plugin talks only to the CC. The following list provides a summary of what each plugin command does.

  • Org Quota Definitions

  • Space Quota Definitions

  • Shared Domains

  • Security Groups

  • Feature Flags

  • Application droplets (zip files holding the staged app)

  • Orgs

    • Spaces

      • Applications

      • Users' references (role in the space)

cf backup-restore reads the cf-backup.json snapshot file found in the current working directory, and then talks to the targeted KubeCF instance to upload the following information, in the specified order:

  • Shared domains

  • Feature flags

  • Quota Definitions (iff --include-quota-definitions)

  • Orgs

    • Space Quotas (iff --include-quota-definitions)

    • UserRoles

    • (private) Domains

    • Spaces

      • UserRoles

      • Applications (+ droplet)

        • Bound Routes

      • Security Groups (iff --include-security-groups)

The following list provides more details of each action.

Shared Domains

Attempts to create domains from the backup. Existing domains are retained, and not overwritten.

Feature Flags

Attempts to update flags from the backup.

Quota Definitions

Existing quotas are overwritten from the backup (deleted, re-created).

Orgs

Attempts to create orgs from the backup. Attempts to update existing orgs from the backup.

Space Quota Definitions

Existing quotas are overwritten from the backup (deleted, re-created).

User roles

Expect the referenced user to exist. Will fail when the user is already associated with the space, in the given role.

(private) Domains

Attempts to create domains from the backup. Existing domains are retained, and not overwritten.

Spaces

Attempts to create spaces from the backup. Attempts to update existing spaces from the backup.

User roles

Expect the referenced user to exist. Will fail when the user is already associated with the space, in the given role.

Apps

Attempts to create apps from the backup. Attempts to update existing apps from the backup (memory, instances, buildpack, state, ...)

Security groups

Existing groups are overwritten from the backup

21.2 Disaster Recovery through Raw Data Backup and Restore

An existing SUSE Cloud Application Platform deployment's data can be migrated to a new SUSE Cloud Application Platform deployment through a backup and restore of its raw data. The process involves performing a backup and restore of the kubecf components respectively. This procedure is agnostic of the underlying Kubernetes infrastructure and can be included as part of your disaster recovery solution.

21.2.1 Prerequisites

In order to complete a raw data backup and restore, the following are required:

21.2.2 Scope of Raw Data Backup and Restore

The following lists the data that is included as part of the backup (and restore) procedure:

21.2.3 Performing a Raw Data Backup

Note
Note: Restore to the Same Version

This process is intended for backing up and restoring to a target deployment with the same version as the source deployment. For example, data from a backup of SUSE Cloud Application Platform 2.1.1 should be restored to a SUSE Cloud Application Platform 2.1.1 deployment.

Perform the following steps to create a backup of your source SUSE Cloud Application Platform deployment.

  1. Export the blobstore into a file.

    tux > kubectl exec --namespace kubecf singleton-blobstore-0 --
    tar cfz - --exclude=/var/vcap/store/shared/tmp /var/vcap/store/shared >
    blob.tgz
  2. The current UAA database configuration does not allow exporting of a mysqldump, so need to be more permissive.

    tux > cat <<EOF | kubectl exec --stdin database-0 --namespace kubecf
    -- mysql
    SET GLOBAL pxc_strict_mode=PERMISSIVE;
    SET GLOBAL
    sql_mode='STRICT_ALL_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
    set GLOBAL innodb_strict_mode='OFF';
    EOF
  3. Export the UAA database into a file.

    tux > kubectl exec --stdin database-0 --namespace kubecf --
    mysqldump uaa > uaadb-src.sql
  4. Export the Cloud Controller Database (CCDB) into a file.

    tux > kubectl exec --stdin database-0 --namespace kubecf --
    mysqldump cloud_controller > ccdb-src.sql
  5. Save the CCDB encryption key(s). Adjust the A flag as needed to include all keys.

    tux > kubectl exec --stdin --tty --namespace kubecf api-0 -- bash
    -c "cat /var/vcap/jobs/cloud_controller_ng/config/cloud_controller_ng.yml | grep
    -A 10 db_encryption" > enc_key

21.2.4 Performing a Raw Data Restore

Important
Important: Ensure Access to the Correct Deployment

Working with multiple Kubernetes clusters simultaneously can be confusing. Ensure you are communicating with the desired cluster by setting $KUBECONFIG correctly.

Perform the following steps to restore your backed up data to the target SUSE Cloud Application Platform deployment.

  1. Deploy the target SUSE Cloud Application Platform cluster following the steps for your platform.

  2. The current UAA database configuration does not allow importing of a mysqldump, so needs to be made more permissive.

    tux > cat <<EOF | kubectl exec --stdin database-0 --namespace kubecf
    -- mysql
    SET GLOBAL pxc_strict_mode=PERMISSIVE;
    SET GLOBAL
    sql_mode='STRICT_ALL_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
    set GLOBAL innodb_strict_mode='OFF';
    EOF
  3. Import the UAA database.

    tux > kubectl exec --stdin database-0 --namespace kubecf -- mysql
    uaa < uaadb-src.sql

    Verify the import is successful. The output should list the users from the deployment the backup was taken from.

    tux > echo "select username from uaa.users;" | kubectl exec -i
    database-0 --namespace kubecf -- mysql
  4. Import the blobstore and restart the pod for changes to take affect.

    tux > kubectl exec --stdin singleton-blobstore-0 --namespace kubecf -- tar xfz - -C
    / < blob.tgz
    
    tux > kubectl delete pod --namespace kubecf singleton-blobstore-0
  5. Drop the current CCDB and create a new instance.

    tux > echo "drop database cloud_controller; create database
    cloud_controller;" | \
         kubectl exec -i database-0 --namespace kubecf -- mysql
  6. Import the CCDB.

    tux > kubectl exec --stdin database-0 --namespace kubecf -- mysql
    cloud_controller < ccdb-src.sql
  7. Update the encryption key.

    1. Create a YAML configuration file containing the encryption key information. The file structure should look similar to the following example, called enc_key_values.yaml. Replace the example values using the values from the enc_key file generated earlier. Depending on the state of the cluster the encryption keys were retrieved from, the key labels may differ and not be encryption_key_0.

      ccdb:
        encryption:
          rotation:
            key_labels:
            - encryption_key_0
            current_key_label: encryption_key_0
      
      credentials:
        cc_db_encryption_key:
      elqdi7TARO6NYELa9cUr6WwMYIvqaG4U0nMyfL1loDYi02C1Rrneov6fxxfd64je
        ccdb_key_label_encryption_key_0:
      tPhZZbMNYVWKs0II8e8pMxsJMokeReUrJAnQNdLaXEheTZVv5OpMe7vdyThhrkEP

      In the above, the key credentials.ccdb_key_label_encryption_key_0 is based on the generic form credentials.ccdb_key_label_XYZ. The XYZ should be replaced with the value of the current_key_label.

      For example, if the current_key_label is new_key, then credentials.ccdb_key_label_new_key should be used.

    2. Perform a helm upgrade for the changes to take affect.

      tux > helm upgrade kubecf suse/kubecf
      \
      --namespace kubecf \
      --values kubecf-config-values.yaml \
      --values enc_key_values.yaml \
      --version 2.7.13
  8. When all pods are fully running, verify the restore is successful. Example commands to run include cf apps, cf marketplace, or cf services.

Print this page