Verrazzano Enterprise Container Platform – Backup and Recovery

Docs: Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides Velero and rancher-backup for backup and recovery at the component and platform level. Use the following instructions to enable and configure these components in your environment.

NOTE: The backup functionality for OpenSearch can be used only if the components are enabled explicitly in the Verrazzano CR.

Enable backup components

To back up and restore persistent data, first you must enable the velero and rancherBackup components. The following configuration shows how to enable the backup components with a prod installation profile.

apiVersion: install.verrazzano.io/v1beta1
kind: Verrazzano
metadata:
  name: example-verrazzano
spec:
  profile: prod
  components:    
    velero:
      enabled: true
    rancherBackup:
      enabled: true

NOTE: rancherBackup will be enabled only in cases when rancher is also enabled.

After they’re enabled, check for Velero pods running in the verrazzano-backup namespace.

# Sample of pods running after enabling the velero component

$ kubectl get pod -n verrazzano-backup
NAME                      READY   STATUS    RESTARTS   AGE
restic-ndxfk              1/1     Running   0          21h
velero-5ff8766fd4-xbn4z   1/1     Running   0          21h

For rancher-backup, the pods will be created in the cattle-resources-system namespace.

# Sample of pods running after enabling the rancherBackup component

$ kubectl get pod -n cattle-resources-system
NAME                              READY   STATUS    RESTARTS   AGE
rancher-backup-5c4b985697-xw7md   1/1     Running   0          2d4h

Configure backup components

Next, meet the following prerequisite requirements for both velero and rancherBackup components:

Object store bucket name.
- Both components require an object store that is Amazon S3 compatible, therefore, you need to have an object storage bucket. This can be an Oracle Cloud Object Storage bucket in any compartment of your Oracle Cloud tenancy.
  - Make a note of the bucket name and tenancy name for reference.
  - For more information about creating a bucket with Object Storage, refer to this page.
- For private clouds, enterprise networks, or air-gapped environments, this could be MinIO or an equivalent object store solution.
Object store prefix name. This will be a child folder under the bucket, which the backup component creates.
Object store region information.
A signing key, which is required to authenticate with the Amazon S3 compatible object store. Follow these steps to create a Customer Secret Key.

Velero CLI (optional)

The Velero CLI helps you access Velero objects in a more descriptive manner; you can also manage them using kubectl.

If desired, install the Velero CLI on Oracle Linux as follows:

$ rpm -ivh https://yum.oracle.com/repo/OracleLinux/OL7/developer/olcne/x86_64/getPackage/velero-1.8.1-1.el7.x86_64.rpm

Component-specific prerequisites

Meet the following component-specific prerequisites:

Velero operator prerequisites

Now, create the following objects:

Create a backup-secret.txt file, which has the object store credentials.

[default]
aws_access_key_id=<object store access key>
aws_secret_access_key=<object store secret key>

In the namespace verrazzano-backup, create a Kubernetes secret verrazzano-backup-creds.

$ kubectl create secret generic -n <backup-namespace> <secret-name> --from-file=<key>=<full_path_to_creds_file>

Example

$ kubectl create secret generic -n verrazzano-backup verrazzano-backup-creds --from-file=cloud=backup-secret.txt

NOTE: To avoid misuse of sensitive data, ensure that the backup-secret.txt file is deleted after the Kubernetes secret is created.

Create BackupStorageLocation, which the backup component will reference for subsequent backups. See the following BackupStorageLocation example. For more information, see here.

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: verrazzano-backup-location
  namespace: verrazzano-backup
spec:
  provider: aws
  objectStorage:
    bucket: example-verrazzano
    prefix: backup-demo
  credential:
    name: verrazzano-backup-creds
    key: cloud
  config:
    region: us-phoenix-1
    s3ForcePathStyle: "true"
    s3Url: https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com

rancher-backup operator prerequisites

Now, in the namespace verrazzano-backup, create a Kubernetes secret rancher-backup-creds.

$ kubectl create secret generic -n <backup-namespace> <secret-name> --from-literal=accessKey=<accesskey> --from-literal=secretKey=<secretKey>

Example

$ kubectl create secret generic -n verrazzano-backup rancher-backup-creds --from-literal=accessKey="s5VLpXwa0xNZQds4UTVV" --from-literal=secretKey="nFFpvyxpQvb0dIQovsl0"

Docs: Backup

Mon, 01 Jan 0001 00:00:00 +0000

Verrazzano provides Velero and rancher-backup for backup and recovery at the component and platform level. First, ensure that the backup component prerequisites are met, as indicated here.

The following sections provide detailed configuration information for:

The rancher-backup operator, to back up persistent data and Rancher-related configuration. See Rancher backup.
Velero hooks, to ensure a consistent backup experience for these components:
- OpenSearch. See OpenSearch backup.
- For all other components, refer to the Velero Backup Reference.

Rancher backup

To initiate a Rancher backup, create the following example custom resource YAML file that uses an Amazon S3 compatible object store as a backend. The operator uses the credentialSecretNamespace value to determine where to look for the Amazon S3 backup secret. Note that in the prerequisites example, you previously created the secret in the verrazzano-backup namespace.

apiVersion: resources.cattle.io/v1
kind: Backup
metadata:
  name: rancher-backup-test
spec:
  storageLocation:
    s3:
      credentialSecretName: rancher-backup-creds
      credentialSecretNamespace: verrazzano-backup
      bucketName: myvz-bucket
      folder: rancher-backup
      region: us-phoenix-1
      endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com
  resourceSetName: rancher-resource-set

The operator creates the backup file, in the *.tar.gz format, and stores it in the location configured in the storageLocation field.

Scheduled backups

Similar to Velero, rancher-backup allows scheduled backups.

OpenSearch backup

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero while invoking a backup. Due to the nature of transient data handled by OpenSearch, the hook invokes OpenSearch snapshot APIs to back up and restore data streams appropriately, thereby ensuring that there is no loss of data and avoids data corruption as well.

The following example shows a sample Velero Backup API object that you can invoke to make an OpenSearch backup.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: verrazzano-opensearch-backup
  namespace: verrazzano-backup
spec:
  includedNamespaces:
    - verrazzano-system
  labelSelector:
    matchLabels:
      verrazzano-component: opensearch
  defaultVolumesToRestic: false
  storageLocation:  verrazzano-backup-location
  hooks:
    resources:
      - name: opensearch-backup-test
        includedNamespaces:
          - verrazzano-system
        labelSelector:
          matchLabels:
            statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
        post:                           
          - exec:
              container: es-master
              command:
                - /usr/share/opensearch/bin/verrazzano-backup-hook
                - -operation
                - backup
                - -velero-backup-name
                - verrazzano-opensearch-backup
              onError: Fail
              timeout: 10m

The preceding example backs up the OpenSearch components:

In this case, you are not backing up the PersistentVolumes directly, rather executing a hook that invokes the OpenSearch APIs to take a snapshot of the data.
So that Velero ignores the associated PVC’s, defaultVolumesToRestic needs to be false.
In this case, the hook can be pre or post.
The command used in the hook requires an operation flag and the Velero backup name as an input.
The container on which the hook needs to be run is identified by the pod label selectors, followed by the container name. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.

After the backup is processed, you can see the hook logs using the velero backup logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch backup logs

# To display the logs from the backup, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-backup-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>

Scheduled backups

Velero supports a Schedule API that is a repeatable request that is sent to the Velero server to perform a backup for a given cron notation. After the Schedule object is created, the Velero server will start the backup process. Then, it will wait for the next valid point in the given cron expression and run the backup process on a repeating basis.

Docs: Restore

Mon, 01 Jan 0001 00:00:00 +0000

Before proceeding, ensure that the backup component prerequisites are met, as indicated here. This document also assumes that a successful backup was previously made using either Velero or rancher-backup, as shown here.

Use the following component-specific instructions to restore application data:

Rancher restore

To initiate a Rancher restore, create the following example custom resource YAML file. When a Restore custom resource is created, the operator accesses the backup *.tar.gz file specified and restores the application data from that file.

apiVersion: resources.cattle.io/v1
kind: Restore
metadata:
  name: s3-restore
spec:
  backupFilename: rancher-backup-test-1111111-2222-3333-2022-07-26T02-44-21Z.tar.gz
  storageLocation:
    s3:
      credentialSecretName: rancher-backup-creds
      credentialSecretNamespace: verrazzano-backup
      bucketName: myvz-bucket
      folder: rancher-backup
      region: us-phoenix-1
      endpoint: mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com

The rancher-backup operator scales down the Rancher deployment during the restore operation and scales it back up after the restoration completes.

Resources are restored in this order:

Custom Resource Definitions (CRDs)
Cluster-scoped resources
Namespace resources

OpenSearch restore

For OpenSearch, Verrazzano provides a custom hook that you can use along with Velero, to perform a restore operation. Due to the nature of transient data handled by OpenSearch, the hook invokes OpenSearch snapshot APIs to back up and restore data streams appropriately, thereby ensuring there is no loss of data and avoids data corruption as well.

To initiate an OpenSearch restore, first delete the existing OpenSearch cluster running on the system and all related data.

Scale down Verrazzano Monitoring Operator.

$ kubectl scale deploy -n verrazzano-system verrazzano-monitoring-operator --replicas=0

Then, clean up the OpenSearch components.

# These are sample commands to demonstrate the OpenSearch restore process

$ kubectl delete sts -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete deploy -n verrazzano-system -l verrazzano-component=opensearch
$ kubectl delete pvc -n verrazzano-system  -l verrazzano-component=opensearch

To perform an OpenSearch restore, you can invoke the following example Velero Restore API object.

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: verrazzano-opensearch-restore
  namespace: verrazzano-backup
spec:
  backupName: verrazzano-opensearch-backup
  includedNamespaces:
    - verrazzano-system
  labelSelector:
    matchLabels:
      verrazzano-component: opensearch
  restorePVs: false
  hooks:
    resources:
      - name: opensearch-test
        includedNamespaces:
          - verrazzano-system       
        labelSelector:
          matchLabels:            
            statefulset.kubernetes.io/pod-name: vmi-system-es-master-0
        postHooks:
          - exec:
              container: es-master
              command:
                - /usr/share/opensearch/bin/verrazzano-backup-hook
                - -operation
                - restore
                - -velero-backup-name
                - verrazzano-opensearch-backup
              waitTimeout: 30m
              execTimeout: 30m
              onError: Fail

The preceding example will restore an OpenSearch cluster from an existing backup.

It will recreate a new OpenSearch cluster (with new indexes).
The postHook will invoke the OpenSearch APIs that restores the snapshot data.
The container on which the hook needs to be run is identified by the pod label selectors, followed by the container name. In this case, it’s statefulset.kubernetes.io/pod-name: vmi-system-es-master-0.

NOTE: The hook needs to be a postHook because it must be applied after the Kubernetes objects are restored.

After the restore operation is processed, you can see the hook logs using the velero restore logs command. Additionally, the hook logs are stored under the /tmp folder in the pod.

OpenSearch restore logs

# To display the logs from the restore, run the following command
$ kubectl logs -n verrazzano-backup -l app.kubernetes.io/name=velero

# Fetch the log file name as shown
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- ls -al /tmp | grep verrazzano-restore-hook | tail -n 1 | awk '{print $NF}'

# To examine the hook logs, exec into the pod as shown, and use the file name retrieved previously
$ kubectl exec -it vmi-system-es-master-0 -n verrazzano-system -- cat /tmp/<log-file-name>