While Red Hat and Amazon Web Services (AWS) manage the ROSA service, the customer has certain responsibilities, including managing their data and their applications.

In a situation where the customer accidentally removes their application on OpenShift, there is no way to recover the application data unless it’s been backed up by the customer. In this post we will demonstrate how to back up an  application project using Velero Backup. The OADP (OpenShift API for Data Protection) operator will be used to set up and install Velero on the OpenShift platform.

Steps to test the backup and recovery of a WordPress application running on ROSA:

  1. Deploy a ROSA cluster
  2. Install the OADP Operator
  3. Create credentials secret
  4. Install Velero using OADP
  5. Deploy a WordPress application
  6. Create a Backup
  7. Delete the application
  8. Restore from the backup 

Install ROSA

Install the ROSA cluster by following the documentation.

Install the OADP Operator

Install the OADP Operator from OperatorHub. After selecting the defaults and clicking on Install, the OADP Operator will be installed in the oadp-operator project (namespace) as shown in the screenshot below.

image4-Sep-23-2021-04-42-29-98-PM Create credentials secret

Create a secret for the AWS credentials, which is used by Velero to access the Velero S3 bucket in your account. Use the “oc create secret” command as shown below to create the secret, where cloud-credentials.txt is the name of the file where the credentials are saved. Make sure that the cloud-credentials.txt file follows the AWS credentials template.

oc create secret generic cloud-credentials --namespace oadp-operator --from-file cloud=cloud-credentials.txt

Install Velero using OADP

Next we will create a Velero instance as follows from the CLI

First, create a file called velero.yaml with your Velero configuration. A sample velero.yaml file is included below:

apiVersion: konveyor.openshift.io/v1alpha1
kind: Velero
metadata:
name: example-velero
namespace: oadp-operator
spec:
olm_managed: true
backup_storage_locations:
       - config:
          profile: "default"
          region: us-west-2
        credentials_secret_ref:
          name: cloud-credentials
          namespace: oadp-operator
        name: default
        object_storage:
          bucket: managed-velero-backups-1dc95bf4-3830-4231-904a-7782bd14f79e
          prefix: "velero"
        provider: aws
default_velero_plugins:
       - aws
       - openshift
enable_restic: true
volume_snapshot_locations:
       - config:
          profile: default
          region: us-west-2
        name: default
        provider: aws

Then create that file by running the following command:

oc apply -f velero.yaml

Check to see if the Velero pods are up and running.

oc get pods -n oadp-operator

The output will look similar to this:

NAME                                             READY   STATUS        RESTARTS   AGE
oadp-default-aws-registry-774b459f89-c7977   1/1         Running   0              19s
oadp-operator-57985ccd67-mndhb                   1/1         Running   0              22h
restic-cws5x                                     1/1         Running   0              21s
restic-dpj29                                     1/1         Running   0              21s
restic-xw8n2                                     1/1         Running   0              21s
velero-655b4bbf95-jssmk                          1/1         Running   0              22s

Deploy a WordPress application

Deploy the WordPress application from https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/ 

If the wordpress-mysql pod does not start, apply the fix mentioned here.

Update application files mysql-deployment.yaml and wordpress-deployment.yaml to use the gp2 StorageClass

mysql-deployment.yaml:
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
       app: wordpress
spec:
ports:
       - port: 3306
selector:
       app: wordpress
       tier: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
       app: wordpress
spec:
accessModes:
       - ReadWriteOnce
storageClassName: gp2
resources:
       requests:
        storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress-mysql
labels:
       app: wordpress
spec:
selector:
       matchLabels:
        app: wordpress
        tier: mysql
strategy:
       type: Recreate
template:
       metadata:
        labels:
          app: wordpress
          tier: mysql
       spec:
        containers:
        - image: mysql:5.6
          name: mysql
          env:
          - name: MYSQL_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql-pass
                key: password
          ports:
          - containerPort: 3306
            name: mysql
          volumeMounts:
          - name: mysql-persistent-storage
            mountPath: /var/lib/mysql
        volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pv-claim
wordpress-deployment.yaml:
apiVersion: v1
kind: Service
metadata:
name: wordpress
labels:
       app: wordpress
spec:
ports:
       - port: 80
selector:
       app: wordpress
       tier: frontend
type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
       app: wordpress
spec:
accessModes:
       - ReadWriteOnce
storageClassName: gp2
resources:
       requests:
        storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress
labels:
       app: wordpress
spec:
selector:
       matchLabels:
        app: wordpress
        tier: frontend
strategy:
       type: Recreate
template:
       metadata:
        labels:
          app: wordpress
          tier: frontend
       spec:
        containers:
        - image: wordpress:4.8-apache
          name: wordpress
          env:
          - name: WORDPRESS_DB_HOST
            value: wordpress-mysql
          - name: WORDPRESS_DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql-pass
                key: password
          ports:
          - containerPort: 80
            name: wordpress
          volumeMounts:
          - name: wordpress-persistent-storage
            mountPath: /var/www/html
        volumes:
        - name: wordpress-persistent-storage
          persistentVolumeClaim:
            claimName: wp-pv-claim
kustomization.yaml:
secretGenerator:
- name: mysql-pass
literals:
- password=test123
resources:
- mysql-deployment.yaml
- wordpress-deployment.yaml

Install the application using “oc apply -k ./” into the wordpress project, and expose the route for the service using “oc expose svc/wordpress”.

oc get pods,pvc,svc,route -n wordpress
NAME                                      READY   STATUS        RESTARTS   AGE
pod/wordpress-6c77879668-vxkxt            1/1         Running   0              11h
pod/wordpress-mysql-8599ff598-psbsf   1/1         Running   0              11h

NAME                                       STATUS   VOLUME                                         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pv-claim   Bound        pvc-0ce99737-ae50-4784-a2ee-7ea0d2776d9b   20Gi           RWO                gp2                11h
persistentvolumeclaim/wp-pv-claim          Bound        pvc-6241249b-2eab-4051-8bdb-948253c8c1fd   20Gi           RWO                gp2                11h

NAME                          TYPE               CLUSTER-IP          EXTERNAL-IP                                                                   PORT(S)            AGE
service/wordpress             LoadBalancer   172.30.183.22   ad67d6c27c3434d41805205b38f2ecc4-1422739893.us-west-2.elb.amazonaws.com   80:30599/TCP   22h
service/wordpress-mysql   ClusterIP          None                <none>                                                                        3306/TCP           22h

NAME                                     HOST/PORT                                                       PATH   SERVICES        PORT   TERMINATION   WILDCARD
route.route.openshift.io/wordpress   wordpress-wordpress.apps.mytest.wgms.p1.openshiftapps.com              wordpress   80                       None

Enter the URL wordpress-wordpress.apps.mytest.wgms.p1.openshiftapps.com in a web browser and configure WordPress. Then create your first blog entry for testing.

image1-Sep-23-2021-04-42-31-11-PM

Create a Backup

The backup.yaml file below instructs Velero to backup the namespace wordpress where our application is running.  The backup will be named backup-1.

apiVersion: velero.io/v1
kind: Backup
metadata:
namespace: oadp-operator
name: backup-1
spec:
includedNamespaces:
       - wordpress

From within the wordpress namespace run the following command:

oc apply -f backup.yaml

To determine when the backup has completed,  check the Status Phase to see if it is marked as Completed

oc get backup -n oadp-operator
NAME           AGE
backup-1   12h

Check the status of the backup as follows

oc describe backup -n oadp-operator



Status:
Completion Timestamp:  2021-09-01T18:26:10Z
Expiration:                2021-10-01T18:25:26Z
Format Version:            1.1.0
Phase:                     Completed
Progress:
       Items Backed Up:  82
       Total Items:          82
Start Timestamp:        2021-09-01T18:25:26Z
Version:                1
Events:                   <none>

Delete the application

To test that the backup was indeed successful, we will simulate a failure by deleting the wordpress project

oc delete project wordpress
project.project.openshift.io "wordpress" deleted

Quick check to see if the project has indeed been deleted

oc get pods,pvc -n wordpress
No resources found in wordpress namespace.

Restore from the backup 

Let restore the backup using Velero as follows using the restore.yaml shown below. Here we are calling our restore restore-1, and under the spec we specify the name of the backup that we want to use ie. backup-1. We specifically call out the from backup-1 restore the namespace wordpress.

apiVersion: velero.io/v1
kind: Restore
metadata:
namespace: oadp-operator
name: restore-1
spec:
backupName: backup-1
includeNamespaces:
       - wordpress

To restore the project run the following command:

oc apply -f restore.yaml
restore.velero.io/restore-1 created

Check if the restore can be seen under the oadp-operator namespace.

oc get restore -n oadp-operator
NAME            AGE
restore-1   33s

Check that restore status has indeed changed to Completed using the oc describe restore restore-5 -n oadp-operator command.

oc describe restore restore-1 -n oadp-operator



Status:
Completion Timestamp:  2021-09-02T17:14:47Z
Phase:                     Completed
Progress:
       Items Restored:  52
       Total Items:         52
Start Timestamp:   2021-09-02T17:14:05Z
Warnings:              19
Events:                  <none>

Check if the pods, pvc, service,route names are seen under the wordpress namespace.

oc get pods,pvc,svc -n wordpress

NAME                                      READY   STATUS        RESTARTS   AGE
pod/wordpress-6c77879668-vxkxt            1/1         Running   0              15m
pod/wordpress-mysql-8599ff598-psbsf   1/1         Running   0              15m

NAME                                       STATUS   VOLUME                                         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pv-claim   Bound        pvc-8e6a38ba-79cd-46fa-ac23-be43fc32c751   20Gi           RWO                gp2                15m
persistentvolumeclaim/wp-pv-claim          Bound        pvc-ae827f20-92b1-4ca5-84e1-657d3cf61166   20Gi           RWO                gp2                15m

NAME                          TYPE               CLUSTER-IP          EXTERNAL-IP                                                                   PORT(S)            AGE
service/wordpress             LoadBalancer   172.30.146.20   a1e4531ad6fd44fefa57469c96db579d-1493697719.us-west-2.elb.amazonaws.com   80:30711/TCP   15m
service/wordpress-mysql   ClusterIP          None                <none>                                                                        3306/TCP           15m

NAME                                     HOST/PORT                                                       PATH   SERVICES        PORT   TERMINATION   WILDCARD
route.route.openshift.io/wordpress   wordpress-wordpress.apps.mytest.wgms.p1.openshiftapps.com              wordpress   80                       None

We can now access the WordPress application using the URL wordpress-wordpress.apps.mytest.wgms.p1.openshiftapps.com As we can see below that the blog post that we had before we deleted the project has been restored.

Scheduling Backups:

Now that we have seen how to backup and restore our application using Velero, we will now look into scheduling the backup using the Schedule API available in the OADP Operator.

As shown below we have created a schedule called test-schedule. For this post we have scheduled a backup at 5 minutes (*/5 * * * *). Under the Template enter the VolumeSnapshotLocations as “default”, IncludedNamespaces as “wordpress”, and StorageLocation as “default”

This will schedule the backup from the OADP Operator as shown below.

Under the Backup tab of the OADP Operator we can see that the backups are being taken at the scheduled time.

Conclusion

In this post we saw how to backup and restore an application running in a ROSA cluster. We also looked into scheduling backups of our application namespace using the OADP Operator.

Please look into the OADP Operator github location for any new features being added to it.


About the authors

Mayur Shetty is a Principal Solution Architect with Red Hat’s Global Partners and Alliances (GPA) organization,  working closely with cloud and system partners. He has been with Red Hat for more than five years and was part of the OpenStack Tiger Team.

Read full bio