This article was written by Sachin Kulkarni of Trilio.

Introduction

Organizations are moving towards Kubernetes as an operating environment, and protecting the data is paramount. It’s their top-most priority to protect the business-critical data, and set up a business continuity plan in case of a disaster. A cloud-native backup and Disaster Recovery (DR) solution is the need of the hour, and the answer is TrilioVault for Kubernetes (TVK).

TVK is a cloud-native, application-centric data protection platform designed to support the scale, performance, and mobility requirements of Kubernetes container environments across any public or hybrid cloud environment. It offers backup and DR solutions for the entire application, including data, metadata, and Kubernetes objects. It is protected and can be restored from any point-in-time.

RHACM is a management solution designed to help organizations extend and scale Red Hat OpenShift, the leading enterprise Kubernetes platform. It enables management consistency across hybrid cloud deployments, including on-premises and public clouds.

Integration between TVK and RHACM provides an automated solution to protect containers, Kubernetes, and cloud infrastructure using the policy framework. This powerful framework allows users to govern multiple clusters in the data center using policies. The TVK policies enable the users to protect their OpenShift clusters and provide continued protection for existing and new clusters.

This blog provides a walk-through on how to deploy the TVK policies on RHACM. It’s an efficient way to protect your cloud workloads.

Prerequisites

Be sure to meet the following prerequisites:

Install RHACM

RHACM can be used to comply with enterprise and industry standards for aspects, such as security and regulatory compliance, resiliency, and software engineering. Deviation from defined values for such standards represents a configuration drift, which can be detected using the built-in configuration policy controller of RHACM.

Install RHACM version 2.6 as outlined in the product documentation.

Deploy TrilioVault for Kubernetes TVK policies

In the RHACM policy collection from the community folder, there are three TVK policies available. View the following table for the TVK policy descriptions:

Policy Description Prerequisites
Policy to install TVK Operator Use this policy to install TrilioVault for Kubernetes Operator and a trial license on OpenShift clusters with the label protected-by=triliovault. Requires OpenShift 4.8 or later, needs CSI Driver with snapshot capabilities, a storageClass and volumeSnapshotClass. For more information, refer to the Trilio documentation.
Policy to create namespace-based backup using TrilioVault for Kubernetes Use this policy to create namespace-based backups using TVK on OpenShift clusters with the label protected-by=triliovault. Requires OpenShift 4.8 or later. Note: TVK must be installed to use this policy. On the hub cluster, create a secret named aws-s3-secret with S3 credentials and a ConfigMap named aws-s3-configmap with S3 bucket name, region name & threshold capacity in the namespace, where this policy is created (details given in the policy). For more information, refer to the Trilio documentation.
Policy to create namespace-based backup using TVK and Kyverno template Use this policy to create namespace-based backup using TKO and Kyverno template on OpenShift clusters with the label protected-by=triliovault. It creates a backup of the namespaces that have the following label, protected-by=tvk-ns-backup. Requires OpenShift 4.8 or later. Note: Kyverno controller must be installed to use the kyverno policy. See the Policy to install Kyverno. TVK must be installed to use this policy. On the hub cluster, create a secret named aws-s3-secret with S3 credentials, and a ConfigMap named aws-s3-configmap with S3 bucket name, region name & threshold capacity in the namespace, where this policy is created (details given in the policy). For more information, refer to the Trilio documentation.

Let’s start with the policy to install TrilioVault for Kubernetes (TVK).

Install TVK using a policy

Let's use the policy to install TVK. This policy installs the latest TVK operator on any Openshift cluster with a label protected-by=triliovault.

For demo purposes, there are two OpenShift Container Platform clusters; one of them runs RHACM and is the hub cluster. You can view the clusters from the RHACM console Clusters page. View the following screen capture:

clusters-page-tvk

Note: Notice the clusters are prelabeled with the label protected-by=triliovault. There are no TVK policies created on the RHACM cluster yet, so let’s start with our first example.

From the navigation menu, select Governance. This is also referred to as the policy engine. Then click the Create policy button as shown in the following screen capture:

gov-page-tvk The Create policy form is displayed. By default, the YAML editor is also displayed alongside the form.

create-policy-tvk Let’s go to the policy from GitHub page, click the Raw button to view the contents of the policy YAML, copy the YAML code from GitHub and paste into the YAML editor of RHACM.

Note: Before pasting the policy contents into RHACM, be sure to clear the YAML editor. Typically, you can use the shortcut, <ctrl>-a <ctrl>-c from the GitHub window, and <ctrl>-a <ctrl>-v from the RHACM window. After you paste the policy into that YAML editor, your console might resemble the following image:

gov-install-tvk In the last line of the policy code of the PlacementRule section, notice that this policy uses the labels vendor=OpenShift and protected-by=triliovault. This indicates that the policy is deployed on all OpenShift clusters with the user-defined label, protected-by=triliovault. Before you create the policy, you need to select a namespace for the policy to be run in. This is for internal organization reasons only; it does NOT affect the results of the policy engine itself. From the policy form, select the default namespace or any other namespace available on the hub cluster. You can create some specific policy-engine namespaces in advance to be able to group them more efficiently. For this example, select Inform as the remediation action.

Before you create the policy, let’s verify that TVK is not installed. Recheck the list of installed operators on the cluster itself, from the OpenShift console:

ocp-installed-operators

It is verified that TVK is not installed, so let’s create the policy by clicking the Submit button in the Create policy dialog box. After a couple of moments, the policy is listed in the policy table. Notice that there is a cluster violation. Select the install-tvk policy to view the policy details and results. View the following image:

acm-tvk-policy Select the Results Tab to view the listed violations:

tvk-policy-results

The required operator elements are missing, which is why the policy failed. Let's change the remediation action by navigating to the Governance Overview tab, select the Actions icon (three vertical dots), and click Enforce:

enforce-tvk-policy The Enforce policy dialog box is displayed. Click Enforce to confirm the action. View the following image:

enforce-dialog It takes a few minutes for enforcement of the policy. Recheck the details of the policy from the Results tab. View the following image:

enforce-results And you can confirm that the operator has been installed on your cluster with the protected-by=triliovault label.

trilio-operator

This concludes how to install TrilioVault for Kubernetes (TVK) on the clusters managed by RHACM. This policy also protects new OpenShift Container Platform clusters with a label protected-by=triliovault.

Protect a namespace in the cluster using the TVK backup creation policy

Now that the TVK is installed and a trial license is created (as seen in the image, where TVK is listed as an installed operator), let's start using TVK. To get started with TVK in your environment, be surethat all prerequisites are fulfilled as listed in the Prerequisites section. Once done, you must perform the following steps:

  • Create a TVK Target - Location where backups are stored
  • Create a backup plan - Details about the backup
  • Create a namespace-based backup

Creating a policy that creates a namespace-based backup using TVK helps to achieve the earlier mentioned steps for a single namespace. Let's take a look at how to apply the policy. For this example, the namespace is mysql, with the MySQL application running. Run the following command to receive a list of mysql applications:

# oc get all -n mysql
NAME READY STATUS RESTARTS AGE
pod/k8s-demo-app-mysql-78dbddc54f-dtpd6 1/1 Running 0 1d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/k8s-demo-app-mysql ClusterIP None <none> 3306/TCP 1d

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/k8s-demo-app-mysql 1/1 1 1 1d

NAME DESIRED CURRENT READY AGE
replicaset.apps/k8s-demo-app-mysql-78dbddc54f 1 1 1 1d
#

Navigate to the create-tvk-backup policy in the policy-collection repository. As mentioned in the comments of the create-tvk-backup policy, you must create a secret and ConfigMap with AWS S3 details for TVK target creation. Your secret and ConfigMap might resemble the following content:

cat <<EOF | oc apply -f - -n default
apiVersion: v1
kind: Secret
metadata:
name: aws-s3-secret
namespace: default
type: Opaque
stringData:
accessKey: "PROVIDE_ACCESS_KEY"
secretKey: "PROVIDE_SECRET_KEY"
EOF
cat <<EOF | oc apply -f - -n default
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-s3-configmap
namespace: default
data:
bucketName: "PROVIDE_S3_BUCKET_NAME"
region: "PROVIDE_REGION"
thresholdCapacity: "100Gi"
EOF

Now, let’s create the policy for TVK backups by clicking the Create policy button. By default, the YAML editor displays the YAML contents, which is helpful for importing policies. Let’s go to the create-tvk-backup policy from GitHub, click the Raw button for the policy YAML contents, and copy the YAML contents. Then paste the contents from GitHub into the YAML section of your RHACM console.

Note: Before pasting into your hub cluster, clear the YAML editor. Typically you select <ctrl>-a <ctrl>-c in the GitHub Window, and a <ctrl>-a <ctrl>-v in the RHACM window. After you paste the policy contents, your YAML editor might resemble the following image:

create-tvk-backup Replace the test namespace used in the policy with mysql to create the backup. You can make this change from the YAML editor in RHACM.

In the PlacementRule section, notice that this policy is defined to be applied on all clusters with labels vendor=OpenShift and protected-by=triliovault. This policy is expected to be deployed on all OpenShift clusters with user-defined label protected-by=triliovault. Before you select the Create button, you still need to select a namespace in which this policy needs to be applied. This is for internal organization reasons only; it does NOT affect the results of the policy engine itself. From the Create policy form, you can select the default namespace on the hub cluster. View the following image:

policy-review Click Submit to create the policy. Afterwards a Policy created banner appears. View the following image:

policy-created-banner

Notice, that RHACM detected a cluster violation. This is because the remediation action must be set to enforce to create a backup for the TVK operator. Return to the Policies tab and select the Actions icon for the create-tvk-backup policy. Select the Enforce option.

tvk-action-list-policies-tab From the Enforce policy box, click Enforce to confirm the remediation action change:

enforce-policy-box It takes a few minutes for the policy to be enforced. Select the policy name to see the details and results. Select the Results tab to view the cluster violations. Based on the following image, there are no longer any violations:

results-without-violation As expected and displayed in the previous screenshot, the policy created a secret that uses AWS S3 credentials, a target to store backup data in the specified S3 bucket, a BackupPlan with a daily backup schedule, and a first full backup of the namespace mysql. With this, the namespace mysql is protected, and a new backup is used daily.

Let's also log in to the TVK console and sign in through your OpenShift cluster. Refer to OpenShift Environments in the Trilio documentation for more details. View the following image:

trilio-ui View the backups created for the mysql namespace from the TrilioVault management console, as shown in the following image:

mysql-backup-trilio This concludes how to create a namespace-based backup using the TVK policy on OpenShift clusters with the label, protected-by=triliovault, which is managed by RHACM.

However, you must update the namespace name to create the backup manually. Note, it is expected that the same namespace is present on all the OpenShift clusters with the label protected-by=triliovault.

Continue reading to learn how to protect multiple namespaces on any OpenShift clusters, matching the placement rule managed by RHACM.

Protect multiple namespaces in any cluster using the TVK backup creation policy

Before starting, be sure that all prerequisites are fulfilled as listed in the previous section, Protect a namespace in the cluster using the TVK backup creation policy.

Creating a policy that creates a namespace-based backup using TVK and a Kyverno template, creates a backup of the namespaces that have the protected-by=tvk-ns-backup label on all the OpenShift clusters with a protected-by=triliovault label. In addition to installing TVK, the Kyverno controller must be installed to use the kyverno policy. See the Policy to install Kyverno from the policy-collection.

Note - Grant additional privileges to the Kyverno service account for the namespaces.

Once the prerequisites are met, this policy performs the following steps for each existing or newly created namespace, with the protected-by=tvk-ns-backup label:

  • Create a TVK Target - Location where backups are stored.
  • Create a backup plan - Details about the backup
  • Create a namespace-based backup

As a reminder, TVK is already installed with the install-tvk policy. Now, let’s create the policy-kyverno-tvk-create-ns-backup policy. Click the Create policy button. Return to the policy example in the policy-collection repository to copy the contents, and paste them in the RHACM YAML editor. View the following image:

kyverno-tvk-policy In the PlacementRule section, notice that this policy is defined to be applied on all clusters with the labels vendor=OpenShift and protected-by=triliovault. This policy is expected to be deployed on all OpenShift clusters with the user-defined label, protected-by=triliovault. Before you select the Create button, you still need to select a namespace in which this policy needs to be applied. This is for internal organization reasons only; it does NOT affect the results of the policy engine itself. From the Create policy form, you can select the default namespace on the hub cluster. View the following image:

kyverno-yaml-editor Click Submit to create the policy. After the policy is created, the policy details are displayed similar to the following image:

kyverno-cluster-violation

Similar to the previous example in the earlier section, RHACM detected a cluster violation. There is a cluster violation because the remediation action must be set to enforce. From the Details tab, select the Actions drop-down menu and click Enforce. View the following image:

kyverno-action-list-policies-tab

Click Enforce to confirm that action from the Enforce policy box.

kyverno-tvk-backup-enforce

It takes a few minutes for the policy to be enforced. After you select the policy name, select the Results tab:

kyverno-results-without-violation

The previous screenshot shows that the policy created a Kyverno policy on the target cluster. This Kyverno policy monitors all the namespaces and creates a target to store backup data in the specified S3 bucket, a BackupPlan with a daily backup schedule, and a first, full backup for all the namespaces that have the protected-by=tvk-ns-backup label. This is also applied to any new namespace created with the same label.

Let’s log in to the TVK console through OpenShift to check if the backups are created:

tvk-console-monitoring

As seen in the previous screenshot, there are no backups from the Monitoring page in the Cluster Dashboard section. This is because there are no namespaces in the target cluster with the label protected-by=tvk-ns-backup.

For this demo, let's create a namespace named postgres with the PostgreSQL application running. Run the following command:

# oc get all -n postgres
NAME READY STATUS RESTARTS AGE
pod/postgres-demo-0 1/1 Running 0 5m28s
pod/postgres-demo-1 1/1 Running 0 5m3s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/postgres NodePort 172.30.17.37 <none> 5432:30992/TCP 5m29s

NAME READY AGE
statefulset.apps/postgres-demo 2/2 5m29s
#

Apply the protected-by=tvk-ns-backup label to the postgres namespace with the following command:

# oc label ns postgres protected-by=tvk-ns-backup
namespace/postgres labeled
#

It takes a few minutes for the Kyverno policy to come into effect. When you check the details, notice that the resource details within the namespace are displayed using the following command:

# oc get target,backupplan,backup -n postgres
NAME TYPE THRESHOLD CAPACITY VENDOR STATUS BROWSING ENABLED
target.triliovault.trilio.io/tvk-postgres-s3-target ObjectStore 100Gi AWS Available

NAME TARGET RETENTION POLICY INCREMENTAL SCHEDULE FULL BACKUP SCHEDULE STATUS
backupplan.triliovault.trilio.io/tvk-postgres-backupplan tvk-postgres-s3-target trilio-daily-schedule-policy Available

NAME BACKUPPLAN BACKUP TYPE STATUS DATA SIZE CREATION TIME START TIME END TIME PERCENTAGE COMPLETED BACKUP SCOPE DURATION
backup.triliovault.trilio.io/tvk-postgres-backup tvk-postgres-backupplan Full Available 112328704 2022-09-29T10:15:03Z 2022-09-29T10:15:03Z 2022-09-29T10:23:22Z 100 Namespace 8m19.479134026s
#

Let’s check the Monitoring page for the Cluster Dashboard in the TVK console. In the following screen capture, notice a backup is created for the postgres namespace:

tvk-console-monitoring-backup

This concludes how to create a namespace-based backup using Kyverno-based, TVK policy for all the namespaces that have the protected-by=tvk-ns-backup label on the OpenShift clusters with the protected-by=triliovault label, which is managed by RHACM. Apply the protected-by=tvk-ns-backup label for any namespace to be protected.

Conclusion

TrilioVault for Kubernetes (TVK) is a cloud-native, application-centric, data protection platform designed to support the scale, performance, and mobility requirements of Kubernetes container environments across any public or hybrid cloud environment. It offers backup and recovery of the entire application, including data, metadata, and Kubernetes objects. It is protected and can be restored from any point-in-time.

TVK provides the following wide range of helpful features:

  • Native Kubernetes application
  • Storage of metadata and all application resources to a specified target
  • Support for Helm, Label, and Operators application deployment types and support for S3 or NFS-based backup targets
  • Provisioning of application hooks to provide data consistent backups
  • Support for RHACM policies to install TVK and create backups for namespaces

TVK provides a strong platform for enterprise database backups. For more information on TVK, request a demo or download your free trial today.

References


About the author

Red Hatter since 2018, tech historian, founder of themade.org, serial non-profiteer.

Read full bio