Applications Here, Applications There! - Part 4 - Disaster Recovery

In the first blog posts of this series, we introduced the basic concepts for Application Lifecycle on Advanced Cluster Management and deployed an application to multiple clusters.

In the second blog post, we showed how a Blue/Green deployment can be made by using Advanced Cluster Management.

In the third blog post, we migrated our application across different regions by using Advanced Cluster Management.

If you haven't read the previous blog posts yet, go ahead and read them. This post is a continuation and you will need the context from the previous ones.

We are using the same infrastructure we used in the previous posts. See the following diagram:

NOTE: Pay special attention to the different labels, as they will be used during the blog posts examples.

Disaster Recovery on Red Hat Advanced Cluster Management

In the previous post we migrated our application across different regions using PlacementRules.

In this blog post we are going to see how PlacementRules can help us with a basic Disaster Recovery scenario.

Configuring required Advanced Cluster Managemen manifests

We will re-use the Red Hat Advanced Cluster Management manifests used in the previous blog post, which means that we will be using the Namespace reverse-words-region and the Subscription reversewords-region-app-subscription.

We will need to create a new PlacementRule. This time we will include new properties to our PlacementRule; let's review them:

apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
  name: us-eu-region-clusters
  namespace: reverse-words-region
spec:
  clusterConditions:
    - type: "ManagedClusterConditionAvailable"
      status: "True"
  clusterSelector:
    matchExpressions:
    - key: region
      operator: In
      values:
      - EU
      - US
    matchLabels: {}
  clusterReplicas: 1

1. We are using the matchExpressions property in order to match any cluster that has a label region with a value of either EU or US.
2. We are using the clusterReplicas property in order to get only one of the clusters that match the previous expression.
3. Additionally, we are matching only healthy clusters.

This new PlacementRule will make sure that in case one of the clusters moves to a non-healthy state, the cluster returned by the PlacementRule changes to one on the list that has a healthy state.

Configuring Subscription to use the new PlacementRule

1. Let's create the PlacementRule discussed in the previous section.

   oc --context hub create -f https://raw.githubusercontent.com/RHsyseng/acm-app-lifecycle-blog/master/acm-manifests/reversewords-region/05_placement_rule_DR.yaml

   If we look at the clusters reported by the PlacementRule, we will only see one cluster (Production in this case).

   oc --context hub -n reverse-words-region get placementrule us-eu-region-clusters -o yaml

   <OMITTED_OUTPUT>
   status:
     decisions:
     - clusterName: managed-cluster1-dev
       clusterNamespace: managed-cluster1-dev

2. Now we can go ahead and update the Subscription we used in the previous blog post. We are going to patch it to use the new PlacementRule we just created.

   oc --context hub -n reverse-words-region patch subscription.apps.open-cluster-management.io/reversewords-region-app-subscription -p '{"spec":{"placement":{"placementRef":{"name":"us-eu-region-clusters"}}}}' --type=merge

3. The application will run in EU cluster (Development). See the following commands and output:

   oc --context dev -n reverse-words-region get deployments,services,pods

   NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.extensions/reverse-words   1/1     1            1           42s
   
   NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP                                                                PORT(S)          AGE
   service/reverse-words   LoadBalancer   172.30.185.94   a520ed21ff982452abeacf63b0b58cc5-31012041.eu-central-1.elb.amazonaws.com   8080:32283/TCP   42s
   
   NAME                                 READY   STATUS    RESTARTS   AGE
   pod/reverse-words-68795d69ff-crzqp   1/1     Running   0          42s

4. Now, I'm going to destroy my EU cluster in order to simulate a disaster; let's see what happens.

   NOTE: We actually destroyed the cluster, if you want to try this without destroying your cluster, you can remove the region: EU label from the cluster

   1. As soon as Red Hat Advanced Cluster Management detects my EU cluster is gone, the PlacementRule gets updated and now it points to the US cluster.

      oc --context hub -n reverse-words-region get placementrule us-eu-region-clusters -o yaml

      The PlacementRule now points to US cluster.

      <OMITTED_OUTPUT>
      status:
        decisions:
        - clusterName: managed-cluster2-prod
          clusterNamespace: managed-cluster2-prod

   2. The application has been moved automatically to the US cluster.

      oc --context pro -n reverse-words-region get deployments,services,pods

      NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
      deployment.extensions/reverse-words   1/1     1            1           76s
      
      NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)          AGE
      service/reverse-words   LoadBalancer   172.30.187.142   a1c7d218d901c40ac98375f4a9474084-1310645059.us-west-2.elb.amazonaws.com   8080:31095/TCP   78s
      
      NAME                                 READY   STATUS    RESTARTS   AGE
      pod/reverse-words-68795d69ff-ttzz5   1/1     Running   0          77s

5. When the EU cluster is online again, it will get added to the PlacementRule again automatically

   

Closing Thoughts

During this blog posts series we have discovered the basics around Application Lifecycle management using Red Hat Advanced Cluster Management.

Feel free to share your comments and keep exploring other Red Hat Advanced Cluster Management related blog posts.