Part 2: Connecting an OCP application to a MySQL instance

By Marko Karg and Annette Clewett

At the end of the first post in this blog series, we deployed a MySQL instance using StatefulSets (STS). Today, we want to use that same deployment method and connect a WordPress instance to the MySQL database and see what happens if the database fails. Remember, we’re using a MySQL pod created from a StatefulSet.

OpenShift on AWS test environment

All the posts in this series use an OCP-on-AWS setup that includes 8 EC2 instances deployed as 1 master node, 1 infra node, and 6 worker nodes that also run OCS gluster and heketi pods. The 6 worker nodes are basically the storage provider (OpenShift Container Storage [OCS]) and persistent storage consumers (MySQL). As the following figure shows, the ocs worker nodes are of instance type m5.2xlarge with 8 vCPUs, 32 GB Mem, and 3x100GB gp2 volumes attached to each node for OCP and 1 1TB gp2 volume for OCS storage cluster. The AWS region us-west-2 has availability zones (AZs) us-west-2a, us-west-2b, us-west-2c, and the 6 worker nodes are spread across the 3 AZs, two nodes in each AZ. This mean the OCS storage cluster is "stretched" across these 3 AZs.

MySQL setup

We’ve created a headless MySQL service already, using an OCS-based persistent volume claim (PVC):

oc get services
NAME        TYPE CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
mysql       ClusterIP             <none>     3306/TCP    16h

The STS we need is also created as described in the first post of this series. Once the STS has been created and the container has been started, we have a running MySQL instance running:

oc get pods
NAME                READY   STATUS    RESTARTS    AGE
mysql-ocs-0         1/1     Running   0           21h

With a service and a database up, we can move forward to get our application deployed.

Although there are templates available that would set up WordPress with a preconfigured database in one shot, we want to take the long way and start from scratch to illustrates the required steps.

WordPress setup

Let’s create a new php application to run WordPress:

# oc new-app php~https://github.com/wordpress/wordpress

After a few seconds, we have the required pods in our project:

# oc get pods
NAME                READY     STATUS      RESTARTS   AGE
mysql-ocs-0         1/1       Running     0          21h
wordpress-1-build   0/1       Completed   0          22h
wordpress-1-q5jts   1/1       Running     0          22h

To make our WordPress instance available to the world, we need to expose it:

# oc expose service wordpress

Now two services are available, one for MySQL and one for WordPress:

# oc get service
NAME                                                     TYPE       
CLUSTER-IP      EXTERNAL-IP  PORT(S)             AGE
glusterfs-dynamic-31a07eb1-3a72-11e9-96fc-02e7350e98d2   ClusterIP  
172.30.165.245  <none>       1/TCP               1m
mysql-ocs                                                ClusterIP  
172.30.210.183  <none>       3306/TCP            1m
wordpress                                                ClusterIP  
172.30.1.139    <none>       8080/TCP,8443/TCP   28s

Let’s connect to the web interface of WordPress now. To do so, we take the HOST / PORT portion from the following command:

oc get route wordpress
NAME        HOST/PORT                                         PATH 
SERVICES    PORT       TERMINATION     WILDCARD
wordpress   wordpress-marko.apps.ocpocs311.ocpgluster.com             
wordpress   8080-tcp None              None

In our case, that’s wordpress-marko.apps.ocpocs311.ocpgluster.com. Take that string and put it into a browser to go to the WordPress interface. The WordPress web interface will guide us through setting up the database connection now.

We have most of the above values pre-defined in our MySQL STS, so the database name is “wordpress”, the username is “admin”, and the password is “secret”.

cat mysql-sts.yaml
….omitted….
   spec:
      terminationGracePeriodSecods: 10
      containers:
      - name: mysql-ocs
        image: mysql:5.7
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        - name: MYSQL_DATABASE
          value: wordpress
        - name: MYSQL_USER
          value: admin
        - name: MYSQL_PASSWORD
          value: secret
….omitted….

Our database host can be found by running this command:

oc get services
NAME        TYPE CLUSTER-IP       EXTERNAL-IP     PORT(S)
AGE
NAME                                                     TYPE
CLUSTER-IP     EXTERNAL-IP      PORT(S)             AGE
glusterfs-dynamic-31a07eb1-3a72-11e9-96fc-02e7350e98d2   ClusterIP
172.30.165.245 <none>           1/TCP               1m
mysql-ocs                                                ClusterIP 
               <none>           3306/TCP            1m
wordpress                                                ClusterIP
172.30.1.139   <none>           8080/TCP,8443/TCP   28s 

So we will use mysql-ocs for the database host.

If information is entered is correct, WordPress will guide us through the rest of the installation process:

We now need to enter some information for the web front end:

The installation takes some time and finally presents this screen:

So now our deployment is done. Next, we'll log into WordPress and create some test content:

Failure scenario

So now that we have everything in place, we want to see what happens when our MySQL container fails. To simulate that, we’ve set up a client that checks the website using the "curl" command.

Because we're only interested in the HTTP response over a longer time, we run it in a loop, trimming the output to what we're interested in:

while true; do date; curl -I 
http://wordpress-marko.apps.ocpocs311.ocpgluster.com/2019/02/26/lorem-ipsum/ 
2>&1 | grep HTTP; sleep 1; done

As a first test we simply kill the MySQL container, watching the preceding loop closely:

# oc get pods
NAME                READY STATUS     RESTARTS  AGE
mysql-ocs-0         1/1   Running    0         21h
wordpress-1-build   0/1   Completed  0         22h
wordpress-1-q5jts   1/1   Running    0         22h

Delete the MySQL pod:

oc delete pod mysql-ocs-0

Here’s the output from the preceding curl loop:

Mi 27. Feb 09:40:44 UTC 2019
HTTP/1.1 200 OK
Mi 27. Feb 09:40:45 UTC 2019
HTTP/1.1 500 Internal Server Error
…
Mi 27. Feb 09:40:59 UTC 2019
HTTP/1.1 500 Internal Server Error
Mi 27. Feb 09:41:03 UTC 2019
HTTP/1.1 200 OK

So the pod failure effectively caused our WordPress instance to be unavailable for 18 seconds (09:40:45 to 09:41;)3), give or take a few seconds for the curl command. We’ve run a larger number of the same test and ended up with an average value of 12 seconds. This is the time that the WordPress application is unavailable due to the MySQL pod being deleted. Once the pod is re-created and the OCS storage is mounted in the pod, then WordPress is available again.

This test only deletes the pod that is running the database. What we cannot be sure of so far is that the MySQL pod actually moves from one node to another. To have that happen, we have to cordon the node on which the pod currently runs and then delete the pod. Cordoning the node means that it will take no new containers and, as a consequence, the new incarnation of our database pod will have to be started on a different node.

As a first step, we need to find the node on which mysql-ocs is currently running:

# ocs get pod mysql-ocs-0 -o wide

ocs get pod mysql-ocs-0 -o wide

NAME          READY   STATUS    RESTARTS      AGE      IP            NODE           
NOMINATED NODE
mysql-ocs-0   1/1     Running   0             5m       10.129.2.44   
ip-172-16-27-161.us-west-2.compute.internal   <none>

Now we cordon the node ip-172-16-27-161.us-west-2.compute.internal and then delete the mysql-ocs-0 pod:

# oc adm cordon ip-172-16-27-161.us-west-2.compute.internal
node/ip-172-16-27-161.us-west-2.compute.internal cordoned
# oc delete pod mysql-ocs-0
pod "mysql-ocs-0" deleted

Again, we’ve run a series of the same test and ended up with an average value of 12 seconds. Therefore, it does not matter if the pod must be relocated to another node or is re-created on the same node again.

Conclusion

The goal behind this post was to show how an application like WordPress in OCP can be connected to a database, as well as how fast a pod can fail-over to another node when it is using Red Hat OCS as a storage platform. An average time of 12 seconds is what we can reproduce persistently for one MySQL pod. The exact time is, of course, specific to every setup, as it depends on a lot of different factors, but the reproducibility and the deterministic time is something that's common to OpenShift deployments.