Replication between data centers with JBoss Data Grid
Bela Ban, Red Hat
Red Hat Summit Boston, June 2013
What is data center replication ?
What problems does it solve ?
How is it done ?
How do I set it up ?
## What is replication between data centers ?
## What problems does it solve ?
## Site failure
An entire site failing ? What are the chances of that ?
Slim. But if it happens it can have a big (negative) impact on a company !
## What can cause a site failure ?
* All nodes crashing at the same time (very unlikely !)
* Switches/routers crashing, causing partitions
* If we favor CP in CAP, the site may turn read-only
* Clients losing access to the site due to connectivity loss
* Catastrophic events
* Floods, hurricanes (Sandy), terrorist attacks (9/11)
* Data centers under water (World Trade One by fire water)
* Sandy flooded Lower Manhattan with corrosive saltwater
## Follow the sun
* Not only data centers, but also client are located around the world
* E.g. clients in LON, BOS, SFO
* First the LON clients are up, then BOS, then SFO
* Clients want to access their local DC
* A LON client accessing the SFO DC has latency of ~30 ms
* Clients are directed to the data center closest to them
* Follow the sun: only one DC is active at any time
* Replication between active and passive DCs
## How is xsite replication done ?
### The whole picture
- Data center (site): a local cluster of N nodes in which data is stored redundantly
on multiple nodes
- Replication: sending modifications to all nodes which have the data
- Site master: a node in the site which is responsible for relaying of modifications to other sites
- xsite: cross-site (traffic), e.g. replication between data centers
- JDG: JBoss Data Grid, a cluster in which data is stored redundantly
Example of async xsite replication
## Drill down
### A look at two JDG nodes in a site
## How do I set this up ?
* Can be downloaded from [https://github.com/belaban/jdg-client](https://github.com/belaban/jdg-client).
* Provides XML configuration for sites LON, BOS and SFO
* Provides scripts to start nodes in each site
* Simply copy the config and start scripts into a JDG installation
* Can be run on the same host
### Running lon1:
./lon.sh lon1 127.0.0.1 0
./standalone.sh -c lon.xml -Dsite=LON -b 127.0.0.1 -Djboss.node.name=lon1 \
-Djboss.socket.binding.port-offset=0 -u 18.104.22.168
### Running lon2:
./lon.sh lon2 127.0.0.1 500
./standalone.sh -c lon.xml -Dsite=LON -b 127.0.0.1 -Djboss.node.name=lon2 \
-Djboss.socket.binding.port-offset=500 -u 22.214.171.124
### Running bos1:
./bos.sh bos1 127.0.0.1 1000
./standalone.sh -c bos.xml -Dsite=BOS -b 127.0.0.1 -Djboss.node.name=bos1 \
-Djboss.socket.binding.port-offset=1000 -u 126.96.36.199
### SiteMaster as bottleneck
* All update requests are handled and applied by the SiteMaster
* This creates a bottleneck, thread pool exhaustion
* ==> SiteMaster forwards update requests to random node within the same site for processing
### State transfer between sites
* When a new site is started, grab the state from another site
* The new site is only operational when the entire state has been transferred
* Pull request submitted, needs to be integrated
* Infinispan Reliable Asynchronous Clustering
* Synchronous transactional replication is not good
* Slow due to latency between sites
* Increased risk of TX rollback due to many involved nodes
* IRAC provides
* (Almost) the speed of async repl with
* the reliability of sync transactional replication
* Smaller bandwidth requirement between sites
* Xsite replication backs up a site to one or more sites
* Can be used to implement follow-the-sun, or as backup for site failures
* Easy to configure
* Try it out for yourself with JDG 6.1
* __JDG__: [http://www.redhat.com/products/jbossenterprisemiddleware/data-grid/](http://www.redhat.com/products/jbossenterprisemiddleware/data-grid)
* __Infinispan__: [http://www.infinispan.org](http://www.infinispan.org)
* __JGroups__: [http://www.jgroups.org](http://www.jgroups.org)
* __Demo__: [https://github.com/belaban/jdg-client](https://github.com/belaban/jdg-client)
* __This talk__: [https://github.com/belaban/Talks](https://github.com/belaban/Talks)
- The TX originator completes TX and forwards changes to the local site master (as one message)
- Site master relays message to backup site master(s)
- Backup site master starts local TX and applies TX