7.4.4. Replicated caches with each node having its own store
Here, each node has its own datastore. Modifications to the cache are (a) replicated across the cluster and (b) persisted using the CacheLoader. This means that all datastores have exactly the same state. When replicating changes synchronously and in a transaction, the two phase commit protocol takes care that all modifications are replicated and persisted in each datastore, or none is replicated and persisted (atomic updates).
Note that currently JBossCache is not an XAResource, that means it doesn't implement recovery. When used with a TransactionManager that supports recovery, this functionality is not available.
The challenge here is state transfer: when a new node starts it needs to do the following:
Tell the coordinator (oldest node in a cluster) to send it the state
The coordinator then needs to wait until all in-flight transactions have completed. During this time, it will not allow for new transactions to be started.
Then the coordinator asks its CacheLoader for the entire state using loadEntireState(). It then sends back that state to the new node.
The new node then tells its CacheLoader to store that state in its store, overwriting the old state. This is the CacheLoader.storeEntireState() method
As an option, the transient (in-memory) state can be transferred as well during the state transfer.
The new node now has the same state in its backend store as everyone else in the cluster, and modifications received from other nodes will now be persisted using the local CacheLoader.