Using JGroups for Hibernate Second-Level Cache in AWS EC2

ehcache.xml

... partial ...

<?xml version="1.0" encoding="UTF-8"?>
<ehcache 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
	xsi:noNamespaceSchemaLocation="ehcache.xsd"
	name="hatch-api">

	<!--  Example config on EhCache doc page is wrong; this URL has some help:
	      http://www-01.ibm.com/support/docview.wss?uid=swg21693135
    -->
	<cacheManagerPeerProviderFactory
		class="net.sf.ehcache.distribution.jgroups.JGroupsCacheManagerPeerProviderFactory"
		properties="connect=TCP(bind_port=7800):
		   com.meltmedia.jgroups.aws.AWS_PING(timeout=3000;port_number=7800;tags=Env,Role;log_aws_error_messages=true):         
		   MERGE2(min_interval=10000;max_interval=30000):
		   FD_SOCK:
		   FD(timeout=3000;max_tries=3):
		   VERIFY_SUSPECT(timeout=1500):
		   pbcast.NAKACK(retransmit_timeout=3000;use_mcast_xmit=false):
		   UNICAST:
		   pbcast.STABLE(stability_delay=1000;desired_avg_gossip=50000;max_bytes=4M):
		   pbcast.GMS(join_timeout=5000;print_local_addr=true;view_bundling=true):
		   pbcast.STATE_TRANSFER"
		propertySeparator="::" />


	<cache name="com.myapp.domain.MyDomainObject"
           maxElementsInMemory="2000"
           eternal="false"
           timeToIdleSeconds="3600"
           timeToLiveSeconds="21600"
           statistics="true"
           memoryStoreEvictionPolicy="LFU"
           overflowToDisk="false">
           <cacheEventListenerFactory
                class="net.sf.ehcache.distribution.jgroups.JGroupsCacheReplicatorFactory"
                properties="replicateAsynchronously=true, replicatePuts=false,
                            replicateUpdates=true, replicateUpdatesViaCopy=false,
                            replicateRemovals=true, 
                            asynchronousReplicationIntervalMillis=500"
           />
    </cache>
</ehcache>		

JGroupsHibernateAWS.md

Configuring a Hibernate second-level cache in AWS is something of a challenge, as the EhCache multicast discovery mechanism doesn't work there. JGroups is another option, but can be difficult to configure. Here's how I got it working.

I'm using the very nice JGroups-AWS project https://github.com/meltmedia/jgroups-aws. In my configuration, you can see that I use "tags=Env,Role". This means that any given server will query EC2 to find out the values of those tags for itself. For example, suppose the server wakes up and finds that it has Env=Production and Role=API_Server. It will look for other servers with the same tag values (using the AWS webservice endpoints) and form a cluster with them. It checks back periodically so that if servers enter or leave the group it will adjust periodically. Very nice.

The 1.3.0 jgroups-aws uses JGroups 3.1.0, which is a bit out of date. I have not tried forcing a later version in the POM yet. I also cannot completely vouch for all the protocols set up in the ehcache.xml file below; I mostly duplicated the example I found in the jgroups-aws repository.

The secret ingredient

What I found was that JGroups (using Java 8) will preferentially bind to the IPv6 network adapter, while EC2 was returning IPv4 addresses, resulting in the inability to form a cluster. I had to add this to the startup options for Tomcat:

    -Djava.net.preferIPv4Stack=true

pom.xml

... partial ...
		<dependency>
			<groupId>com.meltmedia.jgroups</groupId>
			<artifactId>jgroups-aws</artifactId>
			<version>1.3.0</version>
		</dependency>