G1GC / CMS

0000README.md

G1GC v.s. CMS for Cassandra 2.0

Many people have asked me about using the G1 garbage collector with Cassandra. Since most of my customers are running 2.0 in production the test is with 2.0 for now. Once I script it up I'll re-run the numbers with 2.1.

I also need to re-test with Java 7 and the Oracle JDKs.

TL;DR

G1 with "other" settings (see below) can beat CMS on latency and throughput on anemic CPUs if you set -XX:MaxGCPauseMillis=1000.

Settings	Op/s	latency mean	latency median	latency p95	latency p99	latency p99.9	latency max
G1+other	74379	2.7	1.7	4.9	7.5	21.7	2901.2
G1-other	72116	2.8	1.7	5.0	7.7	24.3	1088.2
CMS+other	65681	3.0	1.5	4.7	7.7	276.9	1848.0
CMS-other	62196	3.2	1.6	5.1	8.3	185.8	1615.3
CMS-other	64973	3.1	1.5	4.7	7.4	127.6	1645.7

Hardware / Software

• 6x Intel NUC 16GB RAM, quad-core @ 1.3Ghz, SSD
• Zulu OpenJDK 8
• Ubuntu 14.04 LTS Docker image (tobert/cassandra:2.0.13 w/ zulu openjdk)
• CoreOS 607.0 kernel 3.18.6
• /data is on an Intel mSATA SSD using ext4

cassandra-stress

cassandra-stress \
                write \
                n=50000000 \
                cl=ONE \
                -mode native cql3 \
                -node 192.168.10.12

Other Settings

In the course of tuning for these workloads I've experimented on and off with the following settings. I'll need to break down which is providing what benefit later.

-XX:+AlwaysPreTouch -XX:+ResizeTLAB -XX:-UseBiasedLocking

stress logs

(http://tobert.org/logs/cms-2Gnew-with-other.log.gz) (http://tobert.org/logs/cms-2Gnew-without-other.log.gz) (http://tobert.org/logs/g1-1000ms-with-other.log.gz) (http://tobert.org/logs/g1-1000ms-without-other.log.gz)

010g1-1000ms-with-other.txt

-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=60 -XX:MaxGCPauseMillis=1000 -XX:+AlwaysPreTouch -XX:+ResizeTLAB -XX:-UseBiasedLocking 

op rate                   : 74379 [WRITE:74379]
partition rate            : 74379 [WRITE:74379]
row rate                  : 74379 [WRITE:74379]
latency mean              : 2.7 [WRITE:2.7]
latency median            : 1.7 [WRITE:1.7]
latency 95th percentile   : 4.9 [WRITE:4.9]
latency 99th percentile   : 7.5 [WRITE:7.5]
latency 99.9th percentile : 21.7 [WRITE:21.7]
latency max               : 2901.2 [WRITE:2901.2]
Total partitions          : 50000000 [WRITE:50000000]
Total errors              : 0 [WRITE:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:11:12

020g1-1000ms-without-other.txt

-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=60 -XX:MaxGCPauseMillis=1000

op rate                   : 72116 [WRITE:72116]
partition rate            : 72116 [WRITE:72116]
row rate                  : 72116 [WRITE:72116]
latency mean              : 2.8 [WRITE:2.8]
latency median            : 1.7 [WRITE:1.7]
latency 95th percentile   : 5.0 [WRITE:5.0]
latency 99th percentile   : 7.7 [WRITE:7.7]
latency 99.9th percentile : 24.3 [WRITE:24.3]
latency max               : 1088.2 [WRITE:1088.2]
Total partitions          : 50000000 [WRITE:50000000]
Total errors              : 0 [WRITE:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:11:33
END

030cms-2Gnew-without-other.txt

-XX:+UseParNewGC -XX:+UseConcMarkSweepGC

Run 2: I wanted to add links to the stress logs and lost the one from above so I ran again and got better numbers:

op rate                   : 64973 [WRITE:64973]
partition rate            : 64973 [WRITE:64973]
row rate                  : 64973 [WRITE:64973]
latency mean              : 3.1 [WRITE:3.1]
latency median            : 1.5 [WRITE:1.5]
latency 95th percentile   : 4.7 [WRITE:4.7]
latency 99th percentile   : 7.4 [WRITE:7.4]
latency 99.9th percentile : 127.6 [WRITE:127.6]
latency max               : 1645.7 [WRITE:1645.7]
Total partitions          : 50000000 [WRITE:50000000]

Run 1:

op rate                   : 62196 [WRITE:62196]
partition rate            : 62196 [WRITE:62196]
row rate                  : 62196 [WRITE:62196]
latency mean              : 3.2 [WRITE:3.2]
latency median            : 1.6 [WRITE:1.6]
latency 95th percentile   : 5.1 [WRITE:5.1]
latency 99th percentile   : 8.3 [WRITE:8.3]
latency 99.9th percentile : 185.8 [WRITE:185.8]
latency max               : 1615.3 [WRITE:1615.3]
Total partitions          : 50000000 [WRITE:50000000]
Total errors              : 0 [WRITE:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:13:23

040cms-2Gnew-with-other.txt

-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+AlwaysPreTouch -XX:+ResizeTLAB -XX:-UseBiasedLocking 

op rate                   : 65681 [WRITE:65681]
partition rate            : 65681 [WRITE:65681]
row rate                  : 65681 [WRITE:65681]
latency mean              : 3.0 [WRITE:3.0]
latency median            : 1.5 [WRITE:1.5]
latency 95th percentile   : 4.7 [WRITE:4.7]
latency 99th percentile   : 7.7 [WRITE:7.7]
latency 99.9th percentile : 276.8 [WRITE:276.8]
latency max               : 1848.0 [WRITE:1848.0]
Total partitions          : 50000000 [WRITE:50000000]
Total errors              : 0 [WRITE:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:12:41

110g1-1000ms-with-other.yaml

# G1GC settings for Cassandra 2.0 (or 2.1)
chdir: /data
stdin: /dev/null
stdout: /data/log/console.log
stderr: /data/log/console.log
uid: 1337
gid: 1337
argv:
  - /usr/bin/java
  - -ea
  - -javaagent:{{ glob "" "/opt/cassandra/lib/jamm-*.jar" }}
  - -Xmx8G
  - -Xms8G
  - -Xss256k
  - -XX:+UseG1GC                          # use garbage-first collection
  - -XX:InitiatingHeapOccupancyPercent=60 # don't do mixed mode until the heap is this full
  - -XX:MaxGCPauseMillis=1000             # default is 200ms
  - -XX:+AlwaysPreTouch                   # allocate and zero (force fault) heap memory on startup
  - -XX:+UseTLAB                          # thread local allocation blocks
  - -XX:+ResizeTLAB                       # auto-optimize TLAB size https://blogs.oracle.com/jonthecollector/entry/the_real_thing
  - -XX:-UseBiasedLocking                 # disable biased locking for cassandra
  - -XX:StringTableSize=1000003
  - -XX:CompileCommandFile=/data/conf/hotspot_compiler
  - -Djava.net.preferIPv4Stack=true
  - -Dcom.sun.management.jmxremote.port=7199
  - -Dcom.sun.management.jmxremote.rmi.port=7199
  - -Dcom.sun.management.jmxremote.ssl=false
  - -Dcom.sun.management.jmxremote.authenticate=false
  - -Dlog4j.defaultInitOverride=true
  - -Dlog4j.configuration=log4j-server.properties # cassandra <= 2.0
  - -Dlogback.configurationFile=logback.xml       # cassandra >= 2.1
  - -Dcassandra.logdir=/data/log
  - -Dcassandra-foreground=yes
  - -XX:+HeapDumpOnOutOfMemoryError
  - -XX:+PrintGCDetails
  - -XX:+PrintAdaptiveSizePolicy
  - -XX:+PrintGCApplicationStoppedTime
  - -XX:+PrintPromotionFailure
  - -Xloggc:/data/log/gc.log
  - -XX:+UseGCLogFileRotation
  - -XX:NumberOfGCLogFiles=10
  - -XX:GCLogFileSize=10M
  - -cp
  - {{ glob ":" "/data/conf" "/data/lib" "/opt/cassandra/lib/*.jar" }}
  - org.apache.cassandra.service.CassandraDaemon
{{ range .ExtraArgs }}  - {{ . }}
{{ end }}

120g1-1000ms-without-other.yaml

# G1GC without unrelated settings
chdir: /data
stdin: /dev/null
stdout: /data/log/console.log
stderr: /data/log/console.log
uid: 1337
gid: 1337
argv:
  - /usr/bin/java
  - -ea
  - -javaagent:{{ glob "" "/opt/cassandra/lib/jamm-*.jar" }}
  - -Xmx8G
  - -Xms8G
  - -Xss256k
  - -XX:+UseG1GC                          # use garbage-first collection
  - -XX:InitiatingHeapOccupancyPercent=60 # don't do mixed mode until the heap is this full
  - -XX:MaxGCPauseMillis=1000             # default is 200ms
  - -XX:+UseTLAB                          # thread local allocation blocks
  - -XX:StringTableSize=1000003
  - -XX:CompileCommandFile=/data/conf/hotspot_compiler
  - -Djava.net.preferIPv4Stack=true
  - -Dcom.sun.management.jmxremote.port=7199
  - -Dcom.sun.management.jmxremote.rmi.port=7199
  - -Dcom.sun.management.jmxremote.ssl=false
  - -Dcom.sun.management.jmxremote.authenticate=false
  - -Dlog4j.defaultInitOverride=true
  - -Dlog4j.configuration=log4j-server.properties # cassandra <= 2.0
  - -Dlogback.configurationFile=logback.xml       # cassandra >= 2.1
  - -Dcassandra.logdir=/data/log
  - -Dcassandra-foreground=yes
  - -XX:+HeapDumpOnOutOfMemoryError
  - -XX:+PrintGCDetails
  - -XX:+PrintAdaptiveSizePolicy
  - -XX:+PrintGCApplicationStoppedTime
  - -XX:+PrintPromotionFailure
  - -Xloggc:/data/log/gc.log
  - -XX:+UseGCLogFileRotation
  - -XX:NumberOfGCLogFiles=10
  - -XX:GCLogFileSize=10M
  - -cp
  - {{ glob ":" "/data/conf" "/data/lib" "/opt/cassandra/lib/*.jar" }}
  - org.apache.cassandra.service.CassandraDaemon
{{ range .ExtraArgs }}  - {{ . }}
{{ end }}

130cms-2Gnew-with-other.yaml

# default Cassandra 2.0 CMS settings with the 3 other settings from the
# G1 settings that might make a difference
chdir: {{ .DataDir }}
stdin: /dev/null
stdout: /data/log/console.log
stderr: /data/log/console.log
uid: 1337
gid: 1337
argv:
  - /usr/bin/java
  - -ea
  - -javaagent:{{ glob "" "/opt/cassandra/lib/jamm-*.jar" }}
  - -XX:+CMSClassUnloadingEnabled
  - -XX:+UseThreadPriorities
  - -XX:ThreadPriorityPolicy=42
  - -Xmx8G
  - -Xms8G
  - -Xmn2G
  - -XX:+HeapDumpOnOutOfMemoryError
  - -Xss256k
  - -XX:+AlwaysPreTouch                   # allocate and zero (force fault) heap memory on startup
  - -XX:+UseTLAB                          # thread local allocation blocks
  - -XX:+ResizeTLAB                       # auto-optimize https://blogs.oracle.com/jonthecollector/entry/the_real_thing
  - -XX:-UseBiasedLocking                 # disable biased locking for cassandra
  - -XX:StringTableSize=1000003
  - -XX:+UseParNewGC
  - -XX:+UseConcMarkSweepGC
  - -XX:+CMSParallelRemarkEnabled
  - -XX:SurvivorRatio=8
  - -XX:MaxTenuringThreshold=1
  - -XX:CMSInitiatingOccupancyFraction=75
  - -XX:+UseCMSInitiatingOccupancyOnly
  - -XX:CompileCommandFile={{ .ConfDir }}/hotspot_compiler
  - -XX:+UseCondCardMark
  - -XX:+CMSParallelInitialMarkEnabled
  - -XX:+CMSEdenChunksRecordAlways
  - -Djava.net.preferIPv4Stack=true
  - -Dcom.sun.management.jmxremote.port={{ .JmxPort }}
  - -Dcom.sun.management.jmxremote.rmi.port={{ .JmxPort }}
  - -Dcom.sun.management.jmxremote.ssl=false
  - -Dcom.sun.management.jmxremote.authenticate=false
  - -Dlog4j.configuration=log4j-server.properties
  - -Dlog4j.defaultInitOverride=true
  - -Dcassandra.logdir={{ .LogDir }}
  - -Dcassandra-foreground=yes
  - -cp
  - {{ glob ":" .ConfDir .LibDir "/opt/cassandra/lib/*.jar" }}
  - org.apache.cassandra.service.CassandraDaemon
{{ range .ExtraArgs }}  - {{ . }}
{{ end }}

140cms-2Gnew-without-other.yaml

# default Cassandra 2.0 settings
chdir: {{ .DataDir }}
stdin: /dev/null
stdout: /data/log/console.log
stderr: /data/log/console.log
uid: 1337
gid: 1337
argv:
  - /usr/bin/java
  - -ea
  - -javaagent:{{ glob "" "/opt/cassandra/lib/jamm-*.jar" }}
  - -XX:+CMSClassUnloadingEnabled
  - -XX:+UseThreadPriorities
  - -XX:ThreadPriorityPolicy=42
  - -Xmx8G
  - -Xms8G
  - -Xmn2G
  - -XX:+HeapDumpOnOutOfMemoryError
  - -Xss256k
  - -XX:StringTableSize=1000003
  - -XX:+UseParNewGC
  - -XX:+UseConcMarkSweepGC
  - -XX:+CMSParallelRemarkEnabled
  - -XX:SurvivorRatio=8
  - -XX:MaxTenuringThreshold=1
  - -XX:CMSInitiatingOccupancyFraction=75
  - -XX:+UseCMSInitiatingOccupancyOnly
  - -XX:+UseTLAB
  - -XX:CompileCommandFile={{ .ConfDir }}/hotspot_compiler
  - -XX:+UseCondCardMark
  - -XX:+CMSParallelInitialMarkEnabled
  - -XX:+CMSEdenChunksRecordAlways
  - -Djava.net.preferIPv4Stack=true
  - -Dcom.sun.management.jmxremote.port={{ .JmxPort }}
  - -Dcom.sun.management.jmxremote.rmi.port={{ .JmxPort }}
  - -Dcom.sun.management.jmxremote.ssl=false
  - -Dcom.sun.management.jmxremote.authenticate=false
  - -Dlog4j.configuration=log4j-server.properties
  - -Dlog4j.defaultInitOverride=true
  - -Dcassandra.logdir={{ .LogDir }}
  - -Dcassandra-foreground=yes
  - -cp
  - {{ glob ":" .ConfDir .LibDir "/opt/cassandra/lib/*.jar" }}
  - org.apache.cassandra.service.CassandraDaemon
{{ range .ExtraArgs }}  - {{ . }}
{{ end }}