PR 448 replication2 results

gistfile1.txt

Tests to run: [{replication2,
                   [{id,-1},
                    {platform,<<"local">>},
                    {version,<<"riak_ee-2.0.0pre7-1-g852b8c2-develop">>},
                    {project,<<"riak_ee">>}]}]
09:45:23.994 [info] Application lager started on node nonode@nohost
09:45:24.001 [info] These modules are not runnable tests: []
09:45:24.053 [info] Skipping cover, no modules included
09:45:24.627 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev1/bin/riak stop ~ wasn't running.
09:45:24.818 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop ~ wasn't running.
09:45:25.063 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/bin/riak stop ~ wasn't running.
09:45:25.302 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev2/bin/riak stop ~ wasn't running.
09:45:25.556 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop ~ wasn't running.
09:45:25.933 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev3/bin/riak stop ~ wasn't running.
09:45:26.167 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak stop ~ wasn't running.
09:45:26.421 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop ~ wasn't running.
09:45:26.561 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev4/bin/riak stop ~ wasn't running.
09:45:27.173 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev5/bin/riak stop ~ wasn't running.
09:45:27.240 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop ~ wasn't running.
09:45:27.275 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/bin/riak stop ~ wasn't running.
09:45:27.769 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev6/bin/riak stop ~ wasn't running.
09:45:28.015 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop ~ wasn't running.
09:45:28.320 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/bin/riak stop ~ wasn't running.
09:45:28.752 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak stop ~ wasn't running.
09:45:29.288 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/bin/riak stop ~ wasn't running.
09:45:30.212 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/bin/riak stop ~ wasn't running.
09:45:30.212 [info] Resetting nodes to fresh state
09:45:30.212 [info] Running: git --git-dir="/Users/janderson/rt/riak_ee/.git" --work-tree="/Users/janderson/rt/riak_ee/" reset HEAD --hard
09:45:31.098 [info] Running: git --git-dir="/Users/janderson/rt/riak_ee/.git" --work-tree="/Users/janderson/rt/riak_ee/" clean -fd
09:45:31.542 [info] Cleaning up lingering pipe directories
09:45:31.548 [warning] rt:set_backend doesn't recognize undefined as a legit backend, using the default.
09:45:31.549 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/advanced.config)
09:45:31.549 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak chkconfig
09:45:32.618 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/advanced.config)
09:45:32.618 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak chkconfig
09:45:33.673 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/advanced.config)
09:45:33.673 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak chkconfig
09:45:34.723 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev4/etc/advanced.config)
09:45:34.723 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak chkconfig
09:45:35.751 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev5/etc/advanced.config)
09:45:35.751 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak chkconfig
09:45:36.801 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev6/etc/advanced.config)
09:45:36.802 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak chkconfig
09:45:37.836 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev1/etc/app.config)
09:45:37.840 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev2/etc/app.config)
09:45:37.843 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev3/etc/app.config)
09:45:37.847 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev4/etc/app.config)
09:45:37.850 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev5/etc/app.config)
09:45:37.854 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev6/etc/app.config)
09:45:37.859 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/etc/app.config)
09:45:37.863 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/etc/app.config)
09:45:37.867 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/etc/app.config)
09:45:37.872 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/etc/app.config)
09:45:37.876 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/etc/app.config)
09:45:37.880 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/etc/app.config)
09:45:37.885 [info] replication2 prereqs: []
09:45:37.885 [notice] Running Test replication2
09:45:38.887 [info] Test Runner `uname -a` : Darwin Jons-MacBook-Pro.local 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64

09:45:38.887 [info] rtdev:set_conf(all, [{"buckets.default.siblings","off"}])
09:45:38.891 [info] Deploy 6 nodes
09:45:38.891 [info] Riak path: "/Users/janderson/rt/riak_ee"
09:45:38.945 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
09:45:38.945 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev6/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
09:45:38.945 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
09:45:38.945 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev5/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
09:45:38.945 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
09:45:38.945 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev4/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
09:45:38.992 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
09:45:38.992 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak start
09:45:38.992 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak start
09:45:38.992 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
09:45:38.992 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
09:45:38.992 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak start
09:45:45.623 [info] Wait until '[email protected]' is pingable
09:45:45.642 [info] Wait until '[email protected]' is pingable
09:45:45.688 [info] Wait until '[email protected]' is pingable
09:45:45.703 [info] Wait until '[email protected]' is pingable
09:45:45.746 [info] Wait until '[email protected]' is pingable
09:45:46.622 [info] Wait until '[email protected]' is pingable
09:45:46.809 [info] Wait until '[email protected]' is pingable
09:45:46.810 [info] Wait until '[email protected]' is pingable
09:45:46.810 [info] Wait until '[email protected]' is pingable
09:45:46.811 [info] Wait until '[email protected]' is pingable
09:45:46.811 [info] Wait until '[email protected]' is pingable
09:45:46.811 [info] Wait until '[email protected]' is pingable
09:45:46.812 [info] Wait until riak_core_ring_manager is up on '[email protected]'
09:45:46.813 [info] Wait until riak_core_ring_manager is up on '[email protected]'
09:45:46.814 [info] Wait until riak_core_ring_manager is up on '[email protected]'
09:45:46.815 [info] Wait until riak_core_ring_manager is up on '[email protected]'
09:45:46.815 [info] Wait until riak_core_ring_manager is up on '[email protected]'
09:45:46.816 [info] Wait until riak_core_ring_manager is up on '[email protected]'
09:45:46.817 [info] Check '[email protected]' is a singleton
09:45:46.844 [info] Check '[email protected]' is a singleton
09:45:46.845 [info] Check '[email protected]' is a singleton
09:45:46.846 [info] Check '[email protected]' is a singleton
09:45:46.847 [info] Check '[email protected]' is a singleton
09:45:46.847 [info] Check '[email protected]' is a singleton
09:45:46.848 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]']
09:45:46.848 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]'].
09:45:59.865 [info] ANodes: ['[email protected]','[email protected]','[email protected]']
09:45:59.865 [info] BNodes: ['[email protected]','[email protected]','[email protected]']
09:45:59.865 [info] Build cluster A
09:45:59.867 [info] Wait until nodes are ready : ['[email protected]','[email protected]','[email protected]']
09:45:59.882 [info] [join] '[email protected]' to ('[email protected]'): ok
09:45:59.908 [info] [join] '[email protected]' to ('[email protected]'): ok
09:45:59.908 [info] Wait until no pending changes on ['[email protected]','[email protected]','[email protected]']
09:45:59.910 [info] Build cluster B
09:45:59.910 [info] Wait until nodes are ready : ['[email protected]','[email protected]','[email protected]']
09:46:00.038 [info] [join] '[email protected]' to ('[email protected]'): ok
09:46:00.062 [info] [join] '[email protected]' to ('[email protected]'): ok
09:46:00.062 [info] Wait until no pending changes on ['[email protected]','[email protected]','[email protected]']
09:46:00.067 [info] Writing 100 keys to '[email protected]'
09:46:00.866 [info] Naming cluster "A"
09:46:00.889 [info] Naming cluster "B"
09:46:00.910 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:46:01.922 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:46:05.929 [info] waiting for leader to converge on cluster A
09:46:05.930 [info] waiting for leader to converge on cluster B
09:46:05.931 [info] connect cluster A:'[email protected]' to B on port 10046
09:46:06.950 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:46:15.997 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:46:26.033 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:46:27.035 [info] Wait for cluster connection A:'[email protected]' -> B:'[email protected]':10046
09:46:27.037 [info] Writing 100 more keys to '[email protected]'
09:46:27.301 [info] Reading 100 keys written to '[email protected]' from '[email protected]'
09:46:30.656 [info] Reads: []
09:46:30.656 [info] Check keys written before repl was connected are not present
09:46:30.723 [info] waiting for fullsync count to be 1
09:46:30.723 [info] Starting fullsync on '[email protected]' (current)
09:47:13.712 [info] Fullsync on '[email protected]' complete
09:47:13.715 [info] Check keys written before repl was connected are present
09:47:14.026 [info] Reads: []
09:47:14.026 [info] disconnect the 2 clusters
09:47:14.037 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:47:15.049 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:47:16.052 [info] write 2000 keys
09:47:19.479 [info] reconnect the 2 clusters
09:47:20.481 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:47:20.494 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:47:21.497 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:47:21.515 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:47:25.524 [info] waiting for fullsync count to be 2
09:47:25.524 [info] Starting fullsync on '[email protected]' (current)
09:48:07.168 [info] Fullsync on '[email protected]' complete
09:48:07.168 [info] read 2000 keys
09:48:09.979 [info] Reads: []
09:48:09.982 [info] Testing master failover: stopping '[email protected]'
09:48:09.982 [info] Stopping riak on '[email protected]'
09:48:19.984 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
09:48:23.513 [info] Wait until '[email protected]' is not pingable
09:48:23.515 [info] New leader is '[email protected]'
09:48:23.516 [info] Writing 100 more keys to '[email protected]' now that the old leader is down
09:48:23.699 [info] Reading 100 keys written to '[email protected]' from '[email protected]'
09:48:23.852 [info] Reads: []
09:48:23.854 [info] Testing client failover: stopping '[email protected]'
09:48:23.854 [info] Stopping riak on '[email protected]'
09:48:33.856 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop
09:49:37.576 [info] Wait until '[email protected]' is not pingable
09:49:37.577 [info] New leader is '[email protected]'
09:49:37.578 [info] Writing 100 more keys to '[email protected]' now that the old leader is down
09:49:37.778 [info] Reading 101 keys written to '[email protected]' from '[email protected]'
09:49:37.920 [info] Reads: []
09:49:37.922 [info] Re-running fullsync with '[email protected]' and '[email protected]' down
09:49:37.924 [info] waiting for fullsync count to be 1
09:49:37.924 [info] Starting fullsync on '[email protected]' (current)
09:49:52.491 [info] Fullsync on '[email protected]' complete
09:49:52.493 [info] Restarting down node '[email protected]'
09:49:52.493 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
09:49:55.507 [info] Wait until '[email protected]' is pingable
09:49:55.607 [info] Wait until '[email protected]' is pingable
09:49:55.609 [info] waiting for fullsync count to be 2
09:49:55.609 [info] Starting fullsync on '[email protected]' (current)
09:50:07.176 [info] Fullsync on '[email protected]' complete
09:50:07.177 [info] Starting Joe's Repl Test
09:50:07.178 [info] Leader: '[email protected]'
09:50:07.178 [info] LeaderA: '[email protected]'
09:50:07.178 [info] LeaderA2: '[email protected]'
09:50:07.181 [info] Simulation partition to force leader re-election
09:50:08.184 [info] Interim leader: '[email protected]'
09:50:12.197 [info] Leader: '[email protected]'
09:50:12.197 [info] Writing 2 more keys to '[email protected]'
09:50:12.211 [info] Reading 2 keys written to '[email protected]' from '[email protected]'
09:50:12.216 [info] Reads: []
09:50:12.218 [info] Finished Joe's Section
09:50:12.218 [info] Restarting down node '[email protected]'
09:50:12.219 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak start
09:50:15.267 [info] Wait until '[email protected]' is pingable
09:50:15.384 [info] Wait until '[email protected]' is pingable
09:50:15.384 [info] Nodes restarted
09:50:15.417 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:50:16.437 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:50:26.463 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:50:36.476 [info] disconnect the 2 clusters
09:50:36.492 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:50:46.522 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:50:56.535 [info] write 100 keys to a realtime only bucket
09:50:56.663 [info] reconnect the 2 clusters
09:50:58.666 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:50:58.679 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:51:05.691 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:51:05.704 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:51:06.711 [info] write 100 keys to a {repl, false} bucket
09:51:06.844 [info] write 100 keys to a fullsync only bucket
09:51:06.986 [info] Check the fullsync only bucket didn't replicate the writes
09:51:07.042 [info] Check the realtime only bucket that was written to offline isn't replicated
09:51:07.095 [info] Check the {repl, false} bucket didn't replicate
09:51:07.148 [info] waiting for fullsync count to be 3
09:51:07.148 [info] Starting fullsync on '[email protected]' (current)
09:51:30.733 [info] Fullsync on '[email protected]' complete
09:51:30.733 [info] Check fullsync only bucket is now replicated
09:51:30.869 [info] Reads: []
09:51:30.869 [info] Check realtime only bucket didn't replicate
09:51:30.916 [info] Write 100 more keys into realtime only bucket on '[email protected]'
09:51:36.075 [info] Check the realtime keys replicated
09:51:36.208 [info] Reads: []
09:51:36.208 [info] Check the older keys in the realtime bucket did not replicate
09:51:36.254 [info] Check {repl, false} bucket didn't replicate
09:51:36.302 [info] Testing offline realtime queueing
09:51:36.302 [info] Stopping realtime, queue will build
09:51:36.328 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:51:37.331 [info] Writing 100 keys
09:51:37.448 [info] Starting realtime
09:51:37.475 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:51:41.479 [info] Reading keys written while repl was stopped
09:51:41.619 [info] Reads: []
09:51:41.620 [info] Testing realtime migration on node shutdown
09:51:41.620 [info] Stopping realtime, queue will build
09:51:41.645 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
09:51:42.647 [info] Writing 100 keys
09:51:42.768 [info] queue status: [{percent_bytes_used,0.033},
09:51:42.768 [info]                {bytes,34204},
09:51:42.769 [info]                {max_bytes,104857600},
09:51:42.769 [info]                {consumers,[{"B",
09:51:42.769 [info]                             [{pending,79},{unacked,0},{drops,0},{errs,0}]}]},
09:51:42.769 [info]                {overload_drops,0}]
09:51:42.769 [info] Stopping node '[email protected]'
09:51:42.769 [info] Stopping riak on '[email protected]'
09:51:52.771 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
09:51:57.276 [info] Wait until '[email protected]' is not pingable
09:51:57.277 [info] Starting realtime
09:52:00.338 [info] Reading keys written while repl was stopped
09:52:00.492 [info] Reads: []
09:52:00.493 [info] Restarting node '[email protected]'
09:52:00.493 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
09:52:03.468 [info] Wait until '[email protected]' is pingable
09:52:03.580 [info] Wait until '[email protected]' is pingable
09:52:18.593 [info] Connecting to pb socket "127.0.0.1":10027 on '[email protected]'
09:52:18.598 [info] Writing 10,000 keys
09:52:19.100 [info] Stopping node '[email protected]' again
09:52:19.100 [info] Stopping riak on '[email protected]'
09:52:29.101 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
09:52:32.830 [info] Node stopped
09:53:30.968 [info] got 2410 write failures
09:53:33.969 [info] checking number of read failures on secondary cluster
09:53:40.252 [info] got 2407 read failures
09:53:40.252 [info] pb_write_during_shutdown: Ensure node '[email protected]' is down before restart
09:53:40.252 [info] Wait until '[email protected]' is not pingable
09:53:40.253 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
09:53:43.210 [info] Wait until '[email protected]' is pingable
09:53:43.301 [info] Wait until '[email protected]' is pingable
09:54:00.714 [info] got 2407 read failures on '[email protected]'
09:54:05.715 [info] Connecting to http socket "127.0.0.1":10028 on '[email protected]'
09:54:05.720 [info] Writing 10,000 keys
09:54:06.221 [info] Stopping node '[email protected]' again
09:54:06.221 [info] Stopping riak on '[email protected]'
09:54:16.222 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
09:54:18.057 [error] gen_server <0.278.0> terminated with reason: connection_closed
09:54:18.057 [error] CRASH REPORT Process <0.278.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.057 [error] gen_server <0.294.0> terminated with reason: connection_closed
09:54:18.057 [error] CRASH REPORT Process <0.294.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.057 [error] gen_server <0.292.0> terminated with reason: connection_closed
09:54:18.058 [error] CRASH REPORT Process <0.292.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.058 [error] gen_server <0.295.0> terminated with reason: connection_closed
09:54:18.058 [error] CRASH REPORT Process <0.295.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.058 [error] gen_server <0.296.0> terminated with reason: connection_closed
09:54:18.058 [error] CRASH REPORT Process <0.296.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.058 [error] gen_server <0.288.0> terminated with reason: connection_closed
09:54:18.058 [error] CRASH REPORT Process <0.288.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.058 [error] gen_server <0.290.0> terminated with reason: connection_closed
09:54:18.058 [error] CRASH REPORT Process <0.290.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.058 [error] gen_server <0.291.0> terminated with reason: connection_closed
09:54:18.058 [error] gen_server <0.293.0> terminated with reason: connection_closed
09:54:18.058 [error] CRASH REPORT Process <0.293.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.058 [error] CRASH REPORT Process <0.291.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:18.059 [error] gen_server <0.289.0> terminated with reason: connection_closed
09:54:18.059 [error] CRASH REPORT Process <0.289.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:54:19.935 [info] Node stopped
09:54:50.612 [info] got 5560 write failures to '[email protected]'
09:54:53.614 [info] checking number of read failures on secondary cluster node, '[email protected]'
09:55:05.968 [info] got 5560 read failures from '[email protected]'
09:55:05.968 [info] http: write_during_shutdown: Ensure node '[email protected]' is down before restart
09:55:05.968 [info] Wait until '[email protected]' is not pingable
09:55:05.969 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
09:55:09.195 [info] Wait until '[email protected]' is pingable
09:55:09.305 [info] Wait until '[email protected]' is pingable
09:55:32.264 [info] got 5560 read failures on '[email protected]'
09:55:32.264 [info] Test passed
09:55:32.264 [notice] replication2 Test Run Complete
09:55:32.265 [info] Multiple tests run or no failure
09:55:34.818 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop ~ wasn't running.
09:55:35.102 [error] gen_server <0.5908.0> terminated with reason: connection_closed
09:55:35.102 [error] CRASH REPORT Process <0.5908.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.103 [error] gen_server <0.5906.0> terminated with reason: connection_closed
09:55:35.103 [error] gen_server <0.5907.0> terminated with reason: connection_closed
09:55:35.103 [error] CRASH REPORT Process <0.5906.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.103 [error] CRASH REPORT Process <0.5907.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.103 [error] gen_server <0.5904.0> terminated with reason: connection_closed
09:55:35.103 [error] CRASH REPORT Process <0.5904.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.103 [error] gen_server <0.5905.0> terminated with reason: connection_closed
09:55:35.103 [error] gen_server <0.5902.0> terminated with reason: connection_closed
09:55:35.103 [error] CRASH REPORT Process <0.5902.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.103 [error] CRASH REPORT Process <0.5905.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.103 [error] gen_server <0.5903.0> terminated with reason: connection_closed
09:55:35.103 [error] CRASH REPORT Process <0.5903.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.104 [error] gen_server <0.5901.0> terminated with reason: connection_closed
09:55:35.104 [error] gen_server <0.5890.0> terminated with reason: connection_closed
09:55:35.104 [error] CRASH REPORT Process <0.5901.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.104 [error] CRASH REPORT Process <0.5890.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:35.104 [error] gen_server <0.5900.0> terminated with reason: connection_closed
09:55:35.104 [error] CRASH REPORT Process <0.5900.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
09:55:36.213 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop ~ wasn't running.
09:55:38.971 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop ~ wasn't running.
09:55:41.574 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop ~ wasn't running.
09:55:44.067 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop ~ wasn't running.
09:55:46.568 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak stop ~ wasn't running.
09:55:46.963 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev1/bin/riak stop ~ wasn't running.
09:55:47.347 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev2/bin/riak stop ~ wasn't running.
09:55:47.740 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev3/bin/riak stop ~ wasn't running.
09:55:48.128 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev4/bin/riak stop ~ wasn't running.
09:55:48.521 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev5/bin/riak stop ~ wasn't running.
09:55:48.945 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev6/bin/riak stop ~ wasn't running.
09:55:49.656 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/bin/riak stop ~ wasn't running.
09:55:50.299 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak stop ~ wasn't running.
09:55:50.952 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/bin/riak stop ~ wasn't running.
09:55:51.596 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/bin/riak stop ~ wasn't running.
09:55:52.233 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/bin/riak stop ~ wasn't running.

Test Results:
replication2-bitcask: pass
09:55:52.930 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/bin/riak stop ~ wasn't running.
---------------------------------------------
0 Tests Failed
1 Tests Passed
That's 100.0% for those keeping score