Replication2 results for repl PR487

gistfile1.txt

[ 0 ] janderson::Jons-MacBook-Pro { ~/sandbox/features/riak_test }->  ./riak_test -c rtee -t replication2
Tests to run: [{replication2,
                   [{id,-1},
                    {platform,<<"local">>},
                    {version,<<"riak_ee-2.0.0pre7-1-g852b8c2-develop">>},
                    {project,<<"riak_ee">>}]}]
16:09:04.604 [info] Application lager started on node nonode@nohost
16:09:04.608 [info] These modules are not runnable tests: []
16:09:04.655 [info] Skipping cover, no modules included
16:09:05.179 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev1/bin/riak stop ~ wasn't running.
16:09:05.373 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop ~ wasn't running.
16:09:05.544 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/bin/riak stop ~ wasn't running.
16:09:05.687 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev2/bin/riak stop ~ wasn't running.
16:09:06.121 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop ~ wasn't running.
16:09:06.236 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev3/bin/riak stop ~ wasn't running.
16:09:06.466 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak stop ~ wasn't running.
16:09:06.759 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev4/bin/riak stop ~ wasn't running.
16:09:06.845 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop ~ wasn't running.
16:09:07.340 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev5/bin/riak stop ~ wasn't running.
16:09:07.407 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/bin/riak stop ~ wasn't running.
16:09:07.647 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop ~ wasn't running.
16:09:07.876 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev6/bin/riak stop ~ wasn't running.
16:09:08.329 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/bin/riak stop ~ wasn't running.
16:09:08.392 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop ~ wasn't running.
16:09:09.052 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak stop ~ wasn't running.
16:09:09.193 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/bin/riak stop ~ wasn't running.
16:09:09.943 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/bin/riak stop ~ wasn't running.
16:09:09.943 [info] Resetting nodes to fresh state
16:09:09.943 [info] Running: git --git-dir="/Users/janderson/rt/riak_ee/.git" --work-tree="/Users/janderson/rt/riak_ee/" reset HEAD --hard
16:09:10.774 [info] Running: git --git-dir="/Users/janderson/rt/riak_ee/.git" --work-tree="/Users/janderson/rt/riak_ee/" clean -fd
16:09:11.160 [info] Cleaning up lingering pipe directories
16:09:11.230 [warning] rt:set_backend doesn't recognize undefined as a legit backend, using the default.
16:09:11.232 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/advanced.config)
16:09:11.232 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak chkconfig
16:09:12.141 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/advanced.config)
16:09:12.141 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak chkconfig
16:09:13.016 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/advanced.config)
16:09:13.016 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak chkconfig
16:09:13.983 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev4/etc/advanced.config)
16:09:13.984 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak chkconfig
16:09:14.860 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev5/etc/advanced.config)
16:09:14.860 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak chkconfig
16:09:15.780 [info] get_backend(/Users/janderson/rt/riak_ee/current/dev/dev6/etc/advanced.config)
16:09:15.781 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak chkconfig
16:09:16.714 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev1/etc/app.config)
16:09:16.717 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev2/etc/app.config)
16:09:16.719 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev3/etc/app.config)
16:09:16.722 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev4/etc/app.config)
16:09:16.725 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev5/etc/app.config)
16:09:16.728 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev6/etc/app.config)
16:09:16.733 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/etc/app.config)
16:09:16.736 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/etc/app.config)
16:09:16.740 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/etc/app.config)
16:09:16.744 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/etc/app.config)
16:09:16.748 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/etc/app.config)
16:09:16.752 [info] get_backend(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/etc/app.config)
16:09:16.756 [info] replication2 prereqs: []
16:09:16.756 [notice] Running Test replication2
16:09:17.759 [info] Test Runner `uname -a` : Darwin Jons-MacBook-Pro.local 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64

16:09:17.759 [info] rtdev:set_conf(all, [{"buckets.default.siblings","off"}])
16:09:17.762 [info] Deploy 6 nodes
16:09:17.762 [info] Riak path: "/Users/janderson/rt/riak_ee"
16:09:17.810 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev6/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
16:09:17.810 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev4/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
16:09:17.810 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev5/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
16:09:17.810 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
16:09:17.810 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
16:09:17.810 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/advanced.config, [{riak_kv,[{anti_entropy,{on,[]}},{anti_entropy_build_limit,{100,1000}},{anti_entropy_concurrency,100}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
16:09:17.855 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak start
16:09:17.855 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak start
16:09:17.856 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak start
16:09:17.856 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
16:09:17.856 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
16:09:17.856 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
16:09:24.103 [info] Wait until '[email protected]' is pingable
16:09:24.122 [info] Wait until '[email protected]' is pingable
16:09:24.152 [info] Wait until '[email protected]' is pingable
16:09:24.200 [info] Wait until '[email protected]' is pingable
16:09:24.328 [info] Wait until '[email protected]' is pingable
16:09:24.481 [info] Wait until '[email protected]' is pingable
16:09:24.746 [info] Wait until '[email protected]' is pingable
16:09:24.747 [info] Wait until '[email protected]' is pingable
16:09:24.747 [info] Wait until '[email protected]' is pingable
16:09:24.748 [info] Wait until '[email protected]' is pingable
16:09:24.748 [info] Wait until '[email protected]' is pingable
16:09:24.749 [info] Wait until '[email protected]' is pingable
16:09:24.750 [info] Wait until riak_core_ring_manager is up on '[email protected]'
16:09:24.751 [info] Wait until riak_core_ring_manager is up on '[email protected]'
16:09:24.752 [info] Wait until riak_core_ring_manager is up on '[email protected]'
16:09:24.752 [info] Wait until riak_core_ring_manager is up on '[email protected]'
16:09:24.754 [info] Wait until riak_core_ring_manager is up on '[email protected]'
16:09:24.754 [info] Wait until riak_core_ring_manager is up on '[email protected]'
16:09:24.756 [info] Check '[email protected]' is a singleton
16:09:24.855 [info] Check '[email protected]' is a singleton
16:09:24.923 [info] Check '[email protected]' is a singleton
16:09:24.924 [info] Check '[email protected]' is a singleton
16:09:24.925 [info] Check '[email protected]' is a singleton
16:09:24.926 [info] Check '[email protected]' is a singleton
16:09:24.926 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]']
16:09:24.926 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]'].
16:09:36.177 [info] ANodes: ['[email protected]','[email protected]','[email protected]']
16:09:36.177 [info] BNodes: ['[email protected]','[email protected]','[email protected]']
16:09:36.177 [info] Build cluster A
16:09:36.204 [info] Wait until nodes are ready : ['[email protected]','[email protected]','[email protected]']
16:09:36.245 [info] [join] '[email protected]' to ('[email protected]'): ok
16:09:36.263 [info] [join] '[email protected]' to ('[email protected]'): ok
16:09:36.263 [info] Wait until no pending changes on ['[email protected]','[email protected]','[email protected]']
16:09:36.264 [info] Build cluster B
16:09:36.264 [info] Wait until nodes are ready : ['[email protected]','[email protected]','[email protected]']
16:09:36.314 [info] [join] '[email protected]' to ('[email protected]'): ok
16:09:36.379 [info] [join] '[email protected]' to ('[email protected]'): ok
16:09:36.380 [info] Wait until no pending changes on ['[email protected]','[email protected]','[email protected]']
16:09:36.388 [info] Writing 100 keys to '[email protected]'
16:09:37.421 [info] Naming cluster "A"
16:09:37.448 [info] Naming cluster "B"
16:09:37.458 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:09:38.469 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:09:43.475 [info] waiting for leader to converge on cluster A
16:09:43.476 [info] waiting for leader to converge on cluster B
16:09:43.477 [info] connect cluster A:'[email protected]' to B on port 10046
16:09:44.500 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:09:45.513 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:09:54.558 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:09:55.561 [info] Wait for cluster connection A:'[email protected]' -> B:'[email protected]':10046
16:09:55.563 [info] Writing 100 more keys to '[email protected]'
16:09:55.778 [info] Reading 100 keys written to '[email protected]' from '[email protected]'
16:09:58.518 [info] Reads: []
16:09:58.518 [info] Check keys written before repl was connected are not present
16:09:58.587 [info] waiting for fullsync count to be 1
16:09:58.587 [info] Starting fullsync on '[email protected]' (current)
16:10:51.237 [info] Fullsync on '[email protected]' complete
16:10:51.239 [info] Check keys written before repl was connected are present
16:10:51.519 [info] Reads: []
16:10:51.519 [info] disconnect the 2 clusters
16:10:51.557 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:10:52.573 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:11:03.590 [info] write 2000 keys
16:11:06.136 [info] reconnect the 2 clusters
16:11:07.137 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:11:07.162 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:11:14.173 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:11:14.181 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:11:15.188 [info] waiting for fullsync count to be 2
16:11:15.188 [info] Starting fullsync on '[email protected]' (current)
16:12:01.843 [info] Fullsync on '[email protected]' complete
16:12:01.843 [info] read 2000 keys
16:12:04.816 [info] Reads: []
16:12:04.819 [info] Testing master failover: stopping '[email protected]'
16:12:04.819 [info] Stopping riak on '[email protected]'
16:12:14.821 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
16:12:18.322 [info] Wait until '[email protected]' is not pingable
16:12:18.323 [info] New leader is '[email protected]'
16:12:18.325 [info] Writing 100 more keys to '[email protected]' now that the old leader is down
16:12:18.626 [info] Reading 100 keys written to '[email protected]' from '[email protected]'
16:12:18.757 [info] Reads: []
16:12:18.758 [info] Testing client failover: stopping '[email protected]'
16:12:18.759 [info] Stopping riak on '[email protected]'
16:12:28.761 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop
16:12:32.245 [info] Wait until '[email protected]' is not pingable
16:12:32.247 [info] New leader is '[email protected]'
16:12:32.248 [info] Writing 100 more keys to '[email protected]' now that the old leader is down
16:12:32.395 [info] Reading 101 keys written to '[email protected]' from '[email protected]'
16:12:32.563 [info] Reads: []
16:12:32.565 [info] Re-running fullsync with '[email protected]' and '[email protected]' down
16:12:32.566 [info] waiting for fullsync count to be 1
16:12:32.566 [info] Starting fullsync on '[email protected]' (current)
16:12:47.108 [info] Fullsync on '[email protected]' complete
16:12:47.111 [info] Restarting down node '[email protected]'
16:12:47.111 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
16:12:50.299 [info] Wait until '[email protected]' is pingable
16:12:50.402 [info] Wait until '[email protected]' is pingable
16:12:50.404 [info] waiting for fullsync count to be 2
16:12:50.404 [info] Starting fullsync on '[email protected]' (current)
16:13:04.945 [info] Fullsync on '[email protected]' complete
16:13:04.946 [info] Starting Joe's Repl Test
16:13:04.946 [info] Leader: '[email protected]'
16:13:04.946 [info] LeaderA: '[email protected]'
16:13:04.946 [info] LeaderA2: '[email protected]'
16:13:04.950 [info] Simulation partition to force leader re-election
16:13:05.953 [info] Interim leader: '[email protected]'
16:13:09.966 [info] Leader: '[email protected]'
16:13:09.966 [info] Writing 2 more keys to '[email protected]'
16:13:10.009 [info] Reading 2 keys written to '[email protected]' from '[email protected]'
16:13:10.014 [info] Reads: []
16:13:10.016 [info] Finished Joe's Section
16:13:10.016 [info] Restarting down node '[email protected]'
16:13:10.016 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak start
16:13:13.280 [info] Wait until '[email protected]' is pingable
16:13:13.383 [info] Wait until '[email protected]' is pingable
16:13:13.383 [info] Nodes restarted
16:13:13.430 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:14.447 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:15.467 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:23.478 [info] disconnect the 2 clusters
16:13:23.485 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:24.531 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:30.542 [info] write 100 keys to a realtime only bucket
16:13:30.705 [info] reconnect the 2 clusters
16:13:31.707 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:31.724 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:39.739 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:39.758 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:13:40.763 [info] write 100 keys to a {repl, false} bucket
16:13:40.885 [info] write 100 keys to a fullsync only bucket
16:13:41.004 [info] Check the fullsync only bucket didn't replicate the writes
16:13:41.057 [info] Check the realtime only bucket that was written to offline isn't replicated
16:13:41.104 [info] Check the {repl, false} bucket didn't replicate
16:13:41.153 [info] waiting for fullsync count to be 1
16:13:41.153 [info] Starting fullsync on '[email protected]' (current)
16:14:00.733 [info] Fullsync on '[email protected]' complete
16:14:00.733 [info] Check fullsync only bucket is now replicated
16:14:00.907 [info] Reads: []
16:14:00.907 [info] Check realtime only bucket didn't replicate
16:14:00.955 [info] Write 100 more keys into realtime only bucket on '[email protected]'
16:14:06.121 [info] Check the realtime keys replicated
16:14:06.275 [info] Reads: []
16:14:06.275 [info] Check the older keys in the realtime bucket did not replicate
16:14:06.326 [info] Check {repl, false} bucket didn't replicate
16:14:06.373 [info] Testing offline realtime queueing
16:14:06.374 [info] Stopping realtime, queue will build
16:14:06.427 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:14:07.430 [info] Writing 100 keys
16:14:07.565 [info] Starting realtime
16:14:07.583 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:14:11.587 [info] Reading keys written while repl was stopped
16:14:11.734 [info] Reads: []
16:14:11.736 [info] Testing realtime migration on node shutdown
16:14:11.736 [info] Stopping realtime, queue will build
16:14:11.760 [info] Wait until ring converged on ['[email protected]','[email protected]','[email protected]']
16:14:12.763 [info] Writing 100 keys
16:14:12.886 [info] queue status: [{percent_bytes_used,0.033},
16:14:12.886 [info]                {bytes,35046},
16:14:12.886 [info]                {max_bytes,104857600},
16:14:12.886 [info]                {consumers,[{"B",
16:14:12.886 [info]                             [{pending,81},{unacked,0},{drops,0},{errs,0}]}]},
16:14:12.886 [info]                {overload_drops,0}]
16:14:12.886 [info] Stopping node '[email protected]'
16:14:12.886 [info] Stopping riak on '[email protected]'
16:14:22.888 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
16:15:27.610 [info] Wait until '[email protected]' is not pingable
16:15:27.611 [info] Starting realtime
16:15:30.639 [info] Reading keys written while repl was stopped
16:15:30.785 [info] Reads: []
16:15:30.785 [info] Restarting node '[email protected]'
16:15:30.785 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
16:15:34.049 [info] Wait until '[email protected]' is pingable
16:15:34.175 [info] Wait until '[email protected]' is pingable
16:15:49.188 [info] Connecting to pb socket "127.0.0.1":10027 on '[email protected]'
16:15:49.194 [info] Writing 10,000 keys
16:15:49.695 [info] Stopping node '[email protected]' again
16:15:49.695 [info] Stopping riak on '[email protected]'
16:15:59.696 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
16:16:03.425 [info] Node stopped
16:17:01.561 [info] got 3029 write failures
16:17:04.563 [info] checking number of read failures on secondary cluster
16:17:10.857 [info] got 3026 read failures
16:17:10.857 [info] pb_write_during_shutdown: Ensure node '[email protected]' is down before restart
16:17:10.857 [info] Wait until '[email protected]' is not pingable
16:17:10.858 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
16:17:13.826 [info] Wait until '[email protected]' is pingable
16:17:13.931 [info] Wait until '[email protected]' is pingable
16:17:30.590 [info] got 3026 read failures on '[email protected]'
16:17:35.591 [info] Connecting to http socket "127.0.0.1":10028 on '[email protected]'
16:17:35.597 [info] Writing 10,000 keys
16:17:36.098 [info] Stopping node '[email protected]' again
16:17:36.098 [info] Stopping riak on '[email protected]'
16:17:46.099 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
16:17:47.961 [error] gen_server <0.280.0> terminated with reason: connection_closed
16:17:47.961 [error] CRASH REPORT Process <0.280.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.962 [error] gen_server <0.290.0> terminated with reason: connection_closed
16:17:47.962 [error] CRASH REPORT Process <0.290.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.963 [error] gen_server <0.300.0> terminated with reason: connection_closed
16:17:47.963 [error] CRASH REPORT Process <0.300.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.963 [error] gen_server <0.291.0> terminated with reason: connection_closed
16:17:47.963 [error] CRASH REPORT Process <0.291.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.964 [error] gen_server <0.301.0> terminated with reason: connection_closed
16:17:47.964 [error] CRASH REPORT Process <0.301.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.964 [error] gen_server <0.292.0> terminated with reason: connection_closed
16:17:47.964 [error] CRASH REPORT Process <0.292.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.965 [error] gen_server <0.303.0> terminated with reason: connection_closed
16:17:47.965 [error] CRASH REPORT Process <0.303.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.965 [error] gen_server <0.293.0> terminated with reason: connection_closed
16:17:47.965 [error] CRASH REPORT Process <0.293.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.965 [error] gen_server <0.302.0> terminated with reason: connection_closed
16:17:47.966 [error] CRASH REPORT Process <0.302.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.966 [error] gen_server <0.304.0> terminated with reason: connection_closed
16:17:47.966 [error] CRASH REPORT Process <0.304.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.966 [error] gen_server <0.305.0> terminated with reason: connection_closed
16:17:47.966 [error] CRASH REPORT Process <0.305.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.967 [error] gen_server <0.306.0> terminated with reason: connection_closed
16:17:47.967 [error] CRASH REPORT Process <0.306.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.967 [error] gen_server <0.294.0> terminated with reason: connection_closed
16:17:47.968 [error] CRASH REPORT Process <0.294.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.968 [error] gen_server <0.308.0> terminated with reason: connection_closed
16:17:47.968 [error] CRASH REPORT Process <0.308.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.968 [error] gen_server <0.307.0> terminated with reason: connection_closed
16:17:47.968 [error] CRASH REPORT Process <0.307.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.969 [error] gen_server <0.310.0> terminated with reason: connection_closed
16:17:47.969 [error] CRASH REPORT Process <0.310.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.970 [error] gen_server <0.312.0> terminated with reason: connection_closed
16:17:47.970 [error] CRASH REPORT Process <0.312.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.970 [error] gen_server <0.309.0> terminated with reason: connection_closed
16:17:47.970 [error] CRASH REPORT Process <0.309.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.970 [error] gen_server <0.311.0> terminated with reason: connection_closed
16:17:47.970 [error] CRASH REPORT Process <0.311.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.970 [error] gen_server <0.296.0> terminated with reason: connection_closed
16:17:47.970 [error] CRASH REPORT Process <0.296.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.970 [error] gen_server <0.313.0> terminated with reason: connection_closed
16:17:47.971 [error] CRASH REPORT Process <0.313.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.971 [error] gen_server <0.314.0> terminated with reason: connection_closed
16:17:47.971 [error] CRASH REPORT Process <0.314.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.971 [error] gen_server <0.315.0> terminated with reason: connection_closed
16:17:47.971 [error] CRASH REPORT Process <0.315.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.971 [error] gen_server <0.316.0> terminated with reason: connection_closed
16:17:47.971 [error] CRASH REPORT Process <0.316.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.972 [error] gen_server <0.297.0> terminated with reason: connection_closed
16:17:47.972 [error] CRASH REPORT Process <0.297.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.972 [error] gen_server <0.320.0> terminated with reason: connection_closed
16:17:47.972 [error] CRASH REPORT Process <0.320.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.972 [error] gen_server <0.318.0> terminated with reason: connection_closed
16:17:47.972 [error] CRASH REPORT Process <0.318.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.973 [error] gen_server <0.319.0> terminated with reason: connection_closed
16:17:47.973 [error] CRASH REPORT Process <0.319.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.973 [error] gen_server <0.317.0> terminated with reason: connection_closed
16:17:47.973 [error] CRASH REPORT Process <0.317.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.973 [error] gen_server <0.321.0> terminated with reason: connection_closed
16:17:47.973 [error] CRASH REPORT Process <0.321.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.973 [error] gen_server <0.322.0> terminated with reason: connection_closed
16:17:47.973 [error] CRASH REPORT Process <0.322.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.974 [error] gen_server <0.298.0> terminated with reason: connection_closed
16:17:47.974 [error] CRASH REPORT Process <0.298.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.974 [error] gen_server <0.323.0> terminated with reason: connection_closed
16:17:47.974 [error] CRASH REPORT Process <0.323.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.975 [error] gen_server <0.326.0> terminated with reason: connection_closed
16:17:47.975 [error] CRASH REPORT Process <0.326.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.975 [error] gen_server <0.327.0> terminated with reason: connection_closed
16:17:47.975 [error] CRASH REPORT Process <0.327.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.976 [error] gen_server <0.328.0> terminated with reason: connection_closed
16:17:47.976 [error] CRASH REPORT Process <0.328.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.976 [error] gen_server <0.325.0> terminated with reason: connection_closed
16:17:47.976 [error] CRASH REPORT Process <0.325.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.976 [error] gen_server <0.324.0> terminated with reason: connection_closed
16:17:47.976 [error] CRASH REPORT Process <0.324.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.976 [error] gen_server <0.329.0> terminated with reason: connection_closed
16:17:47.976 [error] CRASH REPORT Process <0.329.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.976 [error] gen_server <0.330.0> terminated with reason: connection_closed
16:17:47.976 [error] CRASH REPORT Process <0.330.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.976 [error] gen_server <0.331.0> terminated with reason: connection_closed
16:17:47.977 [error] CRASH REPORT Process <0.331.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.977 [error] gen_server <0.335.0> terminated with reason: connection_closed
16:17:47.977 [error] CRASH REPORT Process <0.335.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.977 [error] gen_server <0.334.0> terminated with reason: connection_closed
16:17:47.977 [error] CRASH REPORT Process <0.334.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.977 [error] gen_server <0.332.0> terminated with reason: connection_closed
16:17:47.978 [error] CRASH REPORT Process <0.332.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.978 [error] gen_server <0.333.0> terminated with reason: connection_closed
16:17:47.978 [error] CRASH REPORT Process <0.333.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.978 [error] gen_server <0.336.0> terminated with reason: connection_closed
16:17:47.978 [error] CRASH REPORT Process <0.336.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:47.978 [error] gen_server <0.338.0> terminated with reason: connection_closed
16:17:48.000 [warning] lager_error_logger_h dropped 70 messages in the last second that exceeded the limit of 50 messages/sec
16:17:48.000 [error] gen_server <0.430.0> terminated with reason: connection_closed
16:17:48.000 [error] CRASH REPORT Process <0.430.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.001 [error] gen_server <0.431.0> terminated with reason: connection_closed
16:17:48.001 [error] gen_server <0.433.0> terminated with reason: connection_closed
16:17:48.001 [error] CRASH REPORT Process <0.431.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.001 [error] CRASH REPORT Process <0.433.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.001 [error] gen_server <0.434.0> terminated with reason: connection_closed
16:17:48.001 [error] gen_server <0.432.0> terminated with reason: connection_closed
16:17:48.001 [error] CRASH REPORT Process <0.434.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.001 [error] CRASH REPORT Process <0.432.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.002 [error] gen_server <0.428.0> terminated with reason: connection_closed
16:17:48.002 [error] gen_server <0.429.0> terminated with reason: connection_closed
16:17:48.002 [error] CRASH REPORT Process <0.428.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.002 [error] CRASH REPORT Process <0.429.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.002 [error] gen_server <0.435.0> terminated with reason: connection_closed
16:17:48.002 [error] CRASH REPORT Process <0.435.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.003 [error] gen_server <0.443.0> terminated with reason: connection_closed
16:17:48.003 [error] CRASH REPORT Process <0.443.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.003 [error] gen_server <0.442.0> terminated with reason: connection_closed
16:17:48.004 [error] CRASH REPORT Process <0.442.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.004 [error] gen_server <0.436.0> terminated with reason: connection_closed
16:17:48.004 [error] CRASH REPORT Process <0.436.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.004 [error] gen_server <0.440.0> terminated with reason: connection_closed
16:17:48.004 [error] CRASH REPORT Process <0.440.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.004 [error] gen_server <0.438.0> terminated with reason: connection_closed
16:17:48.004 [error] CRASH REPORT Process <0.438.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.004 [error] gen_server <0.437.0> terminated with reason: connection_closed
16:17:48.005 [error] CRASH REPORT Process <0.437.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.005 [error] gen_server <0.441.0> terminated with reason: connection_closed
16:17:48.005 [error] CRASH REPORT Process <0.441.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.005 [error] gen_server <0.439.0> terminated with reason: connection_closed
16:17:48.005 [error] CRASH REPORT Process <0.439.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.005 [error] gen_server <0.444.0> terminated with reason: connection_closed
16:17:48.005 [error] CRASH REPORT Process <0.444.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.006 [error] gen_server <0.445.0> terminated with reason: connection_closed
16:17:48.006 [error] gen_server <0.447.0> terminated with reason: connection_closed
16:17:48.006 [error] CRASH REPORT Process <0.447.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.006 [error] CRASH REPORT Process <0.445.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.006 [error] gen_server <0.449.0> terminated with reason: connection_closed
16:17:48.006 [error] gen_server <0.450.0> terminated with reason: connection_closed
16:17:48.006 [error] gen_server <0.448.0> terminated with reason: connection_closed
16:17:48.007 [error] CRASH REPORT Process <0.450.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.007 [error] CRASH REPORT Process <0.448.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.007 [error] CRASH REPORT Process <0.449.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.007 [error] gen_server <0.446.0> terminated with reason: connection_closed
16:17:48.007 [error] CRASH REPORT Process <0.446.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.007 [error] gen_server <0.451.0> terminated with reason: connection_closed
16:17:48.007 [error] CRASH REPORT Process <0.451.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:48.007 [error] gen_server <0.454.0> terminated with reason: connection_closed
16:17:48.007 [error] CRASH REPORT Process <0.454.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:17:50.840 [info] Node stopped
16:17:52.979 [info] got 5573 write failures to '[email protected]'
16:17:55.980 [info] checking number of read failures on secondary cluster node, '[email protected]'
16:18:08.378 [info] got 5573 read failures from '[email protected]'
16:18:08.378 [info] http: write_during_shutdown: Ensure node '[email protected]' is down before restart
16:18:08.378 [info] Wait until '[email protected]' is not pingable
16:18:08.379 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
16:18:11.379 [info] Wait until '[email protected]' is pingable
16:18:11.498 [info] Wait until '[email protected]' is pingable
16:18:35.140 [info] got 5573 read failures on '[email protected]'
16:18:35.141 [info] Test passed
16:18:35.142 [notice] replication2 Test Run Complete
16:18:35.145 [info] Multiple tests run or no failure
16:18:38.105 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop ~ wasn't running.
16:18:38.389 [warning] lager_error_logger_h dropped 3529 messages in the last second that exceeded the limit of 50 messages/sec
16:18:38.389 [error] gen_server <0.5860.0> terminated with reason: connection_closed
16:18:38.389 [error] CRASH REPORT Process <0.5860.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.389 [error] gen_server <0.5859.0> terminated with reason: connection_closed
16:18:38.389 [error] CRASH REPORT Process <0.5859.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.390 [error] gen_server <0.5857.0> terminated with reason: connection_closed
16:18:38.390 [error] CRASH REPORT Process <0.5857.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.390 [error] gen_server <0.5855.0> terminated with reason: connection_closed
16:18:38.390 [error] CRASH REPORT Process <0.5855.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.390 [error] gen_server <0.5858.0> terminated with reason: connection_closed
16:18:38.390 [error] gen_server <0.5854.0> terminated with reason: connection_closed
16:18:38.390 [error] CRASH REPORT Process <0.5858.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.390 [error] CRASH REPORT Process <0.5854.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.390 [error] gen_server <0.5853.0> terminated with reason: connection_closed
16:18:38.391 [error] CRASH REPORT Process <0.5853.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.391 [error] gen_server <0.5852.0> terminated with reason: connection_closed
16:18:38.391 [error] CRASH REPORT Process <0.5852.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.391 [error] gen_server <0.5856.0> terminated with reason: connection_closed
16:18:38.391 [error] CRASH REPORT Process <0.5856.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:38.391 [error] gen_server <0.5842.0> terminated with reason: connection_closed
16:18:38.391 [error] CRASH REPORT Process <0.5842.0> with 0 neighbours exited with reason: connection_closed in gen_server:terminate/6 line 747
16:18:39.612 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop ~ wasn't running.
16:18:42.125 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop ~ wasn't running.
16:18:44.866 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop ~ wasn't running.
16:18:47.364 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop ~ wasn't running.
16:18:49.942 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak stop ~ wasn't running.
16:18:50.331 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev1/bin/riak stop ~ wasn't running.
16:18:50.714 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev2/bin/riak stop ~ wasn't running.
16:18:51.091 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev3/bin/riak stop ~ wasn't running.
16:18:51.486 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev4/bin/riak stop ~ wasn't running.
16:18:51.871 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev5/bin/riak stop ~ wasn't running.
16:18:52.250 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev6/bin/riak stop ~ wasn't running.
16:18:52.888 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/bin/riak stop ~ wasn't running.
16:18:53.526 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak stop ~ wasn't running.
16:18:54.159 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/bin/riak stop ~ wasn't running.
16:18:54.790 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/bin/riak stop ~ wasn't running.
16:18:55.428 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/bin/riak stop ~ wasn't running.

Test Results:
16:18:56.059 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/bin/riak stop ~ wasn't running.
replication2-bitcask: pass
---------------------------------------------
0 Tests Failed
1 Tests Passed
That's 100.0% for those keeping score