rt_cascading output for OpenX fix on riak_repl branch feature/csm/openx-deliverables

gistfile1.txt

21:14:41.190 [info] ======================== EUnit ========================
21:14:41.204 [info] Riak path: "/Users/janderson/rt/riak_ee"
21:14:41.204 [info] module 'rt_cascading'
21:14:41.221 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:14:41.221 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:14:41.221 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:14:41.245 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
21:14:41.245 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
21:14:41.245 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
21:14:43.298 [info] Wait until '[email protected]' is pingable
21:14:43.298 [info] Wait until '[email protected]' is pingable
21:14:43.305 [info] Wait until '[email protected]' is pingable
21:14:43.444 [info] Wait until '[email protected]' is pingable
21:14:43.444 [info] Wait until '[email protected]' is pingable
21:14:43.444 [info] Wait until '[email protected]' is pingable
21:14:43.445 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:14:43.445 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:14:43.446 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:14:43.446 [info] Check '[email protected]' is a singleton
21:14:43.464 [info] Check '[email protected]' is a singleton
21:14:43.464 [info] Check '[email protected]' is a singleton
21:14:43.465 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]']
21:14:43.465 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]'].
21:14:55.581 [info] Wait until nodes are ready : ['[email protected]']
21:14:55.581 [info] Wait until no pending changes on ['[email protected]']
21:14:55.582 [info] Wait until nodes are ready : ['[email protected]']
21:14:55.582 [info] Wait until no pending changes on ['[email protected]']
21:14:55.583 [info] Wait until nodes are ready : ['[email protected]']
21:14:55.584 [info] Wait until no pending changes on ['[email protected]']
21:14:55.585 [info] Naming cluster "beginning"
21:14:55.612 [info] wait_until_is_leader('[email protected]')
21:14:55.613 [info] Checking: '[email protected]' =:= '[email protected]'
21:14:55.613 [info] wait_until_is_leader('[email protected]')
21:14:55.613 [info] Checking: '[email protected]' =:= '[email protected]'
21:14:55.613 [info] wait_until_is_leader('[email protected]')
21:14:55.614 [info] Checking: '[email protected]' =:= '[email protected]'
21:14:55.614 [info] Naming cluster "middle"
21:14:55.622 [info] Naming cluster "end"
21:14:55.632 [info]
21:14:55.632 [info] rt_cascading: simple_test_ (connecting Beginning to Middle)...
21:14:55.655 [info] [0.020 s] ok
21:14:55.655 [info]
21:14:55.655 [info] rt_cascading: simple_test_ (connection Middle to End)...
21:14:55.695 [info] [0.037 s] ok
21:14:55.695 [info]
21:14:55.695 [info] rt_cascading: simple_test_ (cascade a put from beginning down to ending)...
21:14:55.899 [info] [0.201 s] ok
21:14:55.899 [info]
21:14:55.899 [info] rt_cascading: simple_test_ (disable cascading on middle)...
21:15:05.978 [info] [10.076 s] ok
21:15:05.978 [info]
21:15:05.978 [info] rt_cascading: simple_test_ (re-enable cascading)...
21:15:06.023 [info] [0.042 s] ok
21:15:06.023 [info]
21:15:06.023 [info] rt_cascading: simple_test_ (check pendings)...
tests/rt_cascading.erl:890:<0.157.0>: Got status:[[{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"end",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"middle",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,2106},
             {max_bytes,104857600},
             {consumers,[]},
             {overload_drops,0}]]
tests/rt_cascading.erl:898:<0.157.0>: Got Cs:[{"end",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.157.0>: Got Cs:[{"middle",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:115:<0.157.0>: PENDING:0

21:15:06.027 [info] Stopping riak on '[email protected]'
21:15:06.027 [info] [0.001 s] ok
21:15:16.029 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
21:15:19.487 [info] Wait until '[email protected]' is not pingable
21:15:19.488 [info] Stopping riak on '[email protected]'
21:15:29.490 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
21:15:32.957 [info] Wait until '[email protected]' is not pingable
21:15:32.958 [info] Stopping riak on '[email protected]'
21:15:42.959 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
21:15:46.416 [info] Wait until '[email protected]' is not pingable
21:15:46.417 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev1/data/
21:15:46.450 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev2/data/
21:15:47.021 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev3/data/
21:15:47.721 [info] Riak path: "/Users/janderson/rt/riak_ee"
21:15:47.756 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev5/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:15:47.756 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:15:47.756 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev4/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:15:47.756 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:15:47.756 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:15:47.757 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev6/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:15:47.799 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak start
21:15:47.799 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak start
21:15:47.799 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
21:15:47.799 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
21:15:47.799 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak start
21:15:47.799 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
21:15:51.916 [info] Wait until '[email protected]' is pingable
21:15:51.950 [info] Wait until '[email protected]' is pingable
21:15:51.950 [info] Wait until '[email protected]' is pingable
21:15:51.955 [info] Wait until '[email protected]' is pingable
21:15:51.958 [info] Wait until '[email protected]' is pingable
21:15:51.975 [info] Wait until '[email protected]' is pingable
21:15:52.092 [info] Wait until '[email protected]' is pingable
21:15:52.092 [info] Wait until '[email protected]' is pingable
21:15:52.092 [info] Wait until '[email protected]' is pingable
21:15:52.093 [info] Wait until '[email protected]' is pingable
21:15:52.093 [info] Wait until '[email protected]' is pingable
21:15:52.093 [info] Wait until '[email protected]' is pingable
21:15:52.093 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:15:52.094 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:15:52.094 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:15:52.094 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:15:52.095 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:15:52.095 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:15:52.095 [info] Check '[email protected]' is a singleton
21:15:52.095 [info] Check '[email protected]' is a singleton
21:15:52.096 [info] Check '[email protected]' is a singleton
21:15:52.096 [info] Check '[email protected]' is a singleton
21:15:52.096 [info] Check '[email protected]' is a singleton
21:15:52.096 [info] Check '[email protected]' is a singleton
21:15:52.097 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]']
21:15:52.097 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]'].
21:16:04.134 [info] Wait until nodes are ready : ['[email protected]']
21:16:04.134 [info] Wait until no pending changes on ['[email protected]']
21:16:04.135 [info] Wait until nodes are ready : ['[email protected]']
21:16:04.137 [info] Wait until no pending changes on ['[email protected]']
21:16:04.138 [info] Wait until nodes are ready : ['[email protected]']
21:16:04.139 [info] Wait until no pending changes on ['[email protected]']
21:16:04.139 [info] Wait until nodes are ready : ['[email protected]']
21:16:04.140 [info] Wait until no pending changes on ['[email protected]']
21:16:04.143 [info] Wait until nodes are ready : ['[email protected]']
21:16:04.145 [info] Wait until no pending changes on ['[email protected]']
21:16:04.146 [info] Wait until nodes are ready : ['[email protected]']
21:16:04.147 [info] Wait until no pending changes on ['[email protected]']
21:16:04.147 [info] wait_until_is_leader('[email protected]')
21:16:04.148 [info] Checking: '[email protected]' =:= '[email protected]'
21:16:04.148 [info] wait_until_is_leader('[email protected]')
21:16:04.149 [info] Checking: '[email protected]' =:= '[email protected]'
21:16:04.149 [info] wait_until_is_leader('[email protected]')
21:16:04.149 [info] Checking: '[email protected]' =:= '[email protected]'
21:16:04.149 [info] wait_until_is_leader('[email protected]')
21:16:04.150 [info] Checking: '[email protected]' =:= '[email protected]'
21:16:04.150 [info] wait_until_is_leader('[email protected]')
21:16:04.151 [info] Checking: '[email protected]' =:= '[email protected]'
21:16:04.152 [info] wait_until_is_leader('[email protected]')
21:16:04.152 [info] Checking: '[email protected]' =:= '[email protected]'
21:16:04.152 [info] Naming cluster "1"
21:16:04.160 [info] Naming cluster "2"
21:16:04.206 [info] Naming cluster "3"
21:16:04.214 [info] Naming cluster "4"
21:16:04.254 [info] Naming cluster "5"
21:16:04.272 [info] Naming cluster "6"
tests/rt_cascading.erl:161:<0.126.0>: der res: [ok,ok,ok,ok,ok,ok]
21:16:10.470 [info]
21:16:10.470 [info] rt_cascading: big_circle_test_ (circle it)...
tests/rt_cascading.erl:178:<0.256.0>: Checking '[email protected]'
tests/rt_cascading.erl:178:<0.256.0>: Checking '[email protected]'
tests/rt_cascading.erl:178:<0.256.0>: Checking '[email protected]'
tests/rt_cascading.erl:178:<0.256.0>: Checking '[email protected]'
tests/rt_cascading.erl:178:<0.256.0>: Checking '[email protected]'
tests/rt_cascading.erl:178:<0.256.0>: Checking '[email protected]'
tests/rt_cascading.erl:890:<0.256.0>: Got status:[[{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"2",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"3",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"4",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"5",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"6",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"1",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}]]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"2",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"3",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"4",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"5",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"6",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"1",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:182:<0.256.0>: PENDING:0

21:16:10.885 [info] [0.412 s] ok
21:16:10.885 [info]
21:16:10.885 [info] rt_cascading: big_circle_test_ (2 way repl, and circle it)...
tests/rt_cascading.erl:201:<0.256.0>: Testing '[email protected]'
tests/rt_cascading.erl:201:<0.256.0>: Testing '[email protected]'
tests/rt_cascading.erl:201:<0.256.0>: Testing '[email protected]'
tests/rt_cascading.erl:201:<0.256.0>: Testing '[email protected]'
tests/rt_cascading.erl:201:<0.256.0>: Testing '[email protected]'
tests/rt_cascading.erl:201:<0.256.0>: Testing '[email protected]'
21:16:18.050 [info] [7.162 s] ok
21:16:18.050 [info]
21:16:18.050 [info] rt_cascading: big_circle_test_ (check pendings)...
tests/rt_cascading.erl:890:<0.256.0>: Got status:[[{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"6",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"2",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"1",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"3",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"2",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"4",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,1307},
             {max_bytes,104857600},
             {consumers,[{"5",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"3",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,1275},
             {max_bytes,104857600},
             {consumers,[{"6",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"4",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"1",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"5",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}]]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"6",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"2",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"1",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"3",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"2",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"4",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"5",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"3",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"6",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"4",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.256.0>: Got Cs:[{"1",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"5",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:235:<0.256.0>: PENDING:0

21:16:18.056 [info] Stopping riak on '[email protected]'
21:16:18.056 [info] [0.002 s] ok
21:16:28.058 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
21:16:31.527 [info] Wait until '[email protected]' is not pingable
21:16:31.528 [info] Stopping riak on '[email protected]'
21:16:41.530 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
21:16:44.993 [info] Wait until '[email protected]' is not pingable
21:16:44.994 [info] Stopping riak on '[email protected]'
21:16:54.996 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
21:16:58.457 [info] Wait until '[email protected]' is not pingable
21:16:58.458 [info] Stopping riak on '[email protected]'
21:17:08.459 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop
21:17:11.912 [info] Wait until '[email protected]' is not pingable
21:17:11.913 [info] Stopping riak on '[email protected]'
21:17:21.915 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop
21:17:25.363 [info] Wait until '[email protected]' is not pingable
21:17:25.364 [info] Stopping riak on '[email protected]'
21:17:35.366 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak stop
21:17:38.830 [info] Wait until '[email protected]' is not pingable
21:17:38.832 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev1/data/
21:17:38.861 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev2/data/
21:17:39.374 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev3/data/
21:17:39.764 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev4/data/
21:17:40.181 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev5/data/
21:17:40.979 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev6/data/
21:17:41.524 [info] Riak path: "/Users/janderson/rt/riak_ee"
21:17:41.544 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:17:41.544 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:17:41.544 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:17:41.565 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
21:17:41.565 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
21:17:41.565 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
21:17:43.632 [info] Wait until '[email protected]' is pingable
21:17:43.643 [info] Wait until '[email protected]' is pingable
21:17:43.646 [info] Wait until '[email protected]' is pingable
21:17:43.732 [info] Wait until '[email protected]' is pingable
21:17:43.732 [info] Wait until '[email protected]' is pingable
21:17:43.732 [info] Wait until '[email protected]' is pingable
21:17:43.733 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:17:43.733 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:17:43.733 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:17:43.733 [info] Check '[email protected]' is a singleton
21:17:43.734 [info] Check '[email protected]' is a singleton
21:17:43.734 [info] Check '[email protected]' is a singleton
21:17:43.734 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]']
21:17:43.734 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]'].
21:17:54.747 [info] Wait until nodes are ready : ['[email protected]']
21:17:54.747 [info] Wait until no pending changes on ['[email protected]']
21:17:54.748 [info] Wait until nodes are ready : ['[email protected]']
21:17:54.748 [info] Wait until no pending changes on ['[email protected]']
21:17:54.749 [info] Wait until nodes are ready : ['[email protected]']
21:17:54.749 [info] Wait until no pending changes on ['[email protected]']
21:17:54.750 [info] wait_until_is_leader('[email protected]')
21:17:54.759 [info] Badrpc during is_leader for '[email protected]'. Error: {'EXIT',{noproc,{gen_server,call,[riak_core_cluster_manager,leader_node,infinity]}}}
21:17:55.760 [info] Checking: '[email protected]' =:= '[email protected]'
21:17:55.760 [info] wait_until_is_leader('[email protected]')
21:17:55.760 [info] Checking: '[email protected]' =:= '[email protected]'
21:17:55.760 [info] wait_until_is_leader('[email protected]')
21:17:55.761 [info] Checking: '[email protected]' =:= '[email protected]'
21:17:55.761 [info] Naming cluster "one"
21:17:55.778 [info] Naming cluster "two"
21:17:55.787 [info] Naming cluster "three"
21:17:58.864 [info]
21:17:58.864 [info] rt_cascading: circle_test_ (cascade all the way to the other end, but no further)...
21:18:00.001 [info] [1.134 s] ok
21:18:00.001 [info]
21:18:00.001 [info] rt_cascading: circle_test_ (cascade starting at a different point)...
21:18:01.074 [info] [1.069 s] ok
21:18:01.074 [info]
21:18:01.074 [info] rt_cascading: circle_test_ (check pendings)...
tests/rt_cascading.erl:890:<0.339.0>: Got status:[[{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"two",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"three",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"one",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}]]
tests/rt_cascading.erl:898:<0.339.0>: Got Cs:[{"two",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.339.0>: Got Cs:[{"three",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.339.0>: Got Cs:[{"one",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:301:<0.339.0>: PENDING:0

21:18:01.078 [info] Stopping riak on '[email protected]'
21:18:01.078 [info] [0.001 s] ok
21:18:11.080 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
21:18:14.551 [info] Wait until '[email protected]' is not pingable
21:18:14.552 [info] Stopping riak on '[email protected]'
21:18:24.554 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
21:18:28.013 [info] Wait until '[email protected]' is not pingable
21:18:28.015 [info] Stopping riak on '[email protected]'
21:18:38.018 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
21:18:41.474 [info] Wait until '[email protected]' is not pingable
21:18:41.476 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev1/data/
21:18:41.505 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev2/data/
21:18:42.031 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev3/data/
21:18:42.661 [info] Riak path: "/Users/janderson/rt/riak_ee"
21:18:42.694 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:18:42.694 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev5/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:18:42.694 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:18:42.694 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:18:42.694 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev4/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:18:42.896 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak start
21:18:42.896 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
21:18:42.896 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
21:18:42.896 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
21:18:42.896 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak start
21:18:46.803 [info] Wait until '[email protected]' is pingable
21:18:46.806 [info] Wait until '[email protected]' is pingable
21:18:46.809 [info] Wait until '[email protected]' is pingable
21:18:46.818 [info] Wait until '[email protected]' is pingable
21:18:46.834 [info] Wait until '[email protected]' is pingable
21:18:46.940 [info] Wait until '[email protected]' is pingable
21:18:46.941 [info] Wait until '[email protected]' is pingable
21:18:46.941 [info] Wait until '[email protected]' is pingable
21:18:46.941 [info] Wait until '[email protected]' is pingable
21:18:46.941 [info] Wait until '[email protected]' is pingable
21:18:46.942 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:18:46.942 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:18:46.942 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:18:46.943 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:18:46.943 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:18:46.943 [info] Check '[email protected]' is a singleton
21:18:46.944 [info] Check '[email protected]' is a singleton
21:18:46.944 [info] Check '[email protected]' is a singleton
21:18:46.944 [info] Check '[email protected]' is a singleton
21:18:46.944 [info] Check '[email protected]' is a singleton
21:18:46.945 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]']
21:18:46.945 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]'].
21:18:57.962 [info] Wait until nodes are ready : ['[email protected]']
21:18:57.963 [info] Wait until no pending changes on ['[email protected]']
21:18:57.965 [info] Wait until nodes are ready : ['[email protected]']
21:18:57.966 [info] Wait until no pending changes on ['[email protected]']
21:18:57.969 [info] Wait until nodes are ready : ['[email protected]']
21:18:57.970 [info] Wait until no pending changes on ['[email protected]']
21:18:57.971 [info] Wait until nodes are ready : ['[email protected]']
21:18:57.973 [info] Wait until no pending changes on ['[email protected]']
21:18:57.975 [info] Wait until nodes are ready : ['[email protected]']
21:18:57.977 [info] Wait until no pending changes on ['[email protected]']
21:18:57.979 [info] wait_until_is_leader('[email protected]')
21:18:57.980 [info] Checking: '[email protected]' =:= '[email protected]'
21:18:57.980 [info] wait_until_is_leader('[email protected]')
21:18:57.993 [info] Badrpc during is_leader for '[email protected]'. Error: {'EXIT',{noproc,{gen_server,call,[riak_core_cluster_manager,leader_node,infinity]}}}
21:18:58.994 [info] Checking: '[email protected]' =:= '[email protected]'
21:18:58.994 [info] wait_until_is_leader('[email protected]')
21:18:58.994 [info] Checking: '[email protected]' =:= '[email protected]'
21:18:58.994 [info] wait_until_is_leader('[email protected]')
21:18:58.995 [info] Checking: '[email protected]' =:= '[email protected]'
21:18:58.995 [info] wait_until_is_leader('[email protected]')
21:18:58.995 [info] Checking: '[email protected]' =:= '[email protected]'
21:18:58.995 [info] Naming cluster "top"
21:18:59.013 [info] Naming cluster "left"
21:18:59.027 [info] Naming cluster "left2"
21:18:59.042 [info] Naming cluster "right"
21:18:59.052 [info] Naming cluster "right2"
21:19:03.177 [info]
21:19:03.177 [info] rt_cascading: pyramid_test_ (Cascade to both kids)...
tests/rt_cascading.erl:350:<0.411.0>: Checking '[email protected]'
tests/rt_cascading.erl:350:<0.411.0>: Checking '[email protected]'
tests/rt_cascading.erl:350:<0.411.0>: Checking '[email protected]'
tests/rt_cascading.erl:350:<0.411.0>: Checking '[email protected]'
tests/rt_cascading.erl:350:<0.411.0>: Checking '[email protected]'
21:19:03.525 [info] [0.346 s] ok
21:19:03.525 [info]
21:19:03.525 [info] rt_cascading: pyramid_test_ (check pendings)...
tests/rt_cascading.erl:890:<0.411.0>: Got status:[[{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"right",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"left",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"left2",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,1420},
             {max_bytes,104857600},
             {consumers,[]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"right2",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,1436},
             {max_bytes,104857600},
             {consumers,[]},
             {overload_drops,0}]]
tests/rt_cascading.erl:898:<0.411.0>: Got Cs:[{"right",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"left",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.411.0>: Got Cs:[{"left2",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.411.0>: Got Cs:[{"right2",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:356:<0.411.0>: PENDING:0

21:19:03.530 [info] Stopping riak on '[email protected]'
21:19:03.530 [info] [0.002 s] ok
21:19:13.531 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
21:19:16.990 [info] Wait until '[email protected]' is not pingable
21:19:16.991 [info] Stopping riak on '[email protected]'
21:19:26.993 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
21:19:30.461 [info] Wait until '[email protected]' is not pingable
21:19:30.462 [info] Stopping riak on '[email protected]'
21:19:40.464 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
21:19:43.923 [info] Wait until '[email protected]' is not pingable
21:19:43.924 [info] Stopping riak on '[email protected]'
21:19:53.926 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop
21:19:57.383 [info] Wait until '[email protected]' is not pingable
21:19:57.384 [info] Stopping riak on '[email protected]'
21:20:07.386 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop
21:20:10.873 [info] Wait until '[email protected]' is not pingable
21:20:10.874 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev1/data/
21:20:10.906 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev2/data/
21:20:11.358 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev3/data/
21:20:11.808 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev4/data/
21:20:12.273 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev5/data/
21:20:12.799 [info] Riak path: "/Users/janderson/rt/riak_ee"
21:20:12.822 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:20:12.822 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:20:12.822 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:20:12.822 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev4/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:20:12.849 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
21:20:12.849 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak start
21:20:12.849 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
21:20:12.849 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
21:20:16.389 [info] Wait until '[email protected]' is pingable
21:20:16.398 [info] Wait until '[email protected]' is pingable
21:20:16.660 [info] Wait until '[email protected]' is pingable
21:20:16.692 [info] Wait until '[email protected]' is pingable
21:20:16.787 [info] Wait until '[email protected]' is pingable
21:20:16.787 [info] Wait until '[email protected]' is pingable
21:20:16.787 [info] Wait until '[email protected]' is pingable
21:20:16.787 [info] Wait until '[email protected]' is pingable
21:20:16.789 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:20:16.789 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:20:16.790 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:20:16.790 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:20:16.791 [info] Check '[email protected]' is a singleton
21:20:16.791 [info] Check '[email protected]' is a singleton
21:20:16.791 [info] Check '[email protected]' is a singleton
21:20:16.791 [info] Check '[email protected]' is a singleton
21:20:16.792 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]','[email protected]']
21:20:16.792 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]','[email protected]'].
21:20:27.805 [info] Wait until nodes are ready : ['[email protected]']
21:20:27.806 [info] Wait until no pending changes on ['[email protected]']
21:20:27.807 [info] Wait until nodes are ready : ['[email protected]']
21:20:27.807 [info] Wait until no pending changes on ['[email protected]']
21:20:27.808 [info] Wait until nodes are ready : ['[email protected]']
21:20:27.809 [info] Wait until no pending changes on ['[email protected]']
21:20:27.809 [info] Wait until nodes are ready : ['[email protected]']
21:20:27.810 [info] Wait until no pending changes on ['[email protected]']
21:20:27.811 [info] Naming cluster "top"
21:20:27.821 [info] Naming cluster "midleft"
21:20:27.832 [info] Naming cluster "midright"
21:20:27.850 [info] Naming cluster "bottom"
21:20:27.860 [info] wait_until_is_leader('[email protected]')
21:20:27.860 [info] Checking: '[email protected]' =:= '[email protected]'
21:20:27.860 [info] wait_until_is_leader('[email protected]')
21:20:27.861 [info] Checking: '[email protected]' =:= '[email protected]'
21:20:27.861 [info] wait_until_is_leader('[email protected]')
21:20:27.861 [info] Checking: '[email protected]' =:= '[email protected]'
21:20:27.861 [info] wait_until_is_leader('[email protected]')
21:20:27.861 [info] Checking: '[email protected]' =:= '[email protected]'
21:20:31.986 [info]
21:20:31.986 [info] rt_cascading: diamond_test_ (unfortunate double write)...
21:22:12.180 [info] [100.191 s] ok
21:22:12.180 [info]
21:22:12.180 [info] rt_cascading: diamond_test_ (connect bottom to top)...
21:22:14.211 [info] [2.028 s] ok
21:22:14.211 [info]
21:22:14.211 [info] rt_cascading: diamond_test_ (start at midright)...
tests/rt_cascading.erl:440:<0.487.0>: Checking '[email protected]'
tests/rt_cascading.erl:440:<0.487.0>: Checking '[email protected]'
tests/rt_cascading.erl:440:<0.487.0>: Checking '[email protected]'
21:22:14.253 [info] [0.039 s] ok
21:22:14.253 [info]
21:22:14.253 [info] rt_cascading: diamond_test_ (check pendings)...
tests/rt_cascading.erl:890:<0.487.0>: Got status:[[{bytes,1689},
             {max_bytes,104857600},
             {consumers,[{"midright",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"midleft",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"bottom",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"bottom",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"top",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}]]
tests/rt_cascading.erl:898:<0.487.0>: Got Cs:[{"midright",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"midleft",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.487.0>: Got Cs:[{"bottom",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.487.0>: Got Cs:[{"bottom",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.487.0>: Got Cs:[{"top",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:451:<0.487.0>: PENDING:0

21:22:14.259 [info] Stopping riak on '[email protected]'
21:22:14.259 [info] [0.003 s] ok
21:22:24.261 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
21:22:27.718 [info] Wait until '[email protected]' is not pingable
21:22:27.720 [info] Stopping riak on '[email protected]'
21:22:37.722 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
21:22:41.194 [info] Wait until '[email protected]' is not pingable
21:22:41.195 [info] Stopping riak on '[email protected]'
21:22:51.197 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
21:22:54.659 [info] Wait until '[email protected]' is not pingable
21:22:54.660 [info] Stopping riak on '[email protected]'
21:23:04.662 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop
21:23:08.131 [info] Wait until '[email protected]' is not pingable
21:23:08.132 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev1/data/
21:23:08.165 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev2/data/
21:23:08.523 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev3/data/
21:23:08.787 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev4/data/
21:23:09.350 [info] Riak path: "/Users/janderson/rt/riak_ee"
21:23:09.383 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev5/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:23:09.383 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev4/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:23:09.383 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev2/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:23:09.383 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev6/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:23:09.383 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:23:09.383 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:23:09.422 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak start
21:23:09.422 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak start
21:23:09.422 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
21:23:09.422 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
21:23:09.422 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak start
21:23:09.422 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
21:23:13.478 [info] Wait until '[email protected]' is pingable
21:23:13.482 [info] Wait until '[email protected]' is pingable
21:23:13.483 [info] Wait until '[email protected]' is pingable
21:23:13.495 [info] Wait until '[email protected]' is pingable
21:23:13.505 [info] Wait until '[email protected]' is pingable
21:23:13.512 [info] Wait until '[email protected]' is pingable
21:23:13.628 [info] Wait until '[email protected]' is pingable
21:23:13.628 [info] Wait until '[email protected]' is pingable
21:23:13.629 [info] Wait until '[email protected]' is pingable
21:23:13.629 [info] Wait until '[email protected]' is pingable
21:23:13.629 [info] Wait until '[email protected]' is pingable
21:23:13.629 [info] Wait until '[email protected]' is pingable
21:23:13.630 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:23:13.630 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:23:13.631 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:23:13.631 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:23:13.631 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:23:13.632 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:23:13.632 [info] Check '[email protected]' is a singleton
21:23:13.632 [info] Check '[email protected]' is a singleton
21:23:13.633 [info] Check '[email protected]' is a singleton
21:23:13.633 [info] Check '[email protected]' is a singleton
21:23:13.633 [info] Check '[email protected]' is a singleton
21:23:13.633 [info] Check '[email protected]' is a singleton
21:23:13.634 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]']
21:23:13.634 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]'].
21:23:25.663 [info] Wait until nodes are ready : ['[email protected]']
21:23:25.664 [info] Wait until no pending changes on ['[email protected]']
21:23:25.666 [info] Wait until nodes are ready : ['[email protected]']
21:23:25.667 [info] Wait until no pending changes on ['[email protected]']
21:23:25.668 [info] Wait until nodes are ready : ['[email protected]']
21:23:25.669 [info] Wait until no pending changes on ['[email protected]']
21:23:25.670 [info] Wait until nodes are ready : ['[email protected]']
21:23:25.672 [info] Wait until no pending changes on ['[email protected]']
21:23:25.673 [info] Wait until nodes are ready : ['[email protected]']
21:23:25.674 [info] Wait until no pending changes on ['[email protected]']
21:23:25.675 [info] Wait until nodes are ready : ['[email protected]']
21:23:25.677 [info] Wait until no pending changes on ['[email protected]']
21:23:25.679 [info] Naming cluster "north"
21:23:25.718 [info] Naming cluster "east"
21:23:25.780 [info] Naming cluster "west"
21:23:25.802 [info] Naming cluster "north_spur"
21:23:25.818 [info] Naming cluster "east_spur"
21:23:25.861 [info] Naming cluster "west_spur"
21:23:25.890 [info] wait_until_is_leader('[email protected]')
21:23:25.891 [info] Checking: '[email protected]' =:= '[email protected]'
21:23:25.891 [info] wait_until_is_leader('[email protected]')
21:23:25.892 [info] Checking: '[email protected]' =:= '[email protected]'
21:23:25.892 [info] wait_until_is_leader('[email protected]')
21:23:25.893 [info] Checking: '[email protected]' =:= '[email protected]'
21:23:25.893 [info] wait_until_is_leader('[email protected]')
21:23:25.894 [info] Checking: '[email protected]' =:= '[email protected]'
21:23:25.894 [info] wait_until_is_leader('[email protected]')
21:23:25.896 [info] Checking: '[email protected]' =:= '[email protected]'
21:23:25.896 [info] wait_until_is_leader('[email protected]')
21:23:25.897 [info] Checking: '[email protected]' =:= '[email protected]'
21:23:37.108 [info]
21:23:37.108 [info] rt_cascading: circle_and_spurs_test_ (start at north)...
tests/rt_cascading.erl:496:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:496:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:496:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:496:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:496:<0.579.0>: Checking '[email protected]'
21:23:37.448 [info] [0.337 s] ok
21:23:37.448 [info]
21:23:37.448 [info] rt_cascading: circle_and_spurs_test_ (Start at west)...
tests/rt_cascading.erl:509:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:509:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:509:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:509:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:509:<0.579.0>: Checking '[email protected]'
21:23:37.481 [info] [0.030 s] ok
21:23:37.481 [info]
21:23:37.481 [info] rt_cascading: circle_and_spurs_test_ (spurs don't replicate back)...
tests/rt_cascading.erl:522:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:522:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:522:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:522:<0.579.0>: Checking '[email protected]'
tests/rt_cascading.erl:522:<0.579.0>: Checking '[email protected]'
21:24:27.637 [info] [50.153 s] ok
21:24:27.637 [info]
21:24:27.637 [info] rt_cascading: circle_and_spurs_test_ (check pendings)...
tests/rt_cascading.erl:890:<0.579.0>: Got status:[[{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"north_spur",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"east",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,1921},
             {max_bytes,104857600},
             {consumers,[{"west",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"east_spur",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"north",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]},
                         {"west_spur",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,2356},
             {max_bytes,104857600},
             {consumers,[]},
             {overload_drops,0}],
            [{bytes,2756},
             {max_bytes,104857600},
             {consumers,[]},
             {overload_drops,0}],
            [{bytes,2500},
             {max_bytes,104857600},
             {consumers,[]},
             {overload_drops,0}]]
tests/rt_cascading.erl:898:<0.579.0>: Got Cs:[{"north_spur",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"east",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.579.0>: Got Cs:[{"west",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"east_spur",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.579.0>: Got Cs:[{"north",[{pending,0},{unacked,0},{drops,0},{errs,0}]},
        {"west_spur",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:528:<0.579.0>: PENDING:0

21:24:27.642 [info] Stopping riak on '[email protected]'
21:24:27.642 [info] [0.002 s] ok
21:24:37.644 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
21:24:41.141 [info] Wait until '[email protected]' is not pingable
21:24:41.142 [info] Stopping riak on '[email protected]'
21:24:51.143 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
21:24:54.595 [info] Wait until '[email protected]' is not pingable
21:24:54.596 [info] Stopping riak on '[email protected]'
21:25:04.598 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
21:25:08.081 [info] Wait until '[email protected]' is not pingable
21:25:08.082 [info] Stopping riak on '[email protected]'
21:25:18.084 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop
21:25:21.539 [info] Wait until '[email protected]' is not pingable
21:25:21.540 [info] Stopping riak on '[email protected]'
21:25:31.541 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop
21:25:35.015 [info] Wait until '[email protected]' is not pingable
21:25:35.016 [info] Stopping riak on '[email protected]'
21:25:45.018 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak stop
21:25:48.489 [info] Wait until '[email protected]' is not pingable
21:25:48.490 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev1/data/
21:25:48.522 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev2/data/
21:25:49.241 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev3/data/
21:25:49.602 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev4/data/
21:25:50.036 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev5/data/
21:25:50.816 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev6/data/
21:25:51.294 [info] Riak path: "/Users/janderson/rt/riak_ee"
21:25:51.326 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:25:51.326 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:25:51.326 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:25:51.326 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:25:51.326 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:25:51.326 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:25:51.365 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/bin/riak start
21:25:51.365 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/bin/riak start
21:25:51.365 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/bin/riak start
21:25:51.365 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/bin/riak start
21:25:51.365 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/bin/riak start
21:25:51.365 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak start
21:25:56.637 [info] Wait until '[email protected]' is pingable
21:25:56.685 [info] Wait until '[email protected]' is pingable
21:25:56.692 [info] Wait until '[email protected]' is pingable
21:25:56.771 [info] Wait until '[email protected]' is pingable
21:25:56.779 [info] Wait until '[email protected]' is pingable
21:25:56.792 [info] Wait until '[email protected]' is pingable
21:25:57.926 [info] Wait until '[email protected]' is pingable
21:25:57.926 [info] Wait until '[email protected]' is pingable
21:25:57.927 [info] Wait until '[email protected]' is pingable
21:25:57.928 [info] Wait until '[email protected]' is pingable
21:25:57.928 [info] Wait until '[email protected]' is pingable
21:25:57.928 [info] Wait until '[email protected]' is pingable
21:25:57.938 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:25:57.939 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:25:57.967 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:25:57.968 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:25:57.970 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:25:57.970 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:25:57.971 [info] Check '[email protected]' is a singleton
21:25:57.972 [info] Check '[email protected]' is a singleton
21:25:57.973 [info] Check '[email protected]' is a singleton
21:25:57.973 [info] Check '[email protected]' is a singleton
21:25:57.974 [info] Check '[email protected]' is a singleton
21:25:57.975 [info] Check '[email protected]' is a singleton
21:25:57.976 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]']
21:25:57.976 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]','[email protected]','[email protected]','[email protected]'].
21:25:59.981 [info] Wait until nodes are ready : ['[email protected]','[email protected]']
21:26:00.053 [info] [join] '[email protected]' to ('[email protected]'): ok
21:26:00.053 [info] Wait until no pending changes on ['[email protected]','[email protected]']
21:26:00.099 [info] Wait until nodes are ready : ['[email protected]','[email protected]']
21:26:00.123 [info] [join] '[email protected]' to ('[email protected]'): ok
21:26:00.123 [info] Wait until no pending changes on ['[email protected]','[email protected]']
21:26:00.140 [info] Wait until nodes are ready : ['[email protected]','[email protected]']
21:26:00.164 [info] [join] '[email protected]' to ('[email protected]'): ok
21:26:00.164 [info] Wait until no pending changes on ['[email protected]','[email protected]']
21:26:00.190 [info] Naming cluster "n12"
21:26:00.210 [info] Naming cluster "n34"
21:26:00.253 [info] Naming cluster "n56"
21:26:03.983 [info]
21:26:03.983 [info] no cascading at first
21:26:03.984 [info]
21:26:03.984 [info] rt_cascading: mixed_version_clusters_test_...
21:26:14.491 [info] [10.505 s] ok
21:26:14.491 [info]
21:26:14.491 [info] rt_cascading: mixed_version_clusters_test_...
21:26:24.586 [info] Upgrading '[email protected]' : previous -> current
21:26:24.586 [info] [10.092 s] ok
21:26:24.586 [info]
21:26:24.586 [info] [done in 20.603 s]
21:26:24.586 [info]
21:26:24.586 [info] mixed source can send
21:26:24.586 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/bin/riak stop
21:27:28.483 [info] Wait until '[email protected]' is not pingable
21:27:28.484 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/data" "/Users/janderson/rt/riak_ee/current/dev/dev1"
21:27:28.633 [info] Running: rm -rf /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/data/*
21:27:28.713 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/etc" "/Users/janderson/rt/riak_ee/current/dev/dev1"
21:27:28.722 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
21:27:30.822 [info] Wait until '[email protected]' is pingable
21:27:30.934 [info] Wait until '[email protected]' is pingable
21:27:41.951 [info]
21:27:41.952 [info] rt_cascading: mixed_version_clusters_test_ (node1 put)...
21:29:23.361 [info] [101.407 s] ok
21:29:23.361 [info]
21:29:23.361 [info] rt_cascading: mixed_version_clusters_test_ (node2 put)...
21:29:33.416 [info] [10.052 s] ok
21:29:33.416 [info] Upgrading '[email protected]' : previous -> current
21:29:33.416 [info]
21:29:33.416 [info] [done in 111.465 s]
21:29:33.416 [info]
21:29:33.416 [info] upgrade the world, cascade starts working
21:29:33.416 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak stop
21:29:37.066 [info] Wait until '[email protected]' is not pingable
21:29:37.067 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/data" "/Users/janderson/rt/riak_ee/current/dev/dev2"
21:29:37.155 [info] Running: rm -rf /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/data/*
21:29:37.179 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/etc" "/Users/janderson/rt/riak_ee/current/dev/dev2"
21:29:37.188 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak start
21:29:39.095 [info] Wait until '[email protected]' is pingable
21:29:39.207 [info] Wait until '[email protected]' is pingable
21:29:39.207 [info] Upgrading '[email protected]' : previous -> current
21:29:39.208 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/bin/riak stop
21:29:42.894 [info] Wait until '[email protected]' is not pingable
21:29:42.895 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/data" "/Users/janderson/rt/riak_ee/current/dev/dev3"
21:29:42.983 [info] Running: rm -rf /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/data/*
21:29:43.091 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/etc" "/Users/janderson/rt/riak_ee/current/dev/dev3"
21:29:43.100 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
21:29:44.956 [info] Wait until '[email protected]' is pingable
21:29:45.050 [info] Wait until '[email protected]' is pingable
21:29:45.050 [info] Upgrading '[email protected]' : previous -> current
21:29:45.050 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/bin/riak stop
21:29:48.932 [info] Wait until '[email protected]' is not pingable
21:29:48.933 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/data" "/Users/janderson/rt/riak_ee/current/dev/dev4"
21:29:49.072 [info] Running: rm -rf /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/data/*
21:29:49.195 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/etc" "/Users/janderson/rt/riak_ee/current/dev/dev4"
21:29:49.409 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak start
21:29:51.423 [info] Wait until '[email protected]' is pingable
21:29:51.537 [info] Wait until '[email protected]' is pingable
21:29:51.538 [info] Upgrading '[email protected]' : previous -> current
21:29:51.538 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/bin/riak stop
21:29:55.254 [info] Wait until '[email protected]' is not pingable
21:29:55.255 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/data" "/Users/janderson/rt/riak_ee/current/dev/dev5"
21:29:55.399 [info] Running: rm -rf /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/data/*
21:29:55.430 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/etc" "/Users/janderson/rt/riak_ee/current/dev/dev5"
21:29:55.767 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak start
21:29:57.727 [info] Wait until '[email protected]' is pingable
21:29:57.860 [info] Wait until '[email protected]' is pingable
21:29:57.861 [info] Upgrading '[email protected]' : previous -> current
21:29:57.861 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/bin/riak stop
21:30:01.618 [info] Wait until '[email protected]' is not pingable
21:30:01.619 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/data" "/Users/janderson/rt/riak_ee/current/dev/dev6"
21:30:01.763 [info] Running: rm -rf /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/data/*
21:30:01.795 [info] Running: cp -p -P -R "/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/etc" "/Users/janderson/rt/riak_ee/current/dev/dev6"
21:30:02.052 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak start
21:30:04.129 [info] Wait until '[email protected]' is pingable
21:30:04.245 [info] Wait until '[email protected]' is pingable
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:14.275 [info]
21:30:14.275 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:17.549 [info] [3.271 s] ok
21:30:17.549 [info]
21:30:17.549 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:17.575 [info] [0.023 s] ok
21:30:17.575 [info]
21:30:17.575 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:17.610 [info] [0.032 s] ok
21:30:17.610 [info]
21:30:17.610 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.649 [info] [1.036 s] ok
21:30:18.649 [info]
21:30:18.649 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.679 [info] [0.027 s] ok
21:30:18.679 [info]
21:30:18.679 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.705 [info] [0.023 s] ok
21:30:18.705 [info]
21:30:18.705 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.736 [info] [0.028 s] ok
21:30:18.736 [info]
21:30:18.736 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.769 [info] [0.030 s] ok
21:30:18.769 [info]
21:30:18.769 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.794 [info] [0.022 s] ok
21:30:18.794 [info]
21:30:18.794 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.818 [info] [0.021 s] ok
21:30:18.818 [info]
21:30:18.818 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.838 [info] [0.017 s] ok
21:30:18.838 [info]
21:30:18.838 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.859 [info] [0.018 s] ok
21:30:18.859 [info]
21:30:18.859 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.884 [info] [0.022 s] ok
21:30:18.884 [info]
21:30:18.884 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.903 [info] [0.016 s] ok
21:30:18.903 [info]
21:30:18.903 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.922 [info] [0.016 s] ok
21:30:18.922 [info]
21:30:18.922 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.944 [info] [0.019 s] ok
21:30:18.944 [info]
21:30:18.944 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
tests/rt_cascading.erl:684:<0.1009.0>: Running test "writing [email protected]"
21:30:18.965 [info] [0.018 s] ok
21:30:18.965 [info]
21:30:18.965 [info] rt_cascading: mixed_version_clusters_test_ (writing [email protected])...
tests/rt_cascading.erl:673:<0.1009.0>: Node and it's res:
[{'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>},
 {'[email protected]',<<"[email protected]">>}]
21:30:18.993 [info] [0.025 s] ok
21:30:18.993 [info]
21:30:18.993 [info] [done in 4.718 s]
21:30:18.993 [info]
21:30:18.993 [info] rt_cascading: mixed_version_clusters_test_ (check pendings)...
tests/rt_cascading.erl:890:<0.743.0>: Got status:[[{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"n34",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"n34",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"n56",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"n56",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"n12",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}],
            [{bytes,768},
             {max_bytes,104857600},
             {consumers,[{"n12",
                          [{pending,0},{unacked,0},{drops,0},{errs,0}]}]},
             {overload_drops,0}]]
tests/rt_cascading.erl:898:<0.743.0>: Got Cs:[{"n34",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.743.0>: Got Cs:[{"n34",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.743.0>: Got Cs:[{"n56",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.743.0>: Got Cs:[{"n56",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.743.0>: Got Cs:[{"n12",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:898:<0.743.0>: Got Cs:[{"n12",[{pending,0},{unacked,0},{drops,0},{errs,0}]}]
tests/rt_cascading.erl:699:<0.743.0>: PENDING:0

21:30:18.998 [info] Stopping riak on '[email protected]'
21:30:18.998 [info] [0.002 s] ok
21:30:29.000 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
21:30:32.475 [info] Wait until '[email protected]' is not pingable
21:30:32.476 [info] Stopping riak on '[email protected]'
21:30:42.478 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop
21:30:45.945 [info] Wait until '[email protected]' is not pingable
21:30:45.946 [info] Stopping riak on '[email protected]'
21:30:55.948 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
21:30:59.400 [info] Wait until '[email protected]' is not pingable
21:30:59.401 [info] Stopping riak on '[email protected]'
21:31:09.403 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop
21:31:12.893 [info] Wait until '[email protected]' is not pingable
21:31:12.894 [info] Stopping riak on '[email protected]'
21:31:22.896 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop
21:31:26.347 [info] Wait until '[email protected]' is not pingable
21:31:26.348 [info] Stopping riak on '[email protected]'
21:31:36.350 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak stop
21:31:39.819 [info] Wait until '[email protected]' is not pingable
21:31:39.820 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev1/data/
21:31:39.850 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev2/data/
21:31:39.870 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev3/data/
21:31:40.005 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev4/data/
21:31:40.027 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev5/data/
21:31:40.253 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev6/data/
21:31:40.278 [info] Riak path: "/Users/janderson/rt/riak_ee"
21:31:40.296 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:31:40.296 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev1/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:31:40.296 [info] rtdev:update_app_config_file(/Users/janderson/rt/riak_ee/current/dev/dev3/etc/app.config, [{lager,[{handlers,[{lager_console_backend,info},{lager_file_backend,[{"./log/error.log",error,10485760,"$D0",5},{"./log/console.log",info,10485760,"$D0",5},{"./log/debug.log",debug,10485760,"$D0",5}]}]},{crash_log,"./log/crash.log"},{crash_log_msg_size,65536},{crash_log_size,10485760},{crash_log_date,"$D0"},{crash_log_count,5},{error_logger_redirect,true}]},{riak_repl,[{fullsync_on_connect,false},{fullsync_interval,disabled},{diff_batch_size,10}]}])
21:31:40.315 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak start
21:31:40.315 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak start
21:31:40.315 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak start
21:31:42.352 [info] Wait until '[email protected]' is pingable
21:31:42.356 [info] Wait until '[email protected]' is pingable
21:31:43.012 [info] Wait until '[email protected]' is pingable
21:31:43.108 [info] Wait until '[email protected]' is pingable
21:31:43.108 [info] Wait until '[email protected]' is pingable
21:31:43.109 [info] Wait until '[email protected]' is pingable
21:31:43.109 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:31:43.109 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:31:43.110 [info] Wait until riak_core_ring_manager is up on '[email protected]'
21:31:43.110 [info] Check '[email protected]' is a singleton
21:31:43.110 [info] Check '[email protected]' is a singleton
21:31:43.126 [info] Check '[email protected]' is a singleton
21:31:43.126 [info] Deployed nodes: ['[email protected]','[email protected]','[email protected]']
21:31:43.126 [info] Waiting for services [riak_kv] to start on ['[email protected]','[email protected]','[email protected]'].
21:31:53.140 [info] Wait until nodes are ready : ['[email protected]']
21:31:53.141 [info] Wait until no pending changes on ['[email protected]']
21:31:53.141 [info] Wait until nodes are ready : ['[email protected]']
21:31:53.142 [info] Wait until no pending changes on ['[email protected]']
21:31:53.144 [info] Wait until nodes are ready : ['[email protected]']
21:31:53.144 [info] Wait until no pending changes on ['[email protected]']
21:31:53.145 [info] Naming cluster "new1"
21:31:53.238 [info] Naming cluster "old2"
21:31:53.259 [info] Naming cluster "new3"
21:31:53.284 [info] wait_until_is_leader('[email protected]')
21:31:53.292 [info] Badrpc during is_leader for '[email protected]'. Error: {'EXIT',{noproc,{gen_server,call,[riak_core_cluster_manager,leader_node,infinity]}}}
21:31:54.293 [info] Checking: '[email protected]' =:= '[email protected]'
21:31:54.293 [info] wait_until_is_leader('[email protected]')
21:31:54.293 [info] Checking: '[email protected]' =:= '[email protected]'
21:31:54.293 [info] wait_until_is_leader('[email protected]')
21:31:54.294 [info] Checking: '[email protected]' =:= '[email protected]'
21:31:57.374 [info]
21:31:57.374 [info] rt_cascading: new_to_old_test_ (From new1 to old2)...
21:32:07.553 [info] [10.176 s] ok
21:32:07.553 [info]
21:32:07.553 [info] rt_cascading: new_to_old_test_ (old2 does not cascade at all)...
21:32:17.605 [info] [10.048 s] ok
21:32:17.605 [info]
21:32:17.605 [info] rt_cascading: new_to_old_test_ (from new3 to old2)...
21:32:17.652 [info] [0.044 s] ok
21:32:17.652 [info]
21:32:17.652 [info] rt_cascading: new_to_old_test_ (from old2 to new3 no cascade)...
tests/rt_cascading.erl:890:<0.1223.0>: Got status:[]
tests/rt_cascading.erl:777:<0.1223.0>: PENDING:0

21:32:27.701 [info] [10.046 s] ok
21:32:27.701 [info]
21:32:27.701 [info] rt_cascading: new_to_old_test_ (check pendings)...
21:32:27.704 [info] Stopping riak on '[email protected]'
21:32:27.704 [info] ok
21:32:37.705 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop
21:32:41.171 [info] Wait until '[email protected]' is not pingable
21:32:41.172 [info] Stopping riak on '[email protected]'
21:32:51.174 [info] Running: /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak stop
21:32:54.839 [info] Wait until '[email protected]' is not pingable
21:32:54.840 [info] Stopping riak on '[email protected]'
21:33:04.842 [info] Running: /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop
21:33:08.317 [info] Wait until '[email protected]' is not pingable
21:33:08.319 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev1/data/
21:33:08.356 [info] Removing directory /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/data/
21:33:08.391 [info] Removing directory /Users/janderson/rt/riak_ee/current/dev/dev3/data/
21:33:08.962 [info]
21:33:08.962 [info] [done in 1107.759 s]
21:33:08.962 [info] =======================================================
21:33:08.962 [info]   All 50 tests passed.
21:33:08.963 [notice] rt_cascading Test Run Complete
21:33:08.966 [info] Multiple tests run or no failure
21:33:09.437 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev1/bin/riak stop ~ wasn't running.
21:33:09.952 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev2/bin/riak stop ~ wasn't running.
21:33:10.480 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev3/bin/riak stop ~ wasn't running.
21:33:10.966 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev4/bin/riak stop ~ wasn't running.
21:33:11.427 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev5/bin/riak stop ~ wasn't running.
21:33:11.887 [info] Stopped Node... /Users/janderson/rt/riak_ee/current/dev/dev6/bin/riak stop ~ wasn't running.
21:33:12.274 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev1/bin/riak stop ~ wasn't running.
21:33:12.666 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev2/bin/riak stop ~ wasn't running.
21:33:13.051 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev3/bin/riak stop ~ wasn't running.
21:33:13.478 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev4/bin/riak stop ~ wasn't running.
21:33:13.882 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev5/bin/riak stop ~ wasn't running.
21:33:14.275 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.2.1/dev/dev6/bin/riak stop ~ wasn't running.
21:33:14.925 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev1/bin/riak stop ~ wasn't running.
21:33:15.589 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev2/bin/riak stop ~ wasn't running.
21:33:16.226 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev3/bin/riak stop ~ wasn't running.
21:33:16.868 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev4/bin/riak stop ~ wasn't running.
21:33:17.533 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev5/bin/riak stop ~ wasn't running.

Test Results:
21:33:18.202 [info] Stopped Node... /Users/janderson/rt/riak_ee/riak_ee-1.3.1/dev/dev6/bin/riak stop ~ wasn't running.
rt_cascading-bitcask: pass
---------------------------------------------
0 Tests Failed
1 Tests Passed
That's 100.0% for those keeping score