-
-
Save bcg/1993328 to your computer and use it in GitHub Desktop.
Riak all_nodes_down
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ curl -v -d 'this is a test' -H "Content-Type: text/plain" http://10.32.36.92:8098/riak/test | |
| * About to connect() to 10.32.36.92 port 8098 (#0) | |
| * Trying 10.32.36.92... connected | |
| * Connected to 10.32.36.92 (10.32.36.92) port 8098 (#0) | |
| > POST /riak/test HTTP/1.1 | |
| > User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15 | |
| > Host: 10.32.36.92:8098 | |
| > Accept: */* | |
| > Content-Type: text/plain | |
| > Content-Length: 14 | |
| > | |
| < HTTP/1.1 500 Internal Server Error | |
| < Vary: Accept-Encoding | |
| < Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue) | |
| < Location: /riak/test/Mobuxmt6vIikc40fFnwatTLSS2q | |
| < Date: Wed, 07 Mar 2012 14:05:04 GMT | |
| < Content-Type: text/plain | |
| < Content-Length: 22 | |
| < | |
| Error: | |
| all_nodes_down | |
| * Connection #0 to host 10.32.36.92 left intact | |
| * Closing connection #0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (riak@10.32.36.92)2> riak_core_vnode_manager:force_handoffs(). | |
| ok | |
| (riak@10.32.36.92)3> 08:59:55.728 [error] gen_server riak_kv_vnode_master terminated with reason: no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2 | |
| 08:59:55.730 [error] CRASH REPORT Process <0.8550.6> with 0 neighbours crashed with reason: no match of right hand value {error,enoent} in riak_kv_vnode:update_vnode_status/2 | |
| 08:59:55.732 [error] CRASH REPORT Process riak_kv_vnode_master with 0 neighbours crashed with reason: no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2 | |
| 08:59:55.734 [error] Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.8550.6> exit with reason no match of right hand value {error,enoent} in riak_kv_vnode:update_vnode_status/2 in context child_terminated | |
| 08:59:55.736 [error] Supervisor riak_kv_sup had child riak_kv_vnode_master started with riak_core_vnode_master:start_link(riak_kv_vnode, riak_kv_legacy_vnode) at <0.8543.6> exit with reason no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2 in context child_terminated |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (riak@10.32.36.92)1> riak_core_ring_manager:force_update(). | |
| ok | |
| (riak@10.32.36.92)2> 08:57:01.132 [error] CRASH REPORT Process <0.8493.6> with 0 neighbours crashed with reason: no match of right hand value {error,enoent} in riak_kv_vnode:update_vnode_status/2 | |
| 08:57:01.133 [error] Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.8493.6> exit with reason no match of right hand value {error,enoent} in riak_kv_vnode:update_vnode_status/2 in context child_terminated | |
| 08:57:01.134 [error] gen_server riak_kv_vnode_master terminated with reason: no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2 | |
| 08:57:01.136 [error] CRASH REPORT Process riak_kv_vnode_master with 0 neighbours crashed with reason: no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2 | |
| ... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ sudo riak-admin member_status | |
| Attempting to restart script through sudo -u riak | |
| ================================= Membership ================================== | |
| Status Ring Pending Node | |
| ------------------------------------------------------------------------------- | |
| valid 1.6% 50.0% 'riak@10.24.154.74' | |
| valid 0.0% 0.0% 'riak@10.24.154.76' | |
| valid 0.0% 0.0% 'riak@10.24.154.78' | |
| valid 0.0% 0.0% 'riak@10.24.154.80' | |
| valid 98.4% 50.0% 'riak@10.32.36.92' | |
| valid 0.0% 0.0% 'riak@10.32.36.94' | |
| ------------------------------------------------------------------------------- | |
| Valid:6 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ sudo riak-admin ring_status | |
| Attempting to restart script through sudo -u riak | |
| ================================== Claimant =================================== | |
| Claimant: 'riak@10.32.36.92' | |
| Status: up | |
| Ring Ready: true | |
| ============================== Ownership Handoff ============================== | |
| Owner: riak@10.32.36.92 | |
| Next Owner: riak@10.24.154.74 | |
| Index: 0 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 22835963083295358096932575511191922182123945984 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 91343852333181432387730302044767688728495783936 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 114179815416476790484662877555959610910619729920 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 182687704666362864775460604089535377456991567872 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 205523667749658222872393179600727299639115513856 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 274031556999544297163190906134303066185487351808 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 296867520082839655260123481645494988367611297792 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 388211372416021087647853783690262677096107081728 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 456719261665907161938651510223838443642478919680 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 479555224749202520035584085735030365824602865664 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 548063113999088594326381812268606132370974703616 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 570899077082383952423314387779798054553098649600 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 639406966332270026714112114313373821099470487552 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 662242929415565384811044689824565743281594433536 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 730750818665451459101842416358141509827966271488 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 753586781748746817198774991869333432010090217472 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 822094670998632891489572718402909198556462055424 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 844930634081928249586505293914101120738586001408 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 913438523331814323877303020447676887284957839360 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 936274486415109681974235595958868809467081785344 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1004782375664995756265033322492444576013453623296 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1027618338748291114361965898003636498195577569280 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1096126227998177188652763624537212264741949407232 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1118962191081472546749696200048404186924073353216 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1187470080331358621040493926581979953470445191168 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1210306043414653979137426502093171875652569137152 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1278813932664540053428224228626747642198940975104 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1301649895747835411525156804137939564381064921088 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1370157784997721485815954530671515330927436759040 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| Index: 1392993748081016843912887106182707253109560705024 | |
| Waiting on: [riak_kv_vnode] | |
| Complete: [riak_pipe_vnode] | |
| ------------------------------------------------------------------------------- | |
| ============================== Unreachable Nodes ============================== | |
| All nodes are up and reachable |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment