Skip to content

Instantly share code, notes, and snippets.

@bcg
Created March 7, 2012 14:05
Show Gist options
  • Save bcg/1993328 to your computer and use it in GitHub Desktop.
Save bcg/1993328 to your computer and use it in GitHub Desktop.
Riak all_nodes_down
$ curl -v -d 'this is a test' -H "Content-Type: text/plain" http://10.32.36.92:8098/riak/test
* About to connect() to 10.32.36.92 port 8098 (#0)
* Trying 10.32.36.92... connected
* Connected to 10.32.36.92 (10.32.36.92) port 8098 (#0)
> POST /riak/test HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: 10.32.36.92:8098
> Accept: */*
> Content-Type: text/plain
> Content-Length: 14
>
< HTTP/1.1 500 Internal Server Error
< Vary: Accept-Encoding
< Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
< Location: /riak/test/Mobuxmt6vIikc40fFnwatTLSS2q
< Date: Wed, 07 Mar 2012 14:05:04 GMT
< Content-Type: text/plain
< Content-Length: 22
<
Error:
all_nodes_down
* Connection #0 to host 10.32.36.92 left intact
* Closing connection #0
([email protected])2> riak_core_vnode_manager:force_handoffs().
ok
([email protected])3> 08:59:55.728 [error] gen_server riak_kv_vnode_master terminated with reason: no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2
08:59:55.730 [error] CRASH REPORT Process <0.8550.6> with 0 neighbours crashed with reason: no match of right hand value {error,enoent} in riak_kv_vnode:update_vnode_status/2
08:59:55.732 [error] CRASH REPORT Process riak_kv_vnode_master with 0 neighbours crashed with reason: no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2
08:59:55.734 [error] Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.8550.6> exit with reason no match of right hand value {error,enoent} in riak_kv_vnode:update_vnode_status/2 in context child_terminated
08:59:55.736 [error] Supervisor riak_kv_sup had child riak_kv_vnode_master started with riak_core_vnode_master:start_link(riak_kv_vnode, riak_kv_legacy_vnode) at <0.8543.6> exit with reason no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2 in context child_terminated
([email protected])1> riak_core_ring_manager:force_update().
ok
([email protected])2> 08:57:01.132 [error] CRASH REPORT Process <0.8493.6> with 0 neighbours crashed with reason: no match of right hand value {error,enoent} in riak_kv_vnode:update_vnode_status/2
08:57:01.133 [error] Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.8493.6> exit with reason no match of right hand value {error,enoent} in riak_kv_vnode:update_vnode_status/2 in context child_terminated
08:57:01.134 [error] gen_server riak_kv_vnode_master terminated with reason: no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2
08:57:01.136 [error] CRASH REPORT Process riak_kv_vnode_master with 0 neighbours crashed with reason: no match of right hand value {error,{{badmatch,{error,enoent}},[{riak_kv_vnode,update_vnode_status,2},{riak_kv_vnode,init,1},{riak_core_vnode,init,1},{gen_fsm,init_it,6},{proc_lib,init_p_do_apply,3}]}} in riak_core_vnode_master:get_vnode/2
...
$ sudo riak-admin member_status
Attempting to restart script through sudo -u riak
================================= Membership ==================================
Status Ring Pending Node
-------------------------------------------------------------------------------
valid 1.6% 50.0% '[email protected]'
valid 0.0% 0.0% '[email protected]'
valid 0.0% 0.0% '[email protected]'
valid 0.0% 0.0% '[email protected]'
valid 98.4% 50.0% '[email protected]'
valid 0.0% 0.0% '[email protected]'
-------------------------------------------------------------------------------
Valid:6 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
$ sudo riak-admin ring_status
Attempting to restart script through sudo -u riak
================================== Claimant ===================================
Claimant: '[email protected]'
Status: up
Ring Ready: true
============================== Ownership Handoff ==============================
Owner: [email protected]
Next Owner: [email protected]
Index: 0
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 22835963083295358096932575511191922182123945984
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 91343852333181432387730302044767688728495783936
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 114179815416476790484662877555959610910619729920
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 182687704666362864775460604089535377456991567872
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 205523667749658222872393179600727299639115513856
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 274031556999544297163190906134303066185487351808
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 296867520082839655260123481645494988367611297792
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 388211372416021087647853783690262677096107081728
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 456719261665907161938651510223838443642478919680
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 479555224749202520035584085735030365824602865664
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 548063113999088594326381812268606132370974703616
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 570899077082383952423314387779798054553098649600
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 639406966332270026714112114313373821099470487552
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 662242929415565384811044689824565743281594433536
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 730750818665451459101842416358141509827966271488
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 753586781748746817198774991869333432010090217472
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 822094670998632891489572718402909198556462055424
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 844930634081928249586505293914101120738586001408
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 913438523331814323877303020447676887284957839360
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 936274486415109681974235595958868809467081785344
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1004782375664995756265033322492444576013453623296
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1027618338748291114361965898003636498195577569280
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1096126227998177188652763624537212264741949407232
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1118962191081472546749696200048404186924073353216
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1187470080331358621040493926581979953470445191168
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1210306043414653979137426502093171875652569137152
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1278813932664540053428224228626747642198940975104
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1301649895747835411525156804137939564381064921088
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1370157784997721485815954530671515330927436759040
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
Index: 1392993748081016843912887106182707253109560705024
Waiting on: [riak_kv_vnode]
Complete: [riak_pipe_vnode]
-------------------------------------------------------------------------------
============================== Unreachable Nodes ==============================
All nodes are up and reachable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment