Skip to content

Instantly share code, notes, and snippets.

@pcn

pcn/majority_node_grep_of_minority_events

Created August 30, 2012 15:53
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save pcn/3531536 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save pcn/3531536 to your computer and use it in GitHub Desktop.
Download ZIP
Cassandra gossip has abandoned a node
Raw
majority_node_grep_of_minority_events
@4000000050234dda249f5fb4.s:2012-07-31T20:16:55.00512 INFO 20:16:55,004 Started hinted handoff for token: 85070591730234615865843651859750628463 with IP: /10.192.178.255
@4000000050234dda249f5fb4.s:2012-07-31T20:17:07.32295 INFO 20:17:07,322 Finished hinted handoff of 231 rows to endpoint /10.192.178.255
@4000000050234dda249f5fb4.s:2012-08-01T07:47:17.87990 INFO 07:47:17,878 Started hinted handoff for token: 85070591730234615865843651859750628463 with IP: /10.192.178.255
@4000000050234dda249f5fb4.s:2012-08-01T07:47:18.59174 INFO 07:47:18,591 Finished hinted handoff of 13 rows to endpoint /10.192.178.255
@40000000503e41c804822eb4.s:2012-08-28T23:09:48.58580 INFO 23:09:48,585 InetAddress /10.192.178.255 is now dead.
@40000000503e41c804822eb4.s:2012-08-28T23:09:49.58861 INFO 23:09:49,588 Node /10.192.178.255 has restarted, now UP
@40000000503e41c804822eb4.s:2012-08-28T23:09:49.59137 INFO 23:09:49,590 InetAddress /10.192.178.255 is now UP@40000000503e41c804822eb4.s:2012-08-28T23:09:49.59140 INFO 23:09:49,591 Node /10.192.178.255 state jump to normal
@40000000503e41c804822eb4.s:2012-08-28T23:09:50.49348 INFO 23:09:50,491 Started hinted handoff for token: 85070591730234615865843651859750628463 with IP: /10.192.178.255
@40000000503e41c804822eb4.s:2012-08-28T23:10:00.10458 INFO 23:10:00,104 Finished hinted handoff of 156 rows to endpoint /10.192.178.255
@40000000503e41c804822eb4.s:2012-08-28T23:17:05.48800 INFO 23:17:05,487 Started hinted handoff for token: 85070591730234615865843651859750628463 with IP: /10.192.178.255
@40000000503e41c804822eb4.s:2012-08-28T23:17:05.60807 INFO 23:17:05,607 Finished hinted handoff of 2 rows to endpoint /10.192.178.255
@40000000503e41c804822eb4.s:2012-08-29T16:19:29.37749 INFO 16:19:29,376 InetAddress /10.192.178.255 is now dead.
@40000000503e41c804822eb4.s:2012-08-29T16:19:35.31051 INFO 16:19:35,310 InetAddress /10.192.178.255 is now UP
@40000000503e41c804822eb4.s:2012-08-29T16:19:35.70133 INFO 16:19:35,700 Started hinted handoff for token: 85070591730234615865843651859750628463 with IP: /10.192.178.255
@40000000503e41c804822eb4.s:2012-08-29T16:19:36.27058 INFO 16:19:36,270 Finished hinted handoff of 4 rows to endpoint /10.192.178.255
@40000000503e47f8001d9c14.s:2012-08-29T16:36:09.13869 INFO 16:36:09,138 InetAddress /10.192.178.255 is now dead.
Raw
nodetool_gossipinfo_from_the_majority
(Cassandra 3/6) pn@ip-10-98-179-66:~$ nodetool -h0 gossipinfo
/10.64.19.104
SCHEMA:55dc2760-db55-11e1-0000-e8dbd70370ff
RPC_ADDRESS:10.64.19.104
STATUS:NORMAL,1808575600
RELEASE_VERSION:1.0.10
LOAD:4.8541193686E10
/10.98.179.66
SCHEMA:55dc2760-db55-11e1-0000-e8dbd70370ff
RPC_ADDRESS:10.98.179.66
STATUS:NORMAL,28356863910078205288614550621122593221
RELEASE_VERSION:1.0.10
LOAD:5.3543716896E10
/10.192.178.255
SCHEMA:55dc2760-db55-11e1-0000-e8dbd70370ff
RPC_ADDRESS:10.192.178.255
STATUS:NORMAL,85070591730234615865843651859750628463
RELEASE_VERSION:1.0.10
LOAD:5.0852365234E10
/10.112.221.68
SCHEMA:55dc2760-db55-11e1-0000-e8dbd70370ff
RPC_ADDRESS:10.112.221.68
STATUS:NORMAL,113427455640312821154458202479064646084
RELEASE_VERSION:1.0.10
LOAD:5.5675719369E10
/10.110.26.230
REMOVAL_COORDINATOR:REMOVER,113427455640312821154458202479064646084
SCHEMA:4d395790-d43e-11e1-0000-e8dbd70370ff
RPC_ADDRESS:10.110.26.230
STATUS:removed,1808575600,1343592064542
RELEASE_VERSION:1.0.10
LOAD:7.251761165E9
/10.28.13.30
SCHEMA:55dc2760-db55-11e1-0000-e8dbd70370ff
RPC_ADDRESS:10.28.13.30
STATUS:NORMAL,141784319550391026443072753098378663705
RELEASE_VERSION:1.0.10
LOAD:5.6206567612E10
/10.29.168.72
SCHEMA:55dc2760-db55-11e1-0000-e8dbd70370ff
RPC_ADDRESS:10.29.168.72
STATUS:NORMAL,56713727820156410577229101240436610842
RELEASE_VERSION:1.0.10
LOAD:5.3536843028E10
Raw
nodetool_info
pn@domU-12-31-39-0E-B0-F1:/var/log/cassandra$ nodetool -h0 info
Token : 85070591730234615865843651859750628463
Gossip active : true
Load : 47.66 GB
Generation No : 1346195387
Uptime (seconds) : 146502
Heap Memory (MB) : 2102.22 / 3012.94
Data Center : datacenter1
Rack : rack1
Exceptions : 3
Raw
nodetool_ring_from_the_majority
(Cassandra 3/6) pn@ip-10-98-179-66:~$ nodetool -h0 ring
Address DC Rack Status State Load Owns Token
141784319550391026443072753098378663705
10.64.19.104 datacenter1 rack1 Up Normal 45.21 GB 16.67% 1808575600
10.98.179.66 datacenter1 rack1 Up Normal 49.87 GB 16.67% 28356863910078205288614550621122593221
10.29.168.72 datacenter1 rack1 Up Normal 49.86 GB 16.67% 56713727820156410577229101240436610842
10.192.178.255 datacenter1 rack1 Down Normal 47.36 GB 16.67% 85070591730234615865843651859750628463
10.112.221.68 datacenter1 rack1 Up Normal 51.85 GB 16.67% 113427455640312821154458202479064646084
10.28.13.30 datacenter1 rack1 Up Normal 52.35 GB 16.67% 141784319550391026443072753098378663705
Raw
nodetool_ring_from_the_minority
pn@domU-12-31-39-0E-B0-F1:~$ nodetool -h0 ring
Address DC Rack Status State Load Owns Token
141784319550391026443072753098378663705
10.64.19.104 datacenter1 rack1 Up Normal 40.83 GB 16.67% 1808575600
10.98.179.66 datacenter1 rack1 Up Normal 45.6 GB 16.67% 28356863910078205288614550621122593221
10.29.168.72 datacenter1 rack1 Up Normal 47.15 GB 16.67% 56713727820156410577229101240436610842
10.192.178.255 datacenter1 rack1 Up Normal 47.66 GB 16.67% 85070591730234615865843651859750628463
10.112.221.68 datacenter1 rack1 Up Normal 47.5 GB 16.67% 113427455640312821154458202479064646084
10.28.13.30 datacenter1 rack1 Up Normal 47.85 GB 16.67% 141784319550391026443072753098378663705
@pcn

pcn commented Aug 30, 2012

Copy link
Copy Markdown
Author

It's worth noting that even in this state, the node was overwhelmed by activity this morning until clients started throwing errors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment