David O'Dell dodizzle
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Datacenter: datacenter1 | |
| ======================= | |
| Status=Up/Down | |
| |/ State=Normal/Leaving/Joining/Moving | |
| -- Address Load Tokens Owns (effective) Host ID Rack | |
| UN 10.84.189.183 52.41 MB 256 100.0% 6de6c0aa-55a8-4180-ba97-7c48d7052ee5 rack1 | |
| UN 10.84.189.179 57.01 MB 260 94.3% 166ac779-ff8c-489d-8084-a5309d00e027 rack1 | |
| UN 10.84.189.181 123.39 MB 255 27.4% a09e4785-2ecc-413f-a1ae-b99063284645 rack1 | |
| UN 10.84.189.143 1.14 GB 254 78.3% f58d712b-4a46-4126-b383-51ccfae8080a rack1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| This error on rabbit-hole: | |
| 2014-04-08T21:54:35.338Z - error: [rabbit:14855] [Error: Read Timed Out] 'message handler error' { payload: | |
| { timestamp_id: '1396914077729011088', | |
| hearter_id: 413960, | |
| entry_id: 110252952 }, | |
| type: 'unheart', | |
| ack: [Function], | |
| nack: [Function] } 'Error: Read Timed Out\n at null._onTimeout (/srv/rabbit-hole/releases/20140408214440/node_modules/helenus/lib/connection.js:403:16)\n at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)' | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| daveo@rabbit-hole01:/srv/rabbit-hole/current/config$ while true; do netstat -ant |grep ESTA | grep 9160; date;sleep 1; done | |
| Fri Apr 11 01:19:30 UTC 2014 | |
| tcp 0 0 10.60.186.195:50626 10.84.189.179:9160 ESTABLISHED | |
| Fri Apr 11 01:19:31 UTC 2014 | |
| tcp 0 0 10.60.186.195:50626 10.84.189.179:9160 ESTABLISHED | |
| Fri Apr 11 01:19:32 UTC 2014 | |
| tcp 47 0 10.60.186.195:50626 10.84.189.179:9160 ESTABLISHED | |
| Fri Apr 11 01:19:33 UTC 2014 | |
| tcp 47 0 10.60.186.195:50626 10.84.189.179:9160 ESTABLISHED | |
| Fri Apr 11 01:19:34 UTC 2014 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [2014-05-11T15:55:11.550 INFO (3780) #1F3A4E334A76] Started POST "/join" for 209.49.228.206 at 2014-05-11 15:55:11 +0000 | |
| [2014-05-11T15:55:11.807 WARN (3780) #1F3A4E334A76] Attachment Processing Error | |
| [2014-05-11T15:55:11.807 WARN (3780) #1F3A4E334A76] {} | |
| [2014-05-11T15:55:11.807 WARN (3780) #1F3A4E334A76] RestClient::UnprocessableEntity: 422 Unprocessable Entity | |
| > [#1F3A4E334A76] /srv/whi/shared/bundle/ruby/2.1.0/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!' | |
| > [#1F3A4E334A76] /srv/whi/shared/bundle/ruby/2.1.0/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result' | |
| > [#1F3A4E334A76] /srv/whi/shared/bundle/ruby/2.1.0/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit' | |
| > [#1F3A4E334A76] /home/app/.rbenv/versions/2.1.1/lib/ruby/2.1.0/net/http.rb:853:in `start' | |
| > [#1F3A4E334A76] /srv/whi/shared/bundle/ruby/2.1.0/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit' | |
| > [#1F3A4E334A76] /srv/whi/shared/bundle/ruby/2.1.0/gem |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Davids-MacBook-Air:~ daveo$ ssh cassandra04 | |
| Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-45-generic x86_64) | |
| * Documentation: https://help.ubuntu.com/ | |
| New release '13.10' available. | |
| Run 'do-release-upgrade' to upgrade to it. | |
| __ ___ _ _____ _____ _____ ____ _____ | |
| \ \ / / | | |_ _| | __ \| __ \ / __ \| __ \ | |
| \ \ /\ / /| |__| | | |______| |__) | |__) | | | | | | | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ###################### | |
| GOOD TRACE: | |
| ###################### | |
| cqlsh:system_traces> select * from events where session_id=2dd027f0-1cf1-11e4-bdb7-b77efa82cede; | |
| session_id | event_id | activity | source | source_elapsed | thread | |
| --------------------------------------+--------------------------------------+-----------------------------------------------------------------------------------------------------------+---------------+----------------+--------------------------- | |
| 2dd027f0-1cf1-11e4-bdb7-b77efa82cede | 2dd027f0-1cf1-11e4-8584-938824d1fd36 | Message received from /10.84.189.187 | 10.84.189.146 | 29 | Thread-102375 | |
| 2dd027f0-1cf1-11e4-bdb7-b77efa82cede | 2dd027f0-1cf1-11e4-90d2-c5cb0ca83f8f | |
dodizzle
/ gist:7cb7b1529f4f53faf183
Created
August 12, 2014 16:36
consistency ONE & then ALL tracing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cqlsh:Keyspace1> consistency one; | |
| Consistency level set to ONE. | |
| cqlsh:Keyspace1> update "Counter1" set value = value + 1 where key=0x303031313738343937 AND column1 = 'C0'; | |
| Tracing session: 70a58550-223e-11e4-8d7b-55bc21215380 | |
| activity | timestamp | source | source_elapsed | |
| ----------------------------------------------------------------------------------------------------+--------------+----------------+---------------- | |
| execute_cql3_query | 16:33:50,117 | 10.186.171.105 | 0 | |
| Message received from /10.186.171.105 | 16:33:50,117 | 10.101.159.85 | 32 |
dodizzle
/ gist:3562987b8c28c7594815
Created
August 12, 2014 16:41
ONE and ALL while a node is under high IO
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cqlsh:Keyspace1> consistency one; | |
| Consistency level set to ONE. | |
| cqlsh:Keyspace1> update "Counter1" set value = value + 1 where key=0x303031313738343937 AND column1 = 'C0'; | |
| Request did not complete within rpc_timeout. | |
| Tracing session: 62581020-223f-11e4-8d7b-55bc21215380 | |
| activity | timestamp | source | source_elapsed | |
| ----------------------------------------------------------------------------------------------------+--------------+----------------+---------------- | |
| execute_cql3_query | 16:40:35,618 | 10.186.171.105 | 0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Davids-MacBook-Air:~ daveo$ K "name:fe*" "cat /proc/cpuinfo | grep 'model name' | sort | uniq"| sort | |
| fe01.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz | |
| fe02.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz | |
| fe03.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz | |
| fe04.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz | |
| fe05.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz | |
| fe06.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz | |
| fe07.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz | |
| fe08.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz | |
| fe09.weheartit.com model name : Intel(R) Xeon(R) CPU E7- 4850 @ 2.00GHz |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| WOPR:recipes daveo$ K "chef_environment:prod" "grep 179 /etc/hosts" | |
| /Users/daveo/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/chef-12.0.3/lib/chef/data_bag_item.rb:161: warning: circular argument reference - data_bag | |
| /Users/daveo/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/fog-1.23.0/lib/fog/rackspace/mock_data.rb:42: warning: duplicated key at line 80 ignored: "name" | |
| cassandra14.weheartit.com 10.84.189.179 cassandra08.whi | |
| cassandra19.weheartit.com 10.84.189.179 cassandra08.whi | |
| edge01.weheartit.com 10.84.189.179 cassandra08.whi | |
| fe12.weheartit.com 10.84.189.179 cassandra08.whi | |
| cassandra11.weheartit.com 10.84.189.179 cassandra08.whi | |
| es01.weheartit.com 10.84.189.179 cassandra08.whi | |
| db11.weheartit.com 10.84.189.179 cassandra08.whi |