David O'Dell dodizzle
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| daveo@rabbit-hole01:/srv/rabbit-hole/current/config$ while true; do netstat -ant |grep ESTA | grep 9160; date;sleep 1; done | |
| Fri Apr 11 01:19:30 UTC 2014 | |
| tcp 0 0 10.60.186.195:50626 10.84.189.179:9160 ESTABLISHED | |
| Fri Apr 11 01:19:31 UTC 2014 | |
| tcp 0 0 10.60.186.195:50626 10.84.189.179:9160 ESTABLISHED | |
| Fri Apr 11 01:19:32 UTC 2014 | |
| tcp 47 0 10.60.186.195:50626 10.84.189.179:9160 ESTABLISHED | |
| Fri Apr 11 01:19:33 UTC 2014 | |
| tcp 47 0 10.60.186.195:50626 10.84.189.179:9160 ESTABLISHED | |
| Fri Apr 11 01:19:34 UTC 2014 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| This error on rabbit-hole: | |
| 2014-04-08T21:54:35.338Z - error: [rabbit:14855] [Error: Read Timed Out] 'message handler error' { payload: | |
| { timestamp_id: '1396914077729011088', | |
| hearter_id: 413960, | |
| entry_id: 110252952 }, | |
| type: 'unheart', | |
| ack: [Function], | |
| nack: [Function] } 'Error: Read Timed Out\n at null._onTimeout (/srv/rabbit-hole/releases/20140408214440/node_modules/helenus/lib/connection.js:403:16)\n at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)' | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Datacenter: datacenter1 | |
| ======================= | |
| Status=Up/Down | |
| |/ State=Normal/Leaving/Joining/Moving | |
| -- Address Load Tokens Owns (effective) Host ID Rack | |
| UN 10.84.189.183 52.41 MB 256 100.0% 6de6c0aa-55a8-4180-ba97-7c48d7052ee5 rack1 | |
| UN 10.84.189.179 57.01 MB 260 94.3% 166ac779-ff8c-489d-8084-a5309d00e027 rack1 | |
| UN 10.84.189.181 123.39 MB 255 27.4% a09e4785-2ecc-413f-a1ae-b99063284645 rack1 | |
| UN 10.84.189.143 1.14 GB 254 78.3% f58d712b-4a46-4126-b383-51ccfae8080a rack1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Datacenter: datacenter1 | |
| ======================= | |
| Status=Up/Down | |
| |/ State=Normal/Leaving/Joining/Moving | |
| -- Address Load Tokens Owns (effective) Host ID Rack | |
| DN 10.84.189.183 10.92 MB 256 80.5% 0429410c-111b-4d0f-a71d-4ae7a849be7b rack1 | |
| UN 10.84.189.179 41.01 MB 256 99.0% 166ac779-ff8c-489d-8084-a5309d00e027 rack1 | |
| UN 10.84.189.181 90.83 MB 256 27.4% a09e4785-2ecc-413f-a1ae-b99063284645 rack1 | |
| UN 10.84.189.143 448.2 MB 256 93.1% f58d712b-4a46-4126-b383-51ccfae8080a rack1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"ok":true,"cluster_name":"prod","nodes":{"nXGqcVzQQvKrM7sQ1eoBMg":{"name":"search05.weheartit.com","transport_address":"inet[/10.84.30.30:9300]","hostname":"search05.weheartit.com","version":"0.90.5","http_address":"inet[/10.84.30.30:9200]","attributes":{"max_local_storage_nodes":"1"},"settings":{"path.home":"//elasticsearch","pidfile":"/var/run/elasticsearch/search05_weheartit_com.pid","config":"/etc/elasticsearch/elasticsearch.yml","action.disable_delete_all_indices":"true","gateway.type":"local","node.max_local_storage_nodes":"1","bootstrap.mlockall":"true","path.data":"/var/data/elasticsearch","cluster.name":"prod","index.mapper.dynamic":"true","path.conf":"/etc/elasticsearch","discovery.zen.minimum_master_nodes":"1","network.host":"10.84.30.30","node.data":"true","gateway.expected_nodes":"1","node.name":"search05.weheartit.com","http.enabled":"true","path.logs":"/var/log/elasticsearch","discovery.zen.ping.unicast.hosts":"10.84.30.32,10.84.30.10,10.84.30.56,10.84.30.60,10.84.30.30,10.84.30.58,10.84.30.28 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"ok":true,"_shards":{"total":72,"successful":72,"failed":0},"_all":{"primaries":{"docs":{"count":91923482,"deleted":3455871},"store":{"size":"52.1gb","size_in_bytes":56031408449,"throttle_time":"13.7m","throttle_time_in_millis":826392},"indexing":{"index_total":18756808,"index_time":"4.3h","index_time_in_millis":15677810,"index_current":0,"delete_total":318,"delete_time":"66ms","delete_time_in_millis":66,"delete_current":0},"get":{"total":0,"get_time":"0s","time_in_millis":0,"exists_total":0,"exists_time":"0s","exists_time_in_millis":0,"missing_total":0,"missing_time":"0s","missing_time_in_millis":0,"current":0},"search":{"open_contexts":2170,"query_total":99496761,"query_time":"22.6d","query_time_in_millis":1954587020,"query_current":183,"fetch_total":87698215,"fetch_time":"6.6h","fetch_time_in_millis":24002054,"fetch_current":2},"merges":{"current":2,"current_docs":4787,"current_size":"11.1mb","current_size_in_bytes":11667197,"total":26652,"total_time":"6.1h","total_time_in_millis":22033477,"total_docs":11 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ::: [search02.weheartit.com][BhLFmPTJRYSGh2LrecepMw][inet[/10.84.30.10:9300]]{max_local_storage_nodes=1} | |
| 107.7% (538.3ms out of 500ms) cpu usage by thread 'elasticsearch[search02.weheartit.com][search][T#86]' | |
| 2/10 snapshots sharing following 28 elements | |
| sun.nio.ch.NativeThread.current(Native Method) | |
| sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46) | |
| sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:670) | |
| org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176) | |
| org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:272) | |
| org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:138) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| all.each do |user| | |
| directory "/home/#{user[0]}/" do | |
| owner "#{user[0]}" | |
| group "root" | |
| mode "0755" | |
| action :create | |
| end | |
| directory "/home/#{user[0]}/.ssh" do | |
| owner "#{user[0]}" | |
| group "#{user[0]}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2013-07-14T08:22:32.436 ERROR (81712) #B6761E862510] ActiveRecord::ConnectionTimeoutError: reconnect timed out | |
| > [#B6761E862510] /u/apps/weheartit/shared/bundle/ruby/1.9.1/bundler/gems/seamless_database_pool-5e4a936596f9/lib/seamless_database_pool/connect_timeout.rb:20:in `rescue in reconnect_with_connect_timeout!' | |
| > [#B6761E862510] /u/apps/weheartit/shared/bundle/ruby/1.9.1/bundler/gems/seamless_database_pool-5e4a936596f9/lib/seamless_database_pool/connect_timeout.rb:15:in `reconnect_with_connect_timeout!' | |
| > [#B6761E862510] /u/apps/weheartit/shared/bundle/ruby/1.9.1/gems/activerecord-3.2.13/lib/active_record/connection_adapters/abstract_adapter.rb:219:in `verify!' | |
| > [#B6761E862510] /u/apps/weheartit/shared/bundle/ruby/1.9.1/bundler/gems/seamless_database_pool-5e4a936596f9/lib/active_record/connection_adapters/seamless_database_pool_adapter.rb:224:in `block in verify!' | |
| > [#B6761E862510] /u/apps/weheartit/shared/bundle/ruby/1.9.1/bundler/gems/seamless_database_pool-5e4a936596f9/lib/active_record/conn |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # 5.8s user time, 50ms system time, 29.40M rss, 88.84M vsz | |
| # Current date: Wed Jun 26 17:17:31 2013 | |
| # Hostname: db12 | |
| # Files: slow.log | |
| # Overall: 16.13k total, 98 unique, 0.11 QPS, 0.50x concurrency __________ | |
| # Time range: 2013-06-25 00:02:45 to 2013-06-26 17:11:36 | |
| # Attribute total min max avg 95% stddev median | |
| # ============ ======= ======= ======= ======= ======= ======= ======= | |
| # Exec time 74513s 1s 340s 5s 5s 14s 3s | |
| # Lock time 7s 26us 227ms 411us 98us 4ms 57us |