Last active
August 29, 2015 14:18
-
-
Save apple-corps/ee5fac1cd4bb610ca991 to your computer and use it in GitHub Desktop.
elasticsearch 1.4.1 cluster timeouts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| tail -f es_cluster.log | |
| [2015-04-10 18:29:52,784][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [260248ms] ago, timed out [245248ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277821587] | |
| [2015-04-10 18:29:53,001][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [230465ms] ago, timed out [215465ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277831180] | |
| [2015-04-10 18:29:53,219][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [200683ms] ago, timed out [185683ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277841018] | |
| [2015-04-10 18:29:53,439][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [170902ms] ago, timed out [155902ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277850631] | |
| [2015-04-10 18:29:53,903][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [141366ms] ago, timed out [126366ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277860244] | |
| [2015-04-10 18:29:54,346][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [111809ms] ago, timed out [96809ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277869909] | |
| [2015-04-10 18:29:54,663][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [50751ms] ago, timed out [20751ms] ago, action [internal:cluster/nodes/indices/shard/store[n]], node [[densm2es017][_MMBnIzlTNuUGXtUM-HVbw][densm2es017.prod.local][inet[/10.51.29.17:9300]]{rack=0, group=2, master=false}], id [277897321] | |
| [2015-04-10 18:29:54,744][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [82206ms] ago, timed out [67206ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277879526] | |
| [2015-04-10 18:29:55,128][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [52590ms] ago, timed out [37590ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277889150] | |
| [2015-04-10 18:29:55,629][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [23091ms] ago, timed out [8091ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es002][OM4RguhkTb2Ai9BtHYI7TA][densm2es002.prod.local][inet[/10.51.29.2:9300]]{rack=0, group=2, master=false}], id [277898777] | |
| [2015-04-10 18:30:01,050][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [328515ms] ago, timed out [313515ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277794295] | |
| [2015-04-10 18:30:01,497][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [298962ms] ago, timed out [283962ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277803887] | |
| [2015-04-10 18:30:02,217][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [269681ms] ago, timed out [254681ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277821575] | |
| [2015-04-10 18:30:02,596][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [240060ms] ago, timed out [225060ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277831166] | |
| [2015-04-10 18:30:03,092][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [210556ms] ago, timed out [195556ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277841006] | |
| [2015-04-10 18:30:03,529][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [167419ms] ago, timed out [137419ms] ago, action [internal:cluster/nodes/indices/shard/store[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277859371] | |
| [2015-04-10 18:30:03,989][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [181452ms] ago, timed out [166451ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277850619] | |
| [2015-04-10 18:30:04,270][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [151733ms] ago, timed out [136733ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277860232] | |
| [2015-04-10 18:30:04,687][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [122150ms] ago, timed out [107150ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277869897] | |
| [2015-04-10 18:30:05,377][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [92840ms] ago, timed out [77839ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277879514] | |
| [2015-04-10 18:30:05,898][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [63360ms] ago, timed out [48360ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277889138] | |
| [2015-04-10 18:30:06,104][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [33566ms] ago, timed out [18566ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es021][wE8P_zGmQMKQSxL4vNkpng][densm2es021.prod.local][inet[/10.51.29.21:9300]]{rack=0, group=0, master=false}], id [277898765] | |
| [2015-04-10 18:30:07,175][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [64637ms] ago, timed out [49633ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es022][VRj3UI0SQG-nBXs2gnxG0Q][densm2es022.prod.local][inet[/10.51.29.22:9300]]{rack=0, group=1, master=false}], id [277889168] | |
| [2015-04-10 18:30:07,580][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [35042ms] ago, timed out [20042ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es022][VRj3UI0SQG-nBXs2gnxG0Q][densm2es022.prod.local][inet[/10.51.29.22:9300]]{rack=0, group=1, master=false}], id [277898795] | |
| [2015-04-10 18:30:10,649][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [68111ms] ago, timed out [53111ms] ago, action [cluster:monitor/nodes/stats[n]], node [[us3sm2es015r09][ZVSv_56bSfuQlZ105DVrjg][us3sm2es015r09.comp.prod.local][inet[/10.51.28.150:9300]]{rack=9, group=0, master=false}], id [277889148] | |
| [2015-04-10 18:30:16,987][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [104449ms] ago, timed out [89449ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es025][UjseS4pRRZaljnKUhWMnOA][densm2es025.prod.local][inet[/10.51.29.25:9300]]{rack=0, group=1, master=false}], id [277879541] | |
| [2015-04-10 18:30:16,987][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [74449ms] ago, timed out [59449ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es025][UjseS4pRRZaljnKUhWMnOA][densm2es025.prod.local][inet[/10.51.29.25:9300]]{rack=0, group=1, master=false}], id [277889165] | |
| [2015-04-10 18:30:16,987][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [44449ms] ago, timed out [29449ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es025][UjseS4pRRZaljnKUhWMnOA][densm2es025.prod.local][inet[/10.51.29.25:9300]]{rack=0, group=1, master=false}], id [277898792] | |
| [2015-04-10 18:30:17,538][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [hDkIruinSiKG-SPh6JemsA] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es010][inet[/10.51.29.10:9300]][cluster:monitor/nodes/stats[n]] request_id [277916455] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:30:17,538][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [ZVSv_56bSfuQlZ105DVrjg] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [us3sm2es015r09][inet[/10.51.28.150:9300]][cluster:monitor/nodes/stats[n]] request_id [277916469] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:30:17,538][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [pYTXmEUKQGOuq1--5wclrg] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es014][inet[/10.51.29.14:9300]][cluster:monitor/nodes/stats[n]] request_id [277916461] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:30:17,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [0c-vwPXxQL2tUO3ekPODrA] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es013][inet[/10.51.29.13:9300]][cluster:monitor/nodes/stats[n]] request_id [277916495] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:30:17,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [VRj3UI0SQG-nBXs2gnxG0Q] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es022][inet[/10.51.29.22:9300]][cluster:monitor/nodes/stats[n]] request_id [277916489] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:30:20,287][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [44046ms] ago, timed out [14046ms] ago, action [internal:cluster/nodes/indices/shard/store[n]], node [[densm2es006][4FJnktIfR6moPOTvr5utYg][densm2es006.prod.local][inet[/10.51.29.6:9300]]{rack=0, group=0, master=false}], id [277907107] | |
| [2015-04-10 18:30:24,799][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [22260ms] ago, timed out [7260ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es022][VRj3UI0SQG-nBXs2gnxG0Q][densm2es022.prod.local][inet[/10.51.29.22:9300]]{rack=0, group=1, master=false}], id [277916489] | |
| [2015-04-10 18:30:31,325][WARN ][cluster.action.shard ] [us3sm2zk012r09] [social_content_2015-01-02_live_v1_20150331-022944][1] received shard failed for [social_content_2015-01-02_live_v1_20150331-022944][1], node[qN8t--i8TISIZ1Pnv2FUaQ], [R], s[STARTED], indexUUID [AvelpNRmSl6ZMiV20ZhBsw], reason [Failed to perform [indices:data/write/bulk[s]] on replica, message [SendRequestTransportException[[us3sm2es001r07][inet[/10.51.28.231:9300]][indices:data/write/bulk[s][r]]]; nested: NodeNotConnectedException[[us3sm2es001r07][inet[/10.51.28.231:9300]] Node not connected]; ]] | |
| [2015-04-10 18:30:31,329][WARN ][cluster.action.shard ] [us3sm2zk012r09] [social_content_2015-01-02_live_v1_20150331-022944][1] received shard failed for [social_content_2015-01-02_live_v1_20150331-022944][1], node[qN8t--i8TISIZ1Pnv2FUaQ], [R], s[STARTED], indexUUID [AvelpNRmSl6ZMiV20ZhBsw], reason [Failed to perform [indices:data/write/bulk[s]] on replica, message [SendRequestTransportException[[us3sm2es001r07][inet[/10.51.28.231:9300]][indices:data/write/bulk[s][r]]]; nested: NodeNotConnectedException[[us3sm2es001r07][inet[/10.51.28.231:9300]] Node not connected]; ]] | |
| [2015-04-10 18:30:31,330][WARN ][cluster.action.shard ] [us3sm2zk012r09] [social_content_2015-01-02_live_v1_20150331-022944][1] received shard failed for [social_content_2015-01-02_live_v1_20150331-022944][1], node[qN8t--i8TISIZ1Pnv2FUaQ], [R], s[STARTED], indexUUID [AvelpNRmSl6ZMiV20ZhBsw], reason [Failed to perform [indices:data/write/bulk[s]] on replica, message [SendRequestTransportException[[us3sm2es001r07][inet[/10.51.28.231:9300]][indices:data/write/bulk[s][r]]]; nested: NodeNotConnectedException[[us3sm2es001r07][inet[/10.51.28.231:9300]] Node not connected]; ]] | |
| [2015-04-10 18:30:31,337][WARN ][cluster.action.shard ] [us3sm2zk012r09] [social_content_2015-01-02_live_v1_20150331-022944][1] received shard failed for [social_content_2015-01-02_live_v1_20150331-022944][1], node[qN8t--i8TISIZ1Pnv2FUaQ], [R], s[STARTED], indexUUID [AvelpNRmSl6ZMiV20ZhBsw], reason [Failed to perform [indices:data/write/bulk[s]] on replica, message [SendRequestTransportException[[us3sm2es001r07][inet[/10.51.28.231:9300]][indices:data/write/bulk[s][r]]]; nested: NodeNotConnectedException[[us3sm2es001r07][inet[/10.51.28.231:9300]] Node not connected]; ]] | |
| [2015-04-10 18:30:47,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [hDkIruinSiKG-SPh6JemsA] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es010][inet[/10.51.29.10:9300]][cluster:monitor/nodes/stats[n]] request_id [277926228] timed out after [15001ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:30:47,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [ZVSv_56bSfuQlZ105DVrjg] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [us3sm2es015r09][inet[/10.51.28.150:9300]][cluster:monitor/nodes/stats[n]] request_id [277926242] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:30:47,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [pYTXmEUKQGOuq1--5wclrg] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es014][inet[/10.51.29.14:9300]][cluster:monitor/nodes/stats[n]] request_id [277926234] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:30:47,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [0c-vwPXxQL2tUO3ekPODrA] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es013][inet[/10.51.29.13:9300]][cluster:monitor/nodes/stats[n]] request_id [277926268] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:31:04,280][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [91742ms] ago, timed out [76742ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es010][hDkIruinSiKG-SPh6JemsA][densm2es010.prod.local][inet[/10.51.29.10:9300]]{rack=0, group=1, master=false}], id [277898761] | |
| [2015-04-10 18:31:09,496][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [66958ms] ago, timed out [51958ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es010][hDkIruinSiKG-SPh6JemsA][densm2es010.prod.local][inet[/10.51.29.10:9300]]{rack=0, group=1, master=false}], id [277916455] | |
| [2015-04-10 18:31:09,862][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [185998ms] ago, timed out [155998ms] ago, action [internal:cluster/nodes/indices/shard/store[n]], node [[densm2es022][VRj3UI0SQG-nBXs2gnxG0Q][densm2es022.prod.local][inet[/10.51.29.22:9300]]{rack=0, group=1, master=false}], id [277878072] | |
| [2015-04-10 18:31:11,853][WARN ][transport ] [us3sm2zk012r09] Received response for a request that has timed out, sent [39315ms] ago, timed out [24314ms] ago, action [cluster:monitor/nodes/stats[n]], node [[densm2es010][hDkIruinSiKG-SPh6JemsA][densm2es010.prod.local][inet[/10.51.29.10:9300]]{rack=0, group=1, master=false}], id [277926228] | |
| [2015-04-10 18:31:17,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [pYTXmEUKQGOuq1--5wclrg] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es014][inet[/10.51.29.14:9300]][cluster:monitor/nodes/stats[n]] request_id [277935853] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:31:17,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [NTp0azt0T3GfU3orfHkY7w] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [us3sm2es003r09][inet[/10.51.28.233:9300]][cluster:monitor/nodes/stats[n]] request_id [277935852] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:31:17,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [wE8P_zGmQMKQSxL4vNkpng] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es021][inet[/10.51.29.21:9300]][cluster:monitor/nodes/stats[n]] request_id [277935851] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:31:17,543][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [0c-vwPXxQL2tUO3ekPODrA] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es013][inet[/10.51.29.13:9300]][cluster:monitor/nodes/stats[n]] request_id [277935887] timed out after [15003ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:31:17,539][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [VRj3UI0SQG-nBXs2gnxG0Q] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [densm2es022][inet[/10.51.29.22:9300]][cluster:monitor/nodes/stats[n]] request_id [277935881] timed out after [15000ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
| [2015-04-10 18:31:17,543][DEBUG][action.admin.cluster.node.stats] [us3sm2zk012r09] failed to execute on node [ZVSv_56bSfuQlZ105DVrjg] | |
| org.elasticsearch.transport.ReceiveTimeoutTransportException: [us3sm2es015r09][inet[/10.51.28.150:9300]][cluster:monitor/nodes/stats[n]] request_id [277935861] timed out after [15004ms] | |
| at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) | |
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) | |
| at java.lang.Thread.run(Thread.java:745) | |
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
^^ master logs filled with timeouts
_cat/nodes && _cat/shards && _cat/indices time out across the cluster
_cat/health returns green