Skip to content

Instantly share code, notes, and snippets.

@codyaray
Created January 15, 2016 13:30
Show Gist options
  • Select an option

  • Save codyaray/56eb3997e626e068460b to your computer and use it in GitHub Desktop.

Select an option

Save codyaray/56eb3997e626e068460b to your computer and use it in GitHub Desktop.
pre-market-open hot_threads
::: [svlchi6pesm03.peak6.net][0zQiOjCySx2lc2CjKKAuxQ][svlchi6pesm03][inet[/10.2.4.180:9300]]{data=false, box_type=mgmt, master=true}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
0.0% (196.5micros out of 500ms) cpu usage by thread 'elasticsearch[svlchi6pesm03.peak6.net][keep_alive]'
10/10 snapshots sharing following 3 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.marvel.agent.exporter.ESExporter$ConnectionKeepAliveWorker.run(ESExporter.java:788)
java.lang.Thread.run(Thread.java:745)
0.0% (91.9micros out of 500ms) cpu usage by thread 'elasticsearch[svlchi6pesm03.peak6.net][transport_client_timer][T#1]{Hashed wheel timer #1}'
10/10 snapshots sharing following 5 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
java.lang.Thread.run(Thread.java:745)
::: [svlchi6pes02.peak6.net][7QVV3C62QeyeH9RySjm7Cg][svlchi6pes02][inet[/10.2.4.182:9300]]{box_type=slow, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
0.0% (125.1micros out of 500ms) cpu usage by thread 'elasticsearch[svlchi6pes02.peak6.net][transport_client_timer][T#1]{Hashed wheel timer #1}'
10/10 snapshots sharing following 5 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
java.lang.Thread.run(Thread.java:745)
::: [svlchi6pes01.peak6.net][u3lgZbZoRy-zgTz7jOq33g][svlchi6pes01][inet[/10.2.4.181:9300]]{box_type=slow, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi5pfes08.peak6.net][uaLcTHohTiyaRCHHMtxkUQ][svlchi5pfes08][inet[/10.100.0.179:9300]]{box_type=fast, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
4.3% (21.5ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi5pfes08.peak6.net][bulk][T#35]'
9/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
4.2% (20.8ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi5pfes08.peak6.net][bulk][T#6]'
9/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
::: [svlchi5pfes07.peak6.net][8d9eVbg8SNGyNBCBcqlocQ][svlchi5pfes07][inet[/10.100.0.178:9300]]{box_type=fast, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi6pesm01.peak6.net][YHffHzSlTcKcSDeer8L0hQ][svlchi6pesm01.peak6.net][inet[/10.2.4.186:9300]]{data=false, box_type=mgmt, master=true}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi5pfes03.peak6.net][uWadUHGYSaCPfo3MnG0VsA][svlchi5pfes03][inet[/10.100.0.174:9300]]{box_type=fast, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi5pfes01.peak6.net][E-nLOTFwQFSB0X4HxggcnA][svlchi5pfes01][inet[/10.100.0.172:9300]]{box_type=fast, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
5.1% (25.6ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi5pfes01.peak6.net][marvel.exporters]'
10/10 snapshots sharing following 3 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:248)
java.lang.Thread.run(Thread.java:745)
4.2% (21ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi5pfes01.peak6.net][bulk][T#27]'
5/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
4.1% (20.7ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi5pfes01.peak6.net][bulk][T#17]'
5/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
::: [svlchi6pesm02.peak6.net][U9YcfSMhRc2g-8wOMcp_Nw][svlchi6pesm02.peak6.net][inet[/10.2.4.187:9300]]{data=false, box_type=mgmt, master=true}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi6pes07.peak6.net][wSEY25OARx2t_PYU7ke3dw][svlchi6pes07][inet[/10.2.4.190:9300]]{box_type=slow, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi5pfes05.peak6.net][DHORYEdOQ7y7fmmDzckmKg][svlchi5pfes05][inet[/10.100.0.176:9300]]{box_type=fast, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
5.0% (24.9ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi5pfes05.peak6.net][refresh][T#11]'
10/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
::: [svlchi6pes05.peak6.net][u3OLXtSLQ02OtgHoWhkvMg][svlchi6pes05][inet[/10.2.4.188:9300]]{box_type=slow, master=false}
Hot threads at 2016-01-15T13:21:43.133Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi5pfes02.peak6.net][8DmJ80nsTduNSpgiE6Pw_A][svlchi5pfes02][inet[/10.100.0.173:9300]]{box_type=fast, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi6pes06.peak6.net][D3DDGBe4Qs-7ZL4zBb-Hdg][svlchi6pes06][inet[svlchi6pes06.peak6.net/10.2.4.189:9300]]{box_type=slow, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi5pfes06.peak6.net][V0YnfMScTy2SHBzADhibMQ][svlchi5pfes06][inet[/10.100.0.177:9300]]{box_type=fast, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
3.5% (17.6ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi5pfes06.peak6.net][bulk][T#34]'
6/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
::: [svlchi6pes08.peak6.net][PIdBM5vTRaaBk4TO9-diOA][svlchi6pes08][inet[/10.2.4.191:9300]]{box_type=slow, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi6pes03.peak6.net][blzl2yRCQfy26n20I3p7yw][svlchi6pes03][inet[/10.2.4.183:9300]]{box_type=slow, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
20.1% (100.5ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi6pes03.peak6.net][marvel.exporters]'
10/10 snapshots sharing following 3 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:248)
java.lang.Thread.run(Thread.java:745)
::: [sslchi6pglacieres01.peak6.net][N0B5vl5_RB-HwYTDXEmwvQ][sslchi6pglacieres01][inet[/10.2.4.87:9300]]{box_type=glacier, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi5pfes04.peak6.net][84ZrcLMSSdyKrZFl8JWbIw][svlchi5pfes04][inet[/10.100.0.175:9300]]{box_type=fast, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
::: [svlchi6pes04.peak6.net][-YJ6doXGRGS76qM0r9te3Q][svlchi6pes04][inet[/10.2.4.184:9300]]{box_type=slow, master=false}
Hot threads at 2016-01-15T13:21:43.134Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
3.7% (18.5ms out of 500ms) cpu usage by thread 'elasticsearch[svlchi6pes04.peak6.net][marvel.exporters]'
4/10 snapshots sharing following 14 elements
org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexReader.ramBytesUsed(CompressingStoredFieldsIndexReader.java:176)
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.ramBytesUsed(CompressingStoredFieldsReader.java:529)
org.apache.lucene.index.SegmentCoreReaders.ramBytesUsed(SegmentCoreReaders.java:226)
org.apache.lucene.index.SegmentReader.ramBytesUsed(SegmentReader.java:614)
org.elasticsearch.index.engine.Engine.segmentsStats(Engine.java:309)
org.elasticsearch.index.shard.IndexShard.segmentStats(IndexShard.java:652)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:175)
org.elasticsearch.action.admin.indices.stats.ShardStats.<init>(ShardStats.java:55)
org.elasticsearch.indices.IndicesService.stats(IndicesService.java:231)
org.elasticsearch.indices.IndicesService.stats(IndicesService.java:188)
org.elasticsearch.node.service.NodeService.stats(NodeService.java:138)
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportNodeStats(AgentService.java:342)
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:254)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 19 elements
org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:237)
org.elasticsearch.index.store.fs.DefaultFsDirectoryService$1.listAll(DefaultFsDirectoryService.java:57)
org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1579)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1571)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1558)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:55)
org.elasticsearch.index.store.Store.stats(Store.java:290)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:639)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:139)
org.elasticsearch.action.admin.indices.stats.ShardStats.<init>(ShardStats.java:55)
org.elasticsearch.indices.IndicesService.stats(IndicesService.java:231)
org.elasticsearch.indices.IndicesService.stats(IndicesService.java:188)
org.elasticsearch.node.service.NodeService.stats(NodeService.java:138)
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportNodeStats(AgentService.java:342)
org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:254)
java.lang.Thread.run(Thread.java:745)
3/10 snapshots sharing following 1 elements
java.lang.Thread.run(Thread.java:745)
@ambuc

ambuc commented Oct 2, 2017

Copy link
Copy Markdown

?

@cjohnhanson

Copy link
Copy Markdown

?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment