Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save reedzhao/b6cc5753a4059396bcc6d7e15bc4484c to your computer and use it in GitHub Desktop.
Save reedzhao/b6cc5753a4059396bcc6d7e15bc4484c to your computer and use it in GitHub Desktop.
hot_threads 2018-05-04T10:36:24
# Last GC
2018-05-04T10:36:24.364-0400: 4417.644: [GC (Allocation Failure) 2018-05-04T10:36:24.368-0400: 4417.648: [ParNew: 1602075K->40584K(1763584K), 0.5094672 secs] 2916725K->1355235K(8192704K), 0.5149063 secs] [Times: user=8.88 sys=0.91, real=0.52 secs]
# hot_threads
Fri May 4 10:34:27 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:27.256Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
8.6% (42.8ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#1]'
10/10 snapshots sharing following 14 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:357)
org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:275)
org.elasticsearch.cluster.InternalClusterInfoService.access$500(InternalClusterInfoService.java:68)
org.elasticsearch.cluster.InternalClusterInfoService$SubmitReschedulingClusterInfoUpdatedJob.lambda$run$0(InternalClusterInfoService.java:222)
org.elasticsearch.cluster.InternalClusterInfoService$SubmitReschedulingClusterInfoUpdatedJob$$Lambda$2489/1321448011.run(Unknown Source)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:34:29 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:29.468Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
120.9% (604.2ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#3]'
4/10 snapshots sharing following 37 elements
java.io.UnixFileSystem.canonicalize0(Native Method)
java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
java.io.File.getCanonicalPath(File.java:618)
java.io.FilePermission$1.run(FilePermission.java:215)
java.io.FilePermission$1.run(FilePermission.java:203)
java.security.AccessController.doPrivileged(Native Method)
java.io.FilePermission.init(FilePermission.java:203)
java.io.FilePermission.<init>(FilePermission.java:277)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
6/10 snapshots sharing following 19 elements
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
3.3% (16.6ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#39]'
4/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
3.0% (14.9ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#23]'
2/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:34:31 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:31.734Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:33 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:33.923Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
3.2% (15.9ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#51]'
8/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:34:36 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:36.200Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.2% (761micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:34:38 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:38.400Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:40 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:40.572Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:42 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:42.729Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:44 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:44.873Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:46 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:47.010Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.1% (721.7micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:34:49 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:49.170Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
2.2% (10.9ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#39]'
4/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
0.2% (1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.2% (760.6micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:34:51 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:51.377Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:53 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:53.535Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:55 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:55.679Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:57 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:34:57.848Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:34:59 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:00.032Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
4.6% (22.9ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#13]'
3/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
3.0% (15.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#12]'
9/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:02 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:02.212Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
98.7% (493.5ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#3]'
6/10 snapshots sharing following 29 elements
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
4/10 snapshots sharing following 30 elements
sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:52)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
7.6% (37.8ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#13]'
10/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
6.1% (30.6ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#1]'
10/10 snapshots sharing following 14 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
org.elasticsearch.cluster.InternalClusterInfoService.refresh(InternalClusterInfoService.java:357)
org.elasticsearch.cluster.InternalClusterInfoService.maybeRefresh(InternalClusterInfoService.java:275)
org.elasticsearch.cluster.InternalClusterInfoService.access$500(InternalClusterInfoService.java:68)
org.elasticsearch.cluster.InternalClusterInfoService$SubmitReschedulingClusterInfoUpdatedJob.lambda$run$0(InternalClusterInfoService.java:222)
org.elasticsearch.cluster.InternalClusterInfoService$SubmitReschedulingClusterInfoUpdatedJob$$Lambda$2489/1321448011.run(Unknown Source)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
3.9% (19.4ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#2]'
unique snapshot
org.elasticsearch.common.regex.Regex.simpleMatch(Regex.java:133)
org.elasticsearch.common.settings.SettingsFilter.filterSettings(SettingsFilter.java:111)
org.elasticsearch.common.settings.SettingsFilter.filter(SettingsFilter.java:93)
org.elasticsearch.node.NodeService.info(NodeService.java:93)
org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction.nodeOperation(TransportNodesInfoAction.java:75)
org.elasticsearch.action.admin.cluster.node.info.TransportNodesInfoAction.nodeOperation(TransportNodesInfoAction.java:39)
org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140)
org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:262)
org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:258)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:04 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:04.525Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
120.5% (602.2ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#3]'
4/10 snapshots sharing following 37 elements
java.io.UnixFileSystem.canonicalize0(Native Method)
java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
java.io.File.getCanonicalPath(File.java:618)
java.io.FilePermission$1.run(FilePermission.java:215)
java.io.FilePermission$1.run(FilePermission.java:203)
java.security.AccessController.doPrivileged(Native Method)
java.io.FilePermission.init(FilePermission.java:203)
java.io.FilePermission.<init>(FilePermission.java:277)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
3/10 snapshots sharing following 30 elements
sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:52)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 32 elements
java.security.AccessController.getStackAccessControlContext(Native Method)
java.security.AccessController.checkPermission(AccessController.java:858)
java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
unique snapshot
java.io.UnixFileSystem.canonicalize0(Native Method)
java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
java.io.File.getCanonicalPath(File.java:618)
java.io.FilePermission$1.run(FilePermission.java:215)
java.io.FilePermission$1.run(FilePermission.java:203)
java.security.AccessController.doPrivileged(Native Method)
java.io.FilePermission.init(FilePermission.java:203)
java.io.FilePermission.<init>(FilePermission.java:277)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:394)
java.nio.file.Files.newDirectoryStream(Files.java:457)
org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:215)
org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:234)
org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1418)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:06 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:06.741Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:08 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:08.946Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
3.4% (17ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#23]'
8/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:11 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:11.147Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
89.8% (448.9ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][refresh][T#4]'
3/10 snapshots sharing following 37 elements
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:189)
org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:37)
org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:724)
org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:718)
org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:692)
org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:4979)
org.apache.lucene.index.DocumentsWriter$DeleteNewFilesEvent.process(DocumentsWriter.java:779)
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5057)
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5045)
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:466)
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258)
org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104)
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:292)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:267)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1332)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1314)
org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:855)
org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:695)
org.elasticsearch.index.IndexService.access$400(IndexService.java:97)
org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:899)
org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:809)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
7/10 snapshots sharing following 24 elements
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258)
org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104)
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:292)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:267)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1332)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1314)
org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:855)
org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:695)
org.elasticsearch.index.IndexService.access$400(IndexService.java:97)
org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:899)
org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:809)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.4% (1.8ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
Fri May 4 10:35:13 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:13.414Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:15 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:15.604Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
3.4% (16.9ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#35]'
8/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:17 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:17.813Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:19 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:20.039Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:22 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:22.218Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:24 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:24.387Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (1.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
Fri May 4 10:35:26 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:26.551Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:28 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:28.725Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:30 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:30.884Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
^[[6~^[[6~^[[6~Fri May 4 10:35:32 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:33.044Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (1.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.1% (718.2micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:35 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:35.231Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:37 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:37.408Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
119.6% (597.9ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#3]'
8/10 snapshots sharing following 29 elements
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 30 elements
sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:52)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.3% (1.4ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.1% (660.6micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:39 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:39.643Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
122.4% (611.9ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#3]'
2/10 snapshots sharing following 32 elements
java.security.AccessController.getStackAccessControlContext(Native Method)
java.security.AccessController.checkPermission(AccessController.java:858)
java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
5/10 snapshots sharing following 37 elements
java.io.UnixFileSystem.canonicalize0(Native Method)
java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
java.io.File.getCanonicalPath(File.java:618)
java.io.FilePermission$1.run(FilePermission.java:215)
java.io.FilePermission$1.run(FilePermission.java:203)
java.security.AccessController.doPrivileged(Native Method)
java.io.FilePermission.init(FilePermission.java:203)
java.io.FilePermission.<init>(FilePermission.java:277)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 26 elements
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
unique snapshot
sun.nio.fs.UnixNativeDispatcher.readdir(Native Method)
sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.readNextEntry(UnixDirectoryStream.java:168)
sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator.hasNext(UnixDirectoryStream.java:201)
org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:216)
org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:234)
org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1418)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.6% (3ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#1]'
6/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:41 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:42.003Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:44 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:44.163Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.4% (1.8ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][scheduler][T#1]'
unique snapshot
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:348)
java.lang.management.PlatformComponent.getMXBeanInterface(PlatformComponent.java:366)
java.lang.management.PlatformComponent.getPlatformComponent(PlatformComponent.java:462)
java.lang.management.ManagementFactory.getPlatformMXBeans(ManagementFactory.java:704)
org.elasticsearch.monitor.jvm.JvmStats.jvmStats(JvmStats.java:104)
org.elasticsearch.monitor.jvm.JvmGcMonitorService$JvmMonitor.jvmStats(JvmGcMonitorService.java:482)
org.elasticsearch.monitor.jvm.JvmGcMonitorService$JvmMonitor.monitorGc(JvmGcMonitorService.java:402)
org.elasticsearch.monitor.jvm.JvmGcMonitorService$JvmMonitor.run(JvmGcMonitorService.java:391)
org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:182)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:441)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.2% (1.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.1% (720.1micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:46 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:46.348Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (1.2ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
Fri May 4 10:35:48 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:48.519Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (1.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
Fri May 4 10:35:50 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:50.681Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
4.2% (21.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#21]'
9/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:52 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:52.893Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:54 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:55.136Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:35:57 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:57.304Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (1.2ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.1% (720.8micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:35:59 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:35:59.459Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.4% (1.8ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][scheduler][T#1]'
9/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.3% (1.4ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
Fri May 4 10:36:01 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:01.620Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:03 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:03.797Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:05 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:05.971Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:07 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:08.116Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:10 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:10.302Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:12 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:12.511Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
113.9% (569.6ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#3]'
5/10 snapshots sharing following 30 elements
sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:52)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
4/10 snapshots sharing following 37 elements
java.io.UnixFileSystem.canonicalize0(Native Method)
java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
java.io.File.getCanonicalPath(File.java:618)
java.io.FilePermission$1.run(FilePermission.java:215)
java.io.FilePermission$1.run(FilePermission.java:203)
java.security.AccessController.doPrivileged(Native Method)
java.io.FilePermission.init(FilePermission.java:203)
java.io.FilePermission.<init>(FilePermission.java:277)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
unique snapshot
java.security.AccessController.getStackAccessControlContext(Native Method)
java.security.AccessController.checkPermission(AccessController.java:858)
java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
52.8% (263.8ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][refresh][T#6]'
10/10 snapshots sharing following 24 elements
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268)
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258)
org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104)
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:292)
org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:267)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1332)
org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1314)
org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:855)
org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:695)
org.elasticsearch.index.IndexService.access$400(IndexService.java:97)
org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:899)
org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:809)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.3% (1.6ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.2% (1ms out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:36:14 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:14.900Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
119.3% (596.3ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][management][T#3]'
2/10 snapshots sharing following 30 elements
sun.nio.fs.UnixNativeDispatcher.stat0(Native Method)
sun.nio.fs.UnixNativeDispatcher.stat(UnixNativeDispatcher.java:286)
sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:70)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:52)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
8/10 snapshots sharing following 29 elements
java.lang.SecurityManager.checkRead(SecurityManager.java:888)
sun.nio.fs.UnixPath.checkRead(UnixPath.java:795)
sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49)
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
java.nio.file.Files.readAttributes(Files.java:1737)
java.nio.file.Files.size(Files.java:2332)
org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:243)
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:67)
org.elasticsearch.index.store.Store$StoreStatsCache.estimateSize(Store.java:1421)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1410)
org.elasticsearch.index.store.Store$StoreStatsCache.refresh(Store.java:1399)
org.elasticsearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:54)
org.elasticsearch.index.store.Store.stats(Store.java:349)
org.elasticsearch.index.shard.IndexShard.storeStats(IndexShard.java:947)
org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:178)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:164)
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:45)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:433)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:412)
org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:399)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66)
org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652)
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:36:16 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:17.099Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
3.0% (15.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#22]'
9/10 snapshots sharing following 2 elements
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:36:19 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:19.286Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
118.9% (594.4ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][http_server_worker][T#46]'
2/10 snapshots sharing following 95 elements
org.elasticsearch.common.xcontent.XContentBuilder.ensureNoSelfReferences(XContentBuilder.java:1092)
org.elasticsearch.common.xcontent.XContentBuilder.ensureNoSelfReferences(XContentBuilder.java:1092)
org.elasticsearch.common.xcontent.XContentBuilder.ensureNoSelfReferences(XContentBuilder.java:1071)
org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:872)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$1.writeMappings(RestGetAllMappingsAction.java:101)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$1.buildResponse(RestGetAllMappingsAction.java:85)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$1.buildResponse(RestGetAllMappingsAction.java:76)
org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:38)
org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)
org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onResponse(TransportMasterNodeAction.java:150)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onResponse(TransportMasterNodeAction.java:147)
org.elasticsearch.action.admin.indices.get.TransportGetIndexAction.doMasterOperation(TransportGetIndexAction.java:114)
org.elasticsearch.action.admin.indices.get.TransportGetIndexAction.doMasterOperation(TransportGetIndexAction.java:47)
org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:52)
org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:34)
org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:164)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:127)
org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:105)
org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:55)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
tech.beshu.ror.es.IndexLevelActionFilter$1.onAllow(IndexLevelActionFilter.java:177)
tech.beshu.ror.acl.ACL.lambda$check$2(ACL.java:194)
tech.beshu.ror.acl.ACL$$Lambda$2335/1682755854.apply(Unknown Source)
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614)
java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983)
tech.beshu.ror.acl.ACL.check(ACL.java:182)
tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.java:153)
tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.java:128)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)
org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1253)
org.elasticsearch.client.support.AbstractClient$IndicesAdmin.getIndex(AbstractClient.java:1354)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction.lambda$prepareRequest$0(RestGetAllMappingsAction.java:76)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$$Lambda$2861/1208254472.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97)
tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$3(ReadonlyRestPlugin.java:176)
tech.beshu.ror.es.ReadonlyRestPlugin$$Lambda$2229/54483866.handleRequest(Unknown Source)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:497)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 93 elements
org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:86)
org.elasticsearch.cluster.metadata.MappingMetaData.sourceAsMap(MappingMetaData.java:175)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$1.writeMappings(RestGetAllMappingsAction.java:101)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$1.buildResponse(RestGetAllMappingsAction.java:85)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$1.buildResponse(RestGetAllMappingsAction.java:76)
org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:38)
org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)
org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onResponse(TransportMasterNodeAction.java:150)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onResponse(TransportMasterNodeAction.java:147)
org.elasticsearch.action.admin.indices.get.TransportGetIndexAction.doMasterOperation(TransportGetIndexAction.java:114)
org.elasticsearch.action.admin.indices.get.TransportGetIndexAction.doMasterOperation(TransportGetIndexAction.java:47)
org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:52)
org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:34)
org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:164)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:127)
org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:105)
org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:55)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
tech.beshu.ror.es.IndexLevelActionFilter$1.onAllow(IndexLevelActionFilter.java:177)
tech.beshu.ror.acl.ACL.lambda$check$2(ACL.java:194)
tech.beshu.ror.acl.ACL$$Lambda$2335/1682755854.apply(Unknown Source)
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614)
java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983)
tech.beshu.ror.acl.ACL.check(ACL.java:182)
tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.java:153)
tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.java:128)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)
org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1253)
org.elasticsearch.client.support.AbstractClient$IndicesAdmin.getIndex(AbstractClient.java:1354)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction.lambda$prepareRequest$0(RestGetAllMappingsAction.java:76)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$$Lambda$2861/1208254472.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97)
tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$3(ReadonlyRestPlugin.java:176)
tech.beshu.ror.es.ReadonlyRestPlugin$$Lambda$2229/54483866.handleRequest(Unknown Source)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:497)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
2/10 snapshots sharing following 87 elements
org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)
org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onResponse(TransportMasterNodeAction.java:150)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onResponse(TransportMasterNodeAction.java:147)
org.elasticsearch.action.admin.indices.get.TransportGetIndexAction.doMasterOperation(TransportGetIndexAction.java:114)
org.elasticsearch.action.admin.indices.get.TransportGetIndexAction.doMasterOperation(TransportGetIndexAction.java:47)
org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:52)
org.elasticsearch.action.support.master.info.TransportClusterInfoAction.masterOperation(TransportClusterInfoAction.java:34)
org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:88)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:167)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:164)
org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:127)
org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:105)
org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:55)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)
tech.beshu.ror.es.IndexLevelActionFilter$1.onAllow(IndexLevelActionFilter.java:177)
tech.beshu.ror.acl.ACL.lambda$check$2(ACL.java:194)
tech.beshu.ror.acl.ACL$$Lambda$2335/1682755854.apply(Unknown Source)
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:614)
java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1983)
tech.beshu.ror.acl.ACL.check(ACL.java:182)
tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.java:153)
tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.java:128)
org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)
org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83)
org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72)
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)
org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1253)
org.elasticsearch.client.support.AbstractClient$IndicesAdmin.getIndex(AbstractClient.java:1354)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction.lambda$prepareRequest$0(RestGetAllMappingsAction.java:76)
org.elasticsearch.rest.action.admin.indices.RestGetAllMappingsAction$$Lambda$2861/1208254472.accept(Unknown Source)
org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:97)
tech.beshu.ror.es.ReadonlyRestPlugin.lambda$null$3(ReadonlyRestPlugin.java:176)
tech.beshu.ror.es.ReadonlyRestPlugin$$Lambda$2229/54483866.handleRequest(Unknown Source)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240)
org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336)
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174)
org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:497)
org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
4/10 snapshots sharing following 9 elements
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:752)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:408)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:748)
0.2% (1.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.2% (880.8micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:36:21 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:21.576Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (1.1ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.1% (721.7micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:36:23 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:23.741Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:26 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:27.077Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:29 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:29.231Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:31 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:31.381Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:33 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:33.546Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.2% (961.3micros out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
0.1% (580.7micros out of 500ms) cpu usage by thread 'threadDeathWatcher-2-1'
10/10 snapshots sharing following 4 elements
java.lang.Thread.sleep(Native Method)
io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
java.lang.Thread.run(Thread.java:748)
Fri May 4 10:36:35 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:35.698Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
0.3% (1.6ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][scheduler][T#1]'
8/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.3% (1.3ms out of 500ms) cpu usage by thread 'elasticsearch[EEvTfAV][[timer]]'
10/10 snapshots sharing following 2 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:541)
Fri May 4 10:36:37 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:37.861Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:39 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:40.032Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Fri May 4 10:36:42 EDT 2018
::: {EEvTfAV}{EEvTfAV0QN-RBB8dGd23EA}{qjjV7KZSRwqJqUFAmfWSsQ}{207.38.70.156}{207.38.70.156:9300}
Hot threads at 2018-05-04T14:36:42.230Z, interval=500ms, busiestThreads=10, ignoreIdleThreads=true:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment