Skip to content

Instantly share code, notes, and snippets.

This file has been truncated, but you can view the full file.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 6163 yarn cwd DIR 8,2 4096 1581994 /usr/hdp/2.4.0.0-169/hadoop-yarn
java 6163 yarn rtd DIR 8,2 4096 2 /
java 6163 yarn txt REG 8,2 7734 16912906 /usr/java/jdk1.8.0_72/bin/java
java 6163 yarn mem REG 8,2 157072 11272201 /lib64/ld-2.12.so
java 6163 yarn mem REG 8,2 22536 11272216 /lib64/libdl-2.12.so
java 6163 yarn mem REG 8,2 1926520 11272208 /lib64/libc-2.12.so
java 6163 yarn mem REG 8,2 145896 11272224 /lib64/libpthread-2.12.so
java 6163 yarn mem REG 8,2 47112 11272232 /lib64/librt-2.12.so
java 6163 yarn mem REG 8,2 599392 11272265 /lib64/libm-2.12.so
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000003bb6e7845b, pid=141736, tid=0x00007f637a5ed700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x7845b]
#
2016-06-06 12:12:03,782 WARN util.JvmPauseMonitor (JvmPauseMonitor.java:run(191)) - Detected pause in JVM or host machine (eg GC): pause of approximately 52067ms
GC pool 'ParNew' had collection(s): count=1 time=853ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=51371ms
2016-06-06 12:12:03,804 WARN client.QuorumJournalManager (IPCLoggerChannel.java:call(388)) - Remote journal 10.133.133.195:8485 failed to write txns 1065096713-1065096713. Will try to write to this JN again after the next log roll.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): IPC's epoch 7 is less than the last promised epoch 8
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkRequest(Journal.java:418)
at org.apache.hadoop.hdfs.qjournal.server.Journal.checkWriteRequest(Journal.java:446)
at org.apache.hadoop.hdfs.qjournal.server.Journal.journal(Journal.java:341)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.journal(JournalNodeRpcServer.java:148)
at or
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "*",
"analyze_wildcard": true
}
},
"filter": {
{
"taskStatus" : {
"taskId" : "20160525_080222_03978_8y49s.1.3",
"taskInstanceId" : "6df0347f-0bf8-42f9-b0fe-2b8112c904a5",
"version" : 3998,
"state" : "RUNNING",
"self" : "http://.../v1/task/20160525_080222_03978_8y49s.1.3",
"failures" : [ ],
"queuedPartitionedDrivers" : 43,
"runningPartitionedDrivers" : 1,
2016-05-25 17:06:02
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode):
"async-http-response-56793" #92033 daemon prio=5 os_prio=0 tid=0x00007fbad4032800 nid=0x19a26 waiting on condition [0x00007fba2e3b6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003c00e5760> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
複数サービスをモニタリングする方法
alertmanagerのhistory
$ curl -s http://localhost:9093/api/v1/alerts/groups | jq "."
{
"data": null,
"status": "success"
}
$ curl -s http://localhost:9093/api/v1/alerts/groups | jq "."
{
"data": [
{
"blocks": [
Is query of http://www.robustperception.io/using-group_left-to-calculate-label-proportions/ correct?
I think the following.
* incorrect
sum without (cpu)(rate(node_cpu[1m])) / ignoring(mode) group_left sum without (mode, cpu)(rate(node_cpu[1m]))
* correct
sum without (cpu)(rate(node_cpu[1m])) / on(instance, job) group_left(mode) sum without (mode, cpu)(rate(node_cpu[1m]))
sum(rate(http_requests_total{instance="..."}[10m])) by (code, job) / on(job) group_left(code) (sum(rate(http_requests_total{instance="..."}[10m])) by(job))