wyukawa

2016-03-16 06:11:39,749 ERROR [HiveServer2-Handler-Pool: Thread-94731]: exec.Task (TezTask.java:execute(212)) - Failed to execute tez graph.
java.lang.NullPointerException: Inflater has been closed
        at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
        at java.util.zip.Inflater.inflate(Inflater.java:257)
        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
        at org.apache.xerces.impl.io.UTF8Reader.read(Unknown Source)
        at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
        at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source)

wyukawa

2016-03-13 20:29:36
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode):

"IPC Client (426052262) connection to ...:8020 from hive" #10097 daemon prio=5 os_prio=0 tid=0x00007f23c7247800 nid=0x13ce7 in Object.wait() [0x00007f23693dc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:931)
	- locked <0x000000071685ea10> (a org.apache.hadoop.ipc.Client$Connection)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:976)

wyukawa

2016-03-13 13:15:58,362 INFO  [HiveServer2-Handler-Pool: Thread-98838]: mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting tokens for job: job_1455848418011_44042
2016-03-13 13:15:58,568 INFO  [HiveServer2-Handler-Pool: Thread-98838]: impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted application application_1455848418011_44042
2016-03-13 13:15:58,569 INFO  [HiveServer2-Handler-Pool: Thread-98838]: mapreduce.Job (Job.java:submit(1294)) - The url to track the job: http://resourcemanager:port/proxy/application_1455848418011_44042/
2016-03-13 13:15:58,569 INFO  [HiveServer2-Handler-Pool: Thread-98838]: exec.Task (SessionState.java:printInfo(923)) - Starting Job = job_1455848418011_44042, Tracking URL = http://resourcemanager:port/proxy/application_1455848418011_44042/
2016-03-13 13:15:58,569 INFO  [HiveServer2-Handler-Pool: Thread-98838]: exec.Task (SessionState.java:printInfo(923)) - Kill Command = /usr/hdp/2.3.4.0-3485/hadoop/bin/hadoop job  -kill job_14558484180

wyukawa

Attaching to process ID 144937, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.72-b15
Deadlock Detection:

No deadlocks found.

Thread 45044: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)

wyukawa

$  /usr/java/jdk1.8.0_72/bin/jstack 35161 > 35161.jstack
35161: well-known file is not secure
$  /usr/java/jdk1.8.0_72/bin/jstack -F 35161 > 35161.jstack
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
        at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)
        at sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
        at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)

wyukawa

org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. unable to create new native thread
        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:183)
        at org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:410)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:391)
        at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:261)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:486)
        at org.apache.hive.service.

wyukawa

select
  aaa,
  sum(bbb)
from (
  select
    ...
  from access_log
  where yyyymmdd='20160224' AND ...
)x
group by aaa

wyukawa

java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:157)
        at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)

wyukawa

  <configuration>
    
    <property>
      <name>tez.am.am-rm.heartbeat.interval-ms.max</name>
      <value>250</value>
    </property>
    
    <property>
      <name>tez.am.container.idle.release-timeout-max.millis</name>
      <value>20000</value>

wyukawa

やりたいことは2つのhadoopクラスタへのdouble write

  <match hoge.{aaa.xxx,aaa.yyyy,aaa.zzz}>
    type copy
    <store>
    ...
    </store>
    <store>
    ...
     </store>