Created
May 27, 2016 21:33
-
-
Save msukmanowsky/52ba544e13864691819aba2153723cab to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2016-05-27 21:26:23,209 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Process tree for container: container_1444365544629_1373_01_115320 has processes older than 1 iteration running over the configured limit. Limit=17716740096, current usage = 17761673216 | |
| 2016-05-27 21:26:23,211 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Container [pid=8812,containerID=container_1444365544629_1373_01_115320] is running beyond physical memory limits. Current usage: 16.5 GB of 16.5 GB physical memory used; 31.4 GB of 82.5 GB virtual memory used. Killing container. | |
| Dump of the process-tree for container_1444365544629_1373_01_115320 : | |
| |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE | |
| |- 19475 8930 8930 8812 (python) 122020 9482 648155136 27838 python -m pyspark.daemon | |
| |- 22219 8930 8930 8812 (python) 112689 11552 676610048 34496 python -m pyspark.daemon | |
| |- 19508 8930 8930 8812 (python) 112622 8971 858513408 37003 python -m pyspark.daemon | |
| |- 8821 8812 8812 8812 (java) 247020 70026 18016911360 3579485 /usr/lib/jvm/java-openjdk/bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms15360m -Xmx15360m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -Djava.io.tmpdir=/mnt/yarn/usercache/hadoop/appcache/application_1444365544629_1373/container_1444365544629_1373_01_115320/tmp -Dspark.history.ui.port=18080 -Dspark.driver.port=48712 -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1444365544629_1373/container_1444365544629_1373_01_115320 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url akka.tcp://[email protected]:48712/user/CoarseGrainedScheduler --executor-id 79 --hostname ip-10-21-0-163.ec2.internal --cores 10 --app-id application_1444365544629_1373 --user-class-path file:/mnt/yarn/usercache/hadoop/appcache/application_1444365544629_1373/container_1444365544629_1373_01_115320/__app__.jar | |
| |- 19523 8930 8930 8812 (python) 131277 11226 734552064 32185 python -m pyspark.daemon | |
| |- 19514 8930 8930 8812 (python) 124088 11676 856109056 44679 python -m pyspark.daemon | |
| |- 19505 8930 8930 8812 (python) 124127 11660 965763072 40574 python -m pyspark.daemon | |
| |- 19520 8930 8930 8812 (python) 130656 9852 734928896 31527 python -m pyspark.daemon | |
| |- 19517 8930 8930 8812 (python) 125705 11361 798380032 55966 python -m pyspark.daemon | |
| |- 19478 8930 8930 8812 (python) 115150 10158 607948800 34186 python -m pyspark.daemon | |
| |- 19529 8930 8930 8812 (python) 115916 10414 897613824 39774 python -m pyspark.daemon | |
| |- 8930 8821 8930 8812 (python) 48 72 179798016 5624 python -m pyspark.daemon | |
| |- 19511 8930 8930 8812 (python) 140022 10937 741769216 33782 python -m pyspark.daemon | |
| |- 19502 8930 8930 8812 (python) 129376 10297 737386496 33172 python -m pyspark.daemon | |
| |- 19499 8930 8930 8812 (python) 133399 10717 755941376 36916 python -m pyspark.daemon | |
| |- 19487 8930 8930 8812 (python) 132347 11485 752463872 35968 python -m pyspark.daemon | |
| |- 19496 8930 8930 8812 (python) 128248 12292 806166528 41873 python -m pyspark.daemon | |
| |- 19526 8930 8930 8812 (python) 124980 10369 797589504 39639 python -m pyspark.daemon | |
| |- 8812 8810 8812 8812 (bash) 0 1 9662464 292 /bin/bash -c /usr/lib/jvm/java-openjdk/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms15360m -Xmx15360m '-verbose:gc' '-XX:+PrintGCDetails' '-XX:+PrintGCDateStamps' '-XX:+UseConcMarkSweepGC' '-XX:CMSInitiatingOccupancyFraction=70' '-XX:MaxHeapFreeRatio=70' '-XX:+CMSClassUnloadingEnabled' -Djava.io.tmpdir=/mnt/yarn/usercache/hadoop/appcache/application_1444365544629_1373/container_1444365544629_1373_01_115320/tmp '-Dspark.history.ui.port=18080' '-Dspark.driver.port=48712' -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1444365544629_1373/container_1444365544629_1373_01_115320 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url akka.tcp://[email protected]:48712/user/CoarseGrainedScheduler --executor-id 79 --hostname ip-10-21-0-163.ec2.internal --cores 10 --app-id application_1444365544629_1373 --user-class-path file:/mnt/yarn/usercache/hadoop/appcache/application_1444365544629_1373/container_1444365544629_1373_01_115320/__app__.jar 1> /var/log/hadoop-yarn/containers/application_1444365544629_1373/container_1444365544629_1373_01_115320/stdout 2> /var/log/hadoop-yarn/containers/application_1444365544629_1373/container_1444365544629_1373_01_115320/stderr | |
| |- 19484 8930 8930 8812 (python) 124438 11066 887005184 36955 python -m pyspark.daemon | |
| |- 19481 8930 8930 8812 (python) 127927 11710 799477760 40278 python -m pyspark.daemon | |
| |- 19493 8930 8930 8812 (python) 119666 11695 736460800 41130 python -m pyspark.daemon | |
| |- 19490 8930 8930 8812 (python) 136461 11026 704823296 33004 python -m pyspark.daemon | |
| 2016-05-27 21:26:23,212 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl (Container Monitor): Removed ProcessTree with root 8812 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment