Skip to content

Instantly share code, notes, and snippets.

@akanto

akanto/perf-test-repl2

Created April 11, 2016 14:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save akanto/19066d7c047b03c3baf01dbc76ca6972 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save akanto/19066d7c047b03c3baf01dbc76ca6972 to your computer and use it in GitHub Desktop.
Download ZIP
perf-test-repl2
Raw
perf-test-repl2
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:41:18 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:41:18 INFO fs.TestDFSIO: nrFiles = 1
16/04/11 13:41:18 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:41:18 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:41:18 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:41:19 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 1 files
16/04/11 13:41:19 INFO fs.TestDFSIO: created control files for: 1 files
16/04/11 13:41:19 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:41:19 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:41:20 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:41:20 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:41:20 INFO mapred.FileInputFormat: Total input paths to process : 1
16/04/11 13:41:20 INFO mapreduce.JobSubmitter: number of splits:1
16/04/11 13:41:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0001
16/04/11 13:41:20 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0001
16/04/11 13:41:20 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0001/
16/04/11 13:41:20 INFO mapreduce.Job: Running job: job_1460382022143_0001
16/04/11 13:41:27 INFO mapreduce.Job: Job job_1460382022143_0001 running in uber mode : false
16/04/11 13:41:27 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:41:39 INFO mapreduce.Job: map 67% reduce 0%
^[[A16/04/11 13:41:59 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 13:42:04 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:42:05 INFO mapreduce.Job: Job job_1460382022143_0001 completed successfully
16/04/11 13:42:05 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=96
FILE: Number of bytes written=266918
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=256
HDFS: Number of bytes written=8589934672
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=29838
Total time spent by all reduces in occupied slots (ms)=6556
Total time spent by all map tasks (ms)=29838
Total time spent by all reduce tasks (ms)=3278
Total vcore-seconds taken by all map tasks=29838
Total vcore-seconds taken by all reduce tasks=3278
Total megabyte-seconds taken by all map tasks=106939392
Total megabyte-seconds taken by all reduce tasks=23496704
Map-Reduce Framework
Map input records=1
Map output records=5
Map output bytes=80
Map output materialized bytes=96
Input split bytes=144
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=96
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=794
CPU time spent (ms)=29090
Physical memory (bytes) snapshot=2857730048
Virtual memory (bytes) snapshot=10541359104
Total committed heap usage (bytes)=5114953728
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=112
File Output Format Counters
Bytes Written=80
16/04/11 13:42:05 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:42:05 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:42:05 UTC 2016
16/04/11 13:42:05 INFO fs.TestDFSIO: Number of files: 1
16/04/11 13:42:05 INFO fs.TestDFSIO: Total MBytes processed: 8192.0
16/04/11 13:42:05 INFO fs.TestDFSIO: Throughput mb/sec: 326.79112813148237
16/04/11 13:42:05 INFO fs.TestDFSIO: Average IO rate mb/sec: 326.7911376953125
16/04/11 13:42:05 INFO fs.TestDFSIO: IO rate std deviation: 0.04861786394293855
16/04/11 13:42:05 INFO fs.TestDFSIO: Test exec time sec: 45.997
16/04/11 13:42:05 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:42:06 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:42:06 INFO fs.TestDFSIO: nrFiles = 4
16/04/11 13:42:06 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:42:06 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:42:06 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:42:07 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 4 files
16/04/11 13:42:08 INFO fs.TestDFSIO: created control files for: 4 files
16/04/11 13:42:08 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:42:08 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:42:08 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:42:08 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:42:09 INFO mapred.FileInputFormat: Total input paths to process : 4
16/04/11 13:42:09 INFO mapreduce.JobSubmitter: number of splits:4
16/04/11 13:42:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0002
16/04/11 13:42:09 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0002
16/04/11 13:42:09 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0002/
16/04/11 13:42:09 INFO mapreduce.Job: Running job: job_1460382022143_0002
16/04/11 13:42:15 INFO mapreduce.Job: Job job_1460382022143_0002 running in uber mode : false
16/04/11 13:42:15 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:42:25 INFO mapreduce.Job: map 33% reduce 0%
16/04/11 13:42:26 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:42:48 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 13:42:50 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 13:42:56 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 13:42:57 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:42:57 INFO mapreduce.Job: Job job_1460382022143_0002 completed successfully
16/04/11 13:42:58 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=359
FILE: Number of bytes written=667632
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1024
HDFS: Number of bytes written=34359738449
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Job Counters
Launched map tasks=4
Launched reduce tasks=1
Data-local map tasks=4
Total time spent by all maps in occupied slots (ms)=142139
Total time spent by all reduces in occupied slots (ms)=11272
Total time spent by all map tasks (ms)=142139
Total time spent by all reduce tasks (ms)=5636
Total vcore-seconds taken by all map tasks=142139
Total vcore-seconds taken by all reduce tasks=5636
Total megabyte-seconds taken by all map tasks=509426176
Total megabyte-seconds taken by all reduce tasks=40398848
Map-Reduce Framework
Map input records=4
Map output records=20
Map output bytes=313
Map output materialized bytes=377
Input split bytes=576
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=377
Reduce input records=20
Reduce output records=5
Spilled Records=40
Shuffled Maps =4
Failed Shuffles=0
Merged Map outputs=4
GC time elapsed (ms)=3021
CPU time spent (ms)=119890
Physical memory (bytes) snapshot=10508636160
Virtual memory (bytes) snapshot=21599805440
Total committed heap usage (bytes)=14104395776
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=448
File Output Format Counters
Bytes Written=81
16/04/11 13:42:58 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:42:58 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:42:58 UTC 2016
16/04/11 13:42:58 INFO fs.TestDFSIO: Number of files: 4
16/04/11 13:42:58 INFO fs.TestDFSIO: Total MBytes processed: 32768.0
16/04/11 13:42:58 INFO fs.TestDFSIO: Throughput mb/sec: 257.5412232579342
16/04/11 13:42:58 INFO fs.TestDFSIO: Average IO rate mb/sec: 260.19866943359375
16/04/11 13:42:58 INFO fs.TestDFSIO: IO rate std deviation: 26.600268795397643
16/04/11 13:42:58 INFO fs.TestDFSIO: Test exec time sec: 49.988
16/04/11 13:42:58 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:42:59 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:42:59 INFO fs.TestDFSIO: nrFiles = 8
16/04/11 13:42:59 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:42:59 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:42:59 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:43:00 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 8 files
16/04/11 13:43:00 INFO fs.TestDFSIO: created control files for: 8 files
16/04/11 13:43:01 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:43:01 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:43:01 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:43:01 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:43:01 INFO mapred.FileInputFormat: Total input paths to process : 8
16/04/11 13:43:01 INFO mapreduce.JobSubmitter: number of splits:8
16/04/11 13:43:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0003
16/04/11 13:43:02 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0003
16/04/11 13:43:02 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0003/
16/04/11 13:43:02 INFO mapreduce.Job: Running job: job_1460382022143_0003
16/04/11 13:43:08 INFO mapreduce.Job: Job job_1460382022143_0003 running in uber mode : false
16/04/11 13:43:08 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:43:19 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:43:52 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 13:43:53 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 13:43:55 INFO mapreduce.Job: map 88% reduce 0%
16/04/11 13:43:57 INFO mapreduce.Job: map 96% reduce 0%
16/04/11 13:43:58 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 13:43:59 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:43:59 INFO mapreduce.Job: Job job_1460382022143_0003 completed successfully
16/04/11 13:43:59 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=715
FILE: Number of bytes written=1201928
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2048
HDFS: Number of bytes written=68719476815
HDFS: Number of read operations=35
HDFS: Number of large read operations=0
HDFS: Number of write operations=10
Job Counters
Launched map tasks=8
Launched reduce tasks=1
Data-local map tasks=7
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=356285
Total time spent by all reduces in occupied slots (ms)=9360
Total time spent by all map tasks (ms)=356285
Total time spent by all reduce tasks (ms)=4680
Total vcore-seconds taken by all map tasks=356285
Total vcore-seconds taken by all reduce tasks=4680
Total megabyte-seconds taken by all map tasks=1276925440
Total megabyte-seconds taken by all reduce tasks=33546240
Map-Reduce Framework
Map input records=8
Map output records=40
Map output bytes=629
Map output materialized bytes=757
Input split bytes=1152
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=757
Reduce input records=40
Reduce output records=5
Spilled Records=80
Shuffled Maps =8
Failed Shuffles=0
Merged Map outputs=8
GC time elapsed (ms)=5301
CPU time spent (ms)=244920
Physical memory (bytes) snapshot=20704481280
Virtual memory (bytes) snapshot=36387053568
Total committed heap usage (bytes)=26087522304
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=896
File Output Format Counters
Bytes Written=79
16/04/11 13:43:59 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:43:59 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:43:59 UTC 2016
16/04/11 13:43:59 INFO fs.TestDFSIO: Number of files: 8
16/04/11 13:43:59 INFO fs.TestDFSIO: Total MBytes processed: 65536.0
16/04/11 13:43:59 INFO fs.TestDFSIO: Throughput mb/sec: 201.6908101054679
16/04/11 13:43:59 INFO fs.TestDFSIO: Average IO rate mb/sec: 202.1424560546875
16/04/11 13:43:59 INFO fs.TestDFSIO: IO rate std deviation: 9.50933542255887
16/04/11 13:43:59 INFO fs.TestDFSIO: Test exec time sec: 59.049
16/04/11 13:43:59 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:44:01 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:44:01 INFO fs.TestDFSIO: nrFiles = 12
16/04/11 13:44:01 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:44:01 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:44:01 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:44:02 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 12 files
16/04/11 13:44:02 INFO fs.TestDFSIO: created control files for: 12 files
16/04/11 13:44:03 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:44:03 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:44:03 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:44:03 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:44:03 INFO mapred.FileInputFormat: Total input paths to process : 12
16/04/11 13:44:03 INFO mapreduce.JobSubmitter: number of splits:12
16/04/11 13:44:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0004
16/04/11 13:44:04 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0004
16/04/11 13:44:04 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0004/
16/04/11 13:44:04 INFO mapreduce.Job: Running job: job_1460382022143_0004
16/04/11 13:44:10 INFO mapreduce.Job: Job job_1460382022143_0004 running in uber mode : false
16/04/11 13:44:10 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:44:21 INFO mapreduce.Job: map 28% reduce 0%
16/04/11 13:44:22 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:45:07 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 13:45:08 INFO mapreduce.Job: map 81% reduce 0%
16/04/11 13:45:09 INFO mapreduce.Job: map 86% reduce 0%
16/04/11 13:45:18 INFO mapreduce.Job: map 86% reduce 19%
16/04/11 13:45:23 INFO mapreduce.Job: map 92% reduce 19%
16/04/11 13:45:24 INFO mapreduce.Job: map 92% reduce 25%
16/04/11 13:45:25 INFO mapreduce.Job: map 100% reduce 25%
16/04/11 13:45:26 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:45:26 INFO mapreduce.Job: Job job_1460382022143_0004 completed successfully
16/04/11 13:45:26 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=1071
FILE: Number of bytes written=1736251
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=3076
HDFS: Number of bytes written=103079215188
HDFS: Number of read operations=51
HDFS: Number of large read operations=0
HDFS: Number of write operations=14
Job Counters
Launched map tasks=12
Launched reduce tasks=1
Data-local map tasks=9
Rack-local map tasks=3
Total time spent by all maps in occupied slots (ms)=752217
Total time spent by all reduces in occupied slots (ms)=31328
Total time spent by all map tasks (ms)=752217
Total time spent by all reduce tasks (ms)=15664
Total vcore-seconds taken by all map tasks=752217
Total vcore-seconds taken by all reduce tasks=15664
Total megabyte-seconds taken by all map tasks=2695945728
Total megabyte-seconds taken by all reduce tasks=112279552
Map-Reduce Framework
Map input records=12
Map output records=60
Map output bytes=945
Map output materialized bytes=1137
Input split bytes=1730
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1137
Reduce input records=60
Reduce output records=5
Spilled Records=120
Shuffled Maps =12
Failed Shuffles=0
Merged Map outputs=12
GC time elapsed (ms)=8159
CPU time spent (ms)=392880
Physical memory (bytes) snapshot=30836670464
Virtual memory (bytes) snapshot=51123212288
Total committed heap usage (bytes)=37947441152
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1346
File Output Format Counters
Bytes Written=84
16/04/11 13:45:26 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:45:26 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:45:26 UTC 2016
16/04/11 13:45:26 INFO fs.TestDFSIO: Number of files: 12
16/04/11 13:45:26 INFO fs.TestDFSIO: Total MBytes processed: 98304.0
16/04/11 13:45:26 INFO fs.TestDFSIO: Throughput mb/sec: 139.79760746770424
16/04/11 13:45:26 INFO fs.TestDFSIO: Average IO rate mb/sec: 142.7149200439453
16/04/11 13:45:26 INFO fs.TestDFSIO: IO rate std deviation: 19.92026182622348
16/04/11 13:45:26 INFO fs.TestDFSIO: Test exec time sec: 84.183
16/04/11 13:45:26 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:45:28 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:45:28 INFO fs.TestDFSIO: nrFiles = 16
16/04/11 13:45:28 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:45:28 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:45:28 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:45:29 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 16 files
16/04/11 13:45:29 INFO fs.TestDFSIO: created control files for: 16 files
16/04/11 13:45:30 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:45:30 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:45:30 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:45:30 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:45:30 INFO mapred.FileInputFormat: Total input paths to process : 16
16/04/11 13:45:30 INFO mapreduce.JobSubmitter: number of splits:16
16/04/11 13:45:31 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0005
16/04/11 13:45:31 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0005
16/04/11 13:45:31 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0005/
16/04/11 13:45:31 INFO mapreduce.Job: Running job: job_1460382022143_0005
1316/04/11 13:45:37 INFO mapreduce.Job: Job job_1460382022143_0005 running in uber mode : false
16/04/11 13:45:37 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:45:48 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:46:49 INFO mapreduce.Job: map 71% reduce 0%
16/04/11 13:46:50 INFO mapreduce.Job: map 73% reduce 0%
16/04/11 13:46:51 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 13:46:52 INFO mapreduce.Job: map 77% reduce 0%
16/04/11 13:46:53 INFO mapreduce.Job: map 81% reduce 0%
16/04/11 13:46:55 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 13:46:56 INFO mapreduce.Job: map 90% reduce 0%
16/04/11 13:46:58 INFO mapreduce.Job: map 92% reduce 0%
16/04/11 13:46:59 INFO mapreduce.Job: map 96% reduce 0%
16/04/11 13:47:00 INFO mapreduce.Job: map 98% reduce 29%
16/04/11 13:47:01 INFO mapreduce.Job: map 100% reduce 29%
16/04/11 13:47:02 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:47:02 INFO mapreduce.Job: Job job_1460382022143_0005 completed successfully
16/04/11 13:47:02 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=1429
FILE: Number of bytes written=2270571
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=4108
HDFS: Number of bytes written=137438953557
HDFS: Number of read operations=67
HDFS: Number of large read operations=0
HDFS: Number of write operations=18
Job Counters
Launched map tasks=16
Launched reduce tasks=1
Data-local map tasks=12
Rack-local map tasks=4
Total time spent by all maps in occupied slots (ms)=1216760
Total time spent by all reduces in occupied slots (ms)=19838
Total time spent by all map tasks (ms)=1216760
Total time spent by all reduce tasks (ms)=9919
Total vcore-seconds taken by all map tasks=1216760
Total vcore-seconds taken by all reduce tasks=9919
Total megabyte-seconds taken by all map tasks=4360867840
Total megabyte-seconds taken by all reduce tasks=71099392
Map-Reduce Framework
Map input records=16
Map output records=80
Map output bytes=1263
Map output materialized bytes=1519
Input split bytes=2310
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1519
Reduce input records=80
Reduce output records=5
Spilled Records=160
Shuffled Maps =16
Failed Shuffles=0
Merged Map outputs=16
GC time elapsed (ms)=11436
CPU time spent (ms)=530010
Physical memory (bytes) snapshot=41110106112
Virtual memory (bytes) snapshot=65945767936
Total committed heap usage (bytes)=49943150592
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1798
File Output Format Counters
Bytes Written=85
16/04/11 13:47:02 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:47:02 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:47:02 UTC 2016
16/04/11 13:47:02 INFO fs.TestDFSIO: Number of files: 16
16/04/11 13:47:02 INFO fs.TestDFSIO: Total MBytes processed: 131072.0
16/04/11 13:47:02 INFO fs.TestDFSIO: Throughput mb/sec: 114.07762765324134
16/04/11 13:47:02 INFO fs.TestDFSIO: Average IO rate mb/sec: 114.40523529052734
16/04/11 13:47:02 INFO fs.TestDFSIO: IO rate std deviation: 6.126175703004024
16/04/11 13:47:02 INFO fs.TestDFSIO: Test exec time sec: 93.191
16/04/11 13:47:02 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:47:04 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:47:04 INFO fs.TestDFSIO: nrFiles = 20
16/04/11 13:47:04 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:47:04 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:47:04 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:47:05 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 20 files
16/04/11 13:47:05 INFO fs.TestDFSIO: created control files for: 20 files
16/04/11 13:47:06 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:47:06 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:47:06 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:47:06 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:47:06 INFO mapred.FileInputFormat: Total input paths to process : 20
16/04/11 13:47:06 INFO mapreduce.JobSubmitter: number of splits:20
16/04/11 13:47:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0006
16/04/11 13:47:07 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0006
16/04/11 13:47:07 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0006/
16/04/11 13:47:07 INFO mapreduce.Job: Running job: job_1460382022143_0006
1116/04/11 13:47:13 INFO mapreduce.Job: Job job_1460382022143_0006 running in uber mode : false
16/04/11 13:47:13 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:47:23 INFO mapreduce.Job: map 13% reduce 0%
16/04/11 13:47:24 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:48:14 INFO mapreduce.Job: map 68% reduce 0%
16/04/11 13:48:15 INFO mapreduce.Job: map 70% reduce 0%
16/04/11 13:48:17 INFO mapreduce.Job: map 73% reduce 0%
16/04/11 13:48:18 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 13:48:22 INFO mapreduce.Job: map 77% reduce 0%
16/04/11 13:48:24 INFO mapreduce.Job: map 77% reduce 10%
16/04/11 13:48:27 INFO mapreduce.Job: map 80% reduce 10%
16/04/11 13:48:28 INFO mapreduce.Job: map 82% reduce 10%
16/04/11 13:48:30 INFO mapreduce.Job: map 83% reduce 15%
16/04/11 13:48:31 INFO mapreduce.Job: map 85% reduce 15%
16/04/11 13:48:33 INFO mapreduce.Job: map 85% reduce 18%
16/04/11 13:49:20 INFO mapreduce.Job: map 87% reduce 18%
16/04/11 13:49:21 INFO mapreduce.Job: map 88% reduce 18%
16/04/11 13:49:22 INFO mapreduce.Job: map 90% reduce 22%
16/04/11 13:49:23 INFO mapreduce.Job: map 93% reduce 22%
16/04/11 13:49:24 INFO mapreduce.Job: map 97% reduce 22%
16/04/11 13:49:25 INFO mapreduce.Job: map 98% reduce 30%
16/04/11 13:49:26 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:49:27 INFO mapreduce.Job: Job job_1460382022143_0006 completed successfully
16/04/11 13:49:27 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=1773
FILE: Number of bytes written=2804849
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5140
HDFS: Number of bytes written=171798691925
HDFS: Number of read operations=83
HDFS: Number of large read operations=0
HDFS: Number of write operations=22
Job Counters
Launched map tasks=20
Launched reduce tasks=1
Data-local map tasks=19
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=1890727
Total time spent by all reduces in occupied slots (ms)=139882
Total time spent by all map tasks (ms)=1890727
Total time spent by all reduce tasks (ms)=69941
Total vcore-seconds taken by all map tasks=1890727
Total vcore-seconds taken by all reduce tasks=69941
Total megabyte-seconds taken by all map tasks=6776365568
Total megabyte-seconds taken by all reduce tasks=501337088
Map-Reduce Framework
Map input records=20
Map output records=100
Map output bytes=1567
Map output materialized bytes=1887
Input split bytes=2890
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1887
Reduce input records=100
Reduce output records=5
Spilled Records=200
Shuffled Maps =20
Failed Shuffles=0
Merged Map outputs=20
GC time elapsed (ms)=16205
CPU time spent (ms)=712170
Physical memory (bytes) snapshot=50515066880
Virtual memory (bytes) snapshot=80718311424
Total committed heap usage (bytes)=60998287360
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2250
File Output Format Counters
Bytes Written=85
16/04/11 13:49:27 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:49:27 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:49:27 UTC 2016
16/04/11 13:49:27 INFO fs.TestDFSIO: Number of files: 20
16/04/11 13:49:27 INFO fs.TestDFSIO: Total MBytes processed: 163840.0
16/04/11 13:49:27 INFO fs.TestDFSIO: Throughput mb/sec: 90.44603075849574
16/04/11 13:49:27 INFO fs.TestDFSIO: Average IO rate mb/sec: 101.04875946044922
16/04/11 13:49:27 INFO fs.TestDFSIO: IO rate std deviation: 32.531441650797994
16/04/11 13:49:27 INFO fs.TestDFSIO: Test exec time sec: 141.611
16/04/11 13:49:27 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:49:28 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:49:28 INFO fs.TestDFSIO: nrFiles = 24
16/04/11 13:49:28 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:49:28 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:49:28 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:49:29 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 24 files
16/04/11 13:49:30 INFO fs.TestDFSIO: created control files for: 24 files
16/04/11 13:49:30 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:49:30 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:49:31 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:49:31 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:49:31 INFO mapred.FileInputFormat: Total input paths to process : 24
16/04/11 13:49:31 INFO mapreduce.JobSubmitter: number of splits:24
16/04/11 13:49:31 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0007
16/04/11 13:49:31 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0007
16/04/11 13:49:31 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0007/
16/04/11 13:49:31 INFO mapreduce.Job: Running job: job_1460382022143_0007
16/04/11 13:49:37 INFO mapreduce.Job: Job job_1460382022143_0007 running in uber mode : false
16/04/11 13:49:37 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:49:48 INFO mapreduce.Job: map 22% reduce 0%
16/04/11 13:49:49 INFO mapreduce.Job: map 50% reduce 0%
16/04/11 13:49:50 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:50:51 INFO mapreduce.Job: map 71% reduce 0%
16/04/11 13:50:54 INFO mapreduce.Job: map 72% reduce 0%
16/04/11 13:50:55 INFO mapreduce.Job: map 74% reduce 0%
16/04/11 13:50:56 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 13:50:57 INFO mapreduce.Job: map 76% reduce 0%
16/04/11 13:50:59 INFO mapreduce.Job: map 78% reduce 0%
16/04/11 13:51:00 INFO mapreduce.Job: map 79% reduce 0%
16/04/11 13:51:01 INFO mapreduce.Job: map 82% reduce 0%
16/04/11 13:51:02 INFO mapreduce.Job: map 85% reduce 0%
16/04/11 13:51:03 INFO mapreduce.Job: map 86% reduce 15%
16/04/11 13:51:06 INFO mapreduce.Job: map 86% reduce 19%
16/04/11 13:51:12 INFO mapreduce.Job: map 88% reduce 19%
16/04/11 13:51:14 INFO mapreduce.Job: map 88% reduce 21%
16/04/11 13:51:45 INFO mapreduce.Job: map 89% reduce 21%
16/04/11 13:51:47 INFO mapreduce.Job: map 89% reduce 22%
16/04/11 13:51:48 INFO mapreduce.Job: map 93% reduce 22%
16/04/11 13:51:50 INFO mapreduce.Job: map 94% reduce 26%
16/04/11 13:51:53 INFO mapreduce.Job: map 96% reduce 28%
16/04/11 13:51:54 INFO mapreduce.Job: map 100% reduce 28%
16/04/11 13:51:55 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:51:55 INFO mapreduce.Job: Job job_1460382022143_0007 completed successfully
16/04/11 13:51:55 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=2125
FILE: Number of bytes written=3339149
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=6172
HDFS: Number of bytes written=206158430293
HDFS: Number of read operations=99
HDFS: Number of large read operations=0
HDFS: Number of write operations=26
Job Counters
Launched map tasks=24
Launched reduce tasks=1
Data-local map tasks=23
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2349376
Total time spent by all reduces in occupied slots (ms)=121840
Total time spent by all map tasks (ms)=2349376
Total time spent by all reduce tasks (ms)=60920
Total vcore-seconds taken by all map tasks=2349376
Total vcore-seconds taken by all reduce tasks=60920
Total megabyte-seconds taken by all map tasks=8420163584
Total megabyte-seconds taken by all reduce tasks=436674560
Map-Reduce Framework
Map input records=24
Map output records=120
Map output bytes=1879
Map output materialized bytes=2263
Input split bytes=3470
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=2263
Reduce input records=120
Reduce output records=5
Spilled Records=240
Shuffled Maps =24
Failed Shuffles=0
Merged Map outputs=24
GC time elapsed (ms)=19220
CPU time spent (ms)=862970
Physical memory (bytes) snapshot=60849561600
Virtual memory (bytes) snapshot=95427092480
Total committed heap usage (bytes)=73051144192
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2702
File Output Format Counters
Bytes Written=85
16/04/11 13:51:55 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:51:55 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:51:55 UTC 2016
16/04/11 13:51:55 INFO fs.TestDFSIO: Number of files: 24
16/04/11 13:51:55 INFO fs.TestDFSIO: Total MBytes processed: 196608.0
16/04/11 13:51:55 INFO fs.TestDFSIO: Throughput mb/sec: 87.72271113720772
16/04/11 13:51:55 INFO fs.TestDFSIO: Average IO rate mb/sec: 93.854736328125
16/04/11 13:51:55 INFO fs.TestDFSIO: IO rate std deviation: 22.907345472372686
16/04/11 13:51:55 INFO fs.TestDFSIO: Test exec time sec: 145.45
16/04/11 13:51:55 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:51:57 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:51:57 INFO fs.TestDFSIO: nrFiles = 28
16/04/11 13:51:57 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:51:57 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:51:57 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:51:58 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 28 files
16/04/11 13:51:58 INFO fs.TestDFSIO: created control files for: 28 files
16/04/11 13:51:59 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:51:59 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:51:59 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:51:59 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:51:59 INFO mapred.FileInputFormat: Total input paths to process : 28
16/04/11 13:51:59 INFO mapreduce.JobSubmitter: number of splits:28
16/04/11 13:52:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0008
16/04/11 13:52:00 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0008
16/04/11 13:52:00 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0008/
16/04/11 13:52:00 INFO mapreduce.Job: Running job: job_1460382022143_0008
87.716/04/11 13:52:06 INFO mapreduce.Job: Job job_1460382022143_0008 running in uber mode : false
16/04/11 13:52:06 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:52:17 INFO mapreduce.Job: map 24% reduce 0%
16/04/11 13:52:18 INFO mapreduce.Job: map 64% reduce 0%
16/04/11 13:52:19 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:53:23 INFO mapreduce.Job: map 68% reduce 0%
16/04/11 13:53:25 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 13:53:26 INFO mapreduce.Job: map 70% reduce 0%
16/04/11 13:53:28 INFO mapreduce.Job: map 71% reduce 0%
16/04/11 13:53:29 INFO mapreduce.Job: map 74% reduce 0%
16/04/11 13:53:32 INFO mapreduce.Job: map 77% reduce 0%
16/04/11 13:53:33 INFO mapreduce.Job: map 79% reduce 0%
16/04/11 13:53:34 INFO mapreduce.Job: map 81% reduce 0%
16/04/11 13:53:35 INFO mapreduce.Job: map 82% reduce 0%
16/04/11 13:53:37 INFO mapreduce.Job: map 83% reduce 15%
16/04/11 13:53:40 INFO mapreduce.Job: map 85% reduce 17%
16/04/11 13:53:41 INFO mapreduce.Job: map 89% reduce 17%
16/04/11 13:53:43 INFO mapreduce.Job: map 89% reduce 23%
16/04/11 13:53:47 INFO mapreduce.Job: map 90% reduce 23%
16/04/11 13:53:49 INFO mapreduce.Job: map 90% reduce 24%
16/04/11 13:54:17 INFO mapreduce.Job: map 92% reduce 24%
16/04/11 13:54:19 INFO mapreduce.Job: map 93% reduce 25%
16/04/11 13:54:22 INFO mapreduce.Job: map 93% reduce 26%
16/04/11 13:54:23 INFO mapreduce.Job: map 95% reduce 26%
16/04/11 13:54:24 INFO mapreduce.Job: map 96% reduce 26%
16/04/11 13:54:25 INFO mapreduce.Job: map 96% reduce 29%
16/04/11 13:54:26 INFO mapreduce.Job: map 99% reduce 29%
16/04/11 13:54:27 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:54:28 INFO mapreduce.Job: Job job_1460382022143_0008 completed successfully
16/04/11 13:54:28 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=2458
FILE: Number of bytes written=3873420
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=7204
HDFS: Number of bytes written=240518168661
HDFS: Number of read operations=115
HDFS: Number of large read operations=0
HDFS: Number of write operations=30
Job Counters
Launched map tasks=28
Launched reduce tasks=1
Data-local map tasks=26
Rack-local map tasks=2
Total time spent by all maps in occupied slots (ms)=2803211
Total time spent by all reduces in occupied slots (ms)=118510
Total time spent by all map tasks (ms)=2803211
Total time spent by all reduce tasks (ms)=59255
Total vcore-seconds taken by all map tasks=2803211
Total vcore-seconds taken by all reduce tasks=59255
Total megabyte-seconds taken by all map tasks=10046708224
Total megabyte-seconds taken by all reduce tasks=424739840
Map-Reduce Framework
Map input records=28
Map output records=140
Map output bytes=2172
Map output materialized bytes=2620
Input split bytes=4050
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=2620
Reduce input records=140
Reduce output records=5
Spilled Records=280
Shuffled Maps =28
Failed Shuffles=0
Merged Map outputs=28
GC time elapsed (ms)=22032
CPU time spent (ms)=987180
Physical memory (bytes) snapshot=71146557440
Virtual memory (bytes) snapshot=110259920896
Total committed heap usage (bytes)=85085126656
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=3154
File Output Format Counters
Bytes Written=85
16/04/11 13:54:28 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:54:28 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:54:28 UTC 2016
16/04/11 13:54:28 INFO fs.TestDFSIO: Number of files: 28
16/04/11 13:54:28 INFO fs.TestDFSIO: Total MBytes processed: 229376.0
16/04/11 13:54:28 INFO fs.TestDFSIO: Throughput mb/sec: 85.63292112733686
16/04/11 13:54:28 INFO fs.TestDFSIO: Average IO rate mb/sec: 90.2082290649414
16/04/11 13:54:28 INFO fs.TestDFSIO: IO rate std deviation: 18.84555906485824
16/04/11 13:54:28 INFO fs.TestDFSIO: Test exec time sec: 149.488
16/04/11 13:54:28 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:54:29 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:54:29 INFO fs.TestDFSIO: nrFiles = 32
16/04/11 13:54:29 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:54:29 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:54:29 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:54:30 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 32 files
16/04/11 13:54:31 INFO fs.TestDFSIO: created control files for: 32 files
16/04/11 13:54:31 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:54:31 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:54:32 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:54:32 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:54:32 INFO mapred.FileInputFormat: Total input paths to process : 32
16/04/11 13:54:32 INFO mapreduce.JobSubmitter: number of splits:32
16/04/11 13:54:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0009
16/04/11 13:54:32 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0009
16/04/11 13:54:32 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0009/
16/04/11 13:54:32 INFO mapreduce.Job: Running job: job_1460382022143_0009
16/04/11 13:54:38 INFO mapreduce.Job: Job job_1460382022143_0009 running in uber mode : false
16/04/11 13:54:38 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:54:49 INFO mapreduce.Job: map 40% reduce 0%
16/04/11 13:54:50 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:56:08 INFO mapreduce.Job: map 68% reduce 0%
16/04/11 13:56:13 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 13:56:20 INFO mapreduce.Job: map 71% reduce 0%
16/04/11 13:56:22 INFO mapreduce.Job: map 72% reduce 0%
16/04/11 13:56:24 INFO mapreduce.Job: map 72% reduce 5%
16/04/11 13:56:31 INFO mapreduce.Job: map 74% reduce 5%
16/04/11 13:56:32 INFO mapreduce.Job: map 75% reduce 5%
16/04/11 13:56:33 INFO mapreduce.Job: map 77% reduce 8%
16/04/11 13:56:36 INFO mapreduce.Job: map 77% reduce 10%
16/04/11 13:56:37 INFO mapreduce.Job: map 79% reduce 10%
16/04/11 13:56:38 INFO mapreduce.Job: map 80% reduce 10%
16/04/11 13:56:39 INFO mapreduce.Job: map 81% reduce 14%
16/04/11 13:56:40 INFO mapreduce.Job: map 82% reduce 14%
16/04/11 13:56:41 INFO mapreduce.Job: map 86% reduce 14%
16/04/11 13:56:42 INFO mapreduce.Job: map 88% reduce 20%
16/04/11 13:56:43 INFO mapreduce.Job: map 90% reduce 20%
16/04/11 13:56:44 INFO mapreduce.Job: map 91% reduce 20%
16/04/11 13:56:45 INFO mapreduce.Job: map 91% reduce 24%
16/04/11 13:56:46 INFO mapreduce.Job: map 92% reduce 24%
16/04/11 13:56:47 INFO mapreduce.Job: map 93% reduce 24%
16/04/11 13:56:48 INFO mapreduce.Job: map 93% reduce 26%
16/04/11 13:56:49 INFO mapreduce.Job: map 94% reduce 26%
16/04/11 13:56:51 INFO mapreduce.Job: map 94% reduce 27%
16/04/11 13:57:20 INFO mapreduce.Job: map 95% reduce 27%
16/04/11 13:57:21 INFO mapreduce.Job: map 95% reduce 28%
16/04/11 13:57:22 INFO mapreduce.Job: map 97% reduce 28%
16/04/11 13:57:24 INFO mapreduce.Job: map 98% reduce 30%
16/04/11 13:57:26 INFO mapreduce.Job: map 100% reduce 30%
16/04/11 13:57:27 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 13:57:27 INFO mapreduce.Job: Job job_1460382022143_0009 completed successfully
16/04/11 13:57:27 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=2795
FILE: Number of bytes written=4407678
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=8236
HDFS: Number of bytes written=274877907029
HDFS: Number of read operations=131
HDFS: Number of large read operations=0
HDFS: Number of write operations=34
Job Counters
Launched map tasks=32
Launched reduce tasks=1
Data-local map tasks=31
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3940774
Total time spent by all reduces in occupied slots (ms)=141862
Total time spent by all map tasks (ms)=3940774
Total time spent by all reduce tasks (ms)=70931
Total vcore-seconds taken by all map tasks=3940774
Total vcore-seconds taken by all reduce tasks=70931
Total megabyte-seconds taken by all map tasks=14123734016
Total megabyte-seconds taken by all reduce tasks=508433408
Map-Reduce Framework
Map input records=32
Map output records=160
Map output bytes=2469
Map output materialized bytes=2981
Input split bytes=4630
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=2981
Reduce input records=160
Reduce output records=5
Spilled Records=320
Shuffled Maps =32
Failed Shuffles=0
Merged Map outputs=32
GC time elapsed (ms)=24358
CPU time spent (ms)=1112860
Physical memory (bytes) snapshot=81686364160
Virtual memory (bytes) snapshot=125030821888
Total committed heap usage (bytes)=97207713792
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=3606
File Output Format Counters
Bytes Written=85
16/04/11 13:57:27 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 13:57:27 INFO fs.TestDFSIO: Date & time: Mon Apr 11 13:57:27 UTC 2016
16/04/11 13:57:27 INFO fs.TestDFSIO: Number of files: 32
16/04/11 13:57:27 INFO fs.TestDFSIO: Total MBytes processed: 262144.0
16/04/11 13:57:27 INFO fs.TestDFSIO: Throughput mb/sec: 68.8988494956063
16/04/11 13:57:27 INFO fs.TestDFSIO: Average IO rate mb/sec: 70.98353576660156
16/04/11 13:57:27 INFO fs.TestDFSIO: IO rate std deviation: 11.858304615451853
16/04/11 13:57:27 INFO fs.TestDFSIO: Test exec time sec: 176.645
16/04/11 13:57:27 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 13:57:29 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 13:57:29 INFO fs.TestDFSIO: nrFiles = 36
16/04/11 13:57:29 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 13:57:29 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 13:57:29 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 13:57:30 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 36 files
16/04/11 13:57:30 INFO fs.TestDFSIO: created control files for: 36 files
16/04/11 13:57:31 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:57:31 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:57:31 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 13:57:31 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 13:57:31 INFO mapred.FileInputFormat: Total input paths to process : 36
16/04/11 13:57:32 INFO mapreduce.JobSubmitter: number of splits:36
16/04/11 13:57:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460382022143_0010
16/04/11 13:57:32 INFO impl.YarnClientImpl: Submitted application application_1460382022143_0010
16/04/11 13:57:32 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460382022143_0010/
16/04/11 13:57:32 INFO mapreduce.Job: Running job: job_1460382022143_0010
16/04/11 13:57:39 INFO mapreduce.Job: Job job_1460382022143_0010 running in uber mode : false
16/04/11 13:57:39 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 13:57:50 INFO mapreduce.Job: map 15% reduce 0%
16/04/11 13:57:51 INFO mapreduce.Job: map 65% reduce 0%
16/04/11 13:59:11 INFO mapreduce.Job: map 66% reduce 0%
16/04/11 13:59:19 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 13:59:21 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 13:59:26 INFO mapreduce.Job: map 70% reduce 0%
16/04/11 13:59:28 INFO mapreduce.Job: map 72% reduce 0%
16/04/11 13:59:30 INFO mapreduce.Job: map 73% reduce 6%
16/04/11 13:59:38 INFO mapreduce.Job: map 74% reduce 6%
16/04/11 13:59:39 INFO mapreduce.Job: map 74% reduce 7%
16/04/11 13:59:40 INFO mapreduce.Job: map 77% reduce 7%
16/04/11 13:59:41 INFO mapreduce.Job: map 79% reduce 7%
16/04/11 13:59:42 INFO mapreduce.Job: map 79% reduce 12%
16/04/11 13:59:45 INFO mapreduce.Job: map 80% reduce 12%
16/04/11 13:59:47 INFO mapreduce.Job: map 81% reduce 12%
16/04/11 13:59:48 INFO mapreduce.Job: map 81% reduce 14%
16/04/11 13:59:49 INFO mapreduce.Job: map 82% reduce 14%
16/04/11 13:59:50 INFO mapreduce.Job: map 84% reduce 14%
16/04/11 13:59:51 INFO mapreduce.Job: map 84% reduce 18%
16/04/11 13:59:52 INFO mapreduce.Job: map 85% reduce 18%
16/04/11 13:59:53 INFO mapreduce.Job: map 88% reduce 18%
16/04/11 13:59:54 INFO mapreduce.Job: map 88% reduce 20%
16/04/11 13:59:55 INFO mapreduce.Job: map 89% reduce 20%
16/04/11 13:59:56 INFO mapreduce.Job: map 90% reduce 20%
16/04/11 13:59:57 INFO mapreduce.Job: map 90% reduce 23%
16/04/11 13:59:59 INFO mapreduce.Job: map 91% reduce 23%
16/04/11 14:00:00 INFO mapreduce.Job: map 91% reduce 24%
16/04/11 14:00:23 INFO mapreduce.Job: map 92% reduce 24%
16/04/11 14:00:24 INFO mapreduce.Job: map 92% reduce 25%
16/04/11 14:00:39 INFO mapreduce.Job: map 93% reduce 25%
16/04/11 14:00:40 INFO mapreduce.Job: map 94% reduce 25%
16/04/11 14:00:42 INFO mapreduce.Job: map 94% reduce 27%
16/04/11 14:00:50 INFO mapreduce.Job: map 95% reduce 27%
16/04/11 14:00:51 INFO mapreduce.Job: map 96% reduce 29%
16/04/11 14:00:52 INFO mapreduce.Job: map 98% reduce 29%
16/04/11 14:00:53 INFO mapreduce.Job: map 100% reduce 29%
16/04/11 14:00:54 INFO mapreduce.Job: map 100% reduce 32%
16/04/11 14:00:55 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 14:00:55 INFO mapreduce.Job: Job job_1460382022143_0010 completed successfully
16/04/11 14:00:55 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=3149
FILE: Number of bytes written=4942007
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=9268
HDFS: Number of bytes written=309237645395
HDFS: Number of read operations=147
HDFS: Number of large read operations=0
HDFS: Number of write operations=38
Job Counters
Launched map tasks=36
Launched reduce tasks=1
Data-local map tasks=33
Rack-local map tasks=3
Total time spent by all maps in occupied slots (ms)=4871548
Total time spent by all reduces in occupied slots (ms)=186260
Total time spent by all map tasks (ms)=4871548
Total time spent by all reduce tasks (ms)=93130
Total vcore-seconds taken by all map tasks=4871548
Total vcore-seconds taken by all reduce tasks=93130
Total megabyte-seconds taken by all map tasks=17459628032
Total megabyte-seconds taken by all reduce tasks=667555840
Map-Reduce Framework
Map input records=36
Map output records=180
Map output bytes=2783
Map output materialized bytes=3359
Input split bytes=5210
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=3359
Reduce input records=180
Reduce output records=5
Spilled Records=360
Shuffled Maps =36
Failed Shuffles=0
Merged Map outputs=36
GC time elapsed (ms)=28430
CPU time spent (ms)=1275020
Physical memory (bytes) snapshot=91313713152
Virtual memory (bytes) snapshot=139742748672
Total committed heap usage (bytes)=108788187136
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=4058
File Output Format Counters
Bytes Written=83
16/04/11 14:00:55 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 14:00:55 INFO fs.TestDFSIO: Date & time: Mon Apr 11 14:00:55 UTC 2016
16/04/11 14:00:55 INFO fs.TestDFSIO: Number of files: 36
16/04/11 14:00:55 INFO fs.TestDFSIO: Total MBytes processed: 294912.0
16/04/11 14:00:55 INFO fs.TestDFSIO: Throughput mb/sec: 62.75349604034956
16/04/11 14:00:55 INFO fs.TestDFSIO: Average IO rate mb/sec: 66.68753051757812
16/04/11 14:00:55 INFO fs.TestDFSIO: IO rate std deviation: 16.65385571529417
16/04/11 14:00:55 INFO fs.TestDFSIO: Test exec time sec: 204.801
16/04/11 14:00:55 INFO fs.TestDFSIO:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment