Skip to content

Instantly share code, notes, and snippets.

@akanto

akanto/perf-test-repl1

Last active April 11, 2016 12:44
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save akanto/d4e5585a5beffe12b109c83ed76c7a1d to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save akanto/d4e5585a5beffe12b109c83ed76c7a1d to your computer and use it in GitHub Desktop.
Download ZIP
perf-test-repl1
Raw
perf-test-repl1
[hdfs@docker-ambari ~]# for i in 1 4 8 12 16 20 24 28 32 36; do hadoop jar $JAR_JOBCLIENT TestDFSIO -Ddfs.replication=1 -write -nrFiles $i -fileSize 8192; done
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:31:39 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:31:39 INFO fs.TestDFSIO: nrFiles = 1
16/04/11 12:31:39 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:31:39 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:31:39 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:31:40 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 1 files
16/04/11 12:31:40 INFO fs.TestDFSIO: created control files for: 1 files
16/04/11 12:31:41 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:31:41 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:31:41 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:31:41 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:31:41 INFO mapred.FileInputFormat: Total input paths to process : 1
16/04/11 12:31:41 INFO mapreduce.JobSubmitter: number of splits:1
16/04/11 12:31:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0046
16/04/11 12:31:42 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0046
16/04/11 12:31:42 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0046/
16/04/11 12:31:42 INFO mapreduce.Job: Running job: job_1460364631926_0046
16/04/11 12:31:48 INFO mapreduce.Job: Job job_1460364631926_0046 running in uber mode : false
16/04/11 12:31:48 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:31:59 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:32:05 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 12:32:11 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:32:11 INFO mapreduce.Job: Job job_1460364631926_0046 completed successfully
16/04/11 12:32:11 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=96
FILE: Number of bytes written=266912
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=256
HDFS: Number of bytes written=8589934672
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=15409
Total time spent by all reduces in occupied slots (ms)=5196
Total time spent by all map tasks (ms)=15409
Total time spent by all reduce tasks (ms)=2598
Total vcore-seconds taken by all map tasks=15409
Total vcore-seconds taken by all reduce tasks=2598
Total megabyte-seconds taken by all map tasks=55225856
Total megabyte-seconds taken by all reduce tasks=18622464
Map-Reduce Framework
Map input records=1
Map output records=5
Map output bytes=80
Map output materialized bytes=96
Input split bytes=144
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=96
Reduce input records=5
Reduce output records=5
Spilled Records=10
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=872
CPU time spent (ms)=28200
Physical memory (bytes) snapshot=2853785600
Virtual memory (bytes) snapshot=10530910208
Total committed heap usage (bytes)=5114953728
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=112
File Output Format Counters
Bytes Written=80
16/04/11 12:32:11 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:32:11 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:32:11 UTC 2016
16/04/11 12:32:11 INFO fs.TestDFSIO: Number of files: 1
16/04/11 12:32:11 INFO fs.TestDFSIO: Total MBytes processed: 8192.0
16/04/11 12:32:11 INFO fs.TestDFSIO: Throughput mb/sec: 693.7669376693767
16/04/11 12:32:11 INFO fs.TestDFSIO: Average IO rate mb/sec: 693.7669677734375
16/04/11 12:32:11 INFO fs.TestDFSIO: IO rate std deviation: 0.10873614793009881
16/04/11 12:32:11 INFO fs.TestDFSIO: Test exec time sec: 30.917
16/04/11 12:32:11 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:32:12 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:32:12 INFO fs.TestDFSIO: nrFiles = 4
16/04/11 12:32:12 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:32:12 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:32:12 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:32:13 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 4 files
16/04/11 12:32:14 INFO fs.TestDFSIO: created control files for: 4 files
16/04/11 12:32:14 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:32:14 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:32:15 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:32:15 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:32:15 INFO mapred.FileInputFormat: Total input paths to process : 4
16/04/11 12:32:15 INFO mapreduce.JobSubmitter: number of splits:4
16/04/11 12:32:15 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0047
16/04/11 12:32:15 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0047
16/04/11 12:32:15 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0047/
16/04/11 12:32:15 INFO mapreduce.Job: Running job: job_1460364631926_0047
16/04/11 12:32:21 INFO mapreduce.Job: Job job_1460364631926_0047 running in uber mode : false
16/04/11 12:32:21 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:32:32 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:32:38 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 12:32:40 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 12:32:44 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:32:44 INFO mapreduce.Job: Job job_1460364631926_0047 completed successfully
16/04/11 12:32:44 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=361
FILE: Number of bytes written=667620
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1024
HDFS: Number of bytes written=34359738449
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Job Counters
Launched map tasks=4
Launched reduce tasks=1
Data-local map tasks=4
Total time spent by all maps in occupied slots (ms)=61539
Total time spent by all reduces in occupied slots (ms)=5194
Total time spent by all map tasks (ms)=61539
Total time spent by all reduce tasks (ms)=2597
Total vcore-seconds taken by all map tasks=61539
Total vcore-seconds taken by all reduce tasks=2597
Total megabyte-seconds taken by all map tasks=220555776
Total megabyte-seconds taken by all reduce tasks=18615296
Map-Reduce Framework
Map input records=4
Map output records=20
Map output bytes=315
Map output materialized bytes=379
Input split bytes=576
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=379
Reduce input records=20
Reduce output records=5
Spilled Records=40
Shuffled Maps =4
Failed Shuffles=0
Merged Map outputs=4
GC time elapsed (ms)=3073
CPU time spent (ms)=110920
Physical memory (bytes) snapshot=10522144768
Virtual memory (bytes) snapshot=21620064256
Total committed heap usage (bytes)=14103871488
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=448
File Output Format Counters
Bytes Written=81
16/04/11 12:32:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:32:44 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:32:44 UTC 2016
16/04/11 12:32:44 INFO fs.TestDFSIO: Number of files: 4
16/04/11 12:32:44 INFO fs.TestDFSIO: Total MBytes processed: 32768.0
16/04/11 12:32:44 INFO fs.TestDFSIO: Throughput mb/sec: 690.1141485194389
16/04/11 12:32:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 690.686767578125
16/04/11 12:32:44 INFO fs.TestDFSIO: IO rate std deviation: 19.746242490183704
16/04/11 12:32:44 INFO fs.TestDFSIO: Test exec time sec: 29.894
16/04/11 12:32:44 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:32:45 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:32:45 INFO fs.TestDFSIO: nrFiles = 8
16/04/11 12:32:45 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:32:45 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:32:45 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:32:46 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 8 files
16/04/11 12:32:46 INFO fs.TestDFSIO: created control files for: 8 files
16/04/11 12:32:47 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:32:47 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:32:47 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:32:47 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:32:47 INFO mapred.FileInputFormat: Total input paths to process : 8
16/04/11 12:32:48 INFO mapreduce.JobSubmitter: number of splits:8
16/04/11 12:32:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0048
16/04/11 12:32:48 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0048
16/04/11 12:32:48 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0048/
16/04/11 12:32:48 INFO mapreduce.Job: Running job: job_1460364631926_0048
16/04/11 12:32:54 INFO mapreduce.Job: Job job_1460364631926_0048 running in uber mode : false
16/04/11 12:32:54 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:33:04 INFO mapreduce.Job: map 17% reduce 0%
16/04/11 12:33:05 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:33:12 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 12:33:14 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 12:33:17 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:33:17 INFO mapreduce.Job: Job job_1460364631926_0048 completed successfully
16/04/11 12:33:17 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=718
FILE: Number of bytes written=1201908
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2048
HDFS: Number of bytes written=68719476818
HDFS: Number of read operations=35
HDFS: Number of large read operations=0
HDFS: Number of write operations=10
Job Counters
Launched map tasks=8
Launched reduce tasks=1
Data-local map tasks=7
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=133943
Total time spent by all reduces in occupied slots (ms)=5578
Total time spent by all map tasks (ms)=133943
Total time spent by all reduce tasks (ms)=2789
Total vcore-seconds taken by all map tasks=133943
Total vcore-seconds taken by all reduce tasks=2789
Total megabyte-seconds taken by all map tasks=480051712
Total megabyte-seconds taken by all reduce tasks=19991552
Map-Reduce Framework
Map input records=8
Map output records=40
Map output bytes=632
Map output materialized bytes=760
Input split bytes=1152
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=760
Reduce input records=40
Reduce output records=5
Spilled Records=80
Shuffled Maps =8
Failed Shuffles=0
Merged Map outputs=8
GC time elapsed (ms)=5686
CPU time spent (ms)=224630
Physical memory (bytes) snapshot=20744839168
Virtual memory (bytes) snapshot=36417286144
Total committed heap usage (bytes)=26088046592
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=896
File Output Format Counters
Bytes Written=82
16/04/11 12:33:17 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:33:17 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:33:17 UTC 2016
16/04/11 12:33:17 INFO fs.TestDFSIO: Number of files: 8
16/04/11 12:33:17 INFO fs.TestDFSIO: Total MBytes processed: 65536.0
16/04/11 12:33:17 INFO fs.TestDFSIO: Throughput mb/sec: 622.0197418375095
16/04/11 12:33:17 INFO fs.TestDFSIO: Average IO rate mb/sec: 625.195068359375
16/04/11 12:33:17 INFO fs.TestDFSIO: IO rate std deviation: 44.6601919959646
16/04/11 12:33:17 INFO fs.TestDFSIO: Test exec time sec: 30.915
16/04/11 12:33:17 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:33:19 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:33:19 INFO fs.TestDFSIO: nrFiles = 12
16/04/11 12:33:19 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:33:19 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:33:19 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:33:20 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 12 files
16/04/11 12:33:20 INFO fs.TestDFSIO: created control files for: 12 files
16/04/11 12:33:21 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:33:21 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:33:21 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:33:21 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:33:21 INFO mapred.FileInputFormat: Total input paths to process : 12
16/04/11 12:33:21 INFO mapreduce.JobSubmitter: number of splits:12
16/04/11 12:33:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0049
16/04/11 12:33:22 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0049
16/04/11 12:33:22 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0049/
16/04/11 12:33:22 INFO mapreduce.Job: Running job: job_1460364631926_0049
16/04/11 12:33:29 INFO mapreduce.Job: Job job_1460364631926_0049 running in uber mode : false
16/04/11 12:33:29 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:33:39 INFO mapreduce.Job: map 17% reduce 0%
16/04/11 12:33:40 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:33:46 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 12:33:47 INFO mapreduce.Job: map 75% reduce 0%
16/04/11 12:33:48 INFO mapreduce.Job: map 78% reduce 0%
16/04/11 12:33:54 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 12:33:55 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:33:55 INFO mapreduce.Job: Job job_1460364631926_0049 completed successfully
16/04/11 12:33:55 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1079
FILE: Number of bytes written=1736235
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=3076
HDFS: Number of bytes written=103079215188
HDFS: Number of read operations=51
HDFS: Number of large read operations=0
HDFS: Number of write operations=14
Job Counters
Launched map tasks=12
Launched reduce tasks=1
Data-local map tasks=12
Total time spent by all maps in occupied slots (ms)=249625
Total time spent by all reduces in occupied slots (ms)=15092
Total time spent by all map tasks (ms)=249625
Total time spent by all reduce tasks (ms)=7546
Total vcore-seconds taken by all map tasks=249625
Total vcore-seconds taken by all reduce tasks=7546
Total megabyte-seconds taken by all map tasks=894656000
Total megabyte-seconds taken by all reduce tasks=54089728
Map-Reduce Framework
Map input records=12
Map output records=60
Map output bytes=953
Map output materialized bytes=1145
Input split bytes=1730
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1145
Reduce input records=60
Reduce output records=5
Spilled Records=120
Shuffled Maps =12
Failed Shuffles=0
Merged Map outputs=12
GC time elapsed (ms)=10328
CPU time spent (ms)=362580
Physical memory (bytes) snapshot=30945734656
Virtual memory (bytes) snapshot=51152625664
Total committed heap usage (bytes)=38073794560
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1346
File Output Format Counters
Bytes Written=84
16/04/11 12:33:55 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:33:55 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:33:55 UTC 2016
16/04/11 12:33:55 INFO fs.TestDFSIO: Number of files: 12
16/04/11 12:33:55 INFO fs.TestDFSIO: Total MBytes processed: 98304.0
16/04/11 12:33:55 INFO fs.TestDFSIO: Throughput mb/sec: 496.7206653663861
16/04/11 12:33:55 INFO fs.TestDFSIO: Average IO rate mb/sec: 514.5631713867188
16/04/11 12:33:55 INFO fs.TestDFSIO: IO rate std deviation: 103.41040277671442
16/04/11 12:33:55 INFO fs.TestDFSIO: Test exec time sec: 34.925
16/04/11 12:33:55 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:33:56 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:33:56 INFO fs.TestDFSIO: nrFiles = 16
16/04/11 12:33:56 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:33:56 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:33:56 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:33:57 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 16 files
16/04/11 12:33:58 INFO fs.TestDFSIO: created control files for: 16 files
16/04/11 12:33:58 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:33:58 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:33:59 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:33:59 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:33:59 INFO mapred.FileInputFormat: Total input paths to process : 16
16/04/11 12:33:59 INFO mapreduce.JobSubmitter: number of splits:16
16/04/11 12:33:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0050
16/04/11 12:33:59 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0050
16/04/11 12:33:59 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0050/
16/04/11 12:33:59 INFO mapreduce.Job: Running job: job_1460364631926_0050
16/04/11 12:34:05 INFO mapreduce.Job: Job job_1460364631926_0050 running in uber mode : false
16/04/11 12:34:05 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:34:16 INFO mapreduce.Job: map 21% reduce 0%
16/04/11 12:34:17 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:34:24 INFO mapreduce.Job: map 69% reduce 0%
16/04/11 12:34:25 INFO mapreduce.Job: map 79% reduce 0%
16/04/11 12:34:27 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 12:34:29 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:34:30 INFO mapreduce.Job: Job job_1460364631926_0050 completed successfully
16/04/11 12:34:30 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1432
FILE: Number of bytes written=2270520
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=4108
HDFS: Number of bytes written=137438953555
HDFS: Number of read operations=67
HDFS: Number of large read operations=0
HDFS: Number of write operations=18
Job Counters
Launched map tasks=16
Launched reduce tasks=1
Data-local map tasks=16
Total time spent by all maps in occupied slots (ms)=291957
Total time spent by all reduces in occupied slots (ms)=5260
Total time spent by all map tasks (ms)=291957
Total time spent by all reduce tasks (ms)=2630
Total vcore-seconds taken by all map tasks=291957
Total vcore-seconds taken by all reduce tasks=2630
Total megabyte-seconds taken by all map tasks=1046373888
Total megabyte-seconds taken by all reduce tasks=18851840
Map-Reduce Framework
Map input records=16
Map output records=80
Map output bytes=1266
Map output materialized bytes=1522
Input split bytes=2310
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1522
Reduce input records=80
Reduce output records=5
Spilled Records=160
Shuffled Maps =16
Failed Shuffles=0
Merged Map outputs=16
GC time elapsed (ms)=12143
CPU time spent (ms)=459580
Physical memory (bytes) snapshot=41157726208
Virtual memory (bytes) snapshot=65903616000
Total committed heap usage (bytes)=50061115392
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1798
File Output Format Counters
Bytes Written=83
16/04/11 12:34:30 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:34:30 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:34:30 UTC 2016
16/04/11 12:34:30 INFO fs.TestDFSIO: Number of files: 16
16/04/11 12:34:30 INFO fs.TestDFSIO: Total MBytes processed: 131072.0
16/04/11 12:34:30 INFO fs.TestDFSIO: Throughput mb/sec: 567.2688242778869
16/04/11 12:34:30 INFO fs.TestDFSIO: Average IO rate mb/sec: 571.732666015625
16/04/11 12:34:30 INFO fs.TestDFSIO: IO rate std deviation: 52.50930737179646
16/04/11 12:34:30 INFO fs.TestDFSIO: Test exec time sec: 31.965
16/04/11 12:34:30 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:34:31 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:34:31 INFO fs.TestDFSIO: nrFiles = 20
16/04/11 12:34:31 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:34:31 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:34:31 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:34:32 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 20 files
16/04/11 12:34:33 INFO fs.TestDFSIO: created control files for: 20 files
16/04/11 12:34:33 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:34:33 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:34:34 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:34:34 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:34:34 INFO mapred.FileInputFormat: Total input paths to process : 20
16/04/11 12:34:34 INFO mapreduce.JobSubmitter: number of splits:20
16/04/11 12:34:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0051
16/04/11 12:34:34 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0051
16/04/11 12:34:34 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0051/
16/04/11 12:34:34 INFO mapreduce.Job: Running job: job_1460364631926_0051
16/04/11 12:34:40 INFO mapreduce.Job: Job job_1460364631926_0051 running in uber mode : false
16/04/11 12:34:40 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:34:51 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:34:59 INFO mapreduce.Job: map 72% reduce 0%
16/04/11 12:35:00 INFO mapreduce.Job: map 80% reduce 0%
16/04/11 12:35:01 INFO mapreduce.Job: map 87% reduce 0%
16/04/11 12:35:05 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 12:35:07 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:35:07 INFO mapreduce.Job: Job job_1460364631926_0051 completed successfully
16/04/11 12:35:07 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=1786
FILE: Number of bytes written=2804817
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5140
HDFS: Number of bytes written=171798691925
HDFS: Number of read operations=83
HDFS: Number of large read operations=0
HDFS: Number of write operations=22
Job Counters
Launched map tasks=20
Launched reduce tasks=1
Data-local map tasks=20
Total time spent by all maps in occupied slots (ms)=396447
Total time spent by all reduces in occupied slots (ms)=11042
Total time spent by all map tasks (ms)=396447
Total time spent by all reduce tasks (ms)=5521
Total vcore-seconds taken by all map tasks=396447
Total vcore-seconds taken by all reduce tasks=5521
Total megabyte-seconds taken by all map tasks=1420866048
Total megabyte-seconds taken by all reduce tasks=39574528
Map-Reduce Framework
Map input records=20
Map output records=100
Map output bytes=1580
Map output materialized bytes=1900
Input split bytes=2890
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=1900
Reduce input records=100
Reduce output records=5
Spilled Records=200
Shuffled Maps =20
Failed Shuffles=0
Merged Map outputs=20
GC time elapsed (ms)=15862
CPU time spent (ms)=597030
Physical memory (bytes) snapshot=51424157696
Virtual memory (bytes) snapshot=80640131072
Total committed heap usage (bytes)=62045290496
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2250
File Output Format Counters
Bytes Written=85
16/04/11 12:35:08 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:35:08 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:35:08 UTC 2016
16/04/11 12:35:08 INFO fs.TestDFSIO: Number of files: 20
16/04/11 12:35:08 INFO fs.TestDFSIO: Total MBytes processed: 163840.0
16/04/11 12:35:08 INFO fs.TestDFSIO: Throughput mb/sec: 509.07599475512524
16/04/11 12:35:08 INFO fs.TestDFSIO: Average IO rate mb/sec: 522.7897338867188
16/04/11 12:35:08 INFO fs.TestDFSIO: IO rate std deviation: 83.2373512472243
16/04/11 12:35:08 INFO fs.TestDFSIO: Test exec time sec: 34.932
16/04/11 12:35:08 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:35:09 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:35:09 INFO fs.TestDFSIO: nrFiles = 24
16/04/11 12:35:09 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:35:09 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:35:09 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:35:10 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 24 files
16/04/11 12:35:10 INFO fs.TestDFSIO: created control files for: 24 files
16/04/11 12:35:11 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:35:11 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:35:11 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:35:11 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:35:12 INFO mapred.FileInputFormat: Total input paths to process : 24
16/04/11 12:35:12 INFO mapreduce.JobSubmitter: number of splits:24
16/04/11 12:35:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0052
16/04/11 12:35:12 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0052
16/04/11 12:35:12 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0052/
16/04/11 12:35:12 INFO mapreduce.Job: Running job: job_1460364631926_0052
16/04/11 12:35:19 INFO mapreduce.Job: Job job_1460364631926_0052 running in uber mode : false
16/04/11 12:35:19 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:35:29 INFO mapreduce.Job: map 14% reduce 0%
16/04/11 12:35:30 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:35:39 INFO mapreduce.Job: map 71% reduce 0%
16/04/11 12:35:40 INFO mapreduce.Job: map 81% reduce 0%
16/04/11 12:35:41 INFO mapreduce.Job: map 89% reduce 0%
16/04/11 12:35:43 INFO mapreduce.Job: map 97% reduce 0%
16/04/11 12:35:44 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:35:44 INFO mapreduce.Job: Job job_1460364631926_0052 completed successfully
16/04/11 12:35:44 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2147
FILE: Number of bytes written=3339131
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=6172
HDFS: Number of bytes written=206158430290
HDFS: Number of read operations=99
HDFS: Number of large read operations=0
HDFS: Number of write operations=26
Job Counters
Launched map tasks=24
Launched reduce tasks=1
Data-local map tasks=24
Total time spent by all maps in occupied slots (ms)=490366
Total time spent by all reduces in occupied slots (ms)=5262
Total time spent by all map tasks (ms)=490366
Total time spent by all reduce tasks (ms)=2631
Total vcore-seconds taken by all map tasks=490366
Total vcore-seconds taken by all reduce tasks=2631
Total megabyte-seconds taken by all map tasks=1757471744
Total megabyte-seconds taken by all reduce tasks=18859008
Map-Reduce Framework
Map input records=24
Map output records=120
Map output bytes=1901
Map output materialized bytes=2285
Input split bytes=3470
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=2285
Reduce input records=120
Reduce output records=5
Spilled Records=240
Shuffled Maps =24
Failed Shuffles=0
Merged Map outputs=24
GC time elapsed (ms)=18939
CPU time spent (ms)=729670
Physical memory (bytes) snapshot=61627486208
Virtual memory (bytes) snapshot=95445803008
Total committed heap usage (bytes)=74033659904
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2702
File Output Format Counters
Bytes Written=82
16/04/11 12:35:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:35:44 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:35:44 UTC 2016
16/04/11 12:35:44 INFO fs.TestDFSIO: Number of files: 24
16/04/11 12:35:44 INFO fs.TestDFSIO: Total MBytes processed: 196608.0
16/04/11 12:35:44 INFO fs.TestDFSIO: Throughput mb/sec: 499.82331481740414
16/04/11 12:35:44 INFO fs.TestDFSIO: Average IO rate mb/sec: 503.4390869140625
16/04/11 12:35:44 INFO fs.TestDFSIO: IO rate std deviation: 41.97124929204543
16/04/11 12:35:44 INFO fs.TestDFSIO: Test exec time sec: 33.972
16/04/11 12:35:44 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:35:46 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:35:46 INFO fs.TestDFSIO: nrFiles = 28
16/04/11 12:35:46 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:35:46 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:35:46 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:35:47 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 28 files
16/04/11 12:35:47 INFO fs.TestDFSIO: created control files for: 28 files
16/04/11 12:35:48 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:35:48 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:35:48 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:35:48 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:35:48 INFO mapred.FileInputFormat: Total input paths to process : 28
16/04/11 12:35:48 INFO mapreduce.JobSubmitter: number of splits:28
16/04/11 12:35:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0053
16/04/11 12:35:49 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0053
16/04/11 12:35:49 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0053/
16/04/11 12:35:49 INFO mapreduce.Job: Running job: job_1460364631926_0053
16/04/11 12:35:55 INFO mapreduce.Job: Job job_1460364631926_0053 running in uber mode : false
16/04/11 12:35:55 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:36:06 INFO mapreduce.Job: map 29% reduce 0%
16/04/11 12:36:07 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:36:17 INFO mapreduce.Job: map 73% reduce 0%
16/04/11 12:36:18 INFO mapreduce.Job: map 82% reduce 0%
16/04/11 12:36:19 INFO mapreduce.Job: map 89% reduce 0%
16/04/11 12:36:23 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 12:36:24 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:36:24 INFO mapreduce.Job: Job job_1460364631926_0053 completed successfully
16/04/11 12:36:24 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2509
FILE: Number of bytes written=3873428
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=7204
HDFS: Number of bytes written=240518168661
HDFS: Number of read operations=115
HDFS: Number of large read operations=0
HDFS: Number of write operations=30
Job Counters
Launched map tasks=28
Launched reduce tasks=1
Data-local map tasks=28
Total time spent by all maps in occupied slots (ms)=620311
Total time spent by all reduces in occupied slots (ms)=9370
Total time spent by all map tasks (ms)=620311
Total time spent by all reduce tasks (ms)=4685
Total vcore-seconds taken by all map tasks=620311
Total vcore-seconds taken by all reduce tasks=4685
Total megabyte-seconds taken by all map tasks=2223194624
Total megabyte-seconds taken by all reduce tasks=33582080
Map-Reduce Framework
Map input records=28
Map output records=140
Map output bytes=2223
Map output materialized bytes=2671
Input split bytes=4050
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=2671
Reduce input records=140
Reduce output records=5
Spilled Records=280
Shuffled Maps =28
Failed Shuffles=0
Merged Map outputs=28
GC time elapsed (ms)=23647
CPU time spent (ms)=861430
Physical memory (bytes) snapshot=71858221056
Virtual memory (bytes) snapshot=110168276992
Total committed heap usage (bytes)=86019407872
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=3154
File Output Format Counters
Bytes Written=85
16/04/11 12:36:24 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:36:24 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:36:24 UTC 2016
16/04/11 12:36:24 INFO fs.TestDFSIO: Number of files: 28
16/04/11 12:36:24 INFO fs.TestDFSIO: Total MBytes processed: 229376.0
16/04/11 12:36:24 INFO fs.TestDFSIO: Throughput mb/sec: 455.5377258067079
16/04/11 12:36:24 INFO fs.TestDFSIO: Average IO rate mb/sec: 462.1595764160156
16/04/11 12:36:24 INFO fs.TestDFSIO: IO rate std deviation: 54.29486326503654
16/04/11 12:36:24 INFO fs.TestDFSIO: Test exec time sec: 36.999
16/04/11 12:36:24 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:36:26 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:36:26 INFO fs.TestDFSIO: nrFiles = 32
16/04/11 12:36:26 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:36:26 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:36:26 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:36:27 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 32 files
16/04/11 12:36:27 INFO fs.TestDFSIO: created control files for: 32 files
16/04/11 12:36:28 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:36:28 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:36:28 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:36:28 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:36:28 INFO mapred.FileInputFormat: Total input paths to process : 32
16/04/11 12:36:28 INFO mapreduce.JobSubmitter: number of splits:32
16/04/11 12:36:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0054
16/04/11 12:36:29 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0054
16/04/11 12:36:29 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0054/
16/04/11 12:36:29 INFO mapreduce.Job: Running job: job_1460364631926_0054
16/04/11 12:36:35 INFO mapreduce.Job: Job job_1460364631926_0054 running in uber mode : false
16/04/11 12:36:35 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:36:46 INFO mapreduce.Job: map 33% reduce 0%
16/04/11 12:36:47 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:36:58 INFO mapreduce.Job: map 72% reduce 0%
16/04/11 12:36:59 INFO mapreduce.Job: map 81% reduce 0%
16/04/11 12:37:02 INFO mapreduce.Job: map 83% reduce 0%
16/04/11 12:37:03 INFO mapreduce.Job: map 95% reduce 0%
16/04/11 12:37:04 INFO mapreduce.Job: map 100% reduce 0%
16/04/11 12:37:05 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:37:05 INFO mapreduce.Job: Job job_1460364631926_0054 completed successfully
16/04/11 12:37:05 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2868
FILE: Number of bytes written=4407735
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=8236
HDFS: Number of bytes written=274877907029
HDFS: Number of read operations=131
HDFS: Number of large read operations=0
HDFS: Number of write operations=34
Job Counters
Launched map tasks=32
Launched reduce tasks=1
Data-local map tasks=32
Total time spent by all maps in occupied slots (ms)=770556
Total time spent by all reduces in occupied slots (ms)=7222
Total time spent by all map tasks (ms)=770556
Total time spent by all reduce tasks (ms)=3611
Total vcore-seconds taken by all map tasks=770556
Total vcore-seconds taken by all reduce tasks=3611
Total megabyte-seconds taken by all map tasks=2761672704
Total megabyte-seconds taken by all reduce tasks=25883648
Map-Reduce Framework
Map input records=32
Map output records=160
Map output bytes=2542
Map output materialized bytes=3054
Input split bytes=4630
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=3054
Reduce input records=160
Reduce output records=5
Spilled Records=320
Shuffled Maps =32
Failed Shuffles=0
Merged Map outputs=32
GC time elapsed (ms)=30299
CPU time spent (ms)=1011250
Physical memory (bytes) snapshot=82108014592
Virtual memory (bytes) snapshot=124935184384
Total committed heap usage (bytes)=98008301568
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=3606
File Output Format Counters
Bytes Written=85
16/04/11 12:37:05 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:37:05 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:37:05 UTC 2016
16/04/11 12:37:05 INFO fs.TestDFSIO: Number of files: 32
16/04/11 12:37:05 INFO fs.TestDFSIO: Total MBytes processed: 262144.0
16/04/11 12:37:05 INFO fs.TestDFSIO: Throughput mb/sec: 416.28528144165205
16/04/11 12:37:05 INFO fs.TestDFSIO: Average IO rate mb/sec: 419.5401611328125
16/04/11 12:37:05 INFO fs.TestDFSIO: IO rate std deviation: 37.35773108278558
16/04/11 12:37:05 INFO fs.TestDFSIO: Test exec time sec: 37.999
16/04/11 12:37:05 INFO fs.TestDFSIO:
WARNING: Use "yarn jar" to launch YARN applications.
16/04/11 12:37:06 INFO fs.TestDFSIO: TestDFSIO.1.8
16/04/11 12:37:06 INFO fs.TestDFSIO: nrFiles = 36
16/04/11 12:37:06 INFO fs.TestDFSIO: nrBytes (MB) = 8192.0
16/04/11 12:37:06 INFO fs.TestDFSIO: bufferSize = 1000000
16/04/11 12:37:06 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
16/04/11 12:37:07 INFO fs.TestDFSIO: creating control file: 8589934592 bytes, 36 files
16/04/11 12:37:08 INFO fs.TestDFSIO: created control files for: 36 files
16/04/11 12:37:09 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:37:09 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:37:09 INFO impl.TimelineClientImpl: Timeline service address: http://ip-10-0-71-101.node.dc1.consul:8188/ws/v1/timeline/
16/04/11 12:37:09 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-71-101.node.dc1.consul/10.0.71.101:8050
16/04/11 12:37:10 INFO mapred.FileInputFormat: Total input paths to process : 36
16/04/11 12:37:10 INFO mapreduce.JobSubmitter: number of splits:36
16/04/11 12:37:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460364631926_0055
16/04/11 12:37:10 INFO impl.YarnClientImpl: Submitted application application_1460364631926_0055
16/04/11 12:37:10 INFO mapreduce.Job: The url to track the job: http://ip-10-0-71-101.node.dc1.consul:8088/proxy/application_1460364631926_0055/
16/04/11 12:37:10 INFO mapreduce.Job: Running job: job_1460364631926_0055
16/04/11 12:37:17 INFO mapreduce.Job: Job job_1460364631926_0055 running in uber mode : false
16/04/11 12:37:17 INFO mapreduce.Job: map 0% reduce 0%
16/04/11 12:37:27 INFO mapreduce.Job: map 13% reduce 0%
16/04/11 12:37:28 INFO mapreduce.Job: map 57% reduce 0%
16/04/11 12:37:29 INFO mapreduce.Job: map 67% reduce 0%
16/04/11 12:37:39 INFO mapreduce.Job: map 72% reduce 0%
16/04/11 12:37:40 INFO mapreduce.Job: map 80% reduce 0%
16/04/11 12:37:49 INFO mapreduce.Job: map 81% reduce 0%
16/04/11 12:37:50 INFO mapreduce.Job: map 89% reduce 14%
16/04/11 12:37:51 INFO mapreduce.Job: map 100% reduce 14%
16/04/11 12:37:52 INFO mapreduce.Job: map 100% reduce 100%
16/04/11 12:37:52 INFO mapreduce.Job: Job job_1460364631926_0055 completed successfully
16/04/11 12:37:52 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=3211
FILE: Number of bytes written=4942000
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=9268
HDFS: Number of bytes written=309237645397
HDFS: Number of read operations=147
HDFS: Number of large read operations=0
HDFS: Number of write operations=38
Job Counters
Launched map tasks=36
Launched reduce tasks=1
Data-local map tasks=36
Total time spent by all maps in occupied slots (ms)=997570
Total time spent by all reduces in occupied slots (ms)=21246
Total time spent by all map tasks (ms)=997570
Total time spent by all reduce tasks (ms)=10623
Total vcore-seconds taken by all map tasks=997570
Total vcore-seconds taken by all reduce tasks=10623
Total megabyte-seconds taken by all map tasks=3575290880
Total megabyte-seconds taken by all reduce tasks=76145664
Map-Reduce Framework
Map input records=36
Map output records=180
Map output bytes=2845
Map output materialized bytes=3421
Input split bytes=5210
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=3421
Reduce input records=180
Reduce output records=5
Spilled Records=360
Shuffled Maps =36
Failed Shuffles=0
Merged Map outputs=36
GC time elapsed (ms)=36316
CPU time spent (ms)=1178430
Physical memory (bytes) snapshot=92347961344
Virtual memory (bytes) snapshot=139844718592
Total committed heap usage (bytes)=109997719552
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=4058
File Output Format Counters
Bytes Written=85
16/04/11 12:37:52 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/04/11 12:37:52 INFO fs.TestDFSIO: Date & time: Mon Apr 11 12:37:52 UTC 2016
16/04/11 12:37:52 INFO fs.TestDFSIO: Number of files: 36
16/04/11 12:37:52 INFO fs.TestDFSIO: Total MBytes processed: 294912.0
16/04/11 12:37:52 INFO fs.TestDFSIO: Throughput mb/sec: 353.6259271960727
16/04/11 12:37:52 INFO fs.TestDFSIO: Average IO rate mb/sec: 372.3059997558594
16/04/11 12:37:52 INFO fs.TestDFSIO: IO rate std deviation: 87.68244576761097
16/04/11 12:37:52 INFO fs.TestDFSIO: Test exec time sec: 43.991
16/04/11 12:37:52 INFO fs.TestDFSIO:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment