Skip to content

Instantly share code, notes, and snippets.

@jbenninghoff

jbenninghoff/j-1PL6MK3TL9BA1-XmlExtract.hist

Created April 12, 2023 16:52
Show Gist options
  • Save jbenninghoff/aef4a7d75650ab51d037da804bff2338 to your computer and use it in GitHub Desktop.
Save jbenninghoff/aef4a7d75650ab51d037da804bff2338 to your computer and use it in GitHub Desktop.
Download ZIP
XMLextraction job history, 6:14hrs, scaled down
Raw
j-1PL6MK3TL9BA1-XmlExtract.hist
Hadoop job: job_1681245476823_0001
=====================================
User: hadoop
JobName: XmlExtraction
JobConf: hdfs://ip-10-0-2-30.us-west-2.compute.internal:8020/tmp/hadoop-yarn/staging/hadoop/.staging/job_1681245476823_0001/job.xml
Submitted At: 11-Apr-2023 20:40:08
Launched At: 11-Apr-2023 20:40:14 (6sec)
Finished At: 12-Apr-2023 02:54:52 (6hrs, 14mins, 37sec)
Status: SUCCEEDED
Counters:
|Group Name |Counter name |Map Value |Reduce Value|Total Value|
---------------------------------------------------------------------------------------
|File System Counters |FILE: Number of bytes read |0 |30,056,418,962|30,056,418,962
|File System Counters |FILE: Number of bytes written |31,014,150,546|30,056,833,172|61,070,983,718
|File System Counters |FILE: Number of read operations|0 |0 |0
|File System Counters |FILE: Number of large read operations|0 |0 |0
|File System Counters |FILE: Number of write operations|0 |0 |0
|File System Counters |HDFS: Number of bytes read |262,200 |0 |262,200
|File System Counters |HDFS: Number of bytes written |0 |0 |0
|File System Counters |HDFS: Number of read operations|2,300 |0 |2,300
|File System Counters |HDFS: Number of large read operations|0 |0 |0
|File System Counters |HDFS: Number of write operations|0 |0 |0
|File System Counters |HDFS: Number of bytes read erasure-coded|0 |0 |0
|File System Counters |S3: Number of bytes read |59,073,386,215|0 |59,073,386,215
|File System Counters |S3: Number of bytes written |0 |191,092,854,166|191,092,854,166
|File System Counters |S3: Number of read operations |0 |0 |0
|File System Counters |S3: Number of large read operations|0 |0 |0
|File System Counters |S3: Number of write operations|0 |0 |0
|Job Counters |Launched map tasks |0 |0 |2,300
|Job Counters |Launched reduce tasks |0 |0 |1
|Job Counters |Data-local map tasks |0 |0 |2,300
|Job Counters |Total time spent by all maps in occupied slots (ms)|0 |0 |3,648,628,407,750
|Job Counters |Total time spent by all reduces in occupied slots (ms)|0 |0 |25,301,041,920
|Job Counters |Total time spent by all map tasks (ms)|0 |0 |29,189,027,262
|Job Counters |Total time spent by all reduce tasks (ms)|0 |0 |21,962,710
|Job Counters |Total vcore-milliseconds taken by all map tasks|0 |0 |29,189,027,262
|Job Counters |Total vcore-milliseconds taken by all reduce tasks|0 |0 |21,962,710
|Job Counters |Total megabyte-milliseconds taken by all map tasks|0 |0 |116,756,109,048,000
|Job Counters |Total megabyte-milliseconds taken by all reduce tasks|0 |0 |809,633,341,440
|Map-Reduce Framework |Map input records |1,784,626 |0 |1,784,626
|Map-Reduce Framework |Map output records |523,938,622|0 |523,938,622
|Map-Reduce Framework |Map output bytes |229,795,915,416|0 |229,795,915,416
|Map-Reduce Framework |Map output materialized bytes |30,061,334,074|0 |30,061,334,074
|Map-Reduce Framework |Input split bytes |262,200 |0 |262,200
|Map-Reduce Framework |Combine input records |0 |0 |0
|Map-Reduce Framework |Combine output records |0 |0 |0
|Map-Reduce Framework |Reduce input groups |0 |6,812,347 |6,812,347
|Map-Reduce Framework |Reduce shuffle bytes |0 |30,061,334,074|30,061,334,074
|Map-Reduce Framework |Reduce input records |0 |523,938,622|523,938,622
|Map-Reduce Framework |Reduce output records |0 |523,938,622|523,938,622
|Map-Reduce Framework |Spilled Records |523,938,622|523,938,622|1,047,877,244
|Map-Reduce Framework |Shuffled Maps |0 |2,300 |2,300
|Map-Reduce Framework |Failed Shuffles |0 |0 |0
|Map-Reduce Framework |Merged Map outputs |0 |2,300 |2,300
|Map-Reduce Framework |GC time elapsed (ms) |118,965,998|67,217 |119,033,215
|Map-Reduce Framework |CPU time spent (ms) |30,079,879,050|6,572,080 |30,086,451,130
|Map-Reduce Framework |Physical memory (bytes) snapshot|7,399,899,021,312|17,112,608,768|7,417,011,630,080
|Map-Reduce Framework |Virtual memory (bytes) snapshot|13,088,068,730,880|37,402,832,896|13,125,471,563,776
|Map-Reduce Framework |Total committed heap usage (bytes)|8,146,270,027,776|18,368,430,080|8,164,638,457,856
|Map-Reduce Framework |Peak Map Physical memory (bytes)|4,081,463,296|0 |4,081,463,296
|Map-Reduce Framework |Peak Map Virtual memory (bytes)|5,821,136,896|0 |5,821,136,896
|Map-Reduce Framework |Peak Reduce Physical memory (bytes)|0 |32,165,617,664|32,165,617,664
|Map-Reduce Framework |Peak Reduce Virtual memory (bytes)|0 |37,402,832,896|37,402,832,896
|Shuffle Errors |BAD_ID |0 |0 |0
|Shuffle Errors |CONNECTION |0 |0 |0
|Shuffle Errors |IO_ERROR |0 |0 |0
|Shuffle Errors |WRONG_LENGTH |0 |0 |0
|Shuffle Errors |WRONG_MAP |0 |0 |0
|Shuffle Errors |WRONG_REDUCE |0 |0 |0
|File Input Format Counters |Bytes Read |59,073,386,215|0 |59,073,386,215
|File Output Format Counters |Bytes Written |0 |191,092,854,166|191,092,854,166
=====================================
Task Summary
============================
Kind Total Successful Failed Killed StartTime FinishTime
Setup 0 0 0 0
Map 2300 2300 0 0 11-Apr-2023 20:40:20 12-Apr-2023 01:51:22 (5hrs, 11mins, 2sec)
Reduce 1 1 0 0 11-Apr-2023 20:48:49 12-Apr-2023 02:54:51 (6hrs, 6mins, 2sec)
Cleanup 0 0 0 0
============================
Analysis
=========
Time taken by best performing map task task_1681245476823_0001_m_000309: 2mins, 51sec
Average time taken by map tasks: 3hrs, 31mins, 30sec
Worse performing map tasks:
TaskId Timetaken
task_1681245476823_0001_m_000860 5hrs, 11mins, 0sec
task_1681245476823_0001_m_000980 5hrs, 9mins, 32sec
task_1681245476823_0001_m_001613 5hrs, 7mins, 26sec
task_1681245476823_0001_m_000988 5hrs, 6mins, 34sec
task_1681245476823_0001_m_001807 5hrs, 6mins, 4sec
task_1681245476823_0001_m_001450 5hrs, 5mins, 53sec
task_1681245476823_0001_m_001456 5hrs, 5mins, 47sec
task_1681245476823_0001_m_000708 5hrs, 5mins, 47sec
task_1681245476823_0001_m_000934 5hrs, 5mins, 3sec
task_1681245476823_0001_m_001045 5hrs, 5mins, 1sec
The last map task task_1681245476823_0001_m_000860 finished at (relative to the Job launch time): 12-Apr-2023 01:51:22 (5hrs, 11mins, 7sec)
Time taken by best performing shuffle task task_1681245476823_0001_r_000000: 5hrs, 2mins, 33sec
Average time taken by shuffle tasks: 5hrs, 2mins, 33sec
Worse performing shuffle tasks:
TaskId Timetaken
task_1681245476823_0001_r_000000 5hrs, 2mins, 33sec
The last shuffle task task_1681245476823_0001_r_000000 finished at (relative to the Job launch time): 12-Apr-2023 01:51:22 (5hrs, 11mins, 7sec)
Time taken by best performing reduce task task_1681245476823_0001_r_000000: 1hrs, 3mins, 29sec
Average time taken by reduce tasks: 1hrs, 3mins, 29sec
Worse performing reduce tasks:
TaskId Timetaken
task_1681245476823_0001_r_000000 1hrs, 3mins, 29sec
The last reduce task task_1681245476823_0001_r_000000 finished at (relative to the Job launch time): 12-Apr-2023 02:54:51 (6hrs, 14mins, 37sec)
=========
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment