Skip to content

Instantly share code, notes, and snippets.

@muxueqz
Created May 28, 2014 06:41
Show Gist options
  • Save muxueqz/0d8dae78e4344c30ca1b to your computer and use it in GitHub Desktop.
Save muxueqz/0d8dae78e4344c30ca1b to your computer and use it in GitHub Desktop.
Dpark Fetch failed for shuffle
2014-05-26 00:18:22,040 [INFO] [broadcast] broadcast e96a2f61-fdd6-45d9-9be7-b50931e3fc33: 291 bytes in 1 blocks
2014-05-26 00:18:22,048 [INFO] [broadcast] broadcast 14e902be-8d9b-4a47-809e-589e7a1b6a34: 5 bytes in 1 blocks
2014-05-26 00:18:22,049 [INFO] [broadcast] broadcast 5d791168-2b67-428d-95ce-b7b83981a890: 3218 bytes in 1 blocks
2014-05-26 00:18:22,087 [INFO] [scheduler] Got job 1 with 693 tasks: <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>
2014-05-26 00:19:41,069 [INFO] [job ] Job 1 finished in 79.0s: min=1.4s, avg=5.4s, max=9.2s, maxtry=1
2014-05-26 00:19:41,128 [INFO] [scheduler] Got job 2 with 441 tasks: <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>
2014-05-26 00:19:41,132 [INFO] [scheduler] Got job 3 with 441 tasks: <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>
2014-05-26 00:20:39,751 [INFO] [job ] Job 2 finished in 58.6s: min=6.3s, avg=8.9s, max=11.7s, maxtry=1
2014-05-26 00:21:09,986 [INFO] [job ] Job 3 finished in 88.9s: min=3.1s, avg=4.9s, max=8.3s, maxtry=1
2014-05-26 00:21:10,018 [INFO] [scheduler] Got job 4 with 630 tasks: <FilteredRDD <ShuffledRDD <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>>
2014-05-26 00:21:23,371 [INFO] [job ] Job 4 finished in 13.4s: min=0.5s, avg=1.2s, max=1.7s, maxtry=1
2014-05-26 00:21:23,381 [INFO] [scheduler] Got job 5 with 735 tasks: <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>
[ck-hadoop-m-78.17173.com] 2014-05-26 00:32:02,780 [ERROR] [[email protected]] worker process 24173 of task 5:2284:1 is zombie
2014-05-26 00:32:04,233 [WARNING] [job ] Lost Task 78 (task 5:2284:1) at ck-hadoop-m-78.17173.com
2014-05-26 01:21:17,106 [INFO] [job ] Job 5 finished in 3593.7s: min=176.8s, avg=292.7s, max=2424.7s, maxtry=2
2014-05-26 01:21:17,180 [INFO] [scheduler] Got job 6 with 945 tasks: <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
[ck-hadoop-05.17173.com] 2014-05-26 01:22:28,174 [ERROR] [[email protected]] worker process 6883 of task 6:2942:1 is zombie
2014-05-26 01:22:28,577 [WARNING] [job ] Lost Task 1 (task 6:2942:1) at ck-hadoop-05.17173.com
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:24:54,506 [WARNING] [[email protected]] task 6:2984:1 used too much memory: 4557MB > 3000MB * 1.5, kill it. use -M argument or taskMemory to request more memory.
2014-05-26 01:27:01,699 [WARNING] [job ] Lost Task 104 (task 6:3045:1) at ck-hadoop-05.17173.com
2014-05-26 01:27:01,700 [WARNING] [job ] Lost Task 108 (task 6:3049:1) at ck-hadoop-05.17173.com
2014-05-26 01:27:01,700 [WARNING] [job ] Lost Task 123 (task 6:3064:1) at ck-hadoop-05.17173.com
2014-05-26 01:27:01,700 [WARNING] [job ] Lost Task 96 (task 6:3037:1) at ck-hadoop-05.17173.com
2014-05-26 01:27:01,700 [WARNING] [job ] Lost Task 125 (task 6:3066:1) at ck-hadoop-05.17173.com
2014-05-26 01:27:01,700 [WARNING] [job ] Lost Task 160 (task 6:3101:1) at ck-hadoop-05.17173.com
2014-05-26 01:27:01,701 [WARNING] [job ] Lost Task 165 (task 6:3106:1) at ck-hadoop-05.17173.com
2014-05-26 01:27:01,701 [WARNING] [scheduler] slave 201312080848-2640263946-5050-15521-306 lost
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:01,119 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 218, 717, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/717/218, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:01,424 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 220, 717, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/717/220, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:01,459 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 226, 650, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/650/226, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:28:01,401 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 206, 598, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/598/206, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:28:01,540 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 231, 597, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/597/231, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:01,768 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 232, 440, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/440/232, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:01,768 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 233, 598, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/598/233, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:01,834 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 229, 719, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/719/229, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:01,834 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 230, 729, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/729/230, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:01,853 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 232, 590, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/590/232, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:28:01,932 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 227, 725, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/725/227, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:01,975 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 233, 590, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/590/233, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:28:02,183 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 222, 589, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/589/222, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:02,205 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 217, 294, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/294/217, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:28:02,299 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 234, 729, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/729/234, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:28:02,370 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 216, 514, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/514/216, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:28:02,375 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 219, 583, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/583/219, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:28:02,493 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 223, 328, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/328/223, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:02,520 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 228, 133, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/133/228, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:28:02,534 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 210, 328, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/328/210, not found
2014-05-26 01:28:02,551 [WARNING] [job ] task 3173 failed @ dpark-75.cnc.internal.17173.com: <ResultTask(232) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 440, 232)>
2014-05-26 01:28:02,552 [WARNING] [job ] task 3172 failed @ ck-hadoop-04.17173.com: <ResultTask(231) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 597, 231)>
[ck-hadoop-m-78.17173.com] 2014-05-26 01:28:02,645 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 221, 385, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/385/221, not found
2014-05-26 01:28:02,716 [WARNING] [job ] task 3174 failed @ dpark-75.cnc.internal.17173.com: <ResultTask(233) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 598, 233)>
2014-05-26 01:28:02,748 [WARNING] [job ] task 3171 failed @ dpark-51.ctc.internal.17173.com: <ResultTask(230) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 729, 230)>
[ck-hadoop-01.17173.com] 2014-05-26 01:28:02,553 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 235, 520, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/520/235, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:03,016 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 224, 538, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/538/224, not found
2014-05-26 01:28:03,114 [WARNING] [job ] task 3176 failed @ ck-hadoop-01.17173.com: <ResultTask(235) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 520, 235)>
2014-05-26 01:28:03,114 [WARNING] [job ] task 3170 failed @ dpark-51.ctc.internal.17173.com: <ResultTask(229) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 719, 229)>
2014-05-26 01:28:03,115 [WARNING] [job ] task 3175 failed @ ck-hadoop-m-78.17173.com: <ResultTask(234) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 729, 234)>
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:03,200 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 224, 331, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/331/224, not found
2014-05-26 01:28:03,367 [WARNING] [job ] task 3168 failed @ ck-hadoop-03.17173.com: <ResultTask(227) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 725, 227)>
2014-05-26 01:28:03,481 [WARNING] [job ] task 3167 failed @ dpark-51.ctc.internal.17173.com: <ResultTask(226) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 650, 226)>
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:03,483 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 233, 135, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/135/233, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:28:03,495 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 236, 539, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/539/236, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:28:03,552 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 229, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/12/229, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:28:03,638 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 229, 71, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/71/229, not found
2014-05-26 01:28:03,696 [WARNING] [job ] Task 3174 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:03,699 [INFO] [scheduler] <Stage(5) for <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>> is now unavailable on host http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:03,704 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 234, 538, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/538/234, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:03,732 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 232, 440, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/440/232, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:28:03,727 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 231, 474, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/474/231, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:28:03,722 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 230, 132, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/132/230, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:03,765 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 232, 589, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/589/232, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:03,819 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 234, 440, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/440/234, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:03,834 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 235, 717, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/717/235, not found
2014-05-26 01:28:03,861 [WARNING] [job ] Task 3170 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:03,976 [WARNING] [job ] Task 3173 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-01.17173.com] 2014-05-26 01:28:03,973 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 231, 67, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/67/231, not found
2014-05-26 01:28:04,041 [WARNING] [job ] task 3177 failed @ ck-hadoop-04.17173.com: <ResultTask(236) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 539, 236)>
2014-05-26 01:28:04,115 [WARNING] [job ] Task 3171 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-01.17173.com] 2014-05-26 01:28:04,101 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 225, 527, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/527/225, not found
2014-05-26 01:28:04,262 [WARNING] [job ] Task 3172 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:04,327 [WARNING] [job ] task 3169 failed @ dpark-51.ctc.internal.17173.com: <ResultTask(228) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 133, 228)>
[ck-hadoop-01.17173.com] 2014-05-26 01:28:04,330 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 225, 294, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/294/225, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:04,406 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 236, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/12/236, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:28:04,556 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 228, 353, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/353/228, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:28:04,621 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 228, 136, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/136/228, not found
2014-05-26 01:28:04,655 [WARNING] [job ] Task 3177 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:04,720 [WARNING] [job ] Task 3175 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-04.17173.com] 2014-05-26 01:28:04,794 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 228, 363, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/363/228, not found
2014-05-26 01:28:04,835 [WARNING] [job ] Task 3176 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:04,931 [WARNING] [job ] Task 3169 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:05,828 [WARNING] [job ] task 3161 failed @ dpark-75.cnc.internal.17173.com: <ResultTask(220) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 717, 220)>
2014-05-26 01:28:06,265 [WARNING] [job ] task 3166 failed @ ck-hadoop-01.17173.com: <ResultTask(225) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 527, 225)>
2014-05-26 01:28:06,508 [WARNING] [job ] task 3163 failed @ ck-hadoop-m-78.17173.com: <ResultTask(222) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 589, 222)>
[ck-hadoop-04.17173.com] 2014-05-26 01:28:06,633 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 225, 214, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/214/225, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:28:06,717 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 225, 721, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/721/225, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:28:06,835 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 220, 423, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/423/220, not found
2014-05-26 01:28:06,897 [WARNING] [job ] Task 3166 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-03.17173.com] 2014-05-26 01:28:06,892 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 220, 589, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/589/220, not found
2014-05-26 01:28:07,012 [WARNING] [job ] task 3165 failed @ dpark-75.cnc.internal.17173.com: <ResultTask(224) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 538, 224)>
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:07,036 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 222, 363, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/363/222, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:28:07,041 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 220, 18, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/18/220, not found
2014-05-26 01:28:07,123 [WARNING] [job ] task 3162 failed @ ck-hadoop-m-78.17173.com: <ResultTask(221) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 385, 221)>
2014-05-26 01:28:07,365 [WARNING] [job ] task 3159 failed @ dpark-75.cnc.internal.17173.com: <ResultTask(218) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 717, 218)>
2014-05-26 01:28:07,366 [WARNING] [job ] task 3164 failed @ ck-hadoop-m-78.17173.com: <ResultTask(223) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 328, 223)>
2014-05-26 01:28:07,383 [WARNING] [job ] Task 3163 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:07,471 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 221, 358, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/358/221, not found
2014-05-26 01:28:07,497 [WARNING] [job ] Task 3161 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:07,714 [WARNING] [job ] Task 3162 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-02.17173.com] 2014-05-26 01:28:07,875 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 218, 358, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/358/218, not found
2014-05-26 01:28:08,273 [WARNING] [job ] Task 3159 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:08,281 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 223, 597, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/597/223, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:28:08,340 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 224, 440, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/440/224, not found
2014-05-26 01:28:08,584 [WARNING] [job ] task 3158 failed @ dpark-51.ctc.internal.17173.com: <ResultTask(217) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 294, 217)>
[ck-hadoop-01.17173.com] 2014-05-26 01:28:08,717 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 194, 729, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/729/194, not found
2014-05-26 01:28:08,776 [WARNING] [job ] task 3160 failed @ dpark-76.cnc.internal.17173.com: <ResultTask(219) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 583, 219)>
[ck-hadoop-04.17173.com] 2014-05-26 01:28:08,756 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 30, 723, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/723/30, not found
2014-05-26 01:28:08,790 [WARNING] [job ] Task 3164 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-01.17173.com] 2014-05-26 01:28:08,768 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 194, 67, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/67/194, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:08,841 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 217, 598, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/598/217, not found
2014-05-26 01:28:08,955 [WARNING] [job ] task 3135 failed @ ck-hadoop-01.17173.com: <ResultTask(194) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 729, 194)>
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:28:09,001 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 217, 353, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/353/217, not found
2014-05-26 01:28:09,048 [WARNING] [job ] Task 3165 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:09,049 [WARNING] [job ] task 3157 failed @ dpark-76.cnc.internal.17173.com: <ResultTask(216) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 514, 216)>
2014-05-26 01:28:09,050 [WARNING] [job ] task 2971 failed @ ck-hadoop-04.17173.com: <ResultTask(30) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 723, 30)>
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:09,111 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 219, 589, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/589/219, not found
2014-05-26 01:28:09,143 [WARNING] [job ] Task 3158 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:09,151 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 219, 656, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/656/219, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:09,216 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 194, 239, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/239/194, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:09,281 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 194, 413, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/413/194, not found
2014-05-26 01:28:09,324 [WARNING] [job ] Task 3160 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-03.17173.com] 2014-05-26 01:28:09,375 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 30, 474, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/474/30, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:28:09,421 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 194, 430, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/430/194, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:28:09,484 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 216, 727, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/727/216, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:28:09,498 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 30, 328, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/328/30, not found
2014-05-26 01:28:09,539 [WARNING] [job ] Task 3135 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:09,704 [WARNING] [job ] Task 2971 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:09,736 [WARNING] [job ] Task 3157 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:09,983 [WARNING] [job ] task 3151 failed @ dpark-76.cnc.internal.17173.com: <ResultTask(210) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 328, 210)>
[ck-hadoop-02.17173.com] 2014-05-26 01:28:10,571 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 210, 729, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/729/210, not found
2014-05-26 01:28:10,999 [WARNING] [job ] Task 3151 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:11,197 [WARNING] [job ] task 3147 failed @ ck-hadoop-03.17173.com: <ResultTask(206) of <MultiOutputTextFileRDD /mfs/logs/dpark_result/20140525/day <MappedRDD <FilteredRDD <FlatMappedRDD <GlommedRDD <FilteredRDD <ShuffledRDD <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 5, 598, 206)>
[ck-hadoop-02.17173.com] 2014-05-26 01:28:11,505 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 206, 537, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/537/206, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:28:11,723 [WARNING] [shuffle ] Fetch failed for shuffle 5, reduce 206, 589, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/5/589/206, not found
2014-05-26 01:28:11,970 [WARNING] [job ] Task 3147 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:28:16,314 [INFO] [job ] Job 6 finished in 419.1s: min=0.0s, avg=31.9s, max=410.9s, maxtry=3
2014-05-26 01:29:12,071 [INFO] [scheduler] Resubmitting failed stages: <Stage(5) for <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
2014-05-26 01:29:12,072 [INFO] [scheduler] Got job 7 with 55 tasks: <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:12,496 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 430, 312, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/312/430, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:12,498 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 358, 121, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/121/358, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:12,508 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 358, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/358, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:12,508 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 430, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/430, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:12,515 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 108, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/108, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:12,539 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 239, 218, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/218/239, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:12,560 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 137, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/137, not found
2014-05-26 01:29:12,591 [WARNING] [job ] task 3913 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(5, 430) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 312, 430)>
[ck-hadoop-02.17173.com] 2014-05-26 01:29:12,593 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 108, 210, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/210/108, not found
2014-05-26 01:29:12,706 [WARNING] [job ] task 3905 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(5, 358) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 121, 358)>
2014-05-26 01:29:12,707 [WARNING] [job ] task 3897 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(5, 137) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 309, 137)>
2014-05-26 01:29:12,773 [WARNING] [job ] task 3891 failed @ ck-hadoop-02.17173.com: <ShuffleTask(5, 108) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 396, 108)>
[ck-hadoop-02.17173.com] 2014-05-26 01:29:12,762 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 239, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/239, not found
2014-05-26 01:29:12,886 [WARNING] [job ] task 3899 failed @ ck-hadoop-02.17173.com: <ShuffleTask(5, 239) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 218, 239)>
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:12,896 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 67, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/67, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:12,939 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 18, 404, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/404/18, not found
2014-05-26 01:29:13,072 [WARNING] [job ] task 3888 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(5, 18) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 404, 18)>
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:13,062 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 67, 214, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/214/67, not found
2014-05-26 01:29:13,133 [WARNING] [job ] task 3889 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(5, 67) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 396, 67)>
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,237 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 353, 222, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/222/353, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,256 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 353, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/353, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:13,317 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 18, 312, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/312/18, not found
2014-05-26 01:29:13,373 [WARNING] [job ] task 3904 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(5, 353) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 222, 353)>
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,387 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 133, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/133, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,551 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 387, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/387, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,575 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 387, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/387, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,593 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 133, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/133, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,654 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 294, 222, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/222/294, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,680 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 423, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/423, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,719 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 294, 129, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/129/294, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,758 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 387, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/387, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:13,775 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 723, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/723, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:13,775 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 723, 404, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/404/723, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,783 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 514, 2, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/2/514, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,784 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 721, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/721, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:13,800 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 514, 129, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/129/514, not found
2014-05-26 01:29:13,814 [WARNING] [job ] task 3912 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(5, 423) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 303, 423)>
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:13,808 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 646, 210, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/210/646, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,834 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 108, 392, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/392/108, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,838 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 136, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/136, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,841 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 430, 144, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/144/430, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,846 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 430, 404, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/404/430, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,894 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 604, 22, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/22/604, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:13,924 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 353, 218, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/218/353, not found
2014-05-26 01:29:13,950 [WARNING] [job ] Task 3891 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,934 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 721, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/721, not found
2014-05-26 01:29:13,958 [WARNING] [job ] Task 3888 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:13,966 [WARNING] [job ] task 3937 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(5, 723) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 137, 723)>
2014-05-26 01:29:13,966 [WARNING] [job ] Task 3913 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,952 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 537, 22, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/22/537, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:13,956 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 646, 214, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/214/646, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:13,956 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 537, 312, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/312/537, not found
2014-05-26 01:29:14,018 [INFO] [scheduler] <Stage(2) for <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>> is now unavailable on host http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:14,041 [WARNING] [job ] task 3893 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(5, 133) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 309, 133)>
2014-05-26 01:29:14,042 [WARNING] [job ] task 3929 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(5, 646) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 210, 646)>
2014-05-26 01:29:14,042 [WARNING] [job ] task 3896 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(5, 136) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 32, 136)>
2014-05-26 01:29:14,043 [WARNING] [job ] Task 3904 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:14,059 [WARNING] [job ] task 3936 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(5, 721) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 51, 721)>
2014-05-26 01:29:14,059 [WARNING] [job ] task 3901 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(5, 294) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 222, 294)>
2014-05-26 01:29:14,071 [WARNING] [job ] task 3928 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(5, 604) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 22, 604)>
2014-05-26 01:29:14,075 [WARNING] [job ] task 3909 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(5, 387) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 32, 387)>
2014-05-26 01:29:14,087 [WARNING] [job ] task 3920 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(5, 537) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 22, 537)>
2014-05-26 01:29:14,088 [WARNING] [job ] task 3917 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(5, 514) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 2, 514)>
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:14,084 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 67, 404, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/404/67, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:14,079 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 423, 218, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/218/423, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:14,108 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 423, 210, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/210/423, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:14,142 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 538, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/538, not found
2014-05-26 01:29:14,225 [WARNING] [job ] task 3921 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(5, 538) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 303, 538)>
2014-05-26 01:29:14,226 [WARNING] [job ] Task 3912 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:14,258 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 67, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/67, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:14,303 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 387, 222, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/222/387, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:14,306 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 604, 210, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/210/604, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:14,330 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 721, 210, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/210/721, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:14,333 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 387, 121, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/121/387, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,343 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 590, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/590, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,379 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 537, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/537, not found
2014-05-26 01:29:14,398 [WARNING] [job ] Task 3889 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:14,398 [WARNING] [job ] Task 3909 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-02.17173.com] 2014-05-26 01:29:14,375 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 294, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/294, not found
2014-05-26 01:29:14,430 [WARNING] [job ] Task 3928 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,469 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 537, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/537, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:14,483 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 721, 312, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/312/721, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,505 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 590, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/590, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,528 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 358, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/358, not found
2014-05-26 01:29:14,545 [WARNING] [job ] Task 3901 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,556 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 358, 121, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/121/358, not found
2014-05-26 01:29:14,609 [WARNING] [job ] Task 3936 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:14,609 [WARNING] [job ] Task 3920 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,639 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 657, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/657, not found
2014-05-26 01:29:14,674 [WARNING] [job ] task 3925 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(5, 590) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 137, 590)>
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,676 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 729, 400, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/400/729, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,733 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 239, 214, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/214/239, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,735 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 358, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/358, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,769 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 239, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/12/239, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,784 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 657, 218, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/218/657, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:14,791 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 729, 22, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/22/729, not found
2014-05-26 01:29:14,879 [WARNING] [job ] Task 3905 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:14,879 [WARNING] [job ] Task 3899 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:14,895 [WARNING] [job ] task 3933 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(5, 657) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 32, 657)>
2014-05-26 01:29:14,914 [WARNING] [job ] task 3940 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(5, 729) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 400, 729)>
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:14,914 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 590, 222, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/222/590, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:14,937 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 590, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/590, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:14,949 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 717, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/717, not found
2014-05-26 01:29:14,988 [WARNING] [job ] Task 3925 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,072 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 6, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/6, not found
2014-05-26 01:29:15,119 [WARNING] [job ] task 3934 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(5, 717) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 309, 717)>
2014-05-26 01:29:15,164 [WARNING] [job ] task 3886 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(5, 6) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 306, 6)>
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,141 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 134, 392, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/392/134, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:15,177 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 729, 400, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/400/729, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,171 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 328, 218, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/218/328, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:15,207 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 657, 400, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/400/657, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:15,226 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 657, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/657, not found
2014-05-26 01:29:15,287 [WARNING] [job ] task 3894 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(5, 134) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 392, 134)>
2014-05-26 01:29:15,297 [WARNING] [job ] Task 3933 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:15,329 [WARNING] [job ] Task 3940 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,340 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 514, 312, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/312/514, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,352 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 520, 144, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/144/520, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,360 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 328, 144, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/144/328, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,377 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 328, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/328, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,394 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 597, 121, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/121/597, not found
2014-05-26 01:29:15,443 [WARNING] [job ] Task 3917 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:15,445 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 717, 218, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/218/717, not found
2014-05-26 01:29:15,508 [WARNING] [job ] task 3918 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(5, 520) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 144, 520)>
2014-05-26 01:29:15,508 [WARNING] [job ] task 3902 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(5, 328) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 218, 328)>
2014-05-26 01:29:15,515 [WARNING] [job ] Task 3934 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:15,555 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 6, 312, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/312/6, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:15,571 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 134, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/134, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:15,572 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 134, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/134, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,572 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 597, 400, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/400/597, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,578 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 388, 210, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/210/388, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:15,592 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 6, 404, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/404/6, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,593 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 137, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/137, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:15,629 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 388, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/388, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:15,631 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 135, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/135, not found
2014-05-26 01:29:15,681 [WARNING] [job ] task 3926 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(5, 597) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 121, 597)>
2014-05-26 01:29:15,693 [WARNING] [job ] Task 3886 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:15,697 [WARNING] [job ] Task 3894 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-04.17173.com] 2014-05-26 01:29:15,679 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 135, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/135, not found
2014-05-26 01:29:15,761 [WARNING] [job ] task 3910 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(5, 388) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 210, 388)>
2014-05-26 01:29:15,761 [WARNING] [job ] Task 3897 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-04.17173.com] 2014-05-26 01:29:15,742 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 719, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/719, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:15,843 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 135, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/135, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:15,925 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 331, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/331, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:15,937 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 331, 214, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/214/331, not found
2014-05-26 01:29:15,984 [WARNING] [job ] task 3895 failed @ ck-hadoop-04.17173.com: <ShuffleTask(5, 135) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 137, 135)>
[ck-hadoop-04.17173.com] 2014-05-26 01:29:15,956 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 719, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/719, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:15,977 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 597, 312, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/312/597, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,079 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 527, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/527, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,080 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 527, 392, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/392/527, not found
2014-05-26 01:29:16,125 [WARNING] [job ] task 3935 failed @ ck-hadoop-04.17173.com: <ShuffleTask(5, 719) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 306, 719)>
2014-05-26 01:29:16,125 [WARNING] [job ] Task 3926 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:16,141 [WARNING] [job ] task 3903 failed @ ck-hadoop-04.17173.com: <ShuffleTask(5, 331) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 303, 331)>
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,163 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 598, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/598, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,206 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 413, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/413, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,216 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 12, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/12, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,242 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 12, 129, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/129/12, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:16,250 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 135, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/135, not found
2014-05-26 01:29:16,279 [WARNING] [job ] task 3919 failed @ ck-hadoop-04.17173.com: <ShuffleTask(5, 527) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 137, 527)>
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,270 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 413, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/413, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:16,276 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 135, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/135, not found
2014-05-26 01:29:16,357 [WARNING] [job ] task 3927 failed @ ck-hadoop-04.17173.com: <ShuffleTask(5, 598) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 51, 598)>
2014-05-26 01:29:16,369 [WARNING] [job ] Task 3895 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,369 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 385, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/385, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,371 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 589, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/589, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:29:16,365 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 725, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/725, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:16,366 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 583, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/12/583, not found
2014-05-26 01:29:16,401 [WARNING] [job ] task 3887 failed @ ck-hadoop-04.17173.com: <ShuffleTask(5, 12) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 396, 12)>
[ck-hadoop-03.17173.com] 2014-05-26 01:29:16,374 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 725, 222, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/222/725, not found
2014-05-26 01:29:16,425 [WARNING] [job ] task 3911 failed @ ck-hadoop-04.17173.com: <ShuffleTask(5, 413) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 303, 413)>
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:16,413 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 331, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/331, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:29:16,435 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 440, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/12/440, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:29:16,458 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 440, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/440, not found
2014-05-26 01:29:16,508 [WARNING] [job ] task 3938 failed @ ck-hadoop-03.17173.com: <ShuffleTask(5, 725) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 303, 725)>
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,494 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 288, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/288, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,501 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 288, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/288, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:29:16,516 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 539, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/539, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,525 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 727, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/727, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:16,533 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 331, 129, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/129/331, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:29:16,528 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 539, 2, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/2/539, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,548 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 385, 214, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/214/385, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:16,564 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 527, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/527, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,559 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 727, 22, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/22/727, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:16,603 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 583, 404, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/404/583, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:16,626 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 598, 392, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/392/598, not found
2014-05-26 01:29:16,646 [WARNING] [job ] Task 3903 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:16,646 [WARNING] [job ] task 3914 failed @ ck-hadoop-03.17173.com: <ShuffleTask(5, 440) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 12, 440)>
2014-05-26 01:29:16,653 [WARNING] [job ] Task 3919 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:16,719 [WARNING] [job ] task 3923 failed @ ck-hadoop-02.17173.com: <ShuffleTask(5, 583) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 12, 583)>
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:16,698 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 12, 2, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/2/12, not found
2014-05-26 01:29:16,729 [WARNING] [job ] task 3922 failed @ ck-hadoop-03.17173.com: <ShuffleTask(5, 539) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 32, 539)>
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:16,727 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 598, 129, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/129/598, not found
2014-05-26 01:29:16,803 [WARNING] [job ] task 3900 failed @ ck-hadoop-01.17173.com: <ShuffleTask(5, 288) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 306, 288)>
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,792 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 132, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/132, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,783 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 725, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/725, not found
2014-05-26 01:29:16,818 [WARNING] [job ] Task 3927 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:16,822 [WARNING] [job ] task 3908 failed @ ck-hadoop-01.17173.com: <ShuffleTask(5, 385) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 309, 385)>
[ck-hadoop-02.17173.com] 2014-05-26 01:29:16,804 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 652, 392, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/392/652, not found
2014-05-26 01:29:16,851 [WARNING] [job ] task 3939 failed @ ck-hadoop-01.17173.com: <ShuffleTask(5, 727) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 306, 727)>
[ck-hadoop-03.17173.com] 2014-05-26 01:29:16,822 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 363, 392, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/392/363, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:16,831 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 413, 121, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/121/413, not found
2014-05-26 01:29:16,865 [WARNING] [job ] task 3924 failed @ ck-hadoop-01.17173.com: <ShuffleTask(5, 589) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 51, 589)>
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:16,839 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 12, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/12, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:16,867 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 652, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/652, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:16,892 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 453, 312, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/312/453, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:16,900 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 725, 214, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/214/725, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:16,906 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 12, 404, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/404/12, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:16,912 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 413, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/413, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:16,919 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 381, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/381, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,956 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 132, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/12/132, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:16,964 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 474, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/474, not found
2014-05-26 01:29:16,989 [WARNING] [job ] task 3931 failed @ ck-hadoop-02.17173.com: <ShuffleTask(5, 652) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 392, 652)>
2014-05-26 01:29:16,990 [WARNING] [job ] Task 3887 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,001 [WARNING] [job ] Task 3911 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,001 [WARNING] [job ] task 3906 failed @ ck-hadoop-03.17173.com: <ShuffleTask(5, 363) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 392, 363)>
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:17,014 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 539, 121, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/121/539, not found
2014-05-26 01:29:17,046 [WARNING] [job ] Task 3938 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,046 [WARNING] [job ] task 3915 failed @ ck-hadoop-02.17173.com: <ShuffleTask(5, 453) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 312, 453)>
[ck-hadoop-03.17173.com] 2014-05-26 01:29:17,028 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 214, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/12/214, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:17,045 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 656, 144, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/144/656, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:17,076 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 385, 137, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/137/385, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:29:17,063 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 656, 222, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/222/656, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:17,097 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 385, 218, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/218/385, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:29:17,081 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 288, 222, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/222/288, not found
2014-05-26 01:29:17,120 [WARNING] [job ] task 3892 failed @ ck-hadoop-01.17173.com: <ShuffleTask(5, 132) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 396, 132)>
[ck-hadoop-02.17173.com] 2014-05-26 01:29:17,125 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 381, 303, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/303/381, not found
2014-05-26 01:29:17,180 [WARNING] [job ] task 3916 failed @ ck-hadoop-01.17173.com: <ShuffleTask(5, 474) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 309, 474)>
[ck-hadoop-03.17173.com] 2014-05-26 01:29:17,165 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 214, 2, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/2/214, not found
2014-05-26 01:29:17,195 [WARNING] [job ] Task 3908 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,197 [WARNING] [job ] task 3932 failed @ ck-hadoop-01.17173.com: <ShuffleTask(5, 656) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 144, 656)>
[ck-hadoop-03.17173.com] 2014-05-26 01:29:17,168 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 650, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/650, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:29:17,189 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 71, 2, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/2/71, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:17,219 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 539, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/12/539, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:17,235 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 583, 2, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/2/583, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:29:17,236 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 583, 218, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/218/583, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:17,231 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 589, 121, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/121/589, not found
2014-05-26 01:29:17,261 [WARNING] [job ] task 3907 failed @ ck-hadoop-02.17173.com: <ShuffleTask(5, 381) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 42, 381)>
2014-05-26 01:29:17,274 [WARNING] [job ] task 3898 failed @ ck-hadoop-03.17173.com: <ShuffleTask(5, 214) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 12, 214)>
[ck-hadoop-03.17173.com] 2014-05-26 01:29:17,247 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 288, 129, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/129/288, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:29:17,248 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 71, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/71, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:17,287 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 589, 400, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/400/589, not found
2014-05-26 01:29:17,343 [WARNING] [job ] Task 3922 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,343 [WARNING] [job ] task 3930 failed @ ck-hadoop-03.17173.com: <ShuffleTask(5, 650) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 396, 650)>
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:17,325 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 453, 144, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/144/453, not found
2014-05-26 01:29:17,360 [WARNING] [job ] Task 3923 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:17,352 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 453, 2, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/2/453, not found
2014-05-26 01:29:17,399 [WARNING] [job ] Task 3900 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-04.17173.com] 2014-05-26 01:29:17,423 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 363, 12, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/12/363, not found
2014-05-26 01:29:17,464 [WARNING] [job ] task 3890 failed @ ck-hadoop-03.17173.com: <ShuffleTask(5, 71) of <FlatMappedRDD <FlatMappedRDD <CoGrouped of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <Un>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 3, 2, 71)>
2014-05-26 01:29:17,464 [WARNING] [job ] Task 3924 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-04.17173.com] 2014-05-26 01:29:17,437 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 363, 129, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/129/363, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:17,493 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 381, 144, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/144/381, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:17,468 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 727, 42, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/42/727, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:29:17,500 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 727, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/727, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:17,530 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 656, 144, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/144/656, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:29:17,529 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 453, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/453, not found
2014-05-26 01:29:17,604 [WARNING] [job ] Task 3906 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:17,598 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 656, 306, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/306/656, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:29:17,600 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 381, 404, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/404/381, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:17,580 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 650, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/650, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:17,595 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 214, 309, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/309/214, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:17,586 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 132, 214, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/214/132, not found
2014-05-26 01:29:17,620 [WARNING] [job ] Task 3915 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:17,605 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 474, 51, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/51/474, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:17,608 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 214, 121, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/121/214, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:17,648 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 650, 396, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/396/650, not found
2014-05-26 01:29:17,684 [WARNING] [job ] Task 3898 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,684 [WARNING] [job ] Task 3932 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-04.17173.com] 2014-05-26 01:29:17,674 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 727, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/727, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:29:17,725 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 474, 129, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/129/474, not found
2014-05-26 01:29:17,749 [WARNING] [job ] Task 3907 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,781 [WARNING] [job ] Task 3930 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-02.17173.com] 2014-05-26 01:29:17,777 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 132, 32, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/32/132, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:29:17,791 [WARNING] [shuffle ] Fetch failed for shuffle 3, reduce 132, 400, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/3/400/132, not found
2014-05-26 01:29:17,845 [WARNING] [job ] Task 3939 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,846 [WARNING] [job ] Task 3916 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:29:17,960 [WARNING] [job ] Task 3892 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:18,072 [INFO] [scheduler] Resubmitting failed stages: <Stage(2) for <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
2014-05-26 01:30:18,072 [INFO] [scheduler] Got job 8 with 22 tasks: <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>
[ck-hadoop-02.17173.com] 2014-05-26 01:30:18,386 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 121, 570, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/570/121, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:30:18,434 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 42, 573, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/573/42, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:30:18,421 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 121, 39, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/39/121, not found
2014-05-26 01:30:18,537 [WARNING] [job ] task 3947 failed @ ck-hadoop-02.17173.com: <ShuffleTask(3, 121) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 570, 121)>
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:30:18,525 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 137, 265, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/265/137, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:30:18,593 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 42, 70, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/70/42, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:30:18,578 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 12, 134, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/134/12, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:30:18,593 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 12, 70, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/70/12, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:30:18,624 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 2, 548, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/548/2, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:30:18,635 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 309, 265, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/265/309, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:30:18,640 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 42, 104, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/104/42, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:30:18,657 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 396, 123, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/123/396, not found
[ck-hadoop-02.17173.com] 2014-05-26 01:30:18,665 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 303, 570, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/570/303, not found
2014-05-26 01:30:18,714 [WARNING] [job ] task 3945 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(3, 42) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 573, 42)>
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:18,717 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 222, 657, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/657/222, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:30:18,732 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 306, 570, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/570/306, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:18,738 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 404, 4, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/4/404, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:30:18,755 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 218, 623, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/623/218, not found
2014-05-26 01:30:18,794 [WARNING] [job ] task 3949 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(3, 137) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 265, 137)>
2014-05-26 01:30:18,810 [WARNING] [job ] task 3957 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(3, 309) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 265, 309)>
[ck-hadoop-03.17173.com] 2014-05-26 01:30:18,792 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 12, 39, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/39/12, not found
2014-05-26 01:30:18,837 [WARNING] [job ] task 3941 failed @ dpark-51.ctc.internal.17173.com: <ShuffleTask(3, 2) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 548, 2)>
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:30:18,847 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 218, 265, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/265/218, not found
2014-05-26 01:30:18,880 [WARNING] [job ] task 3955 failed @ ck-hadoop-02.17173.com: <ShuffleTask(3, 303) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 570, 303)>
[ck-hadoop-03.17173.com] 2014-05-26 01:30:18,861 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 144, 4, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/4/144, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:30:18,864 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 396, 573, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/573/396, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:30:18,881 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 32, 573, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/573/32, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:18,886 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 51, 39, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/39/51, not found
2014-05-26 01:30:18,923 [WARNING] [job ] task 3953 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(3, 218) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 623, 218)>
2014-05-26 01:30:18,949 [WARNING] [job ] task 3942 failed @ ck-hadoop-03.17173.com: <ShuffleTask(3, 12) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 134, 12)>
[ck-hadoop-01.17173.com] 2014-05-26 01:30:18,927 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 129, 632, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/632/129, not found
2014-05-26 01:30:18,965 [WARNING] [job ] task 3954 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(3, 222) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 657, 222)>
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:18,942 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 51, 657, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/657/51, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:18,951 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 404, 119, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/119/404, not found
2014-05-26 01:30:18,985 [WARNING] [job ] task 3960 failed @ ck-hadoop-04.17173.com: <ShuffleTask(3, 396) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 123, 396)>
[ck-hadoop-04.17173.com] 2014-05-26 01:30:18,977 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 32, 70, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/70/32, not found
2014-05-26 01:30:19,050 [WARNING] [job ] task 3962 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(3, 404) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 4, 404)>
2014-05-26 01:30:19,054 [WARNING] [job ] task 3946 failed @ ck-hadoop-m-78.17173.com: <ShuffleTask(3, 51) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 39, 51)>
2014-05-26 01:30:19,078 [WARNING] [job ] task 3950 failed @ ck-hadoop-03.17173.com: <ShuffleTask(3, 144) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 4, 144)>
[ck-hadoop-01.17173.com] 2014-05-26 01:30:19,062 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 129, 65, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/65/129, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:30:19,150 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 22, 184, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/184/22, not found
2014-05-26 01:30:19,334 [WARNING] [job ] task 3956 failed @ ck-hadoop-01.17173.com: <ShuffleTask(3, 306) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 570, 306)>
2014-05-26 01:30:19,359 [WARNING] [job ] task 3943 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(3, 22) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 184, 22)>
[ck-hadoop-04.17173.com] 2014-05-26 01:30:19,328 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 404, 119, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/119/404, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:30:19,389 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 312, 570, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/570/312, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:30:19,391 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 214, 548, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/548/214, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:30:19,434 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 121, 265, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/265/121, not found
2014-05-26 01:30:19,467 [WARNING] [job ] task 3944 failed @ ck-hadoop-04.17173.com: <ShuffleTask(3, 32) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 573, 32)>
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:30:19,474 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 210, 4, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/4/210, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:30:19,467 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 400, 641, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/641/400, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:30:19,493 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 396, 65, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/65/396, not found
2014-05-26 01:30:19,515 [WARNING] [job ] task 3948 failed @ ck-hadoop-01.17173.com: <ShuffleTask(3, 129) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 632, 129)>
2014-05-26 01:30:19,522 [WARNING] [job ] Task 3962 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:19,576 [INFO] [scheduler] <Stage(1) for <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>> is now unavailable on host http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-04.17173.com] 2014-05-26 01:30:19,546 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 144, 623, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/623/144, not found
2014-05-26 01:30:19,638 [WARNING] [job ] task 3952 failed @ ck-hadoop-04.17173.com: <ShuffleTask(3, 214) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 548, 214)>
[ck-hadoop-01.17173.com] 2014-05-26 01:30:19,634 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 309, 583, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/583/309, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:19,639 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 42, 184, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/184/42, not found
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:30:19,672 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 392, 134, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/134/392, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:19,683 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 32, 70, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/70/32, not found
2014-05-26 01:30:19,712 [WARNING] [job ] Task 3947 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:19,685 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 32, 123, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/123/32, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:30:19,711 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 396, 170, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/170/396, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:30:19,716 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 309, 209, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/209/309, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:19,746 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 22, 641, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/641/22, not found
2014-05-26 01:30:19,777 [WARNING] [job ] Task 3944 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:19,778 [WARNING] [job ] Task 3960 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-02.17173.com] 2014-05-26 01:30:19,763 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 51, 123, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/123/51, not found
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:19,774 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 137, 209, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/209/137, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:30:19,782 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 222, 641, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/641/222, not found
2014-05-26 01:30:19,842 [WARNING] [job ] task 3958 failed @ ck-hadoop-03.17173.com: <ShuffleTask(3, 312) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 570, 312)>
2014-05-26 01:30:19,855 [WARNING] [job ] task 3961 failed @ dpark-76.cnc.internal.17173.com: <ShuffleTask(3, 400) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 641, 400)>
2014-05-26 01:30:19,866 [WARNING] [job ] task 3951 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(3, 210) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 4, 210)>
2014-05-26 01:30:19,882 [WARNING] [job ] task 3959 failed @ dpark-75.cnc.internal.17173.com: <ShuffleTask(3, 392) of <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>>
<FetchFailed(http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844, 1, 134, 392)>
2014-05-26 01:30:19,882 [WARNING] [job ] Task 3943 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:19,956 [WARNING] [job ] Task 3950 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-02.17173.com] 2014-05-26 01:30:19,950 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 51, 216, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/216/51, not found
2014-05-26 01:30:20,020 [WARNING] [job ] Task 3957 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:20,020 [WARNING] [job ] Task 3949 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:20,021 [WARNING] [job ] Task 3945 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-03.17173.com] 2014-05-26 01:30:20,004 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 303, 184, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/184/303, not found
2014-05-26 01:30:20,053 [WARNING] [job ] Task 3954 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:20,085 [WARNING] [job ] Task 3946 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-01.17173.com] 2014-05-26 01:30:20,120 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 400, 583, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/583/400, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:30:20,127 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 306, 265, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/265/306, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:30:20,203 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 12, 164, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/164/12, not found
2014-05-26 01:30:20,249 [WARNING] [job ] Task 3955 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[dpark-75.cnc.internal.17173.com] 2014-05-26 01:30:20,243 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 214, 170, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/170/214, not found
[ck-hadoop-01.17173.com] 2014-05-26 01:30:20,246 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 400, 184, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/184/400, not found
[dpark-51.ctc.internal.17173.com] 2014-05-26 01:30:20,283 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 306, 570, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/570/306, not found
[dpark-76.cnc.internal.17173.com] 2014-05-26 01:30:20,288 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 12, 657, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/657/12, not found
2014-05-26 01:30:20,378 [WARNING] [job ] Task 3961 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:20,379 [WARNING] [job ] Task 3942 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:20,382 [WARNING] [job ] Task 3956 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:20,398 [WARNING] [job ] Task 3952 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-04.17173.com] 2014-05-26 01:30:20,422 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 2, 184, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/184/2, not found
[ck-hadoop-04.17173.com] 2014-05-26 01:30:20,492 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 2, 632, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/632/2, not found
[ck-hadoop-03.17173.com] 2014-05-26 01:30:20,593 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 218, 632, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/632/218, not found
2014-05-26 01:30:20,713 [WARNING] [job ] Task 3941 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
[ck-hadoop-m-78.17173.com] 2014-05-26 01:30:20,716 [WARNING] [shuffle ] Fetch failed for shuffle 1, reduce 312, 398, http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844/1/398/312, not found
2014-05-26 01:30:20,878 [WARNING] [job ] Task 3953 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:30:20,910 [WARNING] [job ] Task 3958 was Lost due to fetch failure from http://ck-hadoop-05.17173.com:5056/20140526-001821-log-analysis.17173.com-29844
2014-05-26 01:31:21,012 [INFO] [scheduler] Resubmitting failed stages: <Stage(1) for <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>
2014-05-26 01:31:21,012 [INFO] [scheduler] Got job 9 with 24 tasks: <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>
2014-05-26 01:31:27,288 [INFO] [job ] Job 9 finished in 6.3s: min=3.7s, avg=4.7s, max=6.2s, maxtry=1
2014-05-26 01:31:27,324 [INFO] [scheduler] Got job 10 with 22 tasks: <MappedRDD <FilteredRDD <MappedRDD <ShuffledRDD <FlatMappedRDD <UnionRDD 1 <UnionRDD 24 <GZipFileRDD /mfs/logs/out-2014052500.log.gz> ...> ...>>>>>>
2014-05-26 01:31:41,135 [INFO] [job ] Job 10 finished in 13.8s: min=10.9s, avg=12.4s, max=13.4s, maxtry=1
2014-05-26 02:01:41,431 [INFO] [scheduler] stop mesos scheduler after 1800 seconds idle
2014-05-26 09:24:01,063 [ERROR] [context ] got signal 15, exit now
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment