yarn historyserver 使用解析

 

启动服务:

sbin/mr-jobhistory-daemon.sh start historyserver

 

查询历史页面:

http://202.117.10.25:19888/jobhistory

1. job完成后，历史信息存入hdfs下的mr-history/done/中，

执行命令分析历史信息:

 hadoop job -history   all   hdfs://202.117.10.26:9000/mr-history/done/2013/01/31/000000/job_1359637210567_0001-1359637233266-liuqiang-word+count-1359637716126-16-1-SUCCEEDED-default.jhist>result.txt

  即可。

有时候会报莫名其妙的错误，也可直接：

hadoop job -history all  /mr-history/done/2016/03/11/000000/job_1457687511247_0002-1457687640983-liuqiang2-word+count-1457687272398-15-1-SUCCEEDED-default.jhist 

结果如下:

Hadoop job: job_1361498589930_0003
=====================================
User: liuqiang
JobName: word count
JobConf: hdfs://202.117.10.26:9000/tmp/hadoop-yarn/staging/liuqiang/.staging/job_1361498589930_0003/job.xml
Submitted At: 22-Feb-2013 10:23:51
Launched At: 22-Feb-2013 10:27:42 (3mins, 51sec)
Finished At: 22-Feb-2013 10:32:00 (4mins, 17sec)
Status: SUCCEEDED
Counters:

|Group Name                    |Counter name                  |Map Value |Reduce Value|Total Value|
---------------------------------------------------------------------------------------
|File System Counters          |FILE: Number of bytes read    |66,490,968|18,230,892|84,721,860
|File System Counters          |FILE: Number of bytes written |85,744,950|18,294,425|104,039,375
|File System Counters          |FILE: Number of read operations|0         |0         |0        
|File System Counters          |FILE: Number of large read operations|0         |0         |0        
|File System Counters          |FILE: Number of write operations|0         |0         |0        
|File System Counters          |HDFS: Number of bytes read    |1,035,668,512|0         |1,035,668,512
|File System Counters          |HDFS: Number of bytes written |0         |947,869   |947,869  
|File System Counters          |HDFS: Number of read operations|48        |3         |51       
|File System Counters          |HDFS: Number of large read operations|0         |0         |0        
|File System Counters          |HDFS: Number of write operations|0         |2         |2        
|Job Counters                  |Killed map tasks              |0         |0         |10       
|Job Counters                  |Launched map tasks            |0         |0         |26       
|Job Counters                  |Launched reduce tasks         |0         |0         |1        
|Job Counters                  |Data-local map tasks          |0         |0         |20       
|Job Counters                  |Rack-local map tasks          |0         |0         |6        
|Job Counters                  |Total time spent by all maps in occupied slots (ms)|0         |0         |4,317,327
|Job Counters                  |Total time spent by all reduces in occupied slots (ms)|0         |0         |196,704  
|Map-Reduce Framework          |Map input records             |9,383,921 |0         |9,383,921
|Map-Reduce Framework          |Map output records            |177,316,480|0         |177,316,480
|Map-Reduce Framework          |Map output bytes              |1,753,046,400|0         |1,753,046,400
|Map-Reduce Framework          |Map output materialized bytes |18,230,862|0         |18,230,862
|Map-Reduce Framework          |Input split bytes             |1,952     |0         |1,952    
|Map-Reduce Framework          |Combine input records         |181,455,220|0         |181,455,220
|Map-Reduce Framework          |Combine output records        |5,311,383 |0         |5,311,383
|Map-Reduce Framework          |Reduce input groups           |0         |68,979    |68,979   
|Map-Reduce Framework          |Reduce shuffle bytes          |0         |18,230,862|18,230,862
|Map-Reduce Framework          |Reduce input records          |0         |1,172,643 |1,172,643
|Map-Reduce Framework          |Reduce output records         |0         |68,979    |68,979   
|Map-Reduce Framework          |Spilled Records               |5,449,341 |1,172,643 |6,621,984
|Map-Reduce Framework          |Shuffled Maps                 |0         |16        |16       
|Map-Reduce Framework          |Failed Shuffles               |0         |0         |0        
|Map-Reduce Framework          |Merged Map outputs            |0         |16        |16       
|Map-Reduce Framework          |GC time elapsed (ms)          |260,716   |364       |261,080  
|Map-Reduce Framework          |CPU time spent (ms)           |1,808,840 |7,850     |1,816,690
|Map-Reduce Framework          |Physical memory (bytes) snapshot|3,380,711,424|76,144,640|3,456,856,064
|Map-Reduce Framework          |Virtual memory (bytes) snapshot|6,272,446,464|352,477,184|6,624,923,648
|Map-Reduce Framework          |Total committed heap usage (bytes)|2,952,212,480|40,366,080|2,992,578,560
|Shuffle Errors                |BAD_ID                        |0         |0         |0        
|Shuffle Errors                |CONNECTION                    |0         |0         |0        
|Shuffle Errors                |IO_ERROR                      |0         |0         |0        
|Shuffle Errors                |WRONG_LENGTH                  |0         |0         |0        
|Shuffle Errors                |WRONG_MAP                     |0         |0         |0        
|Shuffle Errors                |WRONG_REDUCE                  |0         |0         |0        
|File Input Format Counters    |Bytes Read                    |1,035,666,560|0         |1,035,666,560
|File Output Format Counters   |Bytes Written                 |0         |947,869   |947,869  
|Job Counters                  |Killed map tasks              |0         |0         |10       
|Job Counters                  |Launched map tasks            |0         |0         |26       
|Job Counters                  |Launched reduce tasks         |0         |0         |1        
|Job Counters                  |Data-local map tasks          |0         |0         |20       
|Job Counters                  |Rack-local map tasks          |0         |0         |6        
|Job Counters                  |Total time spent by all maps in occupied slots (ms)|0         |0         |4,317,327
|Job Counters                  |Total time spent by all reduces in occupied slots (ms)|0         |0         |196,704  
|File System Counters          |FILE: Number of bytes read    |66,490,968|18,230,892|84,721,860
|File System Counters          |FILE: Number of bytes written |85,744,950|18,294,425|104,039,375
|File System Counters          |FILE: Number of read operations|0         |0         |0        
|File System Counters          |FILE: Number of large read operations|0         |0         |0        
|File System Counters          |FILE: Number of write operations|0         |0         |0        
|File System Counters          |HDFS: Number of bytes read    |1,035,668,512|0         |1,035,668,512
|File System Counters          |HDFS: Number of bytes written |0         |947,869   |947,869  
|File System Counters          |HDFS: Number of read operations|48        |3         |51       
|File System Counters          |HDFS: Number of large read operations|0         |0         |0        
|File System Counters          |HDFS: Number of write operations|0         |2         |2        
|Map-Reduce Framework          |Map input records             |9,383,921 |0         |9,383,921
|Map-Reduce Framework          |Map output records            |177,316,480|0         |177,316,480
|Map-Reduce Framework          |Map output bytes              |1,753,046,400|0         |1,753,046,400
|Map-Reduce Framework          |Map output materialized bytes |18,230,862|0         |18,230,862
|Map-Reduce Framework          |Input split bytes             |1,952     |0         |1,952    
|Map-Reduce Framework          |Combine input records         |181,455,220|0         |181,455,220
|Map-Reduce Framework          |Combine output records        |5,311,383 |0         |5,311,383
|Map-Reduce Framework          |Reduce input groups           |0         |68,979    |68,979   
|Map-Reduce Framework          |Reduce shuffle bytes          |0         |18,230,862|18,230,862
|Map-Reduce Framework          |Reduce input records          |0         |1,172,643 |1,172,643
|Map-Reduce Framework          |Reduce output records         |0         |68,979    |68,979   
|Map-Reduce Framework          |Spilled Records               |5,449,341 |1,172,643 |6,621,984
|Map-Reduce Framework          |Shuffled Maps                 |0         |16        |16       
|Map-Reduce Framework          |Failed Shuffles               |0         |0         |0        
|Map-Reduce Framework          |Merged Map outputs            |0         |16        |16       
|Map-Reduce Framework          |GC time elapsed (ms)          |260,716   |364       |261,080  
|Map-Reduce Framework          |CPU time spent (ms)           |1,808,840 |7,850     |1,816,690
|Map-Reduce Framework          |Physical memory (bytes) snapshot|3,380,711,424|76,144,640|3,456,856,064
|Map-Reduce Framework          |Virtual memory (bytes) snapshot|6,272,446,464|352,477,184|6,624,923,648
|Map-Reduce Framework          |Total committed heap usage (bytes)|2,952,212,480|40,366,080|2,992,578,560

=====================================

Task Summary
============================
Kind Total Successful Failed Killed StartTime FinishTime

Setup 0 0  0 0  
Map 26 16  0 10 22-Feb-2013 10:27:45 22-Feb-2013 10:31:53 (4mins, 8sec)
Reduce 1 1  0 0 22-Feb-2013 10:28:43 22-Feb-2013 10:32:00 (3mins, 16sec)
Cleanup 0 0  0 0  
============================

Analysis
=========

Time taken by best performing map task task_1361498589930_0003_m_000015: 55sec
Average time taken by map tasks: 2mins, 58sec
Worse performing map tasks:
TaskId  Timetaken
task_1361498589930_0003_m_000004 4mins, 8sec
task_1361498589930_0003_m_000007 4mins, 7sec
task_1361498589930_0003_m_000013 4mins, 4sec
task_1361498589930_0003_m_000003 4mins, 3sec
task_1361498589930_0003_m_000006 3mins, 24sec
task_1361498589930_0003_m_000012 3mins, 23sec
task_1361498589930_0003_m_000009 3mins, 23sec
task_1361498589930_0003_m_000008 3mins, 21sec
task_1361498589930_0003_m_000005 3mins, 20sec
task_1361498589930_0003_m_000001 3mins, 20sec
The last map task task_1361498589930_0003_m_000004 finished at (relative to the Job launch time): 22-Feb-2013 10:31:53 (4mins, 11sec)

Time taken by best performing shuffle task task_1361498589930_0003_r_000000: 3mins, 10sec
Average time taken by shuffle tasks: 3mins, 10sec
Worse performing shuffle tasks:
TaskId  Timetaken
task_1361498589930_0003_r_000000 3mins, 10sec
The last shuffle task task_1361498589930_0003_r_000000 finished at (relative to the Job launch time): 22-Feb-2013 10:31:54 (4mins, 12sec)

Time taken by best performing reduce task task_1361498589930_0003_r_000000: 5sec
Average time taken by reduce tasks: 5sec
Worse performing reduce tasks:
TaskId  Timetaken
task_1361498589930_0003_r_000000 5sec
The last reduce task task_1361498589930_0003_r_000000 finished at (relative to the Job launch time): 22-Feb-2013 10:32:00 (4mins, 17sec)
=========

KILLED task attempts by nodes
Hostname FailedTasks
===============================
node13 task_1361498589930_0003_m_000007, task_1361498589930_0003_m_000012, task_1361498589930_0003_m_000013,
node14 task_1361498589930_0003_m_000003, task_1361498589930_0003_m_000004,
node21 task_1361498589930_0003_m_000000, task_1361498589930_0003_m_000002, task_1361498589930_0003_m_000010, task_1361498589930_0003_m_000011,
node12 task_1361498589930_0003_m_000006,