Hadoop作业优先级控制

最新推荐文章于 2024-06-30 21:23:38 发布

caiandyong

最新推荐文章于 2024-06-30 21:23:38 发布

阅读量580

点赞数

分类专栏： hadoop 文章标签： Hadoop作业优先级控制控制Hadoop作业优先级 Hadoop

本文链接：https://blog.csdn.net/caiandyong/article/details/46276591

版权

hadoop 专栏收录该内容

33 篇文章 0 订阅

订阅专栏

当同时在集群中运行多个作业时，默认情况下，Hadoop将提交的作业放入一个FIFO,一个作业结束后，Hadoop就启动下一个作业。
当一个运行时间长但是优先级较低的作业先于运行时间短而优先级较高的作业提交时，优先级高的作业会长时间排队等待。
为了解决这个问题，Hadoop定义了5种不同的作业优先级，分别是：VERY_HIGH,HIGH,NORMAL,LOW,VERY_LOW,作业的默认优先级是NORMAL，可以通过$hadoop job -set-priority进行修改。

例子：
1.在集群中启动1个运行时间较长的作业
caiyong@caiyong:/opt/hadoop$ bin/hadoop jar hadoop-examples-1.2.1.jar pi 2000 2000

2.查看作业列表
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -list

1 jobs currently running
JobId                 State   StartTime UserName   Priority   SchedulingInfo
job_201503171201_0003   1   1426565671593   caiyong    NORMAL              NA

3.查看作业的运行状态
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -status job_201503171201_0003

Job: job_201503171201_0003
file: hdfs://127.0.0.1:8020/home/caiyong/tmp/mapred/staging/caiyong/.staging/job_201503171201_0003/job.xml
tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201503171201_0003
map() completion: 0.012500001
reduce() completion: 0.0

Counters: 19
   Job Counters
       SLOTS_MILLIS_MAPS=117080
       Launched map tasks=26
       Data-local map tasks=26
   File Input Format Counters
       Bytes Read=2832
   FileSystemCounters
       HDFS_BYTES_READ=5870
       FILE_BYTES_WRITTEN=1316654
   Map-Reduce Framework
       Map output materialized bytes=672
       Map input records=24
       Spilled Records=48
       Map output bytes=432
       Total committed heap usage (bytes)=3815768064
       CPU time spent (ms)=9530
       Map input bytes=576
       SPLIT_RAW_BYTES=3038
       Combine input records=0
       Combine output records=0
       Physical memory (bytes) snapshot=4156928000
       Virtual memory (bytes) snapshot=9500446720
       Map output records=48

4.把作业的优先级提高为VERY_HIGH
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -set-priority job_201503171201_0003 VERY_HIGH

Changed job priority.

5.查看更改后的作业优先级
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -list

1 jobs currently running
JobId                                     State   StartTime              UserName   Priority   SchedulingInfo
job_201503171201_0003   1   1426565671593   caiyong    VERY_HIGH   NA

6.强制结束正在运行的作业
caiyong@caiyong:/opt/hadoop$ bin/hadoop job -kill job_201503171201_0003

Killed job job_201503171201_0003