配置hadoop 使用fair scheduler调度器

hadoop版本为cloudera hadoop cdh3u3

配置步骤为

1. 将$HADOOP_HOME/contrib/fairscheduler/hadoop-fairscheduler-0.20.2-cdh3u3.jar拷贝到$HADOOP_HOME/lib文件夹中

2. 修改$HADOOP_HOME/conf/mapred-site.xml配置文件

[html] view plain copy

1. <property>
2. <name>mapred.jobtracker.taskScheduler</name>
3. <value>org.apache.hadoop.mapred.FairScheduler</value>
4. </property>
5. <property>
6. <name>mapred.fairscheduler.allocation.file</name>
7. <value>/home/hadoop/hadoop-0.20.2-cdh3u3/conf/fair-scheduler.xml</value>
8. </property>
10. <property>
11. <name>mapred.fairscheduler.preemption</name>
12. <value>true</value>
13. </property>
15. <property>
16. <name>mapred.fairscheduler.assignmultiple</name>
17. <value>true</value>
18. </property>
20. <property>
21. <name>mapred.fairscheduler.poolnameproperty</name>
22. <value>mapred.job.queue.name</value>
23. <description>job.set("mapred.job.queue.name",pool);</description>
24. </property>
26. <property>
27. <name>mapred.fairscheduler.preemption.only.log</name>
28. <value>true</value>
29. </property>
31. <property>
32. <name>mapred.fairscheduler.preemption.interval</name>
33. <value>15000</value>
34. </property>
36. <property>
37. <name>mapred.queue.names</name>
38. <value>default,hadoop,hive</value>
39. </property>

3. 在$HADOOP_HOME/conf/新建配置文件fair-scheduler.xml

[html] view plain copy

1. <?xmlversion="1.0"?>
2. <allocations>
3. <poolname="hive">
4. <minMaps>90</minMaps>
5. <minReduces>20</minReduces>
6. <maxRunningJobs>20</maxRunningJobs>
7. <weight>2.0</weight>
8. <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
9. </pool>
11. <poolname="hadoop">
12. <minMaps>9</minMaps>
13. <minReduces>2</minReduces>
14. <maxRunningJobs>20</maxRunningJobs>
15. <weight>1.0</weight>
16. <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
17. </pool>
19. <username="hadoop">
20. <maxRunningJobs>6</maxRunningJobs>
21. </user>
22. <poolMaxJobsDefault>10</poolMaxJobsDefault>
23. <userMaxJobsDefault>8</userMaxJobsDefault>
24. <defaultMinSharePreemptionTimeout>600</defaultMinSharePreemptionTimeout>
25. <fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
26. </allocations>

4. 在集群的各个节点执行以上步骤，然后重启集群，在http://namenode:50030/scheduler 即可查看到调度器运行状态，如果修改调度器配置的话，只需要修改文件fair-scheduler.xml ，不需重启配置即可生效。

5. 在执行hive任务时，设置hive属于的队列set mapred.job.queue.name=hive;

##########

另外，如果在执行MR JOB的时候出现XX用户访问不了YY队列的话，就需要在mapred-queue-acls.xml里配置相应的属性，来对访问权限进行控制，比如：

[html] view plain copy

1. <property>
2. <name>mapred.queue.default.acl-submit-job</name>
3. <value>*</value>
4. <description>Commaseparatedlistofuserandgroupnamesthatareallowed
5. tosubmitjobstothe'default'queue.Theuserlistandthegrouplist
6. areseparatedbyablank.Fore.g.user1,user2group1,group2.
7. Ifsettothespecialvalue'*',itmeansallusersareallowedto
8. submitjobs.Ifsetto''(i.e.space),nouserwillbeallowedtosubmit
9. jobs.
11. ItisonlyusedifauthorizationisenabledinMap/Reducebysettingthe
12. configurationpropertymapred.acls.enabledtotrue.
14. IrrespectiveofthisACLconfiguration,theuserwhostartedtheclusterand
15. clusteradministratorsconfiguredvia
16. mapreduce.cluster.administratorscansubmitjobs.
17. </description>
18. </property>
20. <property>
21. <name>mapred.queue.default.acl-administer-jobs</name>
22. <value>*</value>
23. <description>Commaseparatedlistofuserandgroupnamesthatareallowed
24. toviewjobdetails,killjobsormodifyjob'spriorityforallthejobs
25. inthe'default'queue.Theuserlistandthegrouplist
26. areseparatedbyablank.Fore.g.user1,user2group1,group2.
27. Ifsettothespecialvalue'*',itmeansallusersareallowedtodo
28. thisoperation.Ifsetto''(i.e.space),nouserwillbeallowedtodo
29. thisoperation.
31. ItisonlyusedifauthorizationisenabledinMap/Reducebysettingthe
32. configurationpropertymapred.acls.enabledtotrue.
34. IrrespectiveofthisACLconfiguration,theuserwhostartedtheclusterand
35. clusteradministratorsconfiguredvia
36. mapreduce.cluster.administratorscandotheaboveoperationsonallthejobs
37. inallthequeues.Thejobownercandoalltheaboveoperationsonhis/her
38. jobirrespectiveofthisACLconfiguration.
39. </description>
40. </property>