配置hadoop 使用fair scheduler调度器

hadoop版本为cloudera hadoop cdh3u3

配置步骤为

1.  将$HADOOP_HOME/contrib/fairscheduler/hadoop-fairscheduler-0.20.2-cdh3u3.jar拷贝到$HADOOP_HOME/lib文件夹中

2.  修改$HADOOP_HOME/conf/mapred-site.xml配置文件

 <property>
    <name>mapred.jobtracker.taskScheduler</name>
    <value>org.apache.hadoop.mapred.FairScheduler</value>
  </property>
  <property>
      <name>mapred.fairscheduler.allocation.file</name>
      <value>/home/hadoop/hadoop-0.20.2-cdh3u3/conf/fair-scheduler.xml</value>
   </property>

   <property>
     <name>mapred.fairscheduler.preemption</name>
     <value>true</value>
   </property>

   <property>
     <name>mapred.fairscheduler.assignmultiple</name>
     <value>true</value>
   </property>

   <property>
     <name>mapred.fairscheduler.poolnameproperty</name>
     <value>mapred.queue.name</value>
     <description>job.set("mapred.queue.name",pool); // pool is set to either 'high' or 'low' </description>
   </property>

   <property>
     <name>mapred.fairscheduler.preemption.only.log</name>
     <value>true</value>
   </property>

   <property>
     <name>mapred.fairscheduler.preemption.interval</name>
     <value>15000</value>
   </property>
   
   <property>
     <name>mapred.queue.names</name>
     <value>default,hadoop,hive</value>
   </property>

3.  在$HADOOP_HOME/conf/新建配置文件fair-scheduler.xml

<?xml version="1.0"?>
<allocations>
<pool name="hive">
  <minMaps>90</minMaps>
  <minReduces>20</minReduces>
  <maxRunningJobs>20</maxRunningJobs>
  <weight>2.0</weight>
  <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
</pool>

<pool name="hadoop">
  <minMaps>9</minMaps>
  <minReduces>2</minReduces>
  <maxRunningJobs>20</maxRunningJobs>
  <weight>1.0</weight>
  <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
</pool>

<user name="hadoop">
    <maxRunningJobs>6</maxRunningJobs>
</user>
<poolMaxJobsDefault>10</poolMaxJobsDefault>
<userMaxJobsDefault>8</userMaxJobsDefault>
<defaultMinSharePreemptionTimeout>600</defaultMinSharePreemptionTimeout>
<fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
</allocations>

4.  在集群的各个节点执行以上步骤,然后重启集群,在http://namenode:50030/scheduler  即可查看到调度器运行状态,如果修改调度器配置的话,只需要修改文件fair-scheduler.xml ,不需重启配置即可生效。

5. 在执行hive任务时,设置hive属于的队列set mapred.job.queue.name=hive;

##########

另外,如果在执行MR JOB的时候出现XX用户访问不了YY队列的话,就需要在mapred-queue-acls.xml里配置相应的属性,来对访问权限进行控制,比如:

<property>
  <name>mapred.queue.default.acl-submit-job</name>
  <value>*</value>
  <description> Comma separated list of user and group names that are allowed
    to submit jobs to the 'default' queue. The user list and the group list
    are separated by a blank. For e.g. user1,user2 group1,group2.
    If set to the special value '*', it means all users are allowed to
    submit jobs. If set to ' '(i.e. space), no user will be allowed to submit
    jobs.

    It is only used if authorization is enabled in Map/Reduce by setting the
    configuration property mapred.acls.enabled to true.

    Irrespective of this ACL configuration, the user who started the cluster and
    cluster administrators configured via
    mapreduce.cluster.administrators can submit jobs.
  </description>
</property>

<property>
  <name>mapred.queue.default.acl-administer-jobs</name>
  <value>*</value>
  <description> Comma separated list of user and group names that are allowed
    to view job details, kill jobs or modify job's priority for all the jobs
    in the 'default' queue. The user list and the group list
    are separated by a blank. For e.g. user1,user2 group1,group2.
    If set to the special value '*', it means all users are allowed to do
    this operation. If set to ' '(i.e. space), no user will be allowed to do
    this operation.

    It is only used if authorization is enabled in Map/Reduce by setting the
    configuration property mapred.acls.enabled to true.

    Irrespective of this ACL configuration, the user who started the cluster and
    cluster administrators configured via
    mapreduce.cluster.administrators can do the above operations on all the jobs
    in all the queues. The job owner can do all the above operations on his/her
    job irrespective of this ACL configuration.
  </description>
</property>


评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值