YARN Fair Scheduler配置

启用Fair Scheduler

在${HADOOP_HOME}/etc/hadoop/yarn-site.xml文件中添加如下配置：

   <property>
       <name>yarn.resourcemanager.scheduler.class</name>
       <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
   </property>

队列配置

进入${HADOOP_HOME}/etc/hadoop/目录，新建队列配置文件fair-scheduler.xml，在文件fair-scheduler.xml中做如下队列配置。（此处只是列举一个简单的例子，大家可以根据自己的业务场景来配置）

<?xml version="1.0"?>
<allocations>
    <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>

    <queue name="prod">
        <weight>40</weight>
    </queue>

    <queue name="dev">
        <weight>60</weight>
    </queue>

    <queuePlacementPolicy>
        <rule name="specified" create="false" />
        <rule name="primaryGroup" create="false" />
        <rule name="default" queue="dev" />
    </queuePlacementPolicy>
</allocations>

做好队列配置后，在${HADOOP_HOME}/etc/hadoop/yarn-site.xml文件中添加如下配置：

   <property>
       <name>yarn.scheduler.fair.allocation.file</name>
       <value>/usr/local/hadoop-3.1.0/etc/hadoop/fair-scheduler.xml</value>
   </property>
   <property>
       <name>yarn.scheduler.fair.preemption</name>
       <value>true</value>
   </property>

YARN集群每个节点都做同样的配置，最后重启YARN集群即可。

参考

https://hadoop.apache.org/docs/r3.1.2/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
https://www.cnblogs.com/guochunguang/p/6877518.html
https://yq.aliyun.com/articles/433892