Yarn资源分配示例

Yarn资源分配示例

1 简介

此手册应用于yarn资源按照队列分配的情景。

2 准备

2.1 环境说明

hadoop版本: hadoop-2.5.0-cdh5.3.3

3 配置

3.1 配置说明

1)  Yarn-site.xml里配置yanr能用到的资源总数以及公平调度策略所在路径

#在虚拟机关机情况下，选中后点击设置，在控制器SATA处点击右边的加号以创建虚拟硬盘。

 <property>

<name>yarn.nodemanager.resource.memory-mb</name>

<value>32760</value>

</property>

 

<property>

<name>yarn.scheduler.minimum-allocation-mb</name>

<value>2048</value>

</property>

 

<property>

<name>yarn.scheduler.maximum-allocation-mb</name>

<value>16384</value>

</property>

 

<property>

<name>yarn.nodemanager.resource.cpu-vcores</name>

<value>21</value>

</property>

 

 

 

<property>

         <name>yarn.scheduler.fair.allow-undeclared-pools</name>

         <value>false</value>

         <description>如果这是true,新的队列可以在提交申请时被创建,falsessh使用默认队列</description>

</property>

<property>

         <name>yarn.scheduler.fair.user-as-default-queue</name>

         <value>false</value>

         <description>使用跟用户名相同的队列名,如果设置为false或者未设置,则所有未知队列的应用程序将被提交到default队列中，默认值为true</description>

</property>

<property>

         <name>yarn.resourcemanager.scheduler.class</name>

         <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>

         <description>公平配置队列配置在fair-scheduler</description>

</property>

<property>

         <name>yarn.scheduler.fair.allocation.file</name>

         <value>/home/hadoop/hadoop-2.7.1/etc/hadoop/fair-allocation.xml</value>

         <description>公平配置文件路径</description>

</property>

<property>

         <name>yarn.scheduler.fair.preemption</name>

         <value>true</value>

         <description>是否支持抢占,默认值为false</description>

</property>

<property>

         <name>yarn.scheduler.fair.sizebasedweight</name>

         <value>false</value>

         <description>是否启用按应用程序资源需求分配资源,默认值为false即采用公平轮询的方法分配资源</description>

</property>

<property>

         <name>yarn.scheduler.increment-allocation-mb</name>

         <value>1024</value>

         <description>仅fair有效,内存规整化单位,墨认值1024.(示例一个container请求1.5G,则调度器规整化为2G)</description>

</property>

2)  配置队列的资源限额及应用数量

#队列可指定到用户或则用户组,如果是用户组则需要在aclSubmitApps参数前加空格。

<allocations>

         <queuename="root">

         <aclSubmitApps>hadoop,dpi</aclSubmitApps>

         <aclAdministerApps>hadoop</aclAdministerApps>

                   <queuename="default">

                            <minResources>2000mb,1vcores</minResources>

                            <maxResources>8000mb,8vcores</maxResources>

                            <maxRunningApps>40</maxRunningApps>

                            <schedulingMode>fair</schedulingMode>

                            <aclSubmitApps>*</aclSubmitApps>

                            <aclAdministerApps>hadoop</aclAdministerApps>

                   </queue>

                   <queuename="test">

                            <minResources>2000mb,10vcores</minResources>

                            <maxResources>8000mb,8vcores</maxResources>

                            <maxRunningApps>4</maxRunningApps>

                            <schedulingMode>fair</schedulingMode>

                            <aclSubmitApps>test,hadoop</aclSubmitApps>

                            <aclAdministerApps>hadoop</aclAdministerApps>

                   </queue>

                   <queuename="hadoop">

                            <minResources>2000mb,1vcores</minResources>

                            <maxResources>8000mb,8vcores</maxResources>

                            <maxRunningApps>4</maxRunningApps>

                            <schedulingMode>fair</schedulingMode>

                            <aclSubmitApps>hadoop</aclSubmitApps>

                            <aclAdministerApps>hadoop</aclAdministerApps>

                   </queue>

         </queue>

         <username="hadoop"> <!-- 对于特定用户的配置 -->

          <maxRunningApps>100</maxRunningApps> 

         </user> 

         <userMaxAppsDefault>50</userMaxAppsDefault><!--默认的用户最多可以同时运行的任务 -->

         <queuePlacementPolicy>

                   <rulename="user" create="false" />

                   <rulename="specified" />

                   <rulename="default"/>

         </queuePlacementPolicy>

</allocations>

3.2配置问题

暂无.

#修改队列信息后会动态刷新，以yarn 8088端口里的资源为准。

4 使用

set mapreduce.job.queuename为某个队列到8088端口查看任务是否提交到该队列.

5 总结

暂无.