hadoop capacity schduler 的初步配置了解

先简单了解容量调度器

Capacity Scheduler支持以下特性：

(1) 计算能力保证。支持多个队列，某个作业可被提交到某一个队列中。每个队列会配置一定比例的计算资源，且所有提交到队列中的作业共享该队列中的资源。

(2) 灵活性。空闲资源会被分配给那些未达到资源使用上限的队列，当某个未达到资源的队列需要资源时，一旦出现空闲资源资源，便会分配给他们。

(3) 支持优先级。队列支持作业优先级调度（默认是FIFO）

(4) 多重租赁。综合考虑多种约束防止单个作业、用户或者队列独占队列或者集群中的资源。

(5) 基于资源的调度。 支持资源密集型作业，允许作业使用的资源量高于默认值，进而可容纳不同资源需求的作业。不过，当前仅支持内存资源的调度。

1.首先在yarn-site.xml配置调度器类型：

<configuration>

<!-- Site specific YARN configuration properties -->
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>master</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>

        </property>

       <property>

                 <name>yarn.resourcemanager.scheduler.class</name>
                  <value>org.apache.hadoop.yarn.serve.resourcemanager.scheduler.capacity.CapacityScheduler</value>

        <property>

        <property>
                <name>mapreduce.application.classpath</name>
                <value>/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*</value>
        </property>



</configuration>

其中下面的代码段为此项配置中增添的我们所需要的容量调度的代码：

 <property>

              <name>yarn.resourcemanager.scheduler.class</name>
              <value>org.apache.hadoop.yarn.serve.resourcemanager.scheduler.capacity.CapacityScheduler</value>

  <property>

2.在Capacity Scheduler专属配置文件capacity-scheduler.xml下进行配置，首先来看下这项配置里面的一些常用的配置的相应作用：

<configuration>

  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>//同时处于运行和等待状态的应用程序最大数目
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>//集群中可用于AM的资源比例上限
    <value>0.1</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run 
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare //用于比较调度器中的资源的ResourceCalculator实现。
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
      DominantResourceCalculator uses dominant-resource to compare //默认即DefaultResourceCalculator只使用内存而DominantResourceCalculator使用dominant-resource比较多维资源如内存、CPU等。
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default</value>
    <description>
      The queues at the this level (root is the root queue).//root队列的子队列
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>100</value>
    <description>Default queue target capacity.</description>//default队列的资源容量，总量为100
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.//用户可使用的资源限制
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue. //default队列可使用的资源上限
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.//default队列的状态可以是running或者stopped
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.//限制那些用户可以向default队列提交应用程序，*即任意

    </description>
 </property>
 <property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name> 

<description> The ACL of who can administer jobs on the default queue.//限制那些用户可以向管理default队列提交应用程序

      </description>
  </property>
<property>

<name>yarn.scheduler.capacity.root.default.acl_application_max_priority</name> <value>*</value> <description> The ACL of who can submit applications with configured priority. //谁可以提交具有配置优先级的应用程序的ACL。 For e.g, [user={name} group={name} max_priority={priority} default_priority={priority}] </description> </property> <property> <name>yarn.scheduler.capacity.root.default.maximum-application-lifetime </name> <value>-1</value> <description> Maximum lifetime of an application which is submitted to a queue in seconds. Any value less than or equal to zero will be considered as disabled.//在秒内提交给队列的应用程序的最大生命周期。任何小于或等于零的值都将被视为禁用。 This will be a hard time limit for all applications in this queue. If positive value is configured then any application submitted to this queue will be killed after exceeds the configured lifetime. User can also specify lifetime per application basis in application submission context. But user lifetime will be overridden if it exceeds queue maximum lifetime. It is point-in-time configuration.
//对于这个队列中的所有应用程序来说，他的时间都是很困难的。如果配置了正值，那么提交到该队列的任何应用程序将在超过配置的生命周期后被杀死。
//用户还可以在应用程序上下文中指定应用程序的生命周期。
//但是如果超过了队列最大生命周期，用户的生命周期将被重写。它是时间点配置。
     Note : Configuring too low value will result in killing application sooner. This feature is applicable only for leaf queue. </description> </property> <property> <name>yarn.scheduler.capacity.root.default.default-application-lifetime </name> <value>-1</value> <description> Default lifetime of an application which is submitted to a queue in seconds. Any value less than or equal to zero will be considered as disabled.//应用程序的默认生存期，该应用程序在秒内提交给队列。任何小于或等于零的值将被视为禁用。 If the user has not submitted application with lifetime value then this value will be taken. It is point-in-time configuration. Note : Default lifetime can't exceed maximum lifetime. This feature is applicable only for leaf queue.
//如果用户没有提交具有生命值的应用程序，则该值将被执行。它是时间点配置。注意:默认的生命周期不能超过最大生命周期。这个特性是仅适用于叶子队列。
 </description> </property> <property> <name>yarn.scheduler.capacity.node-locality-delay</name> <value>40</value> <description> Number of missed scheduling opportunities after which the CapacityScheduler attempts to schedule rack-local containers. When setting this parameter, the size of the cluster should be taken into account. We use 40 as the default value, which is approximately the number of nodes in one rack.
//在容量调度器尝试调度本地容器后错过调度机会的数量。在设置此参数时，应该考虑到集群的大小，我们使用40作为默认值，它大约是一个机架中的节点数
 </description> </property> <property> <name>yarn.scheduler.capacity.rack-locality-additional-delay</name> <value>-1</value> <description> Number of additional missed scheduling opportunities over the node-locality-delay ones, after which the CapacityScheduler attempts to schedule off-switch containers, instead of rack-local ones.
//在节点-区域-延迟时间内错过了调度机会的额外数量，在此之后，容量调度器试图安排关闭开关容器，而不是rack-local。
     Example: with node-locality-delay=40 and rack-locality-delay=20, the scheduler will attempt rack-local assignments after 40 missed opportunities, and off-switch assignments after 40+20=60 missed opportunities. When setting this parameter, the size of the cluster should be taken into account. We use -1 as the default value, which disables this feature. In this case, the number of missed opportunities for assigning off-switch containers is calculated based on the number of containers and unique locations specified in the resource request, as well as the size of the cluster.
//在nodelocal- delay=40和rack-local- delay=20的情况下，调度器将在40次错过机会后尝试rack-local任务，并且在40+20=60错过机会后进行切换任务。在设置此参数时，应该考虑到集群的大小。我们使用-1作为默认值，这将禁用该特性。在这种情况下，根据资源请求中指定的容器数量和特定位置，以及集群的大小，计算出分配off-switch容器的错过机会的数量
 </description> </property> <property> <name>yarn.scheduler.capacity.queue-mappings</name> <value></value> <description> A list of mappings that will be used to assign jobs to queues The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]* Typically this list will be used to map users to queues, for example, u:%user:%user maps all users to queues with the same name as the user.
//将用于分配作业到队列的映射列表。此列表的语法为[u|g]:[name]:[queue_name][，下一个映射]*。通常，该列表将用于将用户映射到队列，例如，u:%user:%user将所有用户映射到与用户同名的队列 </description> </property> <property> <name>yarn.scheduler.capacity.queue-mappings-override.enable</name> <value>false</value> <description>

If a queue mapping is present, will it override the value specified
      by the user? This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false.
// 如果队列映射存在，它会覆盖用户指定的值吗?管理员可以使用此方法将作业放置在与用户指定的队列不同的队列中。默认的是假的。

</description> </property> <property> <name>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments</name> <value>1</value> <description> Controls the number of OFF_SWITCH assignments allowed during a node's heartbeat. Increasing this value can improve scheduling rate for OFF_SWITCH containers. Lower values reduce "clumping" of applications on particular nodes. The default is 1. Legal values are 1-MAX_INT. This config is refreshable.
//控制在节点的心跳期间允许的OFF_SWITCH任务的数量。增加这个值可以提高OFF_SWITCH容器的调度速率。较低的值减少了在特定节点上的应用程序的“聚集”。缺省值是1。1-MAX_INT合法值。这个配置是可刷新的。 </description> </property></configuration>