Hadoop (Cloudera manager) Map/Reduce (Hadoop 优化)
yarn.nodemanager.resource.memory-mb 100G
yarn.scheduler.minimum-allocation-mb 9G
yarn.scheduler.maximum-allocation-mb 100G
mapreduce.map.memory.mb 9G
mapreduce.reduce.memory.mb 18G
mapreduce.task.timeout 60 分钟
mapreduce.map.java.opts.max.heap 7G
mapreduce.reduce.java.opts.max.heap 14G
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>default,dayandmonthfirst,dayandmonthsecond,dayandmonththird</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.state</name>
<value>RUNNING</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonthfirst.capacity</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonthfirst.maximum-capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonthfirst.state</name>
<value>RUNNING</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonthsecond.capacity</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonthsecond.maximum-capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonthsecond.state</name>
<value>RUNNING</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonththird.capacity</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonththird.maximum-capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.dayandmonththird.state</name>
<value>RUNNING</value>
</property>
</configuration>
Ambari queue
yarn.scheduler.capacity.maximum-am-resource-percent=0.2yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_administer_jobs=*
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=40
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=2.5
yarn.scheduler.capacity.root.jywz.acl_administer_jobs=*
yarn.scheduler.capacity.root.jywz.acl_submit_applications=*
yarn.scheduler.capacity.root.jywz.capacity=30
yarn.scheduler.capacity.root.jywz.maximum-capacity=100
yarn.scheduler.capacity.root.jywz.state=RUNNING
yarn.scheduler.capacity.root.jywz.user-limit-factor=3.3
yarn.scheduler.capacity.root.queues=default,jywz,wz111
yarn.scheduler.capacity.root.wz111.acl_administer_jobs=*
yarn.scheduler.capacity.root.wz111.acl_submit_applications=*
yarn.scheduler.capacity.root.wz111.capacity=30
yarn.scheduler.capacity.root.wz111.maximum-capacity=100
yarn.scheduler.capacity.root.wz111.state=RUNNING
yarn.scheduler.capacity.root.wz111.user-limit-factor=3.3
yarn.scheduler.capacity.root.wz111.user-limit-factor=3.3
参考链接
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
http://tech.uc.cn/?p=1438
http://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html