Hadoop（Hadoop优化）

为什么80%的码农都做不了架构师？>>>   

Hadoop(Cloudera manager) Map/Reduce （Hadoop优化）
yarn.nodemanager.resource.memory-mb 100G
yarn.scheduler.minimum-allocation-mb 9G
yarn.scheduler.maximum-allocation-mb 100G
mapreduce.map.memory.mb              9G
mapreduce.reduce.memory.mb          18G
mapreduce.task.timeout                      60分钟
mapreduce.map.java.opts.max.heap  7G
mapreduce.reduce.java.opts.max.heap 14G

<?xml version="1.0"?>
<configuration>
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default,dayandmonthfirst,dayandmonthsecond,dayandmonththird</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.capacity</name>
    <value>100</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>25</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dayandmonthfirst.capacity</name>
    <value>25</value>
  </property>
 <property>
    <name>yarn.scheduler.capacity.root.dayandmonthfirst.maximum-capacity</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dayandmonthfirst.state</name>
    <value>RUNNING</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dayandmonthsecond.capacity</name>
    <value>25</value>
  </property>
 <property>
    <name>yarn.scheduler.capacity.root.dayandmonthsecond.maximum-capacity</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dayandmonthsecond.state</name>
    <value>RUNNING</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dayandmonththird.capacity</name>
    <value>25</value>
  </property>
 <property>
    <name>yarn.scheduler.capacity.root.dayandmonththird.maximum-capacity</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dayandmonththird.state</name>
    <value>RUNNING</value>
  </property>
</configuration>

Ambari queue
yarn.scheduler.capacity.maximum-am-resource-percent=0.2yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_administer_jobs=*
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=40
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=2.5
yarn.scheduler.capacity.root.jywz.acl_administer_jobs=*
yarn.scheduler.capacity.root.jywz.acl_submit_applications=*
yarn.scheduler.capacity.root.jywz.capacity=30
yarn.scheduler.capacity.root.jywz.maximum-capacity=100
yarn.scheduler.capacity.root.jywz.state=RUNNING
yarn.scheduler.capacity.root.jywz.user-limit-factor=3.3
yarn.scheduler.capacity.root.queues=default,jywz,wz111
yarn.scheduler.capacity.root.wz111.acl_administer_jobs=*
yarn.scheduler.capacity.root.wz111.acl_submit_applications=*
yarn.scheduler.capacity.root.wz111.capacity=30
yarn.scheduler.capacity.root.wz111.maximum-capacity=100
yarn.scheduler.capacity.root.wz111.state=RUNNING
yarn.scheduler.capacity.root.wz111.user-limit-factor=3.3

yarn.scheduler.capacity.root.wz111.user-limit-factor=3.3

硬盘5T

参考链接
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
http://tech.uc.cn/?p=1438
http://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

转载于:https://my.oschina.net/skyim/blog/465426