环境:
环境: ubuntu 12.04, jdk 1.7
用户名都是:nachuang
域名分别是:mcw-cc-nachuang, mcw-cc-node
注:确定 Hadoop 2.0.3 集群可以正常使用,然后stop-all。
Kerberos配置:
Hadoop 配置:
hadoop-env.sh,hadoopcore-site.xml和hdfs-site.xml配置参考“Hadoop(1.0.4)和Kerberos配置”:
mapred-site.xml:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.job.tracker</name>
<value>hdfs:/mcw-cc-nachuang:9001</value>
<final>true</final>
</property>
<property>
<name>mapred.job.map.memory.mb</name>
<value>1700</value>
</property>
<property>
<name>mapred.job.reduce.memory.mb</name>
<value>1700</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1400m</value>
</property>
<property>
<name>mapreduce.task.io.sort.mb</name>
<value>400</value>
</property>
<property>
<name>mapreduce.task.io.sort.factor</name>
<value>10</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>file:/home/nachuang/Workspace/Hadoop/hadoop-2.0.3-alpha/mapred/system</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>file:/home/nachuang/Workspace/Hadoop/hadoop-2.0.3-alpha/mapred/local</value>
<final>true</final>
</property>
<property>
<name>mapred.reduce.slowstart.completed.maps</name>
<value>1</value>
</property>
<property>
<name>mapreduce.jobtracker.kerberos.principal</name>
<value>nachuang/_HOST@hdfs.server</value>
</property>
<property>
<name>mapreduce.jobtracker.kerberos.https.principal</name>
<value>host/_HOST@hdfs.server</value>
</property>
<property>
<name>mapreduce.jobtracker.keytab.file</name>
<value>/home/nachuang/Workspace/Hadoop/hadoop-2.0.3-alpha/etc/nachuang.keytab</value>
</property>
<property>
<name>mapreduce.tasktracker.kerberos.principal</name>
<value>nachuang/_HOST@hdfs.server</value>
</property>
<property>
<name>mapreduce.tasktracker.kerberos.https.principal</name>
<value>host/_HOST@hdfs.server</value>
</property>
<property>
<name>mapreduce.tasktracker.keytab.file</name>
<value>/home/nachuang/Workspace/Hadoop/hadoop-2.0.3-alpha/etc/nachuang.keytab</value>
</property>
<property>
<name>mapred.task.tracker.task-controller</name>
<value>org.apache.hadoop.mapred.DefaultTaskController</value>
</property>
<property>
<name>mapreduce.tasktracker.group</name>
<value>nachuang</value>
</property>
</configuration>
yarn-site.xml:
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.resourcemanager.address</name>
<value>mcw-cc-nachuang:9080</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>mcw-cc-nachuang:9081</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>mcw-cc-nachuang:9082</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>6</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>10240</value>
</property>
<!-- ResourceManager security configs -->
<property>
<name>yarn.resourcemanager.keytab</name>
<value>/home/nachuang/Workspace/Hadoop/hadoop-2.0.3-alpha/etc/nachuang.keytab</value>
</property>
<property>
<name>yarn.resourcemanager.principal</name>
<value>nachuang/_HOST@hdfs.server</value>
</property>
<!-- NodeManager security configs -->
<property>
<name>yarn.nodemanager.keytab</name>
<value>/home/nachuang/Workspace/Hadoop/hadoop-2.0.3-alpha/etc/nachuang.keytab</value>
</property>
<property>
<name>yarn.nodemanager.principal</name>
<value>nachuang/_HOST@hdfs.server</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
yarn-env.sh:
export HADOOP_YARN_USER=nachuang
hadoop-policy.xml:
${HADOOP_YARN_USER} 改为 *
验证:
启动Hadoop:
sbin/start-all.sh
sudo sbin/start-secure-dns.sh(注意: datanode要用sudo执行)