Hadoop HA MR1配置

<property>
  <name>mapreduce.framework.name</name>
  <value>classic</value>
  <description>The runtime framework for executing MapReduce jobs.
  Can be one of local, classic or yarn.
  </description>
</property>

  <!--property>
    <name>mapred.ha.jobtracker.id</name>
    <value>jt1</value>
  </property-->
--------------------------------关键配置如下-----------------------------
  <property>
    <name>mapred.job.tracker</name>
    <value>logicaljt</value>
    <!-- host:port string is replaced with a logical name -->
  </property>

  <property>
    <name>mapred.jobtrackers.logicaljt</name>
    <value>jt1,jt2</value>
    <description>Comma-separated list of JobTracker IDs.</description>
  </property>

    <property>
    <name>mapred.jobtracker.rpc-address.logicaljt.jt1</name>
    <!-- RPC address for jt1 -->
    <value>namenode01:48021</value>
  </property>

  <property>
    <name>mapred.jobtracker.rpc-address.logicaljt.jt2</name>
    <!-- RPC address for jt2 -->
    <value>namenode02:48021</value>
  </property>
 <property>
    <name>mapred.job.tracker.http.address.logicaljt.jt1</name>
    <!-- HTTP bind address for jt1 -->
    <value>0.0.0.0:50030</value>
  </property>

  <property>
    <name>mapred.job.tracker.http.address.logicaljt.jt2</name>
    <!-- HTTP bind address for jt2 -->
    <value>0.0.0.0:50031</value>
  </property>

  <property>
    <name>mapred.ha.jobtracker.rpc-address.logicaljt.jt1</name>
    <!-- RPC address for jt1 HA daemon -->
    <value>namenode01:48023</value>
  </property>

  <property>
    <name>mapred.ha.jobtracker.rpc-address.logicaljt.jt2</name>
    <!-- RPC address for jt2 HA daemon -->
    <value>namenode02:48023</value>
  </property>

  <property>
    <name>mapred.ha.jobtracker.http-redirect-address.logicaljt.jt1</name>
    <!-- HTTP redirect address for jt1 -->

    <value>namenode01:50030</value>

  </property>

  <property>
    <name>mapred.ha.jobtracker.http-redirect-address.logicaljt.jt2</name>
    <!-- HTTP redirect address for jt2 -->
    <value>namenode02:50030</value>
  </property>

 <property>
    <name>mapred.jobtracker.restart.recover</name>
    <value>true</value>
  </property>

  <property>
    <name>mapred.job.tracker.persist.jobstatus.active</name>
    <value>true</value>

  </property>

----------------------------

MR1启动的时候需要起两个shell。一个是ha-start-mapred.sh，两外一个是start_mrzkfc.sh

ha-start-mapred.sh启动的核心是两个namenode启动 hadoop jobtrackerha，各个datanode启动hadoop tasktracker。

start_mrzkfc.sh启动的核心是两个namenode上启动hadoop mrzkfc

ha-start-mapred.sh

---------------------------

#!/bin/sh
set -x
exec 3<$HOME/mapred/conf/masters
while read -u3 nn
do
echo "-----------$nn start jobtracker-------------"
ssh $nn "setsid $HOME/mapred/bin/hadoop jobtrackerha >>/home/users/mapred/logs/hadoop-jobtracker-$nn-`date +"%F"`.log 2>&1 &"
done
exec 3<&-

exec 3<$HOME/mapred/conf/slaves
while read -u3 dn
do
echo "-----------$dn start tasktracker----------"
ssh $dn "setsid $HOME/mapred/bin/hadoop tasktracker >>/home/users/mapred/logs/hadoop-tasktracker-$dn-`date +"%F"`.log 2>&1 &"
done
exec 3<&-

------------其中HOME/mapred为HADOOP_MAPRED_HOME也就是MR1的目录-------------

start_mrzkfc.sh

----------------------------

#!/bin/sh
set -x
exec 3<$HOME/mapred/conf/masters
while read -u3 mrzkfc
do
echo "-----------$mrzkfc start mrzkfc-------------"
ssh $mrzkfc "setsid $HOME/mapred/bin/hadoop mrzkfc >>$HOME/mapred/logs/hadoop-ocnosql_cdh42-mrzkfc-$mrzkfc-`date +"%F"`.log 2>&1 &"
done
exec 3<&-

如果只启动ha-start-mapred.sh，datanode的tasktracker会报错如下：

ilovers (15) exceeded maximum allowed (15)
java.net.ConnectException: Call From datanode08/10.1.1.242 to namenode02:48021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

配置成功之后，各个datanode的tasktracker会注册到namenode01的JobTrackerHADaemon进程上。

假设namenode1的jobtracker是active状态，如果kil掉这台JT时，namenode02的jobtacker则会由standby会变成active；重新启动namenode01的JT的话，状态不会自动切换为以前的状态，namenode02的JT继续保持active，而namenode01的JT则是standby状态。