hadoop namenode及yarn resourcemanager HA配置

环境信息

三台虚拟机节点（192.168.100.171<debian171>， 192.168.100.172<debian172>， 192.168.100.173<debian173>）

Debian jessie 8.5 

Hadoop 2.6.4

zookeeper集群配置参见“搭建zookeeper集群（Debian 8）”

ssh免密配置参见“Hadoop安装（Debian 8）”

hadoop配置

core-site.xml

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://nxhadoop</value>
</property>
<property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/hadoop-2.6.4/tmp</value>
        <description>Abase for other temporary directories.</description>
</property>

<property>
        <name>ha.zookeeper.quorum</name>
        <value>debian171:2181,debian172:2181,debian173:2181</value>
</property>
<property>
        <name>ha.zookeeper.session-timeout.ms</name>
        <value>6000000</value>
</property>
<property>
        <name>ha.failover-controller.cli-check.rpc-timeout.ms</name>
        <value>6000000</value>
</property>
<property>
        <name>ipc.client.connect.timeout</name>
        <value>6000000</value>
</property>

hdfs-site.xml

<property>
        <name>dfs.nameservices</name>
        <value>nxhadoop</value>
</property>
<property>
        <name>dfs.ha.namenodes.nxhadoop</name>
        <value>nn1,nn2</value>
</property>
<property>
        <name>dfs.namenode.rpc-address.nxhadoop.nn1</name>
        <value>debian171:9000</value>
</property>
<property>
        <name>dfs.namenode.http-address.nxhadoop.nn1</name>
        <value>debian171:50070</value>
</property>
<property>
        <name>dfs.namenode.rpc-address.nxhadoop.nn2</name>
        <value>debian172:9000</value>
</property>
<property>
        <name>dfs.namenode.http-address.nxhadoop.nn2</name>
        <value>debian172:50070</value>
</property>

<property>
        <name>dfs.namenode.servicerpc-address.nxhadoop.nn1</name>
        <value>debian171:53310</value>
</property>
<property>
        <name>dfs.namenode.servicerpc-address.nxhadoop.nn2</name>
        <value>debian172:53310</value>
</property>

<property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://debian171:8485;debian172:8485;debian173:8485/nxhadoop</value>
</property>
<property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/usr/local/hadoop/hadoop-2.6.4/journal</value>
</property>
<property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
</property>
<property>
        <name>dfs.client.failover.proxy.provider.nxhadoop</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
</property>
<property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/home/surfin/.ssh/id_dsa</value>
</property>
<property>
        <name>dfs.ha.fencing.ssh.connect-timeout</name>
        <value>30000</value>
</property>
<property>
        <name>dfs.namenode.handler.count</name>
        <value>8</value>
</property>
<property>
        <name>dfs.qjournal.start-segment.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.prepare-recovery.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.accept-recovery.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.prepare-recovery.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.accept-recovery.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.finalize-segment.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.select-input-streams.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.get-journal-state.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.new-epoch.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>dfs.qjournal.write-txns.timeout.ms</name>
        <value>600000000</value>
</property>
<property>
        <name>ha.zookeeper.session-timeout.ms</name>
        <value>6000000</value>
</property>

<property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>debian171:50090</value>
</property>
<property>
        <name>dfs.replication</name>
        <value>3</value>
</property>
<property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/hadoop-2.6.4/tmp/dfs/name</value>
</property>
<property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/hadoop-2.6.4/tmp/dfs/data</value>
</property>

mapred-site.xml

<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>debian171:10020</value>
</property>
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>debian171:19888</value>
</property>

yarn-site.xml

<property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
</property>
<property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
</property>
<property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
</property>
<property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>debian171:2181,debian172:2181,debian173:2181</value>
</property>
<property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>nxyarn-ha</value>
</property>
<property>
        <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
        <value>true</value>
</property>
<property>
        <name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>
        <value>/yarn-leader-election</value>
</property>
<property>
        <name>yarn.client.failover-proxy-provider</name>
        <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
</property>
<property>
        <name>yarn.resourcemanager.address.rm1</name>
        <value>debian171:8132</value>
</property>


<property>
        <name>yarn.resourcemanager.address.rm2</name>
        <value>debian172:8132</value>
</property>
<property>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>debian171:8130</value>
</property>
<property>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>
        <value>debian172:8130</value>
</property>
<property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
        <value>debian171:8131</value>
</property>
<property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
        <value>debian172:8131</value>
</property>
<property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>debian171:8188</value>
</property>
<property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>debian172:8188</value>
</property>

<property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>debian171</value>
</property>
<property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>debian172</value>
</property>
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>

这里注意：yarn.client.failover-proxy-provider的value为org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider，有些2.2.x的教程使用org.apache.hadoop.yarn.client.RMFailoverProxyProvider，但RMFailoverProxyProvider是个接口类，可能是2.2.x与2.6.4不一样，暂时还没有验证，大家可以看下2.2.x的源码

slaves

debian171
debian172
debian173

启动zookeeper，hadoop，yarn进程

#启动zookeeper（debian171，debian172，debian173）

zkServer.sh start

#格式化zookeeper（仅需第一次）

bin/hdfs zkfc -formatZK

#启动zkfc进程（debian171，debian172）

sbin/hadoop-daemon.sh start zkfc

#启动journalnode进程（debian171，debian172，debian173）

sbin/hadoop-daemon.sh start journalnode

#格式化hdfs

bin/hdfs namenode -format

如果之前非HA环境已经格式化了，可以执行下面脚本，否则会报“Journal Storage Directory not formatted”错误

bin/hdfs namenode -initializeSharedEdits

#启动主节点namenode进程（debian171）

sbin/hadoop-daemon.sh start namenode

#启动备节点同步NN元数据信息（debian172）

bin/hdfs namenode -bootstrapStandby

#启动备节点namenode进程（debian172）

sbin/hadoop-daemon.sh start namenode

#启动所有节点的datanode进程（在debian171上执行）

sbin/hadoop-daemon.sh start datanode

#查看NN状态（http://192.168.100.171（172）:50070/dfshealth.jsp）

可以看到171上状态为active，172上为standby

 

#启动yarn的resourcemanager（debian171）

sbin/yarn-daemons.sh start resourcemanager

#检查resourcemanager状态

surfin@debian172:/usr/local/hadoop/hadoop-2.6.4$ bin/yarn rmadmin -getServiceState rm1
standby
surfin@debian172:/usr/local/hadoop/hadoop-2.6.4$ bin/yarn rmadmin -getServiceState rm2
active

#启动nodemanager（debian171）

sbin/yarn-daemons.sh start nodemanager

注：resourcemanager和nodemanager有些教程上说也可以用start-yarn.sh启动，我这边没有成功

遇到的问题

#如果重新执行hdfs namenode -format后，启动datanode可能会出现“Initialization failed for Block pool <registering> (Datanode Uuid unassigned)”的报错，主要为format namenode 后cluster_id变更导致

修改HADOOP_FILE_SYSTEM/namenode/current/VERSION和HADOOP_FILE_SYSTEM/datanode/current/VERSION文件中的Cluster_ID（保证一致）

然后格式化namenode

$ hdfs namenode -format

参考文档

http://blog.csdn.net/u010967382/article/details/30976935

http://debugo.com/yarn-rm-ha/

http://www.cnblogs.com/liuchangchun/p/4629205.html

转载于:https://my.oschina.net/norxiva/blog/719629