引言
以前写了一个关于
Hadoop2.0 NN HA实验记录 博客,其中记录了hadoop中NameNode出现故障时可以自动切换主备节点,现在hadoop2.5.0版本中实现了RescoureMangager可以自动切换功能,因此下面写的博客是关于 Hadoop RM HA 。
正文
四个节点:SY-0217 SY-0355
SY-0225
SY-0226
SY-0217 : NN RM(主)
SY-0355:NN RM (备)
SY-0225,SY-0226:DN NM (数据节点)
第一步:需要准备好关于hadoop相关的安装包
例如 有如下的安装包,以我的集群为例:
Hadoop:hadoop-2.5.0-cdh5.2.0.tar.gz
Zookeeper:zookeeper-1.4.5-cdh5.2.0.tar.gz
Jdk:jdk-7u71-linux-i586.tar.gz (
CDH5不再支持JDK1.6,必须按照JDK1.7.(1.7.0_45)且只有64位包,无32位包)
假如你需要关于Hadoop CDH版本的安装包,其中包括 Hbase,hive,sqoop,mathout等安装包,你可以去下面地址去下载:
如果有问题,你也可去CDH官网去查找问题,因为上面又比较完善的官网文档做参考。
第二步:安装Hadoop CDH版本
把安装包上传到服务器或者虚拟机上面,并解压。
这里我是采用
以前安装版本hadoop2.2.0的虚拟机,并且主备NameNode出现故障时可以自动切换,在上面解压后,并
修改环境变量就OK了(修改环境变量就再重复了)。
如果你要从零开始的话,你需要去下面链接博客去学习搭建新的集群:
Apache Hadoop 2.2.0 HDFS HA +YARN多机部署
博客地址:
博客地址:
到这里准备工作都做好了,下面开始配置hadoop文件。
第三步:配置$HADOOP_HOME/etc/hadoop/core-stie.xml文件
按照如下配置(
和我以前Hadoop NN HA 中的文件配置一致):
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://fulonghadoop</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/kkzhangtao/hadoopcdh/hdfs/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>SY-0217:2181,SY-0355:2181,SY-0225:2181</value>
</property>
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
<description>The FileSystem for hdfs: uris.</description>
</property>
</configuration>
第四步:配置$HADOOP_HOME/etc/hadoop/hdfs-stie.xml文件
按照如下配置(
和我以前Hadoop NN HA 中的文件配置一致, 但是需要修改标记的地方):
<configuration>
<property>
<name>dfs.nameservices</name>
<value>fulonghadoop</value>
<description>
Comma-separated list of nameservices.
as same as fs.defaultFS in core-site.xml.
</description>
</property>
<property>
<name>dfs.ha.namenodes.fulonghadoop</name>
<value>nn1,nn2</value>
<description>
The prefix for a given nameservice, contains a comma-separated
list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE).
</description>
</property>
<property>
<name>dfs.namenode.rpc-address.fulonghadoop.nn1</name>
<value>SY-0217:8020</value>
<description>
RPC address for nomenode1 of hadoop-test
</description>
</property>
<property>
<name>dfs.namenode.rpc-address.fulonghadoop.nn2</name>
<value>SY-0355:8020</value>
<description>
RPC address for nomenode2 of hadoop-test
</description>
</property>
<property>
<name>dfs.namenode.http-address.fulonghadoop.nn1</name>
<value>SY-0217:50070</value>
<description>
The address and the base port where the dfs namenode1 web ui will listen on.
</description>
</property>
<property>
<name>dfs.namenode.http-address.fulonghadoop.nn2</name>
<value>SY-0355:50070</value>
<description>
The address and the base port where the dfs namenode2 web ui will listen on.
</description>
</property>
<property>
<name>dfs.namenode.servicerpc-address.fulonghadoop.n1</name>
<value>SY-0217:53310</value>
</property>
<property>
<name>dfs.namenode.servicerpc-address.fulonghadoop.n2</name>
<value>SY-0355:53310</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/kkzhangtao/hadoop-2.5.0-cdh5.2.0/hdfs/name</value>
<description>Determines where on the local filesystem the DFS name node
should store the name table(fsimage). If this is a comma-delimited list
of directories then the name table is replicated in all of the
directories, for redundancy. </description>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://SY-0355:8485;SY-0225:8485;SY-0226:8485/hadoop-test</value>
<description>A directory on shared storage between the multiple namenodes
in an HA cluster. This directory will be written by the active and read
by the standby in order to keep the namespaces synchronized. This directory
does not need to be listed in dfs.namenode.edits.dir above. It should be
left empty in a non-HA cluster.
</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/kkzhangtao/hadoop-2.5.0-cdh5.2.0/hdfs/data</value>
<description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited
list of directories, then data will be stored in all named
directories, typically on different devices.
Directories that do not exist are ignored.
</description>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
<description>
Whether automatic failover is enabled. See the HDFS High
Availability documentation for details on automatic HA
configuration.
</description>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/kkzhangtao/hadoop-2.5.0-cdh5.2.0/hdfs/journal/</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.fulonghadoop</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
<description>Configure the name of the Java class which will be used by the DFS Client to determine which NameNode is the current Active, and therefore which NameNode is currently serving client requests.
</description>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
<description>how to communicate in the switch process</description>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/kkzhangtao/.ssh/id_rsa</value>
<description>the location stored ssh key</description>
</property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>1000</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>8</value>
</property>
</configuration>
第五步:配置$HADOOP_HOME/etc/hadoop/mapred-stie.xml文件
按照如下配置(
和我以前Hadoop NN HA 中的文件配置一致):
<configuration>
<!-- MR YARN Application properties -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>The runtime framework for executing MapReduce jobs.
Can be one of local, classic or yarn.
</description>
</property>
<!-- jobhistory properties -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>SY-0355:10020</value>
<description>MapReduce JobHistory Server IPC host:port</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>SY-0355:19888</value>
<description>MapReduce JobHistory Server Web UI host:port</description>
</property>
</configuration>
第六步:配置$HADOOP_HOME/etc/hadoop/yarn-stie.xml文件
按照如下配置(
需要重新配置
):
<configuration>
<!-- nodemanager 配置 ================================================= -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<description>Address where the localizer IPC is.</description>
<name>yarn.nodemanager.localizer.address</name>
<value>0.0.0.0:23344</value>
</property>
<property>
<description>NM Webapp address.</description>
<name>yarn.nodemanager.webapp.address</name>
<value>0.0.0.0:23999</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/kkzhangtao/RMHA/nodemanager/yarn/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/kkzhangtao/RMHA/nodemanager/yarn/logs</value>
</property>
<property>
<name>mapreduce.shuffle.port</name>
<value>23080</value>
</property>
<!-- HA 配置 =============================================================== -->
<!-- Resource Manager Configs -->
<property>
<name>yarn.resourcemanager.connect.retry-interval.ms</name>
<value>2000</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 使嵌入式自动故障转移。HA环境启动,与 ZKRMStateStore 配合 处理fencing -->
<property>
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
<value>true</value>
</property>
<!-- 集群名称,确保HA选举时对应的集群 -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-cluster</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 这里RM主备结点需要单独指定,(可选)-->
<property>
<name>yarn.resourcemanager.ha.id</name>
<value>rm1</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
<value>5000</value>
</property>
<!-- ZKRMStateStore 配置 -->
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property><name>yarn.resourcemanager.zk-address</name>
<value>SY-0217:2181,SY-0355:2181,SY-0225:2181</value>
</property>
<property>
<name>yarn.resourcemanager.zk.state-store.address</name>
<value>SY-0217:2181,SY-0355:2181,SY-0225:2181</value>
</property>
<!-- Client访问RM的RPC地址 (applications manager interface) -->
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>SY-0217:23140</value>
</property>
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>SY-0355:23140</value>
</property>
<!-- AM访问RM的RPC地址(scheduler interface) -->
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>SY-0217:23130</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>SY-0355:23130</value>
</property>
<!-- RM admin interface -->
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>SY-0217:23141</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>SY-0355:23141</value>
</property>
<!--NM访问RM的RPC端口 -->
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>SY-0217:23125</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>SY-0355:23125</value>
</property>
<!-- RM web application 地址 -->
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>SY-0217:23188</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>SY-0355:23188</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm1</name>
<value>SY-0217:23189</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address.rm2</name>
<value>SY-0355:23189</value>
</property>
</configuration>
第七步:配置$HADOOP_HOME/etc/hadoop/slaves,hadoop-env.sh,farscheduler.xml文件
slaves:
SY-0226
SY-0225
hadoop-env.sh:
export JAVA_HOME=/home/kkzhangtao/jdk1.7.0_71
farscheduler.xml:
本集群中的配置文件拷贝以前的Hadoop2.2.0的 你也可以自己配置。
第八步:开始测试
下面是我按照测试需求写的启动和关闭脚本:
Startup.sh
#!/bin/bash
echo "-------------------------Step1----------------------------"
echo "Now,starting Zookeeper."
ssh SY-0217 $ZOOKEEPER_HOME/bin/zkServer.sh start
ssh SY-0355 $ZOOKEEPER_HOME/bin/zkServer.sh start
ssh SY-0225 $ZOOKEEPER_HOME/bin/zkServer.sh start
echo "Now,Zookeeper is started."
echo "-------------------------Step2----------------------------"
echo "Now,starting ZKFC.
ssh SY-0217 $HADOOP_HOME/bin/hdfs zkfc -formatZK
"
ssh SY-0217 $HADOOP_HOME/sbin/hadoop-daemon.sh start zkfc
ssh SY-0355 $HADOOP_HOME/sbin/hadoop-daemon.sh start zkfc
echo "Now,ZKFC started."
echo "-------------------------Step3----------------------------"
echo "Now,starting JN."
ssh SY-0355 $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode
ssh SY-0225 $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode
ssh SY-0226 $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode
echo "Now,JN started."
echo "-------------------------Step4----------------------------"
echo "Now,starting Active,Standby NN."
ssh SY-0217 $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
ssh SY-0355 $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
echo "Now,Active,Standby NN started."
echo "-------------------------Step5----------------------------"
echo "Now,starting DataNodes."
ssh SY-0217 $HADOOP_HOME/sbin/hadoop-daemons.sh start datanode
echo "Now,DataNodes started."
echo "-------------------------Step6----------------------------"
echo "Now,starting Active,Standby RM."
ssh SY-0217 $HADOOP_HOME/sbin/yarn-daemon.sh start resourcemanager
ssh SY-0355 $HADOOP_HOME/sbin/yarn-daemon.sh start resourcemanager
echo "Now,Active,Standby RM started."
echo "-------------------------Step7----------------------------"
echo "Now,starting Nodemanagers."
ssh SY-0217 $HADOOP_HOME/sbin/yarn-daemons.sh start nodemanager
echo "Now,Nodemanagers started."
echo "-------------------------Step8----------------------------"
echo "Now,starting MR JobHistory Server."
ssh SY-0355 $HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver
echo "Now,MR JobHistory Server started."
StopAll.sh
#!/bin/bash
echo "-------------------------Step1----------------------------"
echo "Now,stop JN."
ssh SY-0355 $HADOOP_HOME/sbin/hadoop-daemon.sh stop journalnode
ssh SY-0225 $HADOOP_HOME/sbin/hadoop-daemon.sh stop journalnode
ssh SY-0226 $HADOOP_HOME/sbin/hadoop-daemon.sh stop journalnode
echo "Now,JN stoped."
echo "-------------------------Step2----------------------------"
echo "Now,stoping Active,Standby NN."
ssh SY-0217 $HADOOP_HOME/sbin/hadoop-daemon.sh stop namenode
ssh SY-0355 $HADOOP_HOME/sbin/hadoop-daemon.sh stop namenode
echo "Now,Active,Standby NN stoped."
echo "-------------------------Step3----------------------------"
echo "Now,stop DataNodes."
ssh SY-0217 $HADOOP_HOME/sbin/hadoop-daemons.sh stop datanode
echo "Now,DataNodes stoped."
echo "-------------------------Step4----------------------------"
echo "Now,stop Active,Standby RM."
ssh SY-0217 $HADOOP_HOME/sbin/yarn-daemon.sh stop resourcemanager
ssh SY-0355 $HADOOP_HOME/sbin/yarn-daemon.sh stop resourcemanager
echo "Now,Active,Standby RM stoped."
echo "-------------------------Step5----------------------------"
echo "Now,stop NodeManagers."
ssh SY-0217 $HADOOP_HOME/sbin/yarn-daemons.sh stop nodemanager
echo "Now,NodeManagers stoped."
echo "-------------------------Step6----------------------------"
echo "Now,stop MR JobHistory Server."
ssh SY-0355 $HADOOP_HOME/sbin/mr-jobhistory-daemon.sh stop historyserver
echo "Now,MR stop Server stoped."
第九步:测试结果截图
1.通过在终端采用JSP命令查看,或者通过访问页面来查看。
采用JSP命令查看SY-0217和SY-0355截图
:
采用访问页面查看RescourceManager的截图:(
SY-0217:23188 这里端口号是自己修改的 在yarn-site.xml配置中)
SY-0217访问页面:
SY-0355访问页面:
在hadoop集群中 采用如下命令来运行一个MapReduce程序(
我这里是自己写的一个MapReduce程序,并打好Jar包上传到HDFS上面 ,同时数据也准备好了)。
在HADOOP_HOME目录下面执行: hadoop jar /home/kkzhangtao/Videos/WordCount.jar WordCount /input /output
你也可跑hadoop自带的测试程序 : hadoop jar share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.5.0-cdh5.2.0.jar pi 2 2
通过
Kill -9 14671 命令来杀死
主RescourceManager。
再通过访问监控页面 SY-0355:23188 来查看任务进度,最后并完成 ,效果截图如下:
SY-0217访问页面:
SY-0355访问页面:
在运行的集群中跑个mapreduce程序来测试RM的切换时候保证任务不会中断,并能得到执行结果。
到此RescourceManager主备切换已经成功完成。
第十步:实验总结和目的
本实验的目的主要是来测试hadoop cdh (5.2.0)版本中的RescourceManager的主备切换。次要目的是了解CDH的版本的好处,以后变用于开发中去。
同时这里也涉及到了Hadoop 版本升级的问题,因此下片博客将会写到hadoop 版本之间升级问题和解决hadoop集群因增加节点导致的数据部均衡的问题。
如果有需要请关注............谢谢