Hadoop HA高可用故障转移（hdfs+yarn）

常用集群搭建脚本

• xsync.sh      文件或文件夹同步
• xcall.sh        在所有集群节点执行命令
• xnode.sh     在非master节点执行命令
• xenv.sh        配置环境变量辅助脚本

将脚本放在/usr/bin/或创建软连接 并授予执行权限

HDFS HA

配置文件

core-site.xml

<configuration>
	<property>
		<name>fs.defaultFS</name>
		<value>hdfs://mycluster</value>
		<description>集群nameservice</description>
	</property>

	<property>
		<name>hadoop.tmp.dir</name>
		<value>/opt/module/tmp/ha/hadoop</value>
		<description>临时文件夹</description>
	</property>

	<property>
		<name>io.file.buffer.size</name>
		<value>131072</value>
		<final>4096</final>
		<description>流文件的缓冲区为4K</description>
	</property>

	<property>
		<name>hadoop.http.staticuser.user</name>
		<value>zakza</value>
		<description>用户</description>
	</property>

	<property>
		<name>ha.zookeeper.quorum</name>
		<value>master:2181,node1:2181,node2:2181</value>
	</property>
	
</configuration>

hdfs-site.xml

<configuration>
	<property>
		<name>dfs.nameservices</name>
		<value>mycluster</value>
		<description>名称服务</description>
	</property>

	<property>
		<name>dfs.ha.namenodes.mycluster</name>
		<value>nn1,nn2,nn3</value>
		<description>namenode 集群</description>
	</property>

	<!-- rpc服务 -->
	<property>
		<name>dfs.namenode.rpc-address.mycluster.nn1</name>
		<value>master:9820</value>
	</property>

	<property>
		<name>dfs.namenode.rpc-address.mycluster.nn2</name>
		<value>node1:9820</value>
	</property>

	<property>
		<name>dfs.namenode.rpc-address.mycluster.nn3</name>
		<value>node2:9820</value>
	</property>
	<!-- http服务 -->
	<property>
		<name>dfs.namenode.http-address.mycluster.nn1</name>
		<value>master:9870</value>
	</property>

	<property>
		<name>dfs.namenode.http-address.mycluster.nn2</name>
		<value>node1:9870</value>
	</property>

	<property>
		<name>dfs.namenode.http-address.mycluster.nn3</name>
		<value>node2:9870</value>
	</property>

	<!-- 共享edits目录 -->
	<property>
		<name>dfs.namenode.shared.edits.dir</name>
		<value>qjournal://master:8485;node1:8485;node2:8485/mycluster</value>
	</property>

	<!-- 判断活动namenode -->

	<property>
		<name>dfs.client.failover.proxy.provider.mycluster</name>
		<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
	</property>

	<property>
		<name>dfs.ha.fencing.methods</name>
		<value>sshfence</value>
	</property>

	<property>
		<name>dfs.ha.fencing.ssh.private-key-files</name>
		<value>~/.ssh/id_rsa</value>
	</property>

	<property>
		<name>dfs.journalnode.edits.dir</name>
		<value>${hadoop.tmp.dir}/journalnode/</value>
	</property>

	<property>
		<name>dfs.ha.automatic-failover.enabled</name>
		<value>true</value>
	</property>
</configuration>

xsync 分发同步文件

初始化启动

启动QJM集群

hdfs --workers --daemon start

任一节点格式化namenode

hdfs namenode -format

hdfs --daemon start namenode

其他节点执行命令同步信息

xnode "hdfs namenode -bootstrapStandby"

xnode "hdfs --daemon start namenode"

xcall "hdfs –-daemon start datanode"

启动jobhistory（master）

mapred --daemon start historyserver

格式化zk

stop-dfs.sh

hdfs zkfc -formatZK

start-dfs.sh

Yarn高可用配置

yarn-site.xml

<configuration>

	<property>
		<name>yarn.log-aggregation-enable</name>
		<value>true</value>
		<description>日志聚集功能</description>
	</property>

	<property>
		<name>yarn.log.server.url</name>
		<value>http://master:19888/jobhistory/logs</value>
		<description>日志web地址</description>
	</property>

	<!-- Site specific YARN configuration properties -->
	<!--指定MR走shuffle-->
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>

	<!--环境变量继承-->
	<property>
		<name>yarn.nodemanager.env-whitelist</name>
		<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,HADOOP_MAPRED_HOME</value>
		<description>环境变量继承</description>
	</property>

	<property>
		<name>yarn.log-aggregation.retain-seconds</name>
		<value>604800</value>
		<description>日志聚集功能</description>
	</property>

	<!--启用resourcemanager ha-->
	<property>
		<name>yarn.resourcemanager.ha.enabled</name>
		<value>true</value>
	</property>

	<!--声明两台resourcemanager的地址-->
	<property>
		<name>yarn.resourcemanager.cluster-id</name>
		<value>cluster-yarn1</value>
	</property>

	<property>
		<name>yarn.resourcemanager.ha.rm-ids</name>
		<value>rm1,rm2</value>
	</property>

	<property>
		<name>yarn.resourcemanager.hostname.rm1</name>
		<value>master</value>
	</property>

	<property>
		<name>yarn.resourcemanager.hostname.rm2</name>
		<value>node1</value>
	</property>

	<!--指定zookeeper集群的地址--> 
	<property>
		<name>yarn.resourcemanager.zk-address</name>
		<value>master:2181,node1:2181,node2:2181</value>
	</property>

	<!--启用自动恢复--> 
	<property>
		<name>yarn.resourcemanager.recovery.enabled</name>
		<value>true</value>
	</property>

	<!--指定resourcemanager的状态信息存储在zookeeper集群--> 
	<property>
		<name>yarn.resourcemanager.store.class</name>
		<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
	</property>

</configuration>

xsync分发同步配置文件

start-yarn.sh

查看所有启动服务

xcall jps 

参考

Apache Hadoop 3.3.1 – HDFS High Availability Using the Quorum Journal Manager

Apache Hadoop 3.3.1 – ResourceManager High Availability

配置文件查询参考

core-default.xmlhttps://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml

hdfs-default.xmlhttps://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/hdfs-default.xmlmapred-default.xmlhttps://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xmlyarn-default.xmlhttps://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-common/yarn-default.xml