Hadoop之Yarn集群部署

需要在Hadoop之HDFS高可用集群部署有的基础。

集群部署情况

HOST	JN	NN	SNN	DN	ZKFC	ZK	RM	NM
node01	#	#	-	-	#	-	-	-
node02	#	#	#	#	#	#	#	-
node03	#	-	-	#	-	#	#	-
node04	-	-	-	#	-	#	#	-

修改配置

修改mapred-site.xml

在其中一台修改，通过scp复制到其他服务器。

cd $HADOOP_HOME/etc/hadoop
vim mapred-site.xml
# 添加如下配置
<property>
	<name>mapreduce.framework.name</name>
	<value>yarn</value>
</property>

修改yarn-site.xml

在其中一台修改，通过scp复制到其他服务器。

cd $HADOOP_HOME/etc/hadoop
vim yarn-site.xml
# 添加如下配置
		    <property>
			<name>yarn.nodemanager.aux-services</name>
			<value>mapreduce_shuffle</value>
		    </property>

		 <property>
		   <name>yarn.resourcemanager.ha.enabled</name>
		   <value>true</value>
		 </property>
		 <property>
		   <name>yarn.resourcemanager.zk-address</name>
		   <value>node02:2181,node03:2181,node04:2181</value>
		 </property>

		 <property>
		   <name>yarn.resourcemanager.cluster-id</name>
		   <value>yan</value>
		 </property>

		 <property>
		   <name>yarn.resourcemanager.ha.rm-ids</name>
		   <value>rm1,rm2</value>
		 </property>
		 <property>
		   <name>yarn.resourcemanager.hostname.rm1</name>
		   <value>node03</value>
		 </property>
		 <property>
		   <name>yarn.resourcemanager.hostname.rm2</name>
		   <value>node04</value>
		 </property>

启动

本次在node01

start-yarn.sh

在node03和node04

yarn-daemon.sh start resourcemanager

检查

找一台有ZK的服务器

zkCli.sh
[zk: localhost:2181(CONNECTED) 1] ls /
[hadoop-ha, yarn-leader-election, zookeeper]
[zk: localhost:2181(CONNECTED) 3] ls /yarn-leader-election
[yan]
[zk: localhost:2181(CONNECTED) 5] ls /yarn-leader-election/yan
[ActiveBreadCrumb, ActiveStandbyElectorLock]
[zk: localhost:2181(CONNECTED) 6] get /yarn-leader-election/yan/ActiveStandbyElectorLock

yanrm1