Hadoop搭建yarn环境

1、Hadoop搭建yarn环境

	NameNode01	NameNode02	DateNode	ZKFC	ZooKeeper	JournalNode	ResourceManager	Nodemanager
node01	*		*	*	*	*	*	*
node02		*	*	*	*	*		*
node03			*		*	*	*	*

• yarn环境搭建基于前面HA环境
  如果未安装：
  链接: MySQL安装 ：. https://blog.csdn.net/weixin_43660536/article/details/118033294
  链接: ZooKeeper安装：. https://blog.csdn.net/weixin_43660536/article/details/118369963
  链接: hadoop ha安装：. https://blog.csdn.net/weixin_43660536/article/details/118358298

1. 切换工作目录

[root@node01 ~]# cd /opt/yjx/hadoop-3.1.2/etc/hadoop/

2. 修改集群环境

[root@node01 hadoop]# vim hadoop-env.sh

##继续添加用户信息
export JAVA_HOME=/usr/java/jdk1.8.0_231-amd64
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_ZKFC_USER=root
export HDFS_JOURNALNODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

3. 修改配置文件

• [root@node01 hadoop]# vim mapred-site.xml
  

插入到最后< Config ></ config>中

<!-- 指定mr框架为yarn方式 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!-- 指定mapreduce jobhistory地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>node01:10020</value>
</property>
<!-- 任务历史服务器的web地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>node01:19888</value>
</property>
<!-- 配置运行过的日志存放在hdfs上的存放路径 -->
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/history/done</value>
</property>
<!-- 配置正在运行中的日志在hdfs上的存放路径 -->
<property>
<name>mapreudce.jobhistory.intermediate.done-dir</name>
<value>/history/done/done_intermediate</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>
/opt/yjx/hadoop-3.1.2/etc/hadoop,
/opt/yjx/hadoop-3.1.2/share/hadoop/common/*,
/opt/yjx/hadoop-3.1.2/share/hadoop/common/lib/*,
/opt/yjx/hadoop-3.1.2/share/hadoop/hdfs/*,
/opt/yjx/hadoop-3.1.2/share/hadoop/hdfs/lib/*,
/opt/yjx/hadoop-3.1.2/share/hadoop/mapreduce/*,
/opt/yjx/hadoop-3.1.2/share/hadoop/mapreduce/lib/*,
/opt/yjx/hadoop-3.1.2/share/hadoop/yarn/*,
/opt/yjx/hadoop-3.1.2/share/hadoop/yarn/lib/*
</value>
</property>

• [root@node01 hadoop]# vim yarn-site.xml
  

<!-- 开启RM高可用 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yarn-yjx</value>
</property>
<!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>node01</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>node03</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>node01:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>node03:8088</value>
</property>
<!-- 指定zk集群地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>node01:2181,node02:2181,node03:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 开启日志聚合 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>86400</value>
</property>
<!-- 启用自动恢复 -->
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<!-- 制定resourcemanager的状态信息存储在zookeeper集群上 -->
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateS
tore</value>
</property>
<!-- Whether virtual memory limits will be enforced for containers. -->
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>3</value>
</property>

4. 拷贝到其他节点

[root@node01 hadoop]# scp yarn-site.xml mapred-site.xml root@node02:`pwd`
[root@node01 hadoop]# scp yarn-site.xml mapred-site.xml root@node03:`pwd`

5. 开启集群

[123]# zkServer.sh start
[root@node01 hadoop]# start-dfs.sh
[root@node01 hadoop]# start-yarn.sh 
[root@node01 hadoop]# mr-jobhistory-daemon.sh start historyserver

6. 关机拍摄快照

[1]# stop-all.sh
[1]# mr-jobhistory-daemon.sh stop historyserver
[123]# zkServer.sh stop