Hadoop入门学习笔记-第三天（Yarn高可用集群配置及计算案例）

什么是mapreduce

首先让我们来重温一下 hadoop 的四大组件：
HDFS：分布式存储系统
MapReduce：分布式计算系统
YARN： hadoop 的资源调度系统
Common： 以上三大组件的底层支撑组件，主要提供基础工具包和 RPC 框架等

Mapreduce 是一个分布式运算程序的编程框架，是用户开发“基于 hadoop 的数据分析 应用”的核心框架
Mapreduce 核心功能是将用户编写的业务逻辑代码和自带默认组件整合成一个完整的 分布式运算程序，并发运行在一个 hadoop 集群上

一、yarn单机版配置
1.修改mapred-site.xml 文件指定调度方式为yarn

mapreduce.framework.name
yarn

2.修改yarn-site.xml 配置yarn服务器

yarn.resourcemanager.hostname
nameNode10.com

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

3.在指定机器上面启动./start-yarn.sh

二、yarn ha高可用 集群配置
1.修改yarn-site.xml 配置yarn集群


yarn.resourcemanager.connect.retry-interval.ms
2000



yarn.resourcemanager.ha.enabled
true



yarn.resourcemanager.ha.automatic-failover.enabled
true

<property>
    <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
    <value>true</value>
</property>
<!-- 给yarn cluster 取个名字yarn-rm-cluster -->    
<property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>yarn-rm-cluster</value>
</property>
<!-- 给ResourceManager 取个名字 rm1,rm2 -->
<property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
</property>
<!-- 配置ResourceManager rm1 hostname -->
<property>
    <name>yarn.resourcemanager.hostname.rm1</name>
    <value>nameNode10.com</value>
</property>
<!-- 配置ResourceManager rm2 hostname -->
<property>
    <name>yarn.resourcemanager.hostname.rm2</name>
    <value>dateNode30.com</value>
</property>

<property>
    <name>yarn.resourcemanager.recovery.enabled</name>
    <value>true</value>
</property>

<property>
    <name>yarn.resourcemanager.zk.state-store.address</name>
    <value>nameNode10.com:2181,dataNode20.com:2181,nameNode40.com:2181</value>
</property>
<!-- 配置Zookeeper地址 -->
<property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>nameNode10.com:2181,dataNode20.com:2181,nameNode40.com:2181</value>
</property>

<property>
    <name>yarn.resourcemanager.address.rm1</name>
    <value>nameNode10.com:8032</value>
</property>
<!-- rm1调度器的端口号 -->
<property>
    <name>yarn.resourcemanager.scheduler.address.rm1</name>
    <value>nameNode10.com:8034</value>
</property>
<!-- rm1 webapp端口号 -->
<property>
    <name>yarn.resourcemanager.webapp.address.rm1</name>
    <value>nameNode10.com:8088</value>
</property>

yarn.resourcemanager.address.rm2 nodeNode30.com:8032 yarn.resourcemanager.scheduler.address.rm2 nodeNode30.com:8034 yarn.resourcemanager.webapp.address.rm2 nodeNode30.com:8088

<!-- 执行MapReduce需要配置的shuffle过程 低版本，配置成mapreduces.shuffle -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

2.输入hadoop classpath 会出现以下路径(这步不配置，执行自带案例demo时会报错)
/software/hadoop-3.2.0/etc/hadoop:/software/hadoop-3.2.0/share/hadoop/common/lib/:/software/hadoop-3.2.0/share/hadoop/common/:/software/hadoop-3.2.0/share/hadoop/hdfs:/software/hadoop-3.2.0/share/hadoop/hdfs/lib/:/software/hadoop-3.2.0/share/hadoop/hdfs/:/software/hadoop-3.2.0/share/hadoop/mapreduce/lib/:/software/hadoop-3.2.0/share/hadoop/mapreduce/:/software/hadoop-3.2.0/share/hadoop/yarn:/software/hadoop-3.2.0/share/hadoop/yarn/lib/:/software/hadoop-3.2.0/share/hadoop/yarn/
将上面路径拷贝到 yarn-site.xml文件中

yarn.application.classpath
/software/hadoop-3.2.0/etc/hadoop:/software/hadoop-3.2.0/share/hadoop/common/lib/:/software/hadoop-3.2.0/share/hadoop/common/:/software/hadoop-3.2.0/share/hadoop/hdfs:/software/hadoop-3.2.0/share/hadoop/hdfs/lib/:/software/hadoop-3.2.0/share/hadoop/hdfs/:/software/hadoop-3.2.0/share/hadoop/mapreduce/lib/:/software/hadoop-3.2.0/share/hadoop/mapreduce/:/software/hadoop-3.2.0/share/hadoop/yarn:/software/hadoop-3.2.0/share/hadoop/yarn/lib/:/software/hadoop-3.2.0/share/hadoop/yarn/

3.单步骤启动
yarn-daemon.sh start resourcemanager
yarn-daemon.sh start nodemanager
4.集成启动 yarn-start.sh

5.访问控制台查看(访问备用节点会自动跳转到主节点控制台)
http://datanode30.com:8088/cluster/
http://namenode10.com:8088/cluster/
主备状态查看
yarn rmadmin -getServiceState rm1

6.执行例子
1.计算圆周率 /software/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar pi 10 10
2.统计文中每个单词个数
hadoop fs -mkdir /input
hadoop fs -put -f ./file/file1.txt /input
hadoop jar /software/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /input /output
hadoop fs -ls /output
hadoop fs -cat /output/part-r-00000

/input/hadoop-demo-wordcount.txt