一、集群配置
1. 集群部署规划
2. 配置集群
(1)核心配置文件
配置core-site.xml
[hadoop@hadoop02 hadoop]$ vi core-site.xml
在该文件中编写如下配置
<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop02:9000</value>
</property>
<!-- 指定Hadoop运行时产生文件的存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-2.7.2/data/tmp</value>
</property>
(2)HDFS配置文件
配置hadoop-env.sh
[hadoop@hadoop02 hadoop]$ vi hadoop-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144
配置hdfs-site.xml
[hadoop@hadoop02 hadoop]$ vi hdfs-site.xml
在该文件中编写如下配置
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<!-- 指定Hadoop辅助名称节点主机配置 -->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop04:50090</value>
</property>
(3)YARN配置文件
配置yarn-env.sh
[hadoop@hadoop02 hadoop]$ vi yarn-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144
配置yarn-site.xml
[hadoop@hadoop02 hadoop]$ vi yarn-site.xml
在该文件中增加如下配置
<!-- Reducer获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop03</value>
</property>
(4)MapReduce配置文件
配置mapred-env.sh
[hadoop@hadoop02 hadoop]$ vi mapred-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144
配置mapred-site.xml
[hadoop@hadoop02 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@hadoop02 hadoop]$ vi mapred-site.xml
在该文件中增加如下配置
<!-- 指定MR运行在Yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
2.在集群上分发配置好的Hadoop配置文件
[hadoop@hadoop02 hadoop]$ xsync /opt/module/hadoop-2.7.2/
3.查看文件分发情况
[hadoop@hadoop03 hadoop]$ cat /opt/module/hadoop-2.7.2/etc/hadoop/core-site.xml
二、 集群单点启动
1.如果集群是第一次启动,需要格式化NameNode
[hadoop@hadoop02 hadoop-2.7.2]$ hadoop namenode -format
2.在hadoop02上启动NameNode
[hadoop@hadoop02 hadoop-2.7.2]$ hadoop-daemon.sh start namenode
[hadoop@hadoop02 hadoop-2.7.2]$ jps
3461 NameNode
3.在hadoop02、hadoop03以及hadoop04上分别启动DataNode
[hadoop@hadoop02 hadoop-2.7.2]$ hadoop-daemon.sh start datanode
[hadoop@hadoop02 hadoop-2.7.2]$ jps
3461 NameNode
3608 Jps
3561 DataNode
[hadoop@hadoop03 hadoop-2.7.2]$ hadoop-daemon.sh start datanode
[hadoop@hadoop03 hadoop-2.7.2]$ jps
3190 DataNode
3279 Jps
[hadoop@hadoop04 hadoop-2.7.2]$ hadoop-daemon.sh start datanode
[hadoop@hadoop04 hadoop-2.7.2]$ jps
3237 Jps
3163 DataNode
三、SSH无密登录配置
1. 配置ssh
(1)基本语法
ssh另一台电脑的ip地址
(2)ssh连接时出现Host key verification failed的解决方法
[hadoop@hadoop02 opt] $ ssh 192.168.1.103
The authenticity of host '192.168.1.103 (192.168.1.103)' can't be established.
RSA key fingerprint is cf:1e:de:d7:d0:4c:2d:98:60:b4:fd:ae:b1:2d:ad:06.
Are you sure you want to continue connecting (yes/no)?
Host key verification failed.
(3)解决方案如下:直接输入yes
2. 无密钥配置
(1)免密登录原理
(2)生成公钥和私钥:
[hadoop@hadoop02 .ssh]$ ssh-keygen -t rsa
然后敲(三个回车),就会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥)
(3)将公钥拷贝到要免密登录的目标机器上
[hadoop@hadoop02 .ssh]$ ssh-copy-id hadoop02
[hadoop@hadoop02 .ssh]$ ssh-copy-id hadoop03
[hadoop@hadoop02 .ssh]$ ssh-copy-id hadoop04
3. .ssh文件夹下(~/.ssh)的文件功能解释
四、群起集群
1. 配置slaves
/opt/module/hadoop-2.7.2/etc/hadoop/slaves
[hadoop@hadoop02 hadoop]$ vi slaves
在该文件中增加如下内容:
hadoop02
hadoop03
hadoop04
注意:该文件中添加的内容结尾不允许有空格,文件中不允许有空行。
同步所有节点配置文件
[hadoop@hadoop02 hadoop]$ xsync slaves
2. 启动集群
(1)如果集群是第一次启动,需要格式化NameNode(注意格式化之前,一定要先停止上次启动的所有namenode和datanode进程,然后再删除data和log数据)
[hadoop@hadoop02 hadoop-2.7.2]$ bin/hdfs namenode -format
(2)启动HDFS
[hadoop@hadoop02 hadoop-2.7.2]$ sbin/start-dfs.sh
[hadoop@hadoop02 hadoop-2.7.2]$ jps
4166 NameNode
4482 Jps
4263 DataNode
[hadoop@hadoop03 hadoop-2.7.2]$ jps
3218 DataNode
3288 Jps
[hadoop@hadoop04 hadoop-2.7.2]$ jps
3221 DataNode
3283 SecondaryNameNode
3364 Jps
(3)启动YARN
[hadoop@hadoop03 hadoop-2.7.2]$ sbin/start-yarn.sh
注意:NameNode和ResourceManger如果不是同一台机器,不能在NameNode上启动 YARN,应该在ResouceManager所在的机器上启动YARN。
(4)Web端查看SecondaryNameNode
(a)浏览器中输入:http://hadoop04:50090/status.html
(b)查看SecondaryNameNode信息
五、 集群启动/停止方式总结
1. 各个服务组件逐一启动/停止
(1)分别启动/停止HDFS组件
hadoop-daemon.sh start / stop namenode / datanode / secondarynamenode
(2)启动/停止YARN
yarn-daemon.sh start / stop resourcemanager / nodemanager
2. 各个模块分开启动/停止(配置ssh是前提)常用
(1)整体启动/停止HDFS
start-dfs.sh / stop-dfs.sh
(2)整体启动/停止YARN
start-yarn.sh / stop-yarn.sh