一.集群简介
HADOOP集群具体来说包含两个集群:HDFS集群和YARN集群
HDFS集群:
负责海量数据的存储,集群中的角色主要有 NameNode / DataNode
YARN集群:
负责海量数据运算时的资源调度,集群中的角色主要ResourceManager/NodeManager
二.安装包
yum -y install -y lrzsz
在hadoop01上建文件夹
mkdir -p /home/hadoop/apps
三.上传文件
rz
四.设置域名主机名
vi /etc/hosts
五.配置ssh免密登录
ssh-keygen
六.将生成的秘钥发到hadoop01,hadoop02,hadoop03上
ssh-copy-id hadoop02
ssh-copy-id hadoop03
检验是否免密登录成功
ssh hadoop02
ssh hadoop03
七.解压压缩包配置环境变量
1.1 解压jdk压缩包
tar -zxvf jdk-8u181-linux-x64.tar.gz
1.2 配置环境变量
vi /etc/profile
#进入编辑器在最后输入以下内容
export JAVA_HOME=/home/hadoop/apps/jdk1.8.0_181
export PATH=$JAVA_HOME/bin:$PATH
shift+zz 保存,java -version查看是否存在,出现以下内容jdk配置完成
2.1 解压Hadoop安装包
tar -zxvf hadoop-2.8.0.tar.gz
2.2 配置环境变量
vi /etc/profile
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.8.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
出现以下内容Hadoop配置完成
source /etc/profile
hadoop veision
2.3 修改配置文件
一.hadoop-env.sh
The java implementation to use.
export JAVA_HOME=/home/hadoop/apps/jdk1.8.0_181
二.vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://Hadoop01:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/root/hdptmp</value>
</property>
</configuration>
三.vi hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/root/hdp-meta</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/root/hdp-blocks</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>128m</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>Hadoop01:50090</value>
</property>
</configuration>
四.vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
五.vi yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hdp01</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
六.vi slaves
Hadoop01
Hadoop02
Hadoop03
八.分别在Hadoop02和Hadoo03上创建文件夹
mkdir -p /home/hadoop/apps
九.将Hadoop01上的文件发送到Hadoop02和Hadoop03上
scp -r /home/hadoop/apps/jdk1.8.0_181/ Hadoop02:/home/hadoop/apps/
scp -r /home/hadoop/apps/jdk1.8.0_181/ Hadoop03:/home/hadoop/apps/
scp -r /home/hadoop/apps/hadoop-2.8.0 Hadoop02:/home/hadoop/apps/
scp -r /home/hadoop/apps/hadoop-2.8.0 Hadoop03:/home/hadoop/apps/
scp -r /etc/hosts Hadoop02:/etc
scp -r /etc/hosts Hadoop03:/etc
scp -r /etc/profile Hadoop02:/etc/profile
scp -r /etc/profile Hadoop03:/etc/profile
十.检验是否完成[分别在Hadoo02和Hadoop03上进行]
source /etc/profile
java -version
hadoop version
十一.启动集群
start-all.sh
jps
在浏览器输入192.168.72.110:50070 [Hadoop01的地址]