Hadoop全分布安装步骤

Hadoop全分布模式安装手册

实验开始前的准备：

三台centos7: master-yly：192.168.91.134（IP地址）

    slave01-yly：192.168.91.135

       slave02-yly：192.168.91.136

本实验所使用的两个文件包：

hadoop-2.7.6.tar.gz 

jdk-8u171-linux-x64.tar.gz

可以上传文件的软件（Mac系统可直接用终端进行文件传输，window系统选择合适软件即可）

 

1 .修改主机名

master-yly：

[root@localhost ~]# hostnamectl set-hostname master-yly [root@localhost ~]# bash

slave01-yly：

[root@localhost ~]# hostnamectl set-hostname slave01-yly [root@localhost ~]# bash

slave02-yly：

[root@localhost ~]# hostnamectl set-hostname slave02-yly [root@localhost ~]# bash

2修改/etc/hosts文件，配置主机域名映射

master-yly:

[root@master-yly ~]# vim /etc/hosts 192.168.91.134 master-yly 192.168.91.135 slave01-yly 192.168.91.136 slave02-yly

slave01-yly:

[root@slave01-yly ~]# vim /etc/hosts 192.168.91.134 master-yly 192.168.91.135 slave01-yly 192.168.91.136 slave02-yly

slave02-yly:

[root@slave02-yly ~]# vim /etc/hosts 192.168.91.134 master-yly 192.168.91.135 slave01-yly 192.168.91.136 slave02-yly

3 关闭防火墙（三台同时关闭）

[root@master-yly ~]# systemctl stop firewalld [root@slave01-yly ~]# systemctl stop firewalld [root@slave02-yly ~]# systemctl stop firewalld

4 安装并配置NTP服务，用于时间同步

master-yly：

[root@master-yly ~]# yum install -y ntp   # 修改配置文件，在末尾添加如下两条配置： [root@master-yly ~]# vim /etc/ntp.conf server 127.127.1.0 fudge 127.127.1.0 startum 10 # 启动服务并添加到开机自启动： [root@master-yly ~]# systemctl start ntpd [root@master-yly ~]# systemctl enable ntpd

slave01-yly：

[root@slave01-yly ~]# yum install -y ntpdate   [root@slave01-yly ~]# ntpdate master-yly

slave02-yly：

[root@slave02-yly ~]# yum install -y ntpdate   [root@slave02-yly ~]# ntpdate master-yly

5 配置免密码登录（只需要配置从主节点到从节点即可）

[root@master-yly ~]# ssh-keygen   [root@master-yly ~]# ssh-copy-id master-yly   [root@master-yly ~]# ssh-copy-id slave02-yly   [root@master-yly ~]# ssh-copy-id slave01-yly

6 将安装上传到服务器中

 

通过ls -l命令可以看到，安装已经上传成功

 

7 安装和配置jdk

# 新建一个目录用来存放Hadoop的相关安装包 [root@master-yly ~]# mkdir /opt/hadoop # 将安装包移动到新建的目录下 [root@master-yly ~]# mv hadoop-2.7.6.tar.gz /opt/hadoop/ [root@master-yly ~]# mv jdk-8u171-linux-x64.tar.gz /opt/hadoop/ [root@master-yly ~]# cd /opt/hadoop/ # 将jdk安装包进行解压 [root@master-yly hadoop]# tar -zxvf jdk-8u171-linux-x64.tar.gz # 编辑/etc/profile文件，配置环境变量 [root@master-yly hadoop]# vim /etc/profile export JAVA_HOME=/opt/hadoop/jdk1.8.0_171 export PATH=$PATH:$JAVA_HOME/bin # 刷新环境变量，使修改的变量生效 [root@master-yly hadoop]# source /etc/profile # 验证是否安装成功 [root@master-yly hadoop]# java -version 注：（删除自带jdk）

8 安装Hadoop

# 解压Hadoop的安装包 [root@master-yly hadoop]# tar -zxvf hadoop-2.7.6.tar.gz # 编辑/etc/profile文件，配置环境变量 [root@master-yly hadoop]# vim /etc/profile export HADOOP_HOME=/opt/hadoop/hadoop-2.7.6 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin # 刷新环境变量，使修改的变量生效 [root@master-yly hadoop]# source /etc/profile # 验证是否安装成功 [root@master-yly hadoop]# hadoop version

9 修改Hadoop的配置文件

PS：配置文件在：/opt/hadoop/ hadoop-2.7.6/etc/hadoop/目录下

# 修改core-site.xml

[root@master-yly hadoop]# cd hadoop-2.7.6/etc/hadoop/ [root@master-yly hadoop]# vim core-site.xml <configuration>     <property>         <name>fs.defaultFS</name>         <value>hdfs://master-yly:9000</value>     </property>     <property>         <name>hadoop.tmp.dir</name>         <value>/opt/hadoop/hadoop-2.7.6/hdfs</value>     </property> </configuration>

# 修改hdfs-site.xml

[root@master-yly hadoop]# vim hdfs-site.xml <configuration>     <property>         <name>dfs.replication</name>         <value>3</value>     </property>     <property>         <name>dfs.namenode.secondary.http-address</name>         <value>slave02-yly:9001</value>     </property> </configuration>

# 修改mapred-site.xml

[root@master-yly hadoop]# cp mapred-site.xml.template mapred-site.xml [root@master-yly hadoop]# vim mapred-site.xml <configuration>     <property>         <name>mapreduce.framework.name</name>         <value>yarn</value>     </property> </configuration>

# 修改yarn-site.xml

[root@master-yly hadoop]# vim yarn-site.xml <configuration>     <property>         <name>yarn.resourcemanager.hostname</name>         <value>master-yly</value>     </property>     <property>         <name>yarn.nodemanager.aux-services</name>         <value>mapreduce_shuffle</value>     </property> </configuration>

# 修改slaves文件

[root@master-yly hadoop]# vim slaves  master-yly slave01-yly slave02-yly

# 修改hadoop-env.sh文件

[root@master-yly hadoop]# vim hadoop-env.sh     export JAVA_HOME=/opt/hadoop/jdk1.8.0_171

10 将修改好的环境复制到其他的从节点

# 将jdk复制到其他的从节点 [root@master-yly ~]# scp -r /opt/hadoop/jdk1.8.0_171/  slave01-yly:/opt/hadoop/ [root@master-yly ~]# scp -r /opt/hadoop/jdk1.8.0_171/  slave02-yly:/opt/hadoop/ # 将Hadoop复制到其他从节点 [root@master-yly ~]# scp -r /opt/hadoop/hadoop-2.7.6/  slave01-yly:/opt/hadoop/ [root@master-yly ~]# scp -r /opt/hadoop/hadoop-2.7.6/  slave02-yly:/opt/hadoop/ # 将环境变量复制到其他从节点 [root@master-yly hadoop]# scp /etc/profile slave01-yly:/etc/ [root@master-yly hadoop]# scp /etc/profile slave02-yly:/etc/

11 格式化Hadoop

[root@master-yly hadoop]# hdfs namenode -format

12 启动

[root@master-yly hadoop]# start-all.sh

13 验证启动情况,输入

# master-yly：

[root@master-yly ~]# jps 16626 DataNode 16916 ResourceManager 16468 NameNode 17048 NodeManager 18095 Jps

# slave01-yly:

[root@slave01-yly ~]# jps 16993 NodeManager 16823 DataNode 17320 Jps

# slave02-yly:

[root@slave02-yly ~]# jps 48538 NodeManager 48906 Jps 48204 DataNode 48367 SecondaryNameNode