大数据平台的安装部署——Hadoop伪分布模式篇
1、 先将需要的软件上传至虚拟机中
2、 创建一个目录,并将安装包移动到该目录下,用于存放这两个安装包
[root@localhost ~]# mkdir /opt/Hadoop
[root@localhost ~]# mv hadoop-2.7.6.tar.gz /opt/hadoop/
[root@localhost ~]# mv jdk-8u171-linux-x64.tar.gz /opt/hadoop/
[root@localhost ~]# cd /opt/hadoop/
3、 安装JDK
1)解压JDK安装包
[root@localhost hadoop]# tar -zxvf jdk-8u171-linux-x64.tar.gz
2)进入解压后的jdk目录下
[root@localhost hadoop]# cd jdk1.8.0_171/
3)查看当前所在路径
[root@localhost jdk1.8.0_171]# pwd
/opt/hadoop/jdk1.8.0_171
4)编辑环境变量
[root@localhost jdk1.8.0_171]# vim /etc/profile
export JAVA_HOME=/opt/hadoop/jdk1.8.0_171
export PATH=$PATH:$JAVA_HOME/bin
5)刷新环境变量并查看JAVA的版本
[root@localhost jdk1.8.0_171]# source /etc/profile
[root@localhost jdk1.8.0_171]# java -version
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
4、 安装Hadoop
1) 解压Hadoop安装包
[root@localhost hadoop]# tar -zxvf hadoop-2.7.6.tar.gz
2) 进入解压后的Hadoop目录
[root@localhost hadoop]# cd hadoop-2.7.6/
3) 查看当前目录所在路径
[root@localhost hadoop-2.7.6]# pwd
/opt/hadoop/hadoop-2.7.6
4) 编辑环境变量
[root@localhost hadoop-2.7.6]# vim /etc/profile
export HADOOP_HOME=/opt/hadoop/hadoop-2.7.6
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
5) 刷新环境变量,并查看Hadoop版本
[root@localhost hadoop-2.7.6]# source /etc/profile
[root@localhost hadoop-2.7.6]# hadoop version
Hadoop 2.7.6
Subversion https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r 085099c66cf28be31604560c376fa282e69282b8
Compiled by kshvachk on 2018-04-18T01:33Z
Compiled with protoc 2.5.0
From source with checksum 71e2695531cb3360ab74598755d036
This command was run using /opt/hadoop/hadoop-2.7.6/share/hadoop/common/hadoop-common-2.7.6.jar
5、 配置免密码登录
[root@localhost hadoop-2.7.6]# ssh-keygen
[root@localhost hadoop-2.7.6]# ssh-copy-id 192.168.91.128
[root@localhost hadoop-2.7.6]# ssh 192.168.91.128
此时通过ssh进行测试的时候,不需要再输入密码,则表示成功。
6、 修改core-site.xml文件
[root@localhost hadoop]# vim hadoop-2.7.6/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost</value>
</property>
</configuration>
7、 修改hdfs-site.xml文件
[root@localhost hadoop]# vim hadoop-2.7.6/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
8、 复制mapred-site.xml.template并改名为mapred-site.xml
[root@localhost hadoop]# cd hadoop-2.7.6/etc/hadoop/
[root@localhost hadoop]# cp mapred-site.xml.template mapred-site.xml
9、 修改mapred-site.xml文件
[root@localhost hadoop]# vim mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:8021</value>
</property>
</configuration>
10、修改yarn-site.xml文件
[root@localhost hadoop]# vim yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
11、格式化NameNode
[root@localhost hadoop]# hadoop namenode -format
12、启动服务
[root@localhost hadoop]# start-dfs.sh
提示:
若出现类似的错误,是因为JAVA的路径问题引起的
以上报错的解决方案:
[root@localhost hadoop]# vim hadoop-env.sh
修改后的样式:
修改完成后,再次启动服务
[root@localhost hadoop]# start-dfs.sh
[root@localhost hadoop]# start-yarn.sh
13、关闭防火墙
[root@localhost hadoop]# systemctl stop firewalld
[root@localhost hadoop]# systemctl disable firewalld
14、查看Hadoop的进程