Hadoop安装教程详解

使用VirtualBox安装Hadoop安装详解

一、前言

1、本文使用VirtualBox安装Hadoop，前期的centos7的配置请看：https://blog.csdn.net/qq_48395735/article/details/106816891
（按照博客做完第三步：使用moba连接虚拟机即可）

二、准备

相关安装包下载：链接：https://pan.baidu.com/s/1MOa9GqINlZF4avz0nVyGPg
提取码：w8uh

三、安装Hadoop

1、导入安装包、讲安装包拖入software文件包中

[root@hadoop101 opt]# cd /
[root@hadoop101 /]# mkdir software

2、下载解压缩工具

[root@hadoop101 softeare]# yum install -y unzip

3、解压缩安装包

[root@hadoop101 softeare]# tar -zxvf jdk-8u221-linux-x64.tar.gz -C /opt
[root@hadoop101 softeare]# tar -zxvf hadoop-2.6.0-cdh5.14.2.tar.gz
[root@hadoop101 softeare]# unzip apache-tomcat-8.5.45.zip /opt/tomcat8

4、为文件重名名，新建一个moba窗口2

cd /opt
[root@hadoop101 opt]# mv jdk-8u221-linux-x64/ jdk1.8.0_221
[root@hadoop101 opt]# mv hadoop-2.6.0-cdh5.14.2/ hadoop

5、配置java环境

vi /etc/profile
#添加以下内容
export JAVA_HOME=/opt/jdk1.8.0_221
export CLASSPATH=.:$JAVA_HOME/rt.jar:$JAVA_HOME/tools.jar:$JAVA_HOME/dt.jar
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$NODE_HOME/bin

6、查看java8环境是否配置成功

[root@hadoop101 opt]# source /etc/profile
[root@hadoop101 opt]# java -version
#出现以下代表java配置成功
java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

7、修改主机名

[root@hadoop101 opt]# vi /etc/hostname
#修改为，看你自己取名
hadoop101

8、修改主机列表

[root@hadoop101 opt]# vi /etc/hosts

9、配置免密登陆，在窗口1执行命令

cd ~
ssh-keygen -t rsa -P ""
cat .ssh/id_rsa.pub >> .ssh/authorized_keys

10、配置Hadoop

[root@hadoop101 hadoop]# cd /opt/hadoop/etc/hadoop
#此时是在etc下面的hadoop立马
[root@hadoop101 hadoop]# ll

11、配置hadoop-env.sh
#修改如下：
The java implementation to use.
export JAVA_HOME=/opt/jdk1.8.0_221

[root@localhost hadoop]# vi hadoop-env.sh

12、配置core-site.xml

[root@localhost hadoop]# vi core-site.xml
#修改如下
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://192.168.56.101:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/hadoop/tmp</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
  </property>
</configuration>

13、配置hdfs-site.xml

[root@localhost hadoop]# vi hdfs-size.xml
#修改如下
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop101:50090</value>
  </property>
</configuration>

14、配置mapred-site.xml

[root@localhost hadoop]# mv mapred-site.xml.template mapred-site.xml
[root@localhost hadoop]# vi mapred-site.xm
[root@localhost hadoop]# vi mapred-site.xml
#修改如下：
<configuration>
  <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
  </property>
  <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop101:10020</value>
  </property>
  <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop101:19888</value>
  </property>
</configuration>

15、配置yarn-site.xml

[root@localhost hadoop]# vi yarn-site.xml
#配置如下
<configuration>

<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop101</value>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
  </property>
</configuration>

16、hadoop环境变量配置

[root@localhost hadoop]# vi /etc/profile
#配置如下：shfit+g定位末尾，添加如下内容
export HADOOP_HOME=/opt/greeinstall/hadoop260
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

17、[root@localhost hadoop]# source /etc/profile
18、格式化HDFS

[root@localhost hadoop]# hadoop namenode -format

19、启动hadoop

[root@localhost hadoop]# start-all.sh
[root@localhost hadoop]# jps

20、在网页中输入http://192.168.56.101:50070

21：在网页中输入http://192.168.56.101:8088