搭建hadoop3.1.3环境

使用三台虚拟机进行搭建，分别为node1，node2，node3

配置互信，免密钥登陆

添加java环境

首选在orcle网站下载jdk安装包，传送门
解压安装到/export/servers/目录下，修改/etc/profile文件，添加java路径

export JAVA_HOME=/export/servers/jdk1.8.0_261
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

执行java -verion命令，

到此，java环境配置完成

解压hadoop-3.1.3.tar.gz

这里将解压内容放在`/export/servers/``目录下

tar -zxvf hadoop-3.1.3.tar.gz -C /export/servers/

将hadoop路径配置到/etc/profile

export HADOOP_HOME=/export/servers/hadoop
export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath):$CLASSPATH
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_COMMON_LIB_NATIVE_DIR"

添加完执行source /etc/profile

创建目录

mkdir /export/servers/hadoop/dfs
mkdir /export/servers/hadoop/dfs/data
mkdir /export/servers/hadoop/dfs/name
mkdir /export/servers/hadoop/tmp

修改hadoop配置文件/export/servers/etc/hadoop/hadoop-env.sh，添加Java路径

export JAVA_HOME=/export/servers/jdk1.8.0_261

修改core-site.xml文件,文件所在目录/export/servers/hadoop-3.1.3/etc/hadoop,添加下面内容

<configuration>
        <property>
             <name>fs.defaultFS</name>
             <value>hdfs://node1:9000</value>
        </property>
        <property>
              <name>hadoop.tmp.dir</name>
              <value>/export/servers/hadoop/tmp</value>
         </property>
</configuration>

修改hdfs-site.xml ，添加下面内容

<configuration>

        <property>
            <name>dfs.namenode.http-address</name>
             <!-- Master为当前机器名或者IP地址 -->
             <value>hdfs://node1:9001</value>
        </property>
        <property>
              <name>dfs.namenode.name.dir</name>
              <!-- 以下为存放节点命名的路径 -->
              <value>/export/servers/hadoop/dfs/name</value>
         </property>
        <property>
              <name>dfs.datanode.data.dir</name>
              <!-- 以下为存放数据命名的路径 -->
              <value>/export/servers/hadoop/dfs/data</value>
        </property>
        <property>
              <name>dfs.replication</name>
              <!-- 备份次数，因为有2台DataNode-->
              <value>2</value>
         </property>
        <property>
              <name>dfs.webhdfs.enabled</name>
              <!-- Web HDFS-->
              <value>true</value>
         </property>

</configuration>

修改yarn-site.xml文件，添加下面内容

<configuration>
 <!-- Site specific YARN configuration properties -->
        <property>
            <name>yarn.resourcemanager.hostname</name>
             <!-- Master为当前机器名或者ip号 -->
             <value>node1</value>
        </property>
        <property>
              <name>yarn.nodemanager.aux-services</name>
              <!-- Node Manager辅助服务 -->
              <value>mapreduce_shuffle</value>
         </property>
        <property>
              <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
              <!-- Node Manager辅助服务类 -->
              <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
              <name>yarn.nodemanager.resource.cpu-vcores</name>
              <!-- CPU个数，需要根据当前计算机的CPU设置-->
              <value>1</value>
         </property>
        <property>
              <name>yarn.resourcemanager.admin.address</name>
              <!-- Resource Manager管理地址 -->
              <value>node1:8033</value>
         </property>
        <property>
              <name>yarn.resourcemanager.webapp.address</name>
              <!-- Resource Manager Web地址 -->
              <value>node1:8088</value>
         </property>
</configuration>

修改 mapred-site.xml

<configuration>
        <property>
            <name>mapreduce.framework.name</name>
             <!-- MapReduce Framework -->
             <value>yarn</value>
        </property>
        <property>
              <name>mapreduce.jobhistory.address</name>
              <!-- MapReduce JobHistory, 当前计算机的IP -->
              <value>node1:10020</value>
         </property>
        <property>
              <name>mapreduce.jobhistory.webapp.address</name>
              <!-- MapReduce Web App JobHistory, 当前计算机的IP -->
              <value>node1:19888</value>
        </property>
        <property>
              <name>yarn.app.mapreduce.am.env</name>
              <value>HADOOP_MAPRED_HOME=/export/servers/hadoop</value>
        </property>
        <property>
              <name>mapreduce.map.env</name>
              <value>HADOOP_MAPRED_HOME=/export/servers/hadoop</value>
        </property>
        <property>
              <name>mapreduce.reduce.env</name>
              <value>HADOOP_MAPRED_HOME=/export/servers/hadoop</value>
        </property>
</configuration>

修改sbin/start-yarn.sh和sbin/stop-yarn.sh文件，添加

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

修改./sbin/start-dfs.sh和./sbin/stop-dfs.sh

HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

在sbin/hadoop/workers里删除localhost添加node1 node2 node3

将配置好的文件传递到其他两个节点上

scp -r ./hadoop-3.1.3 node2:$PWD
scp -r ./hadoop-3.1.3 node3:$PWD
scp /etc/profile node2:/etc/
scp /etc/profile node3:/etc/

分别在node2和node3上执行source /etc/profile

启动hadoop

最后在node1上执行hadoop namenode -format格式化文件系统

执行成功后，在node1执行./sbin/start-all.sh，执行结果如下

在node1执行jps显示如下

在node2上执行jps

访问http://172.20.17.132:9001显示

访问http://172.20.17.132:8088/cluster