hadoop3.2集群搭建

前期准备：
环境搭建完成：centos1、centos2、centos3、centos4
SSH免密登录配置完成
JDK已安装

1、下载

下载地址：https://hadoop.apache.org/releases.html

2、上传并解压

[root@centos1 ~]# tar -zxvf hadoop-3.2.0.tar.gz -C ~/soft/app/

3、修改配置文件

进入配置目录

[root@centos1 ~]# /root/soft/app/hadoop/hadoop-3.2.0/etc/hadoop

3.1、配置hadoop-env.sh

export JAVA_HOME=/root/soft/app/jdk1.8.0_221
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

3.2、配置core-site.xml

<configuration>
<!--HDFS的地址-->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://centos1:9000</value>
    </property>
    <!--Hadoop临时存储目录-->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/root/soft/app/hadoop/data/tmp</value>
    </property>
</configuration>

3.3、配置hdfs-site.xml

<configuration>
<!--hdfs副本数量-->
    <name>dfs.replication</name>
    <value>3</value>
	<property>
	      <name>dfs.namenode.name.dir</name>
	      <value>/root/soft/app/hadoop/data/namenode</value>
	</property>
	<property>
	      <name>dfs.datanode.data.dir</name>
	      <value>/root/soft/app/hadoop/data/datanode</value>
	</property>
</configuration>

3.4、配置mapred-site.xml

<configuration>
 <!--MR是有YARN-->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
        <property>
        <name>mapreduce.admin.user.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value>
    </property>
         <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=$HADOOP_COMMON_HOME</value>
    </property>
</configuration>

hadoop3.2要配置mapreduce.admin.user.env、yarn.app.mapreduce.am.env，否则会报错

3.5、配置yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>centos1</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-service</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!--以下可以默认不配置。如果子节点向主节点注册出错，则显示配置-->
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>centos1:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>centos1:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>centos1:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>centos1:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>centos1:8088</value>
    </property>
</configuration>

3.6、配置workers
在hadoop3.2中，原来的slaves变为workers

centos2
centos3
centos4

4、配置环境变量，分发各节点

配置环境变量:

[root@centos1 hadoop]# vim /etc/profile

export HADOOP_HOME=/root/soft/app/hadoop/hadoop-3.2.0
export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

[root@centos1 hadoop]# source /etc/profile

分发各节点：

[root@centos1 app]# scp -r hadoop centos2:/root/soft/app/

5、启动hadoop

5.1、格式化

[root@centos1 hadoop-3.2.0]# bin/hdfs namenode -format 

5.2、启动

[root@bs1 hadoop-3.2.0]# sbin/start-dfs.sh
[root@bs1 hadoop-3.2.0]# sbin/start-yarn.sh

6、验证

[root@centos1 ~]# $HADOOP_HOME/bin/hdfs dfsadmin -report
Configured Capacity: 54716792832 (50.96 GB)
Present Capacity: 32372490240 (30.15 GB)
DFS Remaining: 32372477952 (30.15 GB)
DFS Used: 12288 (12 KB)
DFS Used%: 0.00%
Replicated Blocks:
	Under replicated blocks: 0
	Blocks with corrupt replicas: 0
	Missing blocks: 0
	Missing blocks (with replication factor 1): 0
	Low redundancy blocks with highest priority to recover: 0
	Pending deletion blocks: 0
Erasure Coded Block Groups: 
	Low redundancy block groups: 0
	Block groups with corrupt internal blocks: 0
	Missing block groups: 0
	Low redundancy blocks with highest priority to recover: 0
	Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (3):

Name: 192.168.119.152:9866 (centos2)
Hostname: centos2
Decommission Status : Normal
Configured Capacity: 18238930944 (16.99 GB)
DFS Used: 4096 (4 KB)
Non DFS Used: 2725879808 (2.54 GB)
DFS Remaining: 15513047040 (14.45 GB)
DFS Used%: 0.00%
DFS Remaining%: 85.05%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Aug 25 11:23:09 CST 2019
Last Block Report: Sun Aug 25 11:17:06 CST 2019
Num of Blocks: 0


Name: 192.168.119.153:9866 (centos3)
Hostname: centos3
Decommission Status : Normal
Configured Capacity: 18238930944 (16.99 GB)
DFS Used: 4096 (4 KB)
Non DFS Used: 2726170624 (2.54 GB)
DFS Remaining: 15512756224 (14.45 GB)
DFS Used%: 0.00%
DFS Remaining%: 85.05%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Aug 25 11:23:09 CST 2019
Last Block Report: Sun Aug 25 11:17:06 CST 2019
Num of Blocks: 0


Name: 192.168.119.154:9866 (centos4)
Hostname: centos4
Decommission Status : Normal
Configured Capacity: 18238930944 (16.99 GB)
DFS Used: 4096 (4 KB)
Non DFS Used: 16892252160 (15.73 GB)
DFS Remaining: 1346674688 (1.25 GB)
DFS Used%: 0.00%
DFS Remaining%: 7.38%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Aug 25 11:23:09 CST 2019
Last Block Report: Sun Aug 25 11:17:06 CST 2019
Num of Blocks: 0