0到1搭建hadoop-3.2.2集群

最新推荐文章于 2023-05-29 14:17:31 发布

jackyan163

最新推荐文章于 2023-05-29 14:17:31 发布

阅读量302

点赞数

分类专栏：大数据文章标签： hadoop hdfs big data

本文链接：https://blog.csdn.net/qq_42409495/article/details/122390152

版权

大数据专栏收录该内容

32 篇文章 2 订阅

订阅专栏

1 安装包准备

从官网下载安装包：hadoop-3.2.2.tar.gz
https://archive.apache.org/dist/hadoop/common/

2 集群规划

	Hadoop101	Hadoop102	hadoop103
192.168.56.101	NameNode,DataNode	DataNode	DataNode
192.168.56.102	NodeManager	ResourceManager,NodeManager	NodeManager
192.168.56.103	-	-	SecondaryNameNode,JobHistoryServer

3 创建hadoop用户（非必须）

创建hadoop用户

useradd hadoop
passwd hadoop

4 设置hostname

hostnamectl set-hostname hadoop101

5 配置/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.56.101    hadoop101
192.168.56.102    hadoop102
192.168.56.103    hadoop103

6 配置免密登录

生成密钥

ssh-keygen -t rsa

一直回车键即可
授权，例如授权hadoop101可以免密登录hadoop101,hadoop102,hadoop103

ssh-copy-id hadoop101
ssh-copy-id hadoop102
ssh-copy-id hadoop103

编写xsync同步脚本

#!/bin/bash
#1 获取输入参数个数，如果没有参数，直接退出
pcount=$#
if((pcount==0)); then
echo no args;
exit;
fi

#2 获取文件名称
p1=$1
fname=`basename $p1`
echo fname=$fname

#3 获取上级目录到绝对路径
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir

#4 获取当前用户名称
user=`whoami`

#5 循环
for((host=1; host<4; host++)); do
        echo ------------------- linux$host --------------
        rsync -rvl $pdir/$fname $user@linux$host:$pdir
done

7 安装jdk1.8环境（自行在官网下载安装包）

解压

tar -xf jdk-8u291-linux-x64.tar.gz -C /home/hadoop/modules
# 添加软链接
ln -s jdk1.8.0_291 jdk1.8

在/etc/profile中添加环境变量

# JAVA_HOME
export JAVA_HOME=/opt/modules/jdk1.8
export PATH=$PATH:$JAVA_HOME/bin

使环境变量生效

source /etc/profile

测试java

[root@hadoop101 opt]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

8 安装hadoop

解压hadoop

tar -xf hadoop-3.2.2.tar.gz -C /opt/modules/

配置hadoop环境变量

export HADOOP_HOME=/opt/modules/hadoop-3.2.2
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

使环境变量生效

source /etc/profile

测试查看hadoop版本

[root@hadoop101 opt]# hadoop version
Hadoop 3.2.2

在hadoop安装目录下建立几个文件夹：

mkdir tmp 用来存储临时文件
mkdir -p hdfs/name 作为namenode的数据目录
mkdir -p hdfs/data 作为datanode的数据目录
mkdir logs  日志文件目录
mkdir pid  pid文件目录

配置hadoop核心配置文件

core-site.xml：核心配置文件中指定默认文件系统和临时文件目录

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop101:9000</value>
    </property>
    <!-- 指定Hadoop运行时产生文件的存储目录 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/modules/hadoop-3.2.2/tmp</value>
    </property>
</configuration>

hdfs-site.xml：关于hdfs存储系统相关配置

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop103:50090</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/opt/modules/hadoop-3.2.2/hdfs/name</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.namenode.data.dir</name>
        <value>file:/opt/modules/hadoop-3.2.2/hdfs/data</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
    </property>
</configuration>

mapred-site.xml：MapReduce的相关配置项

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop103:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop103:19888</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>/home/hadoop/modules/hadoop/etc/hadoop:/home/hadoop/modules/hadoop/share/hadoop/common/lib/*:/home/hadoop/modules/hadoop/share/hadoop/common/*:/home/hadoop/modules/hadoop/share/hadoop/hdfs:/home/hadoop/modules/hadoop/share/hadoop/hdfs/lib/*:/home/hadoop/modules/hadoop/share/hadoop/hdfs/*:/home/hadoop/modules/hadoop/share/hadoop/mapreduce/lib/*:/home/hadoop/modules/hadoop/share/hadoop/mapreduce/*:/home/hadoop/modules/hadoop/share/hadoop/yarn:/home/hadoop/modules/hadoop/share/hadoop/yarn/lib/*:/home/hadoop/modules/hadoop/share/hadoop/yarn/*</value>
    </property>
    </property>
</configuration>

yarn-site.xml：yarn相关配置

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop102:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop102:8030</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hadoop102:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>hadoop102:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoop102:8088</value>
    </property>
</configuration>

workers（2.7版本为slaves）：用以指定datanode从节点

10.248.10.69
10.248.10.70
10.248.10.71

hadoop-env.sh：（3.2需要配置，2.7只需要配置java_home）

export JAVA_HOME=/home/hadoop/modules/jdk1.8

export HADOOP_LOG_DIR=/opt/modules/hadoop-3.2.2/logs
export HADOOP_PID_DIR=/opt/modules/hadoop-3.2.2/pid

#### 安装用的是hadoop用户，所以这里都设置为hadoop
export HDFS_NAMENODE_USER=hadoop
export HDFS_DATANODE_USER=hadoop
export HDFS_SECONDARYNAMENODE_USER=hadoop
export YARN_RESOURCEMANAGER_USER=hadoop
export YARN_NODEMANAGER_USER=hadoop

10 格式化hadoop集群

hdfs namenode -format

日志中出行以下日志说明格式化成功：

Storage directory /opt/modules/hadoop-3.2.2/hdfs/name has been successfully formatted.

11 启动验证集群

hadoop101上执行命令

sbin/start-dfs.sh

hadoop102上执行命令

sbin/start-yarn.sh

hadoop103上执行命令

bin/mapred --daemon start historyserver

查看进程

[hadoop@hadoop101 bin]$ xcall jps
--------- hadoop101 ----------
11925 DataNode
13030 NodeManager
11769 NameNode
--------- hadoop102 ----------
82736 NodeManager
81683 DataNode
81948 ResourceManager
--------- hadoop103 ----------
97475 NodeManager
97732 JobHistoryServer
97242 SecondaryNameNode
97101 DataNode

至此，hadoop-3.2.2集群搭建完成！

jackyan163

关注

0
点赞
踩
3

收藏

觉得还不错? 一键收藏
0
评论
0到1搭建hadoop-3.2.2集群

1 安装包准备从官网下载安装包：hadoop-3.2.2.tar.gzhttps://archive.apache.org/dist/hadoop/common/2 集群规划 Hadoop101 Hadoop102 hadoop103 192.168.56.101 NameNode,DataNode DataNode DataNode 192.168.56.102 NodeManager ResourceManager,NodeManage
复制链接

扫一扫