1.1完全分布式Hadoop署集群

完全分布式Hadoop署集群

大家好，我是行不更名，坐不改姓的宋晓刚，下面将带领大家从基础到小白Hadoop学习，跟上我的步伐进入Hadoop的世界。
微信：15319589104
QQ： 2981345658
文章介绍：
在大数据时代，Hadoop已成为处理海量数据的必备工具。但如何从零开始搭建一个完整的Hadoop分布式集群呢？本文将为你详细介绍这一过程，帮助你轻松搭建自己的Hadoop集群，从硬件准备到集群配置，再到优化与维护，每一步都为你详细解读。

1.1部署Hadoop集群

在前面单机模式下克隆出来三台虚拟机分别是HadoopMaster，HadoopSlave，HadoopSlave1

1.0修改主机名

1.1修改主机1的名字为HadoopMaster

[hadoop@master ~]$ hostnamectl set-hostname hadoopmaster
[hadoop@master ~]$ bash
[hadoop@hadoopmaster ~]$ 

1.2修改主机2的名字为HadoopSlave

[hadoop@master sbin]$ hostnamectl set-hostname hadoopslave
[hadoop@master ~]$ bash
[hadoop@hadoopslave ~]$ 

1.3修改主机3的名字为HadoopSlave1

[hadoop@master ~]$ hostnamectl set-hostname hadoopslave1
[hadoop@master ~]$ bash
[hadoop@hadoopslave1 ~]$ 

2.0添加主机映射

2.1添加主机HadoopMaster的映射

[hadoop@hadoopmaster ~]$ sudo vi /etc/hosts
192.168.139.136 hadoopmaster  //主机1
192.168.139.138 hadoopslave  //主机2
192.168.139.1   hadoopslave1  //主机3

2.2将/etc/hosts文件复制给我主机hadoopslave和hadoopslave1上

[hadoop@hadoopmaster ~]$ sudo scp /etc/hosts 192.168.139.138:/etc/hosts  //主机的ip地址
[hadoop@hadoopmaster ~]$ sudo scp /etc/hosts 192.168.139.1:/etc/hosts  

2.3在主机hadoopmaster上查看主机映射：

[hadoop@hadoopmaster ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.139.136 hadoopmaster
192.168.139.138 hadoopslave
192.168.139.1   hadoopslave1

2.4在主机hadoopslave上查看主机映射：

[hadoop@hadoopslave ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.139.136 hadoopmaster
192.168.139.138 hadoopslave
192.168.139.1   hadoopslave1

2.5在主机hadoopslave1上查看主机映射：

[hadoop@hadoopslave1 ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.139.136 hadoopmaster
192.168.139.138 hadoopslave
192.168.139.1   hadoopslave1

3.0配置免密登录

3.1为主机hadoopmaster配置免密登录：

[hadoop@hadoopmaster ~]$ ssh-keygen
[hadoop@hadoopmaster ~]$ ssh-copy-id hadoopmaster
[hadoop@hadoopmaster ~]$ ssh-copy-id hadoopslave
[hadoop@hadoopmaster ~]$ ssh-copy-id hadoopslave1

3.2验证免密登录
各节点之间需要进行验证，通过主机名称判断当前主机

[hadoop@hadoopmaster ~]$ ssh hadoopmaster   //主机1
Last login: Thu Apr  6 20:55:43 2023
[hadoop@hadoopmaster ~]$ exit
登出
Connection to hadoopmaster closed.
[hadoop@hadoopmaster ~]$ ssh hadoopslave   //主机2
Last login: Thu Apr  6 21:23:24 2023 from master
[hadoop@hadoopslave ~]$ exit
登出
Connection to hadoopslave closed.
[hadoop@hadoopmaster ~]$ ssh hadoopslave1  //主机3
Last login: Thu Apr  6 21:43:16 2023
[hadoop@hadoopslave1 ~]$ exit
登出
Connection to hadoopslave1 closed.

4.0设置时间同步

各个主机的时钟与ntp1.aliyum.com时钟保持同步

[hadoop@hadoopmaster ~]$ sudo ntpdate ntp1.aliyum.com
[hadoop@hadoopslave ~]$ sudo ntpdate ntp1.aliyum.com
[hadoop@hadoopslave1 ~]$ sudo ntpdate ntp1.aliyum.com

5.0配置hadoop

部署hadoop集群时，每个节点上的hadoop配置基本相同，只需要在hadoopMaster节点操作，配置完成后复制到hadoopSlave和hadoopSlave1上就可以了
在hadoopmaster中进行以下设置：
5.1设置环境变量hadoop-env.sh的内容如下：

[hadoop@hadoopmaster ~]$ cd hadoop/etc/hadoop/
[hadoop@hadoopmaster hadoop]$ vi hadoop-env.sh
export JAVA_HOME=/home/hadoop/java  //因为我们将jdk的名字修改为java

5.2设置核心组件core-site.xml的内容如下:

[hadoop@hadoopmaster hadoop]$ vi core-site.xml 

<configuration>
        <property>
                <name>fs.defaultFS