Hadoop分布式集群搭建（一）

__Sniper__

已于 2022-05-03 23:20:41 修改

阅读量2.6k

点赞数

分类专栏：大数据学习之路文章标签： hadoop hive mysql zookeeper mapreduce

于 2022-04-30 16:52:02 首次发布

本文链接：https://blog.csdn.net/qq_36461480/article/details/124514721

版权

大数据学习之路专栏收录该内容

12 篇文章 0 订阅

订阅专栏

1.集群部署准备

1.1 集群组件版本

1.2 集群部署规划

使用三台机器搭建高可用分布式集群

1.3 路径规划

/opt/module # 该目录为组件安装目录
/opt/module/software # 该目录为组件安装包上传和管理目录

1.4 上传安装包

使用root账号登录3台服务器节点，并创建目录

[root@hadoop001 ~]# mkdir -p  /opt/module/software

[root@hadoop002 ~]# mkdir -p  /opt/module/software

[root@hadoop003 ~]# mkdir -p  /opt/module/software

上传hive、hadoop和jdk安装包到hadoop001 software目录下

2.安装JDK

hadoop是用Java语言编写，因此我们需要Java运行环境，安装JDK

2.1 解压到指定目录

[root@hadoop001 software]# tar -zxvf jdk-8u231-linux-x64.tar.gz  -C  ../

2.2 配置环境变量

[root@hadoop001 software]# cd ../jdk1.8.0_231/

#配置环境变量，编辑环境变量配置文件
vim  /etc/profile


#加入如下内容
export JAVA_HOME=/opt/module/jdk1.8.0_231
export PATH=.:$PATH:$JAVA_HOME/bin

ESC退出编辑模式，Shift+zz 保存退出


#使更改生效              
[root@hadoop001 ~]# source /etc/profile


#检查JDK环境变量是否配置成功
[root@hadoop001 ~]# java -version

2.3 安装包分发并配置

将jdk安装目录拷贝到其他两个节点，并配置环境变量

[root@hadoop001 module]# scp -r jdk1.8.0_231/ hadoop002:`pwd` 
[root@hadoop001 module]# scp -r jdk1.8.0_231/ hadoop003:`pwd`

#配置环境变量，编辑环境变量配置文件
vim  /etc/profile


#加入如下内容
export JAVA_HOME=/opt/module/jdk1.8.0_231
export PATH=.:$PATH:$JAVA_HOME/bin

ESC退出编辑模式，Shift+zz 保存退出


#使更改生效              
source /etc/profile


#检查JDK环境变量是否配置成功
java -version

3.Zookeeper集群搭建

3.1 下载Zookeeper压缩包

下载地址：https://archive.apache.org/dist/zookeeper/
访问网址，搜索3.4.9 版本，点击进入，找到zookeeper-3.4.9.tar.gz,复制链接

[root@hadoop001 software]# cd /opt/module/software/
[root@hadoop001 software]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz

3.2 解压安装包

[root@hadoop001 software]# tar -zxvf zookeeper-3.4.9.tar.gz  -C ../

3.3 创建zkdata目录

[root@hadoop001 software]# cd ../zookeeper-3.4.9/
[root@hadoop001 zookeeper-3.4.9]# mkdir zkdata


#修改配置文件

[root@hadoop001 zookeeper-3.4.9]# cd conf/
[root@hadoop001 conf]# cp zoo_sample.cfg zoo.cfg

#修改zoo.cfg配置文件
[root@hadoop001 conf]# vim zoo.cfg


#修改dataDir路径
dataDir=/opt/module/zookeeper-3.4.9/zkdata


#添加集群配置
server.1=hadoop001:2888:3888
server.2=hadoop002:2888:3888
server.3=hadoop003:2888:3888

按ESC键退出编辑模式，shift+zz保存退出

3.4 添加myid配置

[root@hadoop001 conf]# cd /opt/module/zookeeper-3.4.9/zkdata/ 
[root@hadoop001 zkdata]# echo 1 >  myid
[root@hadoop001 zkdata]# cat  myid

3.5 安装包分发并修改myid值

[root@hadoop001 zkdata]# cd  /opt/module/
[root@hadoop001 module]# scp -r zookeeper-3.4.9/ hadoop002:`pwd` 
[root@hadoop001 module]# scp -r zookeeper-3.4.9/ hadoop003:`pwd`

#在hadoop002和hadoop003分别操作
[root@hadoop002 zkdata]# echo 2 > myid 
[root@hadoop002 zkdata]# cat myid 
2

[root@hadoop003 ~]#  echo 3 > myid
[root@hadoop003 ~]# cat myid 
3

3.6 启动Zookeeper服务

三台机器都要启动zookeeper服务，依次登录三台机器上执行命令：
在hadoop001节点：

cd /opt/module/zookeeper-3.4.9/bin/
./zkServer.sh start

#查看各节点状态

./zkServer.sh status

#JPS命令查看进程

4.MySQL数据库安装

4.1 上传mysql安装包

下载地址：https://dev.mysql.com/downloads/mysql/5.7.html#downloads

4.2 查看系统是否安装了MariaDB，有则卸载

[root@hadoop002 software]# rpm -qa |grep mysql
[root@hadoop002 software]# rpm -qa |grep postfix
postfix-2.10.1-6.el7.x86_64
[root@hadoop002 software]# rpm -qa |grep mariadb
mariadb-libs-5.5.56-2.el7.x86_64
[root@hadoop002 software]# rpm -e postfix-2.10.1-6.el7.x86_64
[root@hadoop002 software]# rpm -e mariadb-libs-5.5.56-2.el7.x86_64

#配置yum源


#安装MySQL5.7所需要的依赖
[root@hadoop001 mysql-5.7]# rpm -qa |grep libaio
libaio-0.3.109-13.el7.x86_64
[root@hadoop001 mysql-5.7]# rpm -qa |grep net-tools
net-tools-2.0-0.22.20131004git.el7.x86_64
[root@hadoop001 mysql-5.7]# rpm -qa |grep perl
perl-Text-ParseWords-3.29-4.el7.noarch
perl-PathTools-3.40-5.el7.x86_64
perl-Time-Local-1.2300-2.el7.noarch
perl-Data-Dumper-2.145-3.el7.x86_64
perl-DBI-1.627-4.el7.x86_64
perl-podlators-2.5.1-3.el7.noarch
perl-threads-1.87-4.el7.x86_64
perl-macros-5.16.3-292.el7.x86_64
perl-5.16.3-292.el7.x86_64
perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64
perl-Pod-Perldoc-3.20-4.el7.noarch
perl-Pod-Usage-1.63-3.el7.noarch
perl-threads-shared-1.43-6.el7.x86_64
perl-File-Path-2.09-2.el7.noarch
perl-Time-HiRes-1.9725-3.el7.x86_64
perl-Socket-2.010-4.el7.x86_64
perl-Error-0.17020-2.el7.noarch
perl-Net-Daemon-0.48-5.el7.noarch
perl-IO-Compress-2.061-2.el7.noarch
perl-XML-Parser-2.41-10.el7.x86_64
perl-Git-1.8.3.1-11.el7.noarch
perl-parent-0.225-244.el7.noarch
perl-Pod-Escapes-1.04-292.el7.noarch
perl-libs-5.16.3-292.el7.x86_64
perl-Carp-1.26-244.el7.noarch
perl-Exporter-5.68-3.el7.noarch
perl-File-Temp-0.23.01-3.el7.noarch
perl-Pod-Simple-3.28-4.el7.noarch
perl-Thread-Queue-3.02-2.el7.noarch
perl-Compress-Raw-Zlib-2.061-4.el7.x86_64
perl-PlRPC-0.2020-14.el7.noarch
perl-HTTP-Tiny-0.033-3.el7.noarch
perl-Scalar-List-Utils-1.27-248.el7.x86_64
perl-Storable-2.45-3.el7.x86_64
perl-srpm-macros-1-8.el7.noarch
perl-Getopt-Long-2.40-2.el7.noarch
perl-TermReadKey-2.30-20.el7.x86_64
perl-Test-Harness-3.28-3.el7.noarch
perl-Encode-2.51-7.el7.x86_64
perl-Filter-1.49-3.el7.x86_64
perl-constant-1.27-2.el7.noarch
perl-DBD-SQLite-1.39-3.el7.x86_64

#如果没有安装，使用yum源安装mysql所需的依赖包
[root@hadoop002 software]# yum -y  install libaio 
[root@hadoop002 software]# yum -y  install net-tools 
[root@hadoop002 software]# yum -y  install perl

4.3 安装MySQL

4.3.1 解压安装包

#hadoop002节点
[root@hadoop002 software]# mkdir /opt/module/mysql-5.7
[root@hadoop002 software]# tar -xvf  mysql-5.7.36-1.el7.x86_64.rpm-bundle.tar -C 
/opt/module/mysql-5.7/
mysql-community-client-5.7.36-1.el7.x86_64.rpm
mysql-community-common-5.7.36-1.el7.x86_64.rpm
mysql-community-devel-5.7.36-1.el7.x86_64.rpm
mysql-community-embedded-5.7.36-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.36-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.36-1.el7.x86_64.rpm
mysql-community-libs-5.7.36-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.36-1.el7.x86_64.rpm
mysql-community-server-5.7.36-1.el7.x86_64.rpm
mysql-community-test-5.7.36-1.el7.x86_64.rpm

4.3.2 安装 MySQL 包（按以下顺序安装）

[root@hadoop002 mysql-5.7]# rpm  -ivh  mysql-community-common-5.7.36- 1.el7.x86_64.rpm
[root@hadoop002 mysql-5.7]# rpm  -ivh  mysql-community-libs-5.7.36-1.el7.x86_64.rpm 
[root@hadoop002 mysql-5.7]# rpm  -ivh  mysql-community-client-5.7.36-1.el7.x86_64.rpm 
[root@hadoop002 mysql-5.7]# rpm  -ivh  mysql-community-server-5.7.36-1.el7.x86_64.rpm

4.4 启动MySQL服务

[root@hadoop002 mysql-5.7]# systemctl status mysqld
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
[root@hadoop002 mysql-5.7]# systemctl start mysqld

[root@hadoop002 mysql-5.7]# 
[root@hadoop002 mysql-5.7]# systemctl  status mysqld
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2022-04-26 15:14:51 CST; 8s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 45720 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
  Process: 45657 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 45724 (mysqld)
   CGroup: /system.slice/mysqld.service
           └─45724 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/...

4月 26 15:14:33 hadoop002 systemd[1]: Starting MySQL Server...
4月 26 15:14:51 hadoop002 systemd[1]: Started MySQL Server.

#设置开机启动
[root@hadoop002 software]# systemctl enable mysqld 
[root@hadoop002 software]# systemctl daemon-reload

4.4 配置MySQL root密码

MySQL安装完成之后，会在/var/log/mysqld.log 文件中给 root 生成了一个临时的默认密码，查看方式：

[root@hadoop002 mysql-5.7]# cat  /var/log/mysqld.log | grep password
2022-04-26T07:14:49.414725Z 1 [Note] A temporary password is generated for root@localhost: **-;elsW6iwNb1**

[root@hadoop002 mysql-5.7]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.36

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

4.5 修改root密码

# 调整密码的验证强度等级，进行如下操作，如果以上步骤成功则跳过下面步骤 
mysql> set  global  validate_password_policy=LOW;
Query OK, 0 rows affected (0.00 sec) 

# 设置   5 位密码长度策略
mysql> set  global  validate_password_length=5;
Query OK, 0 rows affected (0.00 sec)

# 修改root账户密码为root,自定义
mysql> ALTER  USER 'root'@'localhost' IDENTIFIED BY 'mysql';

# 设置远程登录
mysql> GRANT  ALL PRIVILEGES  ON *.* TO  'root'@'%' IDENTIFIED BY 'mysql'  WITH GRANT  OPTION;

#退出MySQL
mysql> exit;

4.6 配置MySQL默认编码为utf-8

[root@hadoop002 software]# vim /etc/my.cnf 

添加如下代码：
character_set_server=utf8
init_connect='SET NAMES utf8'

esc退出编辑模式，shift+zz 保存退出

4.7 重启MySQL

[root@hadoop002 software]# systemctl restart mysqld

4.8 查看MySQL编码

mysql> show  variables like '%character%';


![image-20220426153129696](https://img-blog.csdnimg.cn/img_convert/1ea936007d0a09e92f6ed24879b3cdbd.png)


[Hadoop分布式集群搭建（二）](https://blog.csdn.net/qq_36461480/article/details/124515285) 
[Hadoop分布式集群搭建（三）](https://blog.csdn.net/qq_36461480/article/details/124515428?spm=1001.2014.3001.5502)