使用docker-19.03.14搭建hadoop-2.9.2完全分布式集群-之一-规划和环境

最新推荐文章于 2024-06-15 23:25:38 发布

szlcy

最新推荐文章于 2024-06-15 23:25:38 发布

阅读量282

点赞数

分类专栏：开船文章标签： hadoop hive spark

本文链接：https://blog.csdn.net/szlcy/article/details/110874443

版权

开船专栏收录该内容

24 篇文章 0 订阅

订阅专栏

集群规划

在宿主机上安装 docker，在 docker 上部署 4 个容器，包括 3 个 hadoop 节点和 1 个客户端。
使用 mysql 来存放 hive 的 metadata 元数据。

宿主机IP: 192.168.109.128
docker 网络: 172.18.0.1/24

Container	IP	Role
hadoop11	172.18.0.11	NameNode,SecondaryNameNode,ResourceManager,JobHistory,DataNode
hadoop12	172.18.0.12	DataNode
hadoop13	172.18.0.13	DataNode
client	172.18.0.10	Client

安装组件

hadoop-2.9.2
hive-2.4.7
spark-2.2.0

环境及软件版本

CentOS-7.4.1708
jdk-8u251
hadoop-2.9.2
apache-hive-2.3.7
scala-2.11.8
spark-2.2.0
mysql-connector-java-5.1.49

系统用户
user: root
pass: 1qt5!QT%

软件包位置
/root/jdk-8u251-linux-x64.tar.gz
/root/apache-hive-2.3.7-bin.tar.gz
/root/hadoop-2.9.2.tar.gz
/root/scala-2.11.8.tgz
/root/spark-2.2.0-bin-hadoop2.7.tgz
/root/mysql-connector-java-5.1.49.tar.gz

准备 Mysql

# mysql 仓库配置
[root@do-hadoop ~]# yum -y install wget
[root@do-hadoop ~]# wget https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm
[root@do-hadoop ~]# yum -y install mysql80-community-release-el7-3.noarch.rpm
[root@do-hadoop ~]# yum makecache fast

# 安装 mysql
[root@do-hadoop ~]# yum list | grep mysql | grep server
mysql-community-server.x86_64               8.0.22-1.el7               mysql80-community

[root@do-hadoop ~]# yum -y install mysql-community-server.x86_64

# 配置 mysql
[root@do-hadoop ~]# systemctl enable mysqld
[root@do-hadoop ~]# systemctl start mysqld
[root@do-hadoop ~]# grep password /var/log/mysqld.log
2020-12-01T10:32:16.773403Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: <5GjpX_mNchV
[root@do-hadoop ~]# mysql -u root -p
Enter password: <5GjpX_mNchV
mysql> alter user 'root'@'localhost' identified by '1qt5!QT%';
mysql> create user 'root'@'%' identified by '1qt5!QT%';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%';
mysql> flush privileges;
mysql> create database hive;
mysql> quit;
[root@do-hadoop ~]#

禁用防火墙

[root@do-hadoop ~]# systemctl stop firewalld
[root@do-hadoop ~]# systemctl disable firewalld
[root@do-hadoop ~]# setenforce 0
[root@do-hadoop ~]# vi /etc/selinux/config
# 找到并按如下项配置
SELINUX=disabled