Docker基于Hadoop HBase Hive 构建三节点集群

一、准备工作

       1、树莓派4B 8g版本

       2、安装Centos7.9系统，并安装Docker环境

       3、下载所需配置文件 

       4、获取Docker镜像文件

            4.1、获取mysql镜像：

docker pull mysql:8.0

            4.2、获取已安装好Hadoop、HBase、Hive、Zookeeper的镜像

                     ARM平台:  

docker pull lmmu00/arm_hive:latest

                     其他平台:  

docker pull lmmu00/hive:latest

二、上传配置文件到系统的opt 目录下并对所有目录更改权限(此操作为必要操作)

chmod -R 777 /opt/mysql/*
chmod -R 777 /opt/master/*
chmod -R 777 /opt/slave01/*
chmod -R 777 /opt/slave02/*

三、启动MySQL

docker run -itd -h mysql --privileged --name mysql -p 3306:3306 -v /opt/mysql/conf/my.cnf:/etc/my.cnf -v /opt/mysql/data:/opt/data -v /opt/mysql/log:/opt/log   -e MYSQL_ROOT_PASSWORD=root  mysql:8.0 

四、启动Hive相关集群

docker run -itd -h master --privileged --name master -v /opt/master/data:/opt/data -v /opt/master/log:/opt/log  -v /opt/master/conf/zookeeper:/usr/local/zookeeper-3.7.1/conf  -v /opt/master/conf/hadoop:/usr/local/hadoop-3.3.1/etc/hadoop  -v /opt/master/conf/hbase:/usr/local/hbase-2.4.16/conf -v /opt/master/conf/hive:/usr/local/hive-3.1.3/conf  lmmu00/arm_hive:latest

docker run -itd -h slave01 --privileged --name slave01 -v /opt/slave01/data:/opt/data -v /opt/master/log:/opt/log -v /opt/slave01/conf/zookeeper:/usr/local/zookeeper-3.7.1/conf  -v /opt/slave01/conf/hadoop:/usr/local/hadoop-3.3.1/etc/hadoop  -v /opt/slave01/conf/hbase:/usr/local/hbase-2.4.16/conf -v /opt/slave01/conf/hive:/usr/local/hive-3.1.3/conf  lmmu00/arm_hive:latest

docker run -itd -h slave02 --privileged --name slave02 -v /opt/slave02/data:/opt/data -v /opt/master/log:/opt/log -v /opt/slave02/conf/zookeeper:/usr/local/zookeeper-3.7.1/conf  -v /opt/slave02/conf/hadoop:/usr/local/hadoop-3.3.1/etc/hadoop  -v /opt/slave02/conf/hbase:/usr/local/hbase-2.4.16/conf -v /opt/slave02/conf/hive:/usr/local/hive-3.1.3/conf  lmmu00/arm_hive:latest

默认ip配置:

      MySQL    172.17.0.2

      master     172.17.0.3

      slave01    172.17.0.4

      slave02    172.17.0.5

 五、进入Hive集群服务器

docker exec -it 7c1959523228 /bin/bash

六、编辑hosts文件

vim /etc/hosts

添加内容(master、slave01、slave02都添加)

172.17.0.3      master
172.17.0.4      slave01
172.17.0.5      slave02

七、启动zookeeper(顺序:master-->slave01-->slave02)

/usr/local/zookeeper-3.7.1/bin/zkServer.sh start

八、Hadoop初始化、启动(master容器中)

/usr/local/hadoop-3.3.1/bin/hdfs namenode -format
/usr/local/hadoop-3.3.1/sbin/start-all.sh

九、启动HBase(master容器中) 

/usr/local/hbase-2.4.16/bin/start-hbase.sh

 

十、初始化Hive的MySQL库

schematool -initSchema -dbType mysql

十一、进入Hive控制台

hive

 

        到此，三节点集群部署完成。如需通过web形式访问容器内部可通过docker 映射端口的方式实现。