spark集群，kafka集群和zookeeper的docker集群快速搭建

最新推荐文章于 2023-11-14 10:48:07 发布

醉与浮

最新推荐文章于 2023-11-14 10:48:07 发布

阅读量481

点赞数

分类专栏： tools 文章标签： docker spark kafka

本文链接：https://blog.csdn.net/weixin_45409055/article/details/127359323

版权

tools 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

准备操作

安装docker(宿主机)

yum install docker (宿主机时centos用yum管理包)

systemctl daemon-reload(重加载deamon服务，是docker的核心)

systemctl restart docker.service (重启docker服务)

docker常用命令

docker ps -a 查看所有容器
docker search Ubuntu 寻找Ubuntu的镜像
docker pull ubuntu 拉取ubuntu的镜像
docker images 查看所有镜像
docker run -it -v /home:/MyData -h kafka1 -p 9092:9092 --privileged=true --name kafka1 ubuntu
在本机运行容器，共享空间是物理机的/home和容器的/Mydata hostname是kafka1 端口固定9092，给交换空间赋权,容器名是kafka1 镜像是Ubuntu(这是初次启动的命令)
docker run -it -v /home:/mydata -h hadoop1 -p 50070:50070 -p 8088:8088 --privileged=true --name hadoop1 ubuntu
开放多个端口
docker start kafka1 启动kafka1
docker exec -it kafka1 bash
docker kill kafka1 关闭kafka1
docker rm kafka1 删除kafka1
docker commit 6e17cba41586 kafka:v0 提交该容器为镜像
docker image rm 镜像名 删除镜像

apt换源

deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-security main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse

安装zookeeper和kafka

/usr/lib/jvm/java-18-openjdk-amd64  JAVA目录
换源报错apt install ca-certificates 
export JAVA_HOME=/usr/lib/jvm/java-18-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME CLASSPATH PATH

sources.list在/etc/apt
rm sources.list 删除源文件
touch sources.list 创建源文件
mv回去
apt-get update 更新列表
apt-get upgrade 更新包
apt-get install vim
apt-get install openjdk-18-jdk

apt-get install net-tools 可以用ifconfig看ip
apt-get install wget 一会儿用wget下载文件
wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz 下载zookeeper
tar -zxvf axxxx 解压到当前目录
修改conf目录zoo——sampe为zoo，配置数据存储地址和日志存储地址
ps -aux |grep zookeeper 查看zookeeper运行情况
./zkServer.sh start
./zkServer.sh stop
./kafka-server-start.sh ../config/server.properties & 启动kafka
在zookeeper bin目录zkCli查看节点 ls /bin/brokers/ids 可以看到节点号
创建topic
./kafka-topics.sh --bootstrap-server kafka1:9092 --topic bigdata --create --partitions 1 --replication-factor 1
从kafka1.0开始，bootstrap参数写要操作的kafka机器的地址。不是zookeeper的地址。
./kafka-topics.sh --list --bootstrap-server kafka1:9092 查看topic list
./kafka-console-producer.sh --bootstrap-server kafka1:9092 --topic bigdata
./kafka-console-consumer.sh --bootstrap-server kafka1:9092 --topic bigdata

安装hadoop

先安装java环境和基础组件和wget和ssh
修改hosts
172.17.0.2      kafka1
172.17.0.3      kafka2
172.17.0.4      kafka3
172.17.0.5      hadoop1
172.17.0.6      hadoop2
172.17.0.7      hadoop3

wget https://downloads.apache.org/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz
vim /etc/profile
export HADOOP_HOME=
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin


vim ./etc/hadoop/hadoop.env.sh
添加javahome

修改四个配置文件
ssh免密登录是密码bug，手动设置密码就行。用passwd命令查看并设置新密码
三个docker密码都是root
重新修改配置yarn-env.sh
export YARN_RESOURCEMANAGER_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED"
export YARN_NODEMANAGER_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED"
修改yarn-site.xml
修改mappred-site.xml

spark

wget https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3.tgz
tar -zxvf xxx

export SPARK_HOME=/soft/spark
export PATH=$SPARK_HOME/bin:$HBASE_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$PATH
bin/shpark-shell.sh 启动