Kafka模拟集群构建(Docker篇)

版权声明:欢迎转载,转载请说明出处https://csdn.yanxml.com。大数据Github项目地址https://github.com/SeanYanxml/bigdata。 https://blog.csdn.net/u010416101/article/details/84332794

前言

周一,送走了之前的小伙伴离开了上海。哎,在上海能够说话的人又少了一个。好了,不多说,我们开始今天的话题。


前置要求

之前,我们成功的在Docker内构建了ZooKeeper镜像,今天我们在这个镜像的基础上进行操作。

  • docker
  • yanxml/zookeeper
  • docker compose(compose命令需要)

正式操作

  • 首先,从官方下载Kafka安装包。
wget "https://archive.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz"
  • 解压安装包,并且修改server.properties文件。
tar -zxvf kafka_2.11-1.0.0.tgz
mv kafka_2.11-1.0.0 app
mkdir kafka1
cp -rp app kafka1/

主要修改server.properties内的如下内容,其余按照默认文件为准:

# kafka1 server.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
port=9092
num.partitions=3
# kafka存储本地data的地址
log.dirs=/opt/apps/kafka/kafka1/data
advertised.host.name=192.168.1.16(本地IP为准)
advertised.host.port=9092

# kafka2 server.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
port=9093
log.dirs=/opt/apps/kafka/kafka2/data
num.partitions=3
advertised.host.name=192.168.1.16(本地IP为准)
advertised.host.port=9093

# kafka3 server.properties
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2
port=9094
log.dirs=/opt/apps/kafka/kafka3/data
num.partitions=3
advertised.host.name=192.168.1.16(本地IP为准)
advertised.host.port=9094
  • 顺便编写kafka启动脚本
# start.sh
#!/bin/bash

source /etc/profile

nohup /opt/apps/kafka/kafka1/app/bin/kafka-server-start.sh /opt/apps/kafka/kafka1/app/config/server.properties >  /dev/null 2>&1 &

nohup /opt/apps/kafka/kafka2/app/bin/kafka-server-start.sh /opt/apps/kafka/kafka2/app/config/server.properties >  /dev/null 2>&1 &

nohup /opt/apps/kafka/kafka3/app/bin/kafka-server-start.sh /opt/apps/kafka/kafka3/app/config/server.properties >  /dev/null 2>&1 &
  • 自动化配置脚本

我想在启动的时候传递docker环境变量进入,随后能够修改本地server.properties文件的内容。

### cat env-config.sh

#!/bin/bash

if [ "${advertised_host_name}" != "" ]; then
	sed -i 's/^advertised.host.name=.*/advertised.host.name='$advertised_host_name'/' ../app/config/server.properties;
fi
  • 随后更新下docker的启动脚本
#!/bin/bash

# source jdk
source /etc/profile

# zookeeper
/opt/apps/zookeeper/bin/start.sh

# kafka
/opt/apps/kafka/bin/env-config.sh
/opt/apps/kafka/bin/start.sh

tail -f /opt/apps/bin/tmp.log

Questions

如何在docker外部访问docker内的kafka?
对于这个问题,需要单独解答。(篇幅过长)
主要的解决办法主要为2种:

  1. 使用两个端口进行配置
    listeners=OUTSIDE://:9093,INSIDE://:9092
    advertised.listeners=OUTSIDE://192.168.1.16:9093,INSIDE://:9092
    listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
    inter.broker.listener.name=INSIDE

  2. 配置advertised.host.name (可能已经过期)
    advertised.host.name=192.168.1.16(本地IP为准)
    advertised.host.port=9094

PS:(Each listener must have a different port, listeners: OUTSIDE://:9092,INSIDE://:9092)


Use the image

  • 下载并,启动镜像。(注意ip:192.168.1.16 改成本地ip)
 docker run -d -p 2181:2181 -p 2182:2182 -p 2183:2183  -p 9092:9092 -p 9093:9093 -p 9094:9094  --env advertised_host_name=192.168.1.16 yanxml/kafka-cluster:0.0.5  /opt/apps/bin/start.sh

Reference

[1]. 阿里云构建Kafka单机集群环境
[2]. DockerHub: kafka-cluster
[3].

没有更多推荐了,返回首页