kafka内外网配置

看此文档，需要首先会安装kafka集群，zookeeper集群；熟悉组件命令；

三台kafka集群（设置加密通道）：

     

192.168.1.86:9092	外网IP:19092
192.168.1.87:9092	外网IP:19093
192.168.1.88:9092	外网IP:19094

修改kafka server.properties  其中listeners 和 advertised.listeners 中 EXTERNAL的端口必须一致

1. 修改86 kafka 内外网配置

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19092
advertised.listeners=INTERNAL://192.168.1.86:9092,EXTERNAL://外网IP:19092
inter.broker.listener.name=INTERNAL
listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT

2. 修改87 kafka 内外网配置

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19093
advertised.listeners=INTERNAL://192.168.1.87:9092,EXTERNAL://外网IP:19093
inter.broker.listener.name=INTERNAL
listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT

3. 修改88 kafka 内外网配置

listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19094
advertised.listeners=INTERNAL://192.168.1.88:9092,EXTERNAL://外网IP:19094
inter.broker.listener.name=INTERNAL
listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT

4. 注释配置

# security.inter.broker.protocol=SASL_PLAINTEXT
# listeners=SASL_PLAINTEXT://0.0.0.0:9092
# advertised.listeners=SASL_PLAINTEXT://192.168.1.86:9092

5. 最终配置demo

broker.id=0
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
# security.inter.broker.protocol=INTERNAL
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
super.users=User:admin
# listeners=SASL_PLAINTEXT://0.0.0.0:9092
# advertised.listeners=SASL_PLAINTEXT://192.168.1.86:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.cleanup.policy=delete
log.retention.hours=72
log.segment.bytes=1073741824
zookeeper.connect=192.168.1.86:2181,192.168.1.87:2181,192.168.1.88:2181
delete.topic.enable=true
zookeeper.connection.timeout.ms=60000


listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:19092
advertised.listeners=INTERNAL://192.168.1.86:9092,EXTERNAL://外网IP:19092
inter.broker.listener.name=INTERNAL

# 这个协议底层使用了哪种安全协议 PLAINTEXT  不加密；SASL_PLAINTEXT 普通加密；SSL 高级加密
listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,INTERNAL:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT

操作kafka集群命令集合：

# 如何进入docker：
docker ps -a 查看所有进程
docker exec -it 容器id bash / 使用管理员权限进入容器）
docker exec -it -u root b667f9cfa164 bash
docker exec -it test bash

#杀掉zk和kafka
ps -ef|grep -E "kafka|zookeeper"|grep -v grep |awk '{print $2}'|while read pid;do kill -9 $pid;done
ps -ef|grep -E "kafka" |grep -v grep |awk '{print $2}'|while read pid;do kill -9 $pid;done
ps -ef|grep -E "zookeeper"|grep -v grep |awk '{print $2}'|while read pid;do kill -9 $pid;done

#启动zk （清除数据：需要删除 zoo.cfg 中 dataDir指向的文件夹下所有的非 myid的文件和文件夹）
/usr/local/zookeeper/bin/zkServer.sh start /usr/local/zookeeper/conf/zoo.cfg

# 查看zk的状态 （启动状态不正确时，查看zoo.cfg 中 dataDir 下的 zookeeper.out日志）
/usr/local/zookeeper/bin/zkServer.sh status /usr/local/zookeeper/conf/zoo.cfg

# 临时启动kafka （查看是否有异常，根据日志调试；查看server.properties中log.dirs指向 删除干净）
/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties

# 启动kafka 
nohup /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties >>/usr/local/kafka/logs/kafka.log 2>1 &

# 查看topic列表
/usr/local/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list

# 删除topic
/usr/local/kafka/bin/kafka-topics.sh --delete --zookeeper 127.0.0.1:2181 --topic douzi-test

# 创建topic
/usr/local/kafka/bin/kafka-topics.sh --create --topic douzi-test --replication-factor 3 --partitions 3 --zookeeper localhost:2181

# 接收消息：（普通模式）
./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic douzi-test

# 接收消息：（加密情况下的命令）
/usr/local/kafka/bin/reader-kafka-console-consumer.sh --bootstrap-server 0.0.0.0:9092 --topic douzi-test --from-beginning --consumer.config /usr/local/kafka/producer.config

# 发送消息：
./kafka-console-producer.sh --broker-list localhost:9092 --topic douzi-test

# 发送消息：(加密情况下的命令)
/usr/local/kafka/bin/writer-kafka-console-producer.sh --broker-list localhost:9092 --topic douzi-test --producer.config /usr/local/kafka/producer.config