基于:阿里云centos8,关闭了防火墙,限定ip开放全端口
利用docker compose构建多服务项目
- zookeeper–协调、配置管理
- kafka–消息中间件
- kafka-manager–kafka可视化
#docker-compose.yml
version: "3.3"
services:
zookeeper:
image: zookeeper:3.5.5
restart: always
container_name: zookeeper
ports:
- "2181:2181"
expose:
- "2181"
environment:
- ZOO_MY_ID=1
kafka:
image: wurstmeister/kafka
restart: always
container_name: kafka
environment:
- KAFKA_BROKER_ID=1
- KAFKA_LISTENERS=PLAINTEXT://kafka:9090
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_MESSAGE_MAX_BYTES=2000000
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9090
- KAFKA_ADVERTISED_HOST_NAME=kafka
ports:
- "9090:9090"
depends_on:
- zookeeper
kafka-manager:
image: sheepkiller/kafka-manager ## 镜像:开源的web管理kafka集群的界面
environment:
ZK_HOSTS: zookeeper ## 修改:宿主机IP
ports:
- "9000:9000" ## 暴露端口
#启动服务
docker-compose -f docker-compose.yml up -d
可视化访问界面url
http://ip:9000/
防火墙设置(可选)
firewall-cmd --zone=trusted --add-source=172.18.0.1/16 --permanent
firewall-cmd --reload
容器相关操作
#进入kafka容器
docker exec -ti kafka bash
#进入bin目录
#cd opt/kafa_<版本>/bin
#创建topic
kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic topic1
#查看topic
kafka-topics.sh --list --zookeeper zookeeper:2181
//创建生产者
kafka-console-producer.sh --broker-list zookeeper:9090 --topic topic1
//创建消费者
kafka-console-consumer.sh --bootstrap-server zookeeper:9090 --topic topic1 --from-beginning
spring boot配置文件(demo代码略)
spring:
kafka.:
bootstrap-servers: 47.106.13.167:9090
listener:
concurrency: 10
ack-mode: MANUAL_IMMEDIATE
poll-timeout: 1500
consumer:
group-id: cluster1
enable-auto-commit: false
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
properties: {session.timeout.ms: 6000, auto.offset.reset: earliest}
producer:
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
batch-size: 65536
buffer-memory: 524288
本地配置
本地连接kafka需要配置host,若不配置,本地测试spring boot项目会报unknowhost异常
Windows:C:\Windows\System32\drivers\etc\hosts
Linux:/etc/hosts
增加一行:kafka宿主机ip kafka服务名
如:47.106.13.167 kafka
refs:
https://www.cnblogs.com/Lambquan/p/13649715.html
https://blog.csdn.net/qq_39141239/article/details/108508194