docker kafka实战

基于：阿里云centos8,关闭了防火墙，限定ip开放全端口

利用docker compose构建多服务项目

• zookeeper–协调、配置管理
• kafka–消息中间件
• kafka-manager–kafka可视化

#docker-compose.yml
version: "3.3"
services:
  zookeeper:
      image: zookeeper:3.5.5
      restart: always
      container_name: zookeeper
      ports:
          - "2181:2181"
      expose:
          - "2181"
      environment:
          - ZOO_MY_ID=1
  kafka:
      image: wurstmeister/kafka
      restart: always
      container_name: kafka
      environment:
          - KAFKA_BROKER_ID=1
          - KAFKA_LISTENERS=PLAINTEXT://kafka:9090
          - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
          - KAFKA_MESSAGE_MAX_BYTES=2000000
          - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9090
          - KAFKA_ADVERTISED_HOST_NAME=kafka
      ports:
          - "9090:9090"
      depends_on:
          - zookeeper
  kafka-manager:
    image: sheepkiller/kafka-manager                ## 镜像：开源的web管理kafka集群的界面
    environment:
        ZK_HOSTS: zookeeper                         ## 修改:宿主机IP
    ports:
      - "9000:9000"                                 ## 暴露端口

#启动服务
docker-compose -f docker-compose.yml up -d

可视化访问界面url

http://ip:9000/

防火墙设置（可选）

firewall-cmd  --zone=trusted --add-source=172.18.0.1/16 --permanent
firewall-cmd  --reload

容器相关操作

#进入kafka容器
docker exec -ti kafka bash

#进入bin目录
#cd opt/kafa_<版本>/bin

#创建topic
kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic topic1

#查看topic
kafka-topics.sh --list --zookeeper zookeeper:2181

//创建生产者
kafka-console-producer.sh --broker-list zookeeper:9090 --topic topic1

//创建消费者 
kafka-console-consumer.sh --bootstrap-server zookeeper:9090 --topic topic1 --from-beginning

spring boot配置文件(demo代码略）

spring:
  kafka.:
    bootstrap-servers: 47.106.13.167:9090
    listener:
      concurrency: 10
      ack-mode: MANUAL_IMMEDIATE
      poll-timeout: 1500
    consumer:
      group-id: cluster1
      enable-auto-commit: false
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      properties: {session.timeout.ms: 6000, auto.offset.reset: earliest}
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      batch-size: 65536
      buffer-memory: 524288

本地配置

本地连接kafka需要配置host，若不配置，本地测试spring boot项目会报unknowhost异常

Windows:C:\Windows\System32\drivers\etc\hosts
Linux:/etc/hosts
增加一行：kafka宿主机ip kafka服务名
如：47.106.13.167 kafka

refs:

https://www.cnblogs.com/Lambquan/p/13649715.html

https://blog.csdn.net/qq_39141239/article/details/108508194