docker-compose构建kafka集群

一、基本概念

二、集群安装部署

1、环境准备：

• 安装docker环境
• 安装docker-compose

kafka集群：
192.168.10.11
192.168.10.12
192.168.10.13

2、部署

kafka需要依赖zookeeper，所以要先启动zookeeper

vim kafka-compose

version: '2.0'
services:
  zookeeper:
    image: zookeeper
    container_name: zk
    network_mode: "host"
    restart: always
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=192.168.10.11:2888:3888;2181 server.2=192.168.10.12:2888:3888;2181 server.3=192.168.10.13:2888:3888;2181
      ZOO_AUTOPURGE_PURGEINTERVAL: 72
      TZ: Asia/Shanghai
    volumes:
    - /data/zookeeper/data:/data
    - /data/zookeeper/logs:/datalog
    - /etc/localtime:/etc/localtime
    ports:
    - "2181:2181"
  
  kafka:
    image: wurstmeister/kafka
    depends_on:
    - zookeeper
    container_name: kafka
    network_mode: "host"
    restart: always
    ports:
    - "9092:9092"
    environment:
      CLUSTER_NAME: kafka1
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_HOST_NAME: 192.168.10.11
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.10.11:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: 192.168.10.11:2181,10.3.4.144:2181,192.168.10.12:2181,192.168.10.13:2181,10.3.4.147:2181
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_LOG_RETENTION_HOURS: 72
      TZ: Asia/Shanghai
    volumes:
    - /data/kafka/docker.sock:/var/run/docker.sock
    - /data/kafka/logs/:/kafka
    - /etc/localtime:/etc/localtime

kafka配置文件关键参数详解

broker.id    //broker的全局唯一编号，不能重复
KAFKA_ADVERTISED_HOST_NAME //为本机ip
KAFKA_LISTENERS  //kafka的broker将监听地址表，只有与该ip匹配正确的客户端能成功连接到kafka
KAFKA_ADVERTISED_LISTENERS  //指向Broker的可用地址列表，kafka 将会在初始连接时将地址发送给client

其他节点部署需要修改zookeeper的ZOO_MY_ID，kafka的KAFKA_BROKER_ID，KAFKA_ADVERTISED_HOST_NAME，KAFKA_ADVERTISED_LISTENERS

访问过程：
producer连接kafka, 如果producer ip在监听的范围内，会返回ADVERTISED_LISTENERS的地址给producer，然后利用这个地址去访问.