一、基本概念
二、集群安装部署
1、环境准备:
- 安装docker环境
- 安装docker-compose
kafka集群:
192.168.10.11
192.168.10.12
192.168.10.13
2、部署
kafka需要依赖zookeeper,所以要先启动zookeeper
vim kafka-compose
version: '2.0'
services:
zookeeper:
image: zookeeper
container_name: zk
network_mode: "host"
restart: always
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=192.168.10.11:2888:3888;2181 server.2=192.168.10.12:2888:3888;2181 server.3=192.168.10.13:2888:3888;2181
ZOO_AUTOPURGE_PURGEINTERVAL: 72
TZ: Asia/Shanghai
volumes:
- /data/zookeeper/data:/data
- /data/zookeeper/logs:/datalog
- /etc/localtime:/etc/localtime
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
depends_on:
- zookeeper
container_name: kafka
network_mode: "host"
restart: always
ports:
- "9092:9092"
environment:
CLUSTER_NAME: kafka1
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_HOST_NAME: 192.168.10.11
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.10.11:9092
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ZOOKEEPER_CONNECT: 192.168.10.11:2181,10.3.4.144:2181,192.168.10.12:2181,192.168.10.13:2181,10.3.4.147:2181
KAFKA_NUM_PARTITIONS: 3
KAFKA_LOG_RETENTION_HOURS: 72
TZ: Asia/Shanghai
volumes:
- /data/kafka/docker.sock:/var/run/docker.sock
- /data/kafka/logs/:/kafka
- /etc/localtime:/etc/localtime
kafka配置文件关键参数详解
broker.id //broker的全局唯一编号,不能重复
KAFKA_ADVERTISED_HOST_NAME //为本机ip
KAFKA_LISTENERS //kafka的broker将监听地址表,只有与该ip匹配正确的客户端能成功连接到kafka
KAFKA_ADVERTISED_LISTENERS //指向Broker的可用地址列表,kafka 将会在初始连接时将地址发送给client
其他节点部署需要修改zookeeper的ZOO_MY_ID,kafka的KAFKA_BROKER_ID,KAFKA_ADVERTISED_HOST_NAME,KAFKA_ADVERTISED_LISTENERS
访问过程:
producer连接kafka, 如果producer ip在监听的范围内,会返回ADVERTISED_LISTENERS的地址给producer,然后利用这个地址去访问.