Kafka集群简介
- Kafka天然支持集群
- 依赖于ZK进行协调
- Kafka主要通过brokerId区分不同节点
Docker-Compose部署Kafka集群
version: '3'
services:
zoo1:
image: zookeeper:3.5.5
hostname: zoo1
ports:
- "2181:2181"
container_name: zoo1
kafka1:
image: wurstmeister/kafka:2.12-2.4.0
ports:
version: '3'
services:
zoo1:
image: zookeeper:3.5.5
hostname: zoo1
ports:
- "2181:2181"
container_name: zoo1
kafka1:
image: wurstmeister/kafka:2.12-2.4.0
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.108
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.108:9092
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
KAFKA_ADVERTISED_PORT: 9092
KAFKA_BROKER_ID: 1
depends_on:
- zoo1
container_name: kafka1
kafka2:
image: wurstmeister/kafka:2.12-2.4.0
ports:
- "9093:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.108
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.108:9093
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
KAFKA_ADVERTISED_PORT: 9093
KAFKA_BROKER_ID: 2
depends_on:
- zoo1
container_name: kafka2
kafka3:
image: wurstmeister/kafka:2.12-2.4.0
ports:
- "9094:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.1.108
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.108:9094
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
KAFKA_ADVERTISED_PORT: 9094
KAFKA_BROKER_ID: 3
depends_on:
- zoo1
container_name: kafka3
kafka-manager:
image: kafkamanager/kafka-manager
environment:
ZK_HOSTS: zoo1:2181
ports:
- "9000:9000"
Kafka节点故障
- Kafka与ZK心跳未保持视为节点故障
- follower消息落后leader太多也视为节点故障
- Kafka会移除故障节点
Kafka Leader选举机制
- Kafka并没有采用多数投票来选举leader
- Kafka会动态维护一组leader数据的副本(ISR)
- Kafka会在ISR中选择一个速度比较快的设为leader
- 当ISR中副本全部宕机的时候,Kafka会进行unclean leader选举
- 建议禁用unclean leader选举和手动指定最小ISR
集群安全(一般不用)
- Kafka提供了SSL或SASL机制
- Kafka提供了Broker到ZK链接的安全机制
- Kafka支持Client的读写验证