09.集群

Kafka集群简介

• Kafka天然支持集群
• 依赖于ZK进行协调
• Kafka主要通过brokerId区分不同节点

Docker-Compose部署Kafka集群

version: '3'
services:
  zoo1:
    image: zookeeper:3.5.5
    hostname: zoo1
    ports:
      - "2181:2181"
    container_name: zoo1
  kafka1:
    image: wurstmeister/kafka:2.12-2.4.0
    ports:
version: '3'
services:
  zoo1:
    image: zookeeper:3.5.5
    hostname: zoo1
    ports:
      - "2181:2181"
    container_name: zoo1
  kafka1:
    image: wurstmeister/kafka:2.12-2.4.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.108                     ## 修改:宿主机IP
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.108:9092    ## 修改:宿主机IP
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_BROKER_ID: 1
    depends_on:
      - zoo1
    container_name: kafka1
  kafka2:
    image: wurstmeister/kafka:2.12-2.4.0
    ports:
      - "9093:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.108                    ## 修改:宿主机IP
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.108:9093   ## 修改:宿主机IP
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_BROKER_ID: 2
    depends_on:
      - zoo1
    container_name: kafka2
  kafka3:
    image: wurstmeister/kafka:2.12-2.4.0
    ports:
      - "9094:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.108                    ## 修改:宿主机IP
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.108:9094   ## 修改:宿主机IP
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_BROKER_ID: 3
    depends_on:
      - zoo1
    container_name: kafka3
  kafka-manager:
   # image: sheepkiller/kafka-manager              ## 镜像：开源的web管理kafka集群的界面
    image: kafkamanager/kafka-manager
    environment:
        ZK_HOSTS: zoo1:2181                   ## 修改:宿主机IP
    ports:
      - "9000:9000"

Kafka节点故障

1. Kafka与ZK心跳未保持视为节点故障
2. follower消息落后leader太多也视为节点故障
3. Kafka会移除故障节点

Kafka Leader选举机制

1. Kafka并没有采用多数投票来选举leader
2. Kafka会动态维护一组leader数据的副本（ISR）
3. Kafka会在ISR中选择一个速度比较快的设为leader
4. 当ISR中副本全部宕机的时候，Kafka会进行unclean leader选举
5. 建议禁用unclean leader选举和手动指定最小ISR

集群安全(一般不用)

1. Kafka提供了SSL或SASL机制
2. Kafka提供了Broker到ZK链接的安全机制
3. Kafka支持Client的读写验证