浅谈MQ-Kafka部署

Kafka的单机安装部署

1.原生安装

下载kafka

https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz

安装过程

安装过程非常简单，只需要解压就行，因为这个是编译好之后的可执行程序
tar -zxvf kafka_2.11-2.0.0.tgz 解压

配置zookeeper

因为kafka依赖于zookeeper来做master选举一起其他数据的维护，所以需要先启动zookeeper节点
kafka内置了zookeeper的服务，所以在bin目录下提供了这些脚本

zookeeper-server-start.sh
zookeeper-server-stop.sh

在config目录下，存在一些配置文件

zookeeper.properties
server.properties

所以我们可以通过下面的脚本来启动zk服务，当然，也可以自己搭建zk的集群来实现

sh zookeeper-server-start.sh -daemon ../config/zookeeper.properties

启动和停止kafka

修改server.properties, 增加zookeeper的配置

zookeeper.connect=localhost:2181

## 集群或者夸宿主机配置注意点
broker.id=0

## 注册到zk是告诉zk kafka的应用地址
listeners=PLAINTEXT://<当前宿主机IP>:9092

启动kafka

sh kafka-server-start.sh -damoen config/server.properties

停止kafka

sh kafka-server-stop.sh -daemon config/server.properties

2.Docker安装

docker run --name zookeeper -p 2181:2181 -d zookeeper:3.5.8

docker run -d --name kafka -p 9092:9092 -v /data/kafka:/kafka \
-v /var/run/docker.sock:/var/run/docker.sock \
-e KAFKA_BROKER_ID=0 \
-e KAFKA_ZOOKEEPER_CONNECT=<内网ip,不能是容器内ip>:2181 \ 
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://<公网ip>:9092 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_LOG_DIRS=/kafka/kafka-logs \
wurstmeister/kafka

3.window 启动单机kafka

docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=172.18.30.181:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092 -e KAFKA_LISTENERS=PLAINTEXT://127.0.0.1:9092 -e KAFKA_LOG_DIRS=/kafka/kafka-logs wurstmeister/kafka

Kafka的集群部署

1.原生部署

配置server.properties文件

分别修改三台机器的server.properties配置，同一个集群中的每个机器的id必须唯一

broker.id=0
broker.id=1
broker.id=2

修改zookeeper的连接配置

zookeeper.connect=192.168.13.106:2181   # 内网ip,不能是容器内ip

修改listeners配置

如果配置了listeners，那么消息生产者和消费者会使用listeners的配置来进行消息的收发，否则，会使用localhost

PLAINTEXT表示协议，默认是明文，可以选择其他加密协议

listeners=PLAINTEXT://192.168.13.102:9092

分别启动三台服务器

sh kafka-server-start.sh -daemon ../config/server.properties

2.docker部署kafka集群

docker run --name zookeeper -p 2181:2181 -d wurstmeister/zookeeper:latest
docker run --name zookeeper -p 2181:2181 -d zookeeper:3.5.8

# -e KAFKA_ADVERTISED_LISTENERS 作用就是暴露给外部的listeners，如果没有设置，会用listeners

# 搭建集群
docker run -d --name kafka1 -p 9092:9092 -v /data/kafka/broker1:/kafka \
-v /var/run/docker.sock:/var/run/docker.sock \
-e KAFKA_BROKER_ID=0 \
-e KAFKA_ZOOKEEPER_CONNECT=172.19.71.214:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://101.132.117.5:9092 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_LOG_DIRS=/kafka/kafka-logs \
wurstmeister/kafka

docker run -d --name kafka2 -p 9095:9092 -v /data/kafka/broker2:/kafka \
-v /var/run/docker.sock:/var/run/docker.sock \
-e KAFKA_BROKER_ID=1 \
-e KAFKA_ZOOKEEPER_CONNECT=172.19.71.214:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://101.132.117.5:9095 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_LOG_DIRS=/kafka/kafka-logs \
wurstmeister/kafka

docker run -d --name kafka3 -p 9096:9092 -v /data/kafka/broker3:/kafka \
-v /var/run/docker.sock:/var/run/docker.sock \
-e KAFKA_BROKER_ID=2 \
-e KAFKA_ZOOKEEPER_CONNECT=172.19.71.214:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://101.132.117.5:9096 \
-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
-e KAFKA_LOG_DIRS=/kafka/kafka-logs \
wurstmeister/kafka

3.docker-compose部署kafka集群

kafka-docker-compose.yml

version: '2'
services:
  zookeeper:
    container_name: "zookeeper"
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    restart: always
  kafka1:
    container_name: "kafka1"
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 0
      KAFKA_ZOOKEEPER_CONNECT: 172.19.71.214:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://101.132.117.5:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_HEAP_OPTS: -Xmx1G -Xms1G
      KAFKA_LOG_DIRS: /kafka/kafka-logs
    volumes:
      - /data/kafka/broker1:/kafka
      - /var/run/docker.sock:/var/run/docker.sock

  kafka2:
    container_name: "kafka2"
    image: wurstmeister/kafka:latest
    ports:
      - "9095:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 172.19.71.214:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://101.132.117.5:9095
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_HEAP_OPTS: -Xmx1G -Xms1G
      KAFKA_LOG_DIRS: /kafka/kafka-logs
    volumes:
      - /data/kafka/broker2:/kafka
      - /var/run/docker.sock:/var/run/docker.sock

  kafka3:
    container_name: "kafka3"
    image: wurstmeister/kafka:latest
    ports:
      - "9096:9092"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: 172.19.71.214:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://101.132.117.5:9096
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_HEAP_OPTS: -Xmx1G -Xms1G
      KAFKA_LOG_DIRS: /kafka/kafka-logs
    volumes:
      - /data/kafka/broker3:/kafka
      - /var/run/docker.sock:/var/run/docker.sock

在kafka-docker-compose.yml文件目录下运行：

docker-compose -f kafka-docker-compose.yml up -d