前言
本小节是我们关于kafka集群搭建的系列教程,也是我们前面设计的整个ELK架构方案中的重要组成部分。由于kafka集群是依赖zookeeper实现通信的,我们也需要搭建一套zookeeper集群,本节中我们依旧采用docker-compose的方式搭建我们的kafka集群,同时为了便于管理我们的kafka集群,我们也会安装俩个辅助的工具管理、监控我们的kafka集群,分别为kafka-manager与kafka-offset-monitor,话不多说,开始我们的正文。
正文
拉取镜像:
命令:docker pull <镜像>:<版本号>
docker-compose启动容器超时
在使用docker-compose启动多个容器时,在其默认的启动时间60s内无法全部启动完成,容器就会整个启动失败。这里我们将此参数调大到1000s。使用vi编辑器修改系统变量文件/etc/profile,在文件的末尾添加俩个参数,然后更新系统参数,使新添加的参数配置生效。
export DOCKER_CLIENT_TIMEOUT=1000
export COMPOSE_HTTP_TIMEOUT=1000
- 创建kafka-cluster.yml启动脚本
#kafka-cluster.yml文件 version: '3.1' services: #zk集群 zk1: image: zookeeper:3.7.0 restart: always container_name: elk-cluster-zk1 hostname: zk1 ports: - 2181:2181 volumes: - "/elk/zookeeper/zk1/data:/data" - "/elk/zookeeper/zk1/logs:/datalog" environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181 zk2: image: zookeeper:3.7.0 restart: always container_name: elk-cluster-zk2 hostname: zk2 ports: - 2182:2181 volumes: - "/elk/zookeeper/zk2/data:/data" - "/elk/zookeeper/zk2/logs:/datalog" environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zk3:2888:3888;2181 zk3: image: zookeeper:3.7.0 restart: always container_name: elk-cluster-zk3 hostname: zk3 ports: - 2183:2181 volumes: - "/elk/zookeeper/zk3/data:/data" - "/elk/zookeeper/zk3/logs:/datalog" environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181 #kafka集群 kafka1: image: wurstmeister/kafka:2.13-2.7.0 restart: always container_name: elk-cluster-kafka1 hostname: kafka1 ports: - "9091:9092" - "9991:9991" depends_on: - zk1 - zk2 - zk3 environment: KAFKA_BROKER_ID: 1 KAFKA_ADVERTISED_HOST_NAME: kafka1 KAFKA_ADVERTISED_PORT: 9091 KAFKA_HOST_NAME: kafka1 KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181 KAFKA_LISTENERS: PLAINTEXT://kafka1:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.23.134:9091 JMX_PORT: 9991 KAFKA_JMX_OPTS: "-Djava.rmi.server.hostname=kafka1 -Dcom.sun.management.jmxremote.port=9991 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.managementote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" volumes: - "/elk/kafka/kafka1/:/kafka" kafka2: image: wurstmeister/kafka:2.13-2.7.0 restart: always container_name: elk-cluster-kafka2 hostname: kafka2 ports: - "9092:9092" - "9992:9992" depends_on: - zk1 - zk2 - zk3 environment: KAFKA_BROKER_ID: 2 KAFKA_ADVERTISED_HOST_NAME: kafka2 KAFKA_ADVERTISED_PORT: 9092 KAFKA_HOST_NAME: kafka2 KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181 KAFKA_LISTENERS: PLAINTEXT://kafka2:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.23.134:9092 JMX_PORT: 9992 KAFKA_JMX_OPTS: "-Djava.rmi.server.hostname=kafka2 -Dcom.sun.management.jmxremote.port=9992 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.managementote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" volumes: - "/elk/kafka/kafka2/:/kafka" kafka3: image: wurstmeister/kafka:2.13-2.7.0 restart: always container_name: elk-cluster-kafka3 hostname: kafka3 ports: - "9093:9092" - "9993:9993" depends_on: - zk1 - zk2 - zk3 environment: KAFKA_BROKER_ID: 3 KAFKA_ADVERTISED_HOST_NAME: kafka3 KAFKA_ADVERTISED_PORT: 9093 KAFKA_HOST_NAME: kafka3 KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181 KAFKA_LISTENERS: PLAINTEXT://kafka3:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.23.134:9093 JMX_PORT: 9993 KAFKA_JMX_OPTS: "-Djava.rmi.server.hostname=kafka3 -Dcom.sun.management.jmxremote.port=9993 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.managementote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false" volumes: - "/elk/kafka/kafka3/:/kafka" #kafka管理工具 'kafka-manager': container_name: kafka-manager image: sheepkiller/kafka-manager:stable restart: always ports: - 9000:9000 depends_on: - kafka1 - kafka2 - kafka3 environment: KM_VERSION: 1.3.3.18 ZK_HOSTS: zk1:2181,zk2:2181,zk3:2181 #kafka监控工具 'kafka-offset-monitor': container_name: kafka-offset-monitor image: 564239555/kafkaoffsetmonitor:latest restart: always volumes: - /elk/kafkaoffsetmonitor/conf:/kafkaoffsetmonitor ports: - 9001:8080 depends_on: - kafka1 - kafka2 - kafka3 environment: ZK_HOSTS: zk1:2181,zk2:2181,zk3:2181 KAFKA_BROKERS: kafka1:9092,kafka2:9092,kafka3:9092 REFRESH_SECENDS: 10 RETAIN_DAYS: 2
- 启动kafka集群
使用vi编辑器创建一个kafka-cluster.yml启动文件,文件内容如上,然后我们根据kafka-cluster.yml文件启动我们的kafka集群。
命令:docker-compose -f kafka-cluster.yml -p kafka up -d
说明:
(1)-f :指定我们的启动文件
(2)-p:指定容器启动的项目目录,如果我们有多个docker-compose启动项目,如果不指定项目目录,会有项目目录冲突的警告
(3)-d:后台启动
- 验证kafka集群
(1)访问kafka集群管理工具kafka-manager
访问工具kafka-manager地址:http://192.168.23.134:9000,这里可以实现简单的kafka集群管理,进行主题创建等简单操作。我们这里创建一个atp主题,后续通过filebeat将数据收集到atp主题上,然后让logstash消费这些主题信息。
(2)访问kafka集群监控工具kafka-offset-monitor
访问http://192.168.23.134:9001/工具kafka-offset-monitor地址,这里可以实现简单的kafka集群监控。
结语
ok,到这里我们的kafka集群就搭建好,关于kakfa日志的收集和消费的配置,我们在后续的章节中会详细介绍。下期见咯。。。