kafka（二）：单节点多broker的部署和使用，以及监控

一、前提准备工作

包括zk安装配置，以及kafka安装配置，请参考：kafka（一）：单节点broker的部署和使用

二、多broker配置

1.配置server.properties，server1.properties，server2.properties三个文件的关键信息：

#1.server.properties
broker.id=0
# The port the socket server listens on
port=9092
# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=hadoop
# A comma seperated list of directories under which to store log files
log.dirs=/opt/modules/kafka_2.11-0.10.2.1/data/0
# root directory for all kafka znodes.
zookeeper.connect=hadoop:2181/kafka08

#2.server.properties
broker.id=1
# The port the socket server listens on
port=9093
# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=hadoop
# A comma seperated list of directories under which to store log files
log.dirs=/opt/modules/kafka_2.11-0.10.2.1/data/1
# root directory for all kafka znodes.
zookeeper.connect=hadoop:2181/kafka08

#3.server.properties
broker.id=2
# The port the socket server listens on
port=9094
# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=hadoop
# A comma seperated list of directories under which to store log files
log.dirs=/opt/modules/kafka_2.11-0.10.2.1/data/2
# root directory for all kafka znodes.
zookeeper.connect=hadoop:2181/kafka08

2.启动kafka

bin/kafka-server-start.sh -daemon config/server.properties &
bin/kafka-server-start.sh -daemon config/server1.properties &
bin/kafka-server-start.sh -daemon config/server2.properties &

备注："&"这个符合千万不要少！

3.创建topic

bin/kafka-topics.sh --create --zookeeper hadoop:2181/kafka08 --replication-factor 3 --partitions 1 --topic my-replicated-topic

三、测试

1.生产者

bin/kafka-console-producer.sh --broker-list hadoop:9092,hadoop:9093,hadoop:9094 --topic my-replicated-topic

2.消费者

bin/kafka-console-consumer.sh --zookeeper hadoop:2181/kafka08 --topic my-replicated-topic --from-beginning

四、Kafka集群管理监控

1Kafka Web Console

      监控信息比较全面，但是存在bug：不会释放KAFKA连接，最终可能会导致一个连接不够的问题，生产环境一般不用
      https://github.com/claudemamo/kafka-web-console

2.Kafka Manager

      雅虎开源，偏向Kafka集群管理，同时实现了集群监控的功能
      https://github.com/yahoo/kafka-manager

3.KafkaOffsetMonitor

     一个非常简单的监控工具，以jar的形式运行，部署非常简单，只有监控功能
      https://github.com/quantifind/KafkaOffsetMonitor