apache kafka性能测试命令使用和构建kafka-perf

转自：http://blog.csdn.net/lizhitao/article/details/35847927

本来想用kafka官方提供的工具做性能测试的。但事与愿违，当我执行官方提供的kafka测试脚本，却报错没有找到ProducerPerformance，后来浏览一些代码文件，才发现没有把perf性能测试程序打包到kafka_2.x.0-0.8.x.x.jar发行版本中。

现在来教您如何打包做测试。

1.准备工作：

安装gradle

2.下载kafka源代码

kafka-0.8.1源代码

3.编译kafka-perf_2.x-0.8.1.x.jar

编译注意事项：默认情况下是编译为2.8.0版本，也可以指定版本编译。目前编译高版本的kafka-perf(2.8.0以上版本)是由问题的，因为build.gradle配置参数有问题(版本不同，会报如下错误，版本不兼容错误)，如果要构建高版本kafka-perf多版本修改内容如下：

下载build.gradle   替换掉kafka-0.8.1.1-src根目录下文件即可

编译构建执行命令：

gradle jar   			默认生成2.8.0版本的kafka和kafka-perf的jar
gradle jar_core_2_8_0		生成2.8.0版本的kafka的jar
gradle jar_core_2_8_2		生成2.8.2版本的kafka的jar
gradle jar_core_2_9_1		生成2.9.1版本的kafka的jar
gradle jar_core_2_9_2		生成2.9.2版本的kafka的jar
gradle jar_core_2_10_1		生成2.10.1版本的kafka的jar
gradle perf:jar			生成2.8.0版本的kafka和kafka-perf的jar
gradle perf_2_9_1		生成2.9.1版本的kafka和kafka-perf的jar
gradle perf_2_10_1		生成2.10.1版本的kafka和kafka-perf的jar
gradle -PscalaVersion=2.8.0 jar			编译scala 2.8.0版本编译所有jar
gradle -PscalaVersion=2.8.2 jar			编译scala 2.8.2版本编译所有jar
gradle -PscalaVersion=2.9.1 jar			编译scala 2.9.1版本编译所有jar
gradle -PscalaVersion=2.10.1 jar 编译scala 2.10.1版本编译所有jar

如果不想编译jar，可以直接下载：kafka-perf_2.x.x-0.8.1.jar 

lizhitao@users-MacBook-Pro:~/mt_wp/tmp$ cd kafka-0.8.1.1-src

lizhitao@users-MacBook-Pro:~/mt_wp/tmp/kafka-0.8.1.1-src$gradle jar

lizhitao@users-MacBook-Pro:~/mt_wp/tmp/kafka-0.8.1.1-src$gradle perf:jar
The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.
Building project 'core' with Scala version 2.8.0
Building project 'perf' with Scala version 2.8.0
:core:compileJava UP-TO-DATE
:core:compileScala
/Users/lizhitao/mt_wp/tmp/kafka-0.8.1.1-src/core/src/main/scala/kafka/admin/AdminUtils.scala:243: non variable type-argument String in type pattern scala.collection.Map[String,_] is unchecked since it is eliminated by erasure
        case Some(map: Map[String, _]) =>
                       ^
/Users/lizhitao/mt_wp/tmp/kafka-0.8.1.1-src/core/src/main/scala/kafka/admin/AdminUtils.scala:246: non variable type-argument String in type pattern scala.collection.Map[String,String] is unchecked since it is eliminated by erasure
            case Some(config: Map[String, String]) =>
                              ^
/Users/lizhitao/mt_wp/tmp/kafka-0.8.1.1-src/core/src/main/scala/kafka/api/LeaderAndIsrResponse.scala:66: non variable type-argument String in type pattern (String, Int) is unchecked since it is eliminated by erasure
    for ((key:(String, Int), value) <- responseMap) {
              ^
/Users/lizhitao/mt_wp/tmp/kafka-0.8.1.1-src/core/src/main/scala/kafka/utils/Utils.scala:363: non variable type-argument V in type pattern List[V] is unchecked since it is eliminated by erasure
        case Some(l: List[V]) => m.put(k, v :: l)
                     ^
four warnings found
:core:processResources UP-TO-DATE
:core:classes
:core:copyDependantLibs UP-TO-DATE
:core:jar UP-TO-DATE
:perf:compileJava UP-TO-DATE
:perf:compileScala
:perf:processResources UP-TO-DATE
:perf:classes
:perf:jar UP-TO-DATE

BUILD SUCCESSFUL
Total time: 54.41 secs

编译jar包目录如下：

a. kafka_2.x-0.8.1.1.jar

kafka-0.8.1.1-src/core/build

b.kafka-perf_2.x-0.8.1.x.jar

kafka-0.8.1.1-src/perf/build/libs

kafka多版本jar：

4. kafka性能测试命令用法：

4.1 创建topic

bin/kafka-topics.sh --zookeeper 192.168.2.225:2182,192.168.2.225:2183/config/mobile/mq/mafka02 --create --topic test-rep-one --partitions 6 --replication-factor 1

4.2 kafka-producer-perf-test.sh中参数说明：

messages		生产者发送总的消息数量
message-size		每条消息大小
batch-size		每次批量发送消息的数量
topics			生产者发送的topic
threads			生产者使用几个线程同时发送
broker-list		安装kafka服务的机器ip:port列表
producer-num-retries	一个消息失败发送重试次数
request-timeout-ms	一个消息请求发送超时时间

4.3 bin/kafka-consumer-perf-test.sh中参数说明：

zookeeperzk		配置
messages		消费者消费消息总数量
topic			消费者需要消费的topic
threads			消费者使用几个线程同时消费
group			消费者组名称
socket-buffer-sizesocket	缓冲大小
fetch-size		每次向kafka broker请求消费大小
consumer.timeout.ms	消费者去kafka broker拿去一条消息超时时间

  
  

   
   

  
  

4.4 生产者发送数据：

lizhitao@users-MacBook-Pro:~/mt_wp/tmp/kafka-0.8.1.1-src$ bin/kafka-producer-perf-test.sh --messages 5000000 --message-size 5000  --batch-size 5000 --topics test-rep-one --threads 8 --broker-list mobile-esb03:9092,mobile-esb04:9092,mobile-esb05:9092

start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
[2014-07-06 12:52:36,139] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-07-06 12:52:36,199] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-07-06 12:52:36,202] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-07-06 12:52:36,204] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-07-06 12:52:36,206] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-07-06 12:52:36,207] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-07-06 12:52:36,209] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-07-06 12:52:36,214] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)

4.5 消费者消费数据

lizhitao@users-MacBook-Pro:~/mt_wp/tmp/kafka-0.8.1.1-src$ bin/kafka-consumer-perf-test.sh --zookeeper

192.168.2.225:2182,192.168.2.225:2183/config/mobile/mq/mafka02 --messages 50000000 --topic test-rep-one --threads 1

start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.