kafka0.9.0.1 生产消费性能测试

bin/kafka-topics.sh --create --zookeeper 172.16.4.41 --topic test-rep-one --partitions 6 --replication-factor 1 

3 台kafka  8core 

3 台zk 8core

#生产消费同时进行测试  

生产测试:

#方式1

bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic test-rep-one --num-records 10000000 --record-size 1024 --throughput 10000000 --producer-props bootstrap.servers=kafka-srv4:9092,kafka-srv5:9092,kafka-srv6:9092 

#方式2

bin/kafka-producer-perf-test.sh --record-size 1024 --num-records 1000000 --throughput 1000000 --topic test-rep-one  --producer-props   bootstrap.servers=kafka-srv1:9092,kafka-srv2:9092,kafka-srv3:9092

结果如下:

574707 records sent, 114941.4 records/sec (112.25 MB/sec), 215.1 ms avg latency, 1127.0 max latency.
634303 records sent, 126860.6 records/sec (123.89 MB/sec), 258.4 ms avg latency, 1102.0 max latency.
679014 records sent, 135802.8 records/sec (132.62 MB/sec), 224.0 ms avg latency, 752.0 max latency.
559776 records sent, 111955.2 records/sec (109.33 MB/sec), 270.3 ms avg latency, 899.0 max latency.
520671 records sent, 104134.2 records/sec (101.69 MB/sec), 294.6 ms avg latency, 957.0 max latency.
521347 records sent, 104269.4 records/sec (101.83 MB/sec), 291.3 ms avg latency, 927.0 max latency.
487594 records sent, 97518.8 records/sec (95.23 MB/sec), 309.9 ms avg latency, 1029.0 max latency.
514576 records sent, 102915.2 records/sec (100.50 MB/sec), 305.2 ms avg latency, 1014.0 max latency.
501866 records sent, 100373.2 records/sec (98.02 MB/sec), 307.6 ms avg latency, 854.0 max latency.
515502 records sent, 103100.4 records/sec (100.68 MB/sec), 294.3 ms avg latency, 896.0 max latency.
561284 records sent, 112256.8 records/sec (109.63 MB/sec), 261.1 ms avg latency, 1209.0 max latency.
707638 records sent, 139052.5 records/sec (135.79 MB/sec), 220.2 ms avg latency, 1099.0 max latency.
686422 records sent, 137284.4 records/sec (134.07 MB/sec), 240.2 ms avg latency, 934.0 max latency.
427714 records sent, 85542.8 records/sec (83.54 MB/sec), 360.8 ms avg latency, 2162.0 max latency.
640378 records sent, 128075.6 records/sec (125.07 MB/sec), 236.7 ms avg latency, 1025.0 max latency.
642196 records sent, 128439.2 records/sec (125.43 MB/sec), 240.3 ms avg latency, 1009.0 max latency.
565316 records sent, 111107.7 records/sec (108.50 MB/sec), 256.7 ms avg latency, 1095.0 max latency.
10000000 records sent, 114243.933647 records/sec (111.57 MB/sec), 266.53 ms avg latency, 2162.00 ms max latency, 64 ms 50th, 891 ms 95th, 1043 ms 99th, 1965 ms 99.9th.

#生产瓶颈 427714 records sent, 85542.8 records/sec (83.54 MB/sec), 360.8 ms avg latency, 2162.0 max latency.

消费测试:

#方式1

bin/kafka-consumer-perf-test.sh --batch-size 1000 --messages 10000000 --broker-list=kafka-srv4:9092,kafka-srv5:9092,kafka-srv6:9092 --topic test-rep-one --group test-rep-one-group --new-consumer --num-fetch-threads 10

#方式2 

bin/kafka-consumer-perf-test.sh  --broker-list kafka-srv1:9092,kafka-srv2:9092,kafka-srv3:9092 --message-size 1024 --messages  10000000 --num-fetch-threads 6 --topic test-rep-one  --new-consumer  

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec

2018-04-23 17:43:37:950, 2018-04-23 17:44:23:675, 9766.3809, 213.5895, 10000774, 218715.6698

# 消费瓶颈 213.5895 MB.sec

##TODO 优化 

参考

https://blog.csdn.net/stark_summer/article/details/50203133

https://www.cnblogs.com/xiaodf/p/6023531.html

https://github.com/JNachtwey/kafka-cheat-sheets/blob/master/kafka-performance.md

https://gist.github.com/jkreps/c7ddb4041ef62a900e6c

https://github.com/jkorab/ameliant-tools/tree/master/kafka/kafka-perf-tool