kafka 生产消费性能测试

生产数据压力测试

kafka-producer-perf-test.sh 脚本命令的参数为：
--topic topic名称，比如test
--num-records 总共需要发送的消息数，比如为1000000
--record-size 每个记录的字节数，比如1000
--throughput 每秒钟发送的最大记录数，比如20000
--producer-props bootstrap.servers=kafka-01:9092,kafka-02:9092,kafka-03:9092 发送端的配置信息，本例只指定了kafka的链接信息

[bigdataservice@yhadoop001]$ kafka-producer-perf-test --topic test --num-records 1000000 --record-size 1000 --throughput 20000 --producer-props bootstrap.servers=kafka-01:9092,kafka-02:9092,kafka-03:9092     
99833 records sent, 19966.6 records/sec (19.04 MB/sec), 9.2 ms avg latency, 83.0 ms max latency.
99970 records sent, 19994.0 records/sec (19.07 MB/sec), 8.8 ms avg latency, 56.0 ms max latency.
100006 records sent, 20001.2 records/sec (19.07 MB/sec), 9.9 ms avg latency, 38.0 ms max latency.
100051 records sent, 20010.2 records/sec (19.08 MB/sec), 9.2 ms avg latency, 34.0 ms max latency.
99986 records sent, 19997.2 records/sec (19.07 MB/sec), 9.0 ms avg latency, 39.0 ms max latency.
100022 records sent, 20004.4 records/sec (19.08 MB/sec), 8.7 ms avg latency, 31.0 ms max latency.
100015 records sent, 20003.0 records/sec (19.08 MB/sec), 8.5 ms avg latency, 31.0 ms max latency.
100060 records sent, 20012.0 records/sec (19.08 MB/sec), 8.5 ms avg latency, 30.0 ms max latency.
99947 records sent, 19989.4 records/sec (19.06 MB/sec), 8.5 ms avg latency, 29.0 ms max latency.
100003 records sent, 20000.6 records/sec (19.07 MB/sec), 8.3 ms avg latency, 27.0 ms max latency.
1000000 records sent, 19992.802591 records/sec (19.07 MB/sec), 8.88 ms avg latency, 83.00 ms max latency, 9 ms 50th, 12 ms 95th, 16 ms 99th, 33 ms 99.9th.

 

 

消费性能压力测试

kafka-consumer-perf-test.sh 脚本命令的参数为：
--broker-list 指定kafka节点信息，比如为kafka-01:9092,kafka-02:9092,kafka-03:9092
--topic 指定topic的名称，比如为test
--fetch-size 指定每次fetch的数据的大小，比如为1048576，也就是1M
--messages 总共要消费的消息个数，比如为1000000

--threads 指定消费线程数

[bigdataservice@hadoop001]$ kafka-consumer-perf-test --broker-list kafka-01:9092,kafka-02:9092,kafka-03:9092 --topic test --fetch-size 50000 --messages 500000 --threads 4    
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-08 10:43:57:695, 2020-10-08 10:44:00:022, 477.2282, 205.0830, 500410, 215045.1225, 25, 2302, 207.3102, 217380.5387