kafka性能测试1.0.0

最新推荐文章于 2021-11-05 16:41:45 发布

weixin_33841503

最新推荐文章于 2021-11-05 16:41:45 发布

阅读量134

点赞数

文章标签：大数据 python java

原文链接：https://my.oschina.net/xiaominmin/blog/1794774

版权

2019独角兽企业重金招聘Python工程师标准>>>

-- 生产者

./bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=ip:prot,ip:prot --topic s1 --num-records 1000000 --record-size 4096 --throughput 10000
46276 records sent, 9255.2 records/sec (36.15 MB/sec), 262.9 ms avg latency, 784.0 max latency.
53641 records sent, 10728.2 records/sec (41.91 MB/sec), 183.8 ms avg latency, 1412.0 max latency.
22274 records sent, 3758.7 records/sec (14.68 MB/sec), 160.2 ms avg latency, 3910.0 max latency.
66837 records sent, 13367.4 records/sec (52.22 MB/sec), 710.3 ms avg latency, 4194.0 max latency.
24689 records sent, 4937.8 records/sec (19.29 MB/sec), 1007.6 ms avg latency, 4397.0 max latency.
66509 records sent, 13301.8 records/sec (51.96 MB/sec), 539.3 ms avg latency, 4304.0 max latency.
23842 records sent, 4768.4 records/sec (18.63 MB/sec), 1212.9 ms avg latency, 4812.0 max latency.
58159 records sent, 11050.5 records/sec (43.17 MB/sec), 503.1 ms avg latency, 1940.0 max latency.
54902 records sent, 10980.4 records/sec (42.89 MB/sec), 628.2 ms avg latency, 3473.0 max latency.
31102 records sent, 6220.4 records/sec (24.30 MB/sec), 962.8 ms avg latency, 3480.0 max latency.
71948 records sent, 13994.9 records/sec (54.67 MB/sec), 445.6 ms avg latency, 3166.0 max latency.
33544 records sent, 6708.8 records/sec (26.21 MB/sec), 950.8 ms avg latency, 3973.0 max latency.
48542 records sent, 9593.3 records/sec (37.47 MB/sec), 443.1 ms avg latency, 2336.0 max latency.
66070 records sent, 13214.0 records/sec (51.62 MB/sec), 593.0 ms avg latency, 3456.0 max latency.
34979 records sent, 6995.8 records/sec (27.33 MB/sec), 919.7 ms avg latency, 3231.0 max latency.
58977 records sent, 8760.7 records/sec (34.22 MB/sec), 409.4 ms avg latency, 3468.0 max latency.
76514 records sent, 15302.8 records/sec (59.78 MB/sec), 624.8 ms avg latency, 3504.0 max latency.
40847 records sent, 8169.4 records/sec (31.91 MB/sec), 721.3 ms avg latency, 3405.0 max latency.
48458 records sent, 9691.6 records/sec (37.86 MB/sec), 545.1 ms avg latency, 2650.0 max latency.
70828 records sent, 14165.6 records/sec (55.33 MB/sec), 498.7 ms avg latency, 2648.0 max latency.
1000000 records sent, 9679.605072 records/sec (37.81 MB/sec), 577.25 ms avg latency, 4812.00 ms max latency, 229 ms 50th, 2336 ms 95th, 3542 ms 99th, 4531 ms 99.9th.

--消费者

./bin/kafka-consumer-perf-test.sh --messages 1000000 --threads 1 --broker-list ip:prot,ip:prot --num-fetch-threads 12 --topic s1

[zhangzm@zk-16-228 kafka]$ ./bin/kafka-consumer-perf-test.sh --messages 1000000 --threads 1 --broker-list ip:prot,ip:prot --num-fetch-threads 12 --topic s1
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2018-04-12 19:49:17:686, 2018-04-12 19:50:02:630, 3906.3750, 86.9165, 1000032, 22250.6230, 3344, 41600, 93.9032, 24039.2308

[zhangzm@zk-16-228 kafka]$ ./bin/kafka-consumer-perf-test.sh --messages 1000000 --threads 1 --broker-list ip:prot,ip:prot --num-fetch-threads 12 --topic s1
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2018-04-12 19:51:59:553, 2018-04-12 19:52:40:536, 3906.3672, 95.3168, 1000030, 24401.0931, 3064, 37919, 103.0187, 26372.7946

[zhangzm@zk-16-228 kafka]$ ./bin/kafka-consumer-perf-test.sh --messages 2000000 --threads 1 --broker-list ip:prot,ip:prot --num-fetch-threads 12 --topic s1
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2018-04-12 19:54:22:298, 2018-04-12 19:55:42:047, 7812.5000, 97.9636, 2000000, 25078.6844, 3168, 76581, 102.0162, 26116.1385

kafkaJVM参数配置
java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
-XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/usr/local/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M

转载于:https://my.oschina.net/xiaominmin/blog/1794774