压测实操--kafka-consumer压测方案

作者：九月

环境信息：

操作系统centos7.9，kafka版本为hdp集群中的2.0版本。

Consumer相关参数

使用Kafka自带的kafka-consumer-perf-test.sh脚本进行压测，该脚本参数为：

thread：测试时的单机线程数；

fetch-size：抓取数据量；

partition:分区数，主要是和线程复合来测试；

replication:副本数；

througout:我们所需要的吞吐量，单位时间内处理消息的数量，可能对我们处理消息的延迟有影响；

consumer测试

fetch-size

fetch-size=524288

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  524288  --threads 4

fetch-size=1048576

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  1048576  --threads 4

fetch-size=2097152

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  2097152  --threads 4

fetch-size=3145728

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  3145728  --threads 4

fetch-size=4194304

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  4194304  --threads 4

fetch-size=5242880

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 1000000 --fetch-size  5242880  --threads 4

partitions

创建不同partitions的topic

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create  --zookeeper mutong01:2181,mutong02:2181,mutong03:2181  --topic test_kafka_consumer1  --partitions 1 --replication-factor 1

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create  --zookeeper mutong01:2181,mutong02:2181,mutong03:2181  --topic test_kafka_consumer2  --partitions 2 --replication-factor 1

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create  --zookeeper mutong01:2181,mutong02:2181,mutong03:2181  --topic test_kafka_consumer3  --partitions 3 --replication-factor 1

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create  --zookeeper mutong01:2181,mutong02:2181,mutong03:2181  --topic test_kafka_consumer4  --partitions 4 --replication-factor 1

topic生产数据

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh  --topic test_kafka_consumer1 --num-records  100000000 --record-size 687  --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667  batch.size=5000 --throughput 30000

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh  --topic test_kafka_consumer2 --num-records  100000000 --record-size 687  --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667  batch.size=5000 --throughput 30000

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh  --topic test_kafka_consumer3 --num-records  100000000 --record-size 687  --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667  batch.size=5000 --throughput 30000

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh  --topic test_kafka_consumer4 --num-records  100000000 --record-size 687  --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667  batch.size=5000 --throughput 30000

消费不同partitions的topic

消费不同partitions的topic，获取不同partition下的吞吐量

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer1 --messages 10000000 --fetch-size  1048576  --threads 4 

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer2 --messages 10000000 --fetch-size  1048576  --threads 4 

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer3 --messages 10000000 --fetch-size  1048576  --threads 4 

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer4 --messages 10000000 --fetch-size  1048576  --threads 4

fetch-threads

fetch-thread=1

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576  --messages 10000000  --threads 1  --num-fetch-threads 1

fetch-thread=4

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576  --messages 10000000  --threads 1  --num-fetch-threads 4

fetch-thread=7

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576  --messages 10000000  --threads 1  --num-fetch-threads 7

fetch-thread=10

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576  --messages 10000000  --threads 1  --num-fetch-threads 10

总结

使用kafka自带的kafka-consumer-perf-test.sh脚本对集群消费情况进行压测，根据业务情况设置要测试的messager条数。不同的变量因素，通过修改其中一个变量条件，固定其余变量值，获取对应的吞吐速率。整理不同条件下的指标结果，形成曲线就可以观察具体的消费趋势。