Kafka 性能测试

Kafka 性能测试(基于Kafka 2.2.1版本)

Producer 测试

1. 不同压缩算法对数据发送性能的影响

测试命令

3 broker , 2 replication , numRecords 500w , throughput 100w , recordSize 256byte

bin/kafka-producer-perf-test.sh --producer-props bootstrap.servers=master:9092 compression.type=none --topic compression_kafka_none --throughput 1000000 --record-size 256 --num-records 5000000

创建Topic

bin/kafka-topics.sh --create --zookeeper master:2181,node1:2181,node2:2181 --replication-factor 2 --partitions 3 --topic compression_kafka_none

bin/kafka-topics.sh --create --zookeeper master:2181,node1:2181,node2:2181 --replication-factor 2 --partitions 3 --topic compression_kafka_gzip

bin/kafka-topics.sh --create --zookeeper master:2181,node1:2181,node2:2181 --replication-factor 2 --partitions 3 --topic compression_kafka_snappy

bin/kafka-topics.sh --create --zookeeper master:2181,node1:2181,node2:2181 --replication-factor 2 --partitions 3 --topic compression_kafka_lz4

bin/kafka-topics.sh --create --zookeeper master:2181,node1:2181,node2:2181 --replication-factor 2 --partitions 3 --topic compression_kafka_zstd

# 删除Topic
bin/kafka-topics.sh --delete --zookeeper master:2181 --topic compression_kafka_none

compression algorithm	records/sec	MB/sec	avg latency(ms)	max latency(ms)
none(启动一瞬间较慢，后续速度飙升424M)	435995.814440	106.44	263.70	1590.00
gzip（压缩后log文件大小明显降低18M）	256055.717724	62.51	62.80	862.00
snappy（