kafka nfs 挂载性能对比测试

软件版本：

系统	kafka	zookeeper
CentOS7.9	2.12-3.1.0	3.7.0

测试方法

测试使用Kafka自带的测试脚本，通过命令对Kafka发起写入MQ消息和Kafka消费MQ消息的请求。模拟10w、级别的消息处理结果

./bin/kafka-producer-perf-test.sh --topic test_perf --num-records 100000 --record-size 1000  --throughput 2000 --producer-props bootstrap.servers=localhost:9092

本地集群测试

10000 records sent, 2000.0 records/sec (1.91 MB/sec), 37.9 ms avg latency, 981.0 ms max latency.
10006 records sent, 2001.2 records/sec (1.91 MB/sec), 0.8 ms avg latency, 11.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 13.0 ms max latency.
10005 records sent, 2000.2 records/sec (1.91 MB/sec), 0.6 ms avg latency, 8.0 ms max latency.
10001 records sent, 2000.2 records/sec (1.91 MB/sec), 0.6 ms avg latency, 4.0 ms max latency.
10005 records sent, 2000.6 records/sec (1.91 MB/sec), 0.6 ms avg latency, 12.0 ms max latency.
10002 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 4.0 ms max latency.
10001 records sent, 2000.2 records/sec (1.91 MB/sec), 0.7 ms avg latency, 5.0 ms max latency.
10001 records sent, 2000.2 records/sec (1.91 MB/sec), 0.7 ms avg latency, 8.0 ms max latency.
100000 records sent, 1999.320231 records/sec (1.91 MB/sec), 4.39 ms avg latency, 981.00 ms max latency, 1 ms 50th, 2 ms 95th, 171 ms 99th, 224 ms 99.9th.




10000 records sent, 2000.0 records/sec (1.91 MB/sec), 17.9 ms avg latency, 562.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.8 ms avg latency, 14.0 ms max latency.
10004 records sent, 2000.8 records/sec (1.91 MB/sec), 0.7 ms avg latency, 10.0 ms max latency.
10002 records sent, 2000.0 records/sec (1.91 MB/sec), 0.7 ms avg latency, 11.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 4.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 5.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 4.0 ms max latency.
10002 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 12.0 ms max latency.
10004 records sent, 2000.8 records/sec (1.91 MB/sec), 0.6 ms avg latency, 7.0 ms max latency.
100000 records sent, 1999.360205 records/sec (1.91 MB/sec), 2.40 ms avg latency, 562.00 ms max latency, 1 ms 50th, 2 ms 95th, 97 ms 99th, 161 ms 99.9th.


10002 records sent, 2000.0 records/sec (1.91 MB/sec), 4.1 ms avg latency, 466.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 17.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 4.0 ms max latency.
10002 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 4.0 ms max latency.
10005 records sent, 2001.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 7.0 ms max latency.
9997 records sent, 1999.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 14.0 ms max latency.
10002 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 4.0 ms max latency.
10002 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 5.0 ms max latency.
10003 records sent, 2000.6 records/sec (1.91 MB/sec), 0.6 ms avg latency, 4.0 ms max latency.
100000 records sent, 1999.480135 records/sec (1.91 MB/sec), 0.98 ms avg latency, 466.00 ms max latency, 1 ms 50th, 1 ms 95th, 22 ms 99th, 39 ms 99.9th.

挂载nfs 后测试

10002 records sent, 1999.6 records/sec (1.91 MB/sec), 18.9 ms avg latency, 497.0 ms max latency.
10007 records sent, 2001.4 records/sec (1.91 MB/sec), 0.8 ms avg latency, 11.0 ms max latency.
10001 records sent, 1999.8 records/sec (1.91 MB/sec), 0.7 ms avg latency, 15.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 3.0 ms max latency.
9998 records sent, 1999.6 records/sec (1.91 MB/sec), 0.7 ms avg latency, 13.0 ms max latency.
10004 records sent, 2000.0 records/sec (1.91 MB/sec), 0.7 ms avg latency, 5.0 ms max latency.
10006 records sent, 2001.2 records/sec (1.91 MB/sec), 0.6 ms avg latency, 10.0 ms max latency.
10002 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 5.0 ms max latency.
10002 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 12.0 ms max latency.
100000 records sent, 1999.560097 records/sec (1.91 MB/sec), 2.50 ms avg latency, 497.00 ms max latency, 1 ms 50th, 2 ms 95th, 117 ms 99th, 180 ms 99.9th.




10002 records sent, 2000.0 records/sec (1.91 MB/sec), 4.6 ms avg latency, 459.0 ms max latency.
10010 records sent, 2001.2 records/sec (1.91 MB/sec), 0.7 ms avg latency, 11.0 ms max latency.
10003 records sent, 2000.6 records/sec (1.91 MB/sec), 0.7 ms avg latency, 5.0 ms max latency.
9999 records sent, 1999.8 records/sec (1.91 MB/sec), 0.7 ms avg latency, 18.0 ms max latency.
10002 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 6.0 ms max latency.
10002 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 10.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 3.0 ms max latency.
10003 records sent, 2000.6 records/sec (1.91 MB/sec), 0.7 ms avg latency, 9.0 ms max latency.
10003 records sent, 2000.2 records/sec (1.91 MB/sec), 0.6 ms avg latency, 3.0 ms max latency.
100000 records sent, 1999.280259 records/sec (1.91 MB/sec), 1.06 ms avg latency, 459.00 ms max latency, 1 ms 50th, 1 ms 95th, 23 ms 99th, 45 ms 99.9th.



10002 records sent, 2000.0 records/sec (1.91 MB/sec), 19.8 ms avg latency, 561.0 ms max latency.
10008 records sent, 2001.2 records/sec (1.91 MB/sec), 0.8 ms avg latency, 12.0 ms max latency.
10004 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 15.0 ms max latency.
10002 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 12.0 ms max latency.
10005 records sent, 2001.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 4.0 ms max latency.
10003 records sent, 2000.2 records/sec (1.91 MB/sec), 0.6 ms avg latency, 5.0 ms max latency.
10002 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 6.0 ms max latency.
10002 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 13.0 ms max latency.
10003 records sent, 2000.6 records/sec (1.91 MB/sec), 0.6 ms avg latency, 2.0 ms max latency.
100000 records sent, 1999.240289 records/sec (1.91 MB/sec), 2.58 ms avg latency, 561.00 ms max latency, 1 ms 50th, 2 ms 95th, 105 ms 99th, 182 ms 99.9th.

10w 写入测试结果分析

用例	耗时 结果
不挂载nfs	100000 records sent, 1999.480135 records/sec (1.91 MB/sec), 0.98 ms avg latency, 466.00 ms max latency, 1 ms 50th, 1 ms 95th, 22 ms 99th, 39 ms 99.9th.
挂载nfs	100000 records sent, 1999.240289 records/sec (1.91 MB/sec), 2.58 ms avg latency, 561.00 ms max latency, 1 ms 50th, 2 ms 95th, 105 ms 99th, 182 ms 99.9th.