kafka指令

kafka指令

docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.1.4:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.4:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka

{“datas”:[{“channel”:"",“metric”:“temperature”,“producer”:“ijinus”,“sn”:“IJA0101-00002245”,“time”:“1543207156000”,“value”:“80”}],“ver”:“1.0”}

主题查看
./kafka-topics.sh --list --bootstrap-server 192.168.1.4:9092

/opt/kafka_2.13-2.7.0/bin
生产消息
./kafka-console-producer.sh --broker-list localhost:9092 --topic test

订阅消息
./kafka-console-consumer.sh --bootstrap-server 192.168.1.4:9092 --topic test --from-beginning

扩展topic为test的分区为2
kafka-topics.sh --zookeeper 192.168.1.4:2181 -alter --partitions 2 --topic test

基准测试
//创建topic
kafka-topics.sh --zookeeper 192.168.1.4:2181 --create --topic benchmark --partitions 1 --replication-factor 1

测试环境500w消息做基准测试，生产环境建议五千万数据
生产者
kafka-producer-perf-test.sh --topic benchmark --num-records 5000000 --throughput -1 --record-size 1000 --producer-props bootstrap.servers=192.168.1.4:
9092 acks=1

result:
5000000 records sent, 4319.908211 records/sec (4.12 MB/sec), 7527.17 ms avg latency, 11419.00 ms max latency, 7371 ms 50th, 9195 ms 95th, 9748 ms 99th, 11300 ms 99.9th

消费者
kafka-consumer-perf-test.sh --broker-list 192.168.1.4:9092 --topic benchmark --fetch-size 1048576 --messages 5000000

result:
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2021-06-17 07:35:42:532, 2021-06-17 07:36:16:071, 477.0393, 14.2234, 500212, 14914.3385, 1623915344009, -1623915310470, -0.0000, -0.0003