kafka练习测试

一、创建topic为albert，partition数目为30，replication-factor为2.如截图所示：

 

[root@hd1 bin]# ./kafka-topics.sh --zookeeper hd1.bigdata:2181,master1.bigdata:2181,master2.bigdata:2181 --create --topic albert --partitions 30  --replication-factor 2
 

二、kafka数据存放的路径是：/insight/data1/kafka-logs,/insight/data2/kafka-logs,/insight/data5/kafka-logs,/insight/data12/kafka-logs,/insight/data7/kafka-logs,/insight/data9/kafka-logs,/insight/data6/kafka-logs,/insight/data4/kafka-logs,/insight/data3/kafka-logs,/insight/data10/kafka-logs,/insight/data8/kafka-logs,/insight/data11/kafka-logs

 

 

三、kafka的broker数目为三个

 

hd1.bigdata:2181  hd2.bigdata:2181  hd3.bigdata:2181 

四、topic albert 在 30个partitions在3个kafka broker的分配情况：

      第一个broker：

   第二个broker：

 

第三个broker：

 

五、查看topic的分区情况：

[root@hd1 bin]# ./kafka-topics.sh --zookeeper hd1.bigdata:2181,master1.bigdata:2181,master2.bigdata:2181 --describe --topic albert

Topic:albert PartitionCount:30 ReplicationFactor:2 Configs:
 Topic: albert Partition: 0 Leader: 1003 Replicas: 1003,1002 Isr: 1003,1002
 Topic: albert Partition: 1 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
 Topic: albert Partition: 2 Leader: 1002 Replicas: 1002,1001 Isr: 1002,1001
 Topic: albert Partition: 3 Leader: 1003 Replicas: 1003,1001 Isr: 1003,1001
 Topic: albert Partition: 4 Leader: 1001 Replicas: 1001,1002 Isr: 1001,1002
 Topic: albert Partition: 5 Leader: 1002 Replicas: 1002,1003 Isr: 1002,1003
 Topic: albert Partition: 6 Leader: 1003 Replicas: 1003,1002 Isr: 1003,1002
 Topic: albert Partition: 7 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
 Topic: albert Partition: 8 Leader: 1002 Replicas: 1002,1001 Isr: 1002,1001
 Topic: albert Partition: 9 Leader: 1003 Replicas: 1003,1001 Isr: 1003,1001
 Topic: albert Partition: 10 Leader: 1001 Replicas: 1001,1002 Isr: 1001,1002
 Topic: albert Partition: 11 Leader: 1002 Replicas: 1002,1003 Isr: 1002,1003
 Topic: albert Partition: 12 Leader: 1003 Replicas: 1003,1002 Isr: 1003,1002
 Topic: albert Partition: 13 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
 Topic: albert Partition: 14 Leader: 1002 Replicas: 1002,1001 Isr: 1002,1001
 Topic: albert Partition: 15 Leader: 1003 Replicas: 1003,1001 Isr: 1003,1001
 Topic: albert Partition: 16 Leader: 1001 Replicas: 1001,1002 Isr: 1001,1002
 Topic: albert Partition: 17 Leader: 1002 Replicas: 1002,1003 Isr: 1002,1003
 Topic: albert Partition: 18 Leader: 1003 Replicas: 1003,1002 Isr: 1003,1002
 Topic: albert Partition: 19 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
 Topic: albert Partition: 20 Leader: 1002 Replicas: 1002,1001 Isr: 1002,1001
 Topic: albert Partition: 21 Leader: 1003 Replicas: 1003,1001 Isr: 1003,1001
 Topic: albert Partition: 22 Leader: 1001 Replicas: 1001,1002 Isr: 1001,1002
 Topic: albert Partition: 23 Leader: 1002 Replicas: 1002,1003 Isr: 1002,1003
 Topic: albert Partition: 24 Leader: 1003 Replicas: 1003,1002 Isr: 1003,1002
 Topic: albert Partition: 25 Leader: 1001 Replicas: 1001,1003 Isr: 1001,1003
 Topic: albert Partition: 26 Leader: 1002 Replicas: 1002,1001 Isr: 1002,1001
 Topic: albert Partition: 27 Leader: 1003 Replicas: 1003,1001 Isr: 1003,1001
 Topic: albert Partition: 28 Leader: 1001 Replicas: 1001,1002 Isr: 1001,1002
 Topic: albert Partition: 29 Leader: 1002 Replicas: 1002,1003 Isr: 1002,1003

 

由第四点和第五点可以看出：albert0 ~albert29 并不是对应partition0 ~ partition29.

 

六：调试

TopicMetadataResponse(Vector(id:1001,host:hd1.bigdata,port:6667, id:1003,host:hd2.bigdata,port:6667, id:1002,host:hd3.bigdata,port:6667),Vector({TopicMetadata for topic albert ->
Metadata for partition [albert,0] is  partition 0 leader: 1003 (hd2.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,1] is  partition 1 leader: 1001 (hd1.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,2] is  partition 2 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,3] is  partition 3 leader: 1003 (hd2.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,4] is  partition 4 leader: 1001 (hd1.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,5] is  partition 5 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,6] is  partition 6 leader: 1003 (hd2.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,7] is  partition 7 leader: 1001 (hd1.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,8] is  partition 8 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,9] is  partition 9 leader: 1003 (hd2.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,10] is  partition 10 leader: 1001 (hd1.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,11] is  partition 11 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,12] is  partition 12 leader: 1003 (hd2.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,13] is  partition 13 leader: 1001 (hd1.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,14] is  partition 14 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,15] is  partition 15 leader: 1003 (hd2.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,16] is  partition 16 leader: 1001 (hd1.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,17] is  partition 17 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,18] is  partition 18 leader: 1003 (hd2.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,19] is  partition 19 leader: 1001 (hd1.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,20] is  partition 20 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,21] is  partition 21 leader: 1003 (hd2.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,22] is  partition 22 leader: 1001 (hd1.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,23] is  partition 23 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,24] is  partition 24 leader: 1003 (hd2.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,25] is  partition 25 leader: 1001 (hd1.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,26] is  partition 26 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,27] is  partition 27 leader: 1003 (hd2.bigdata:6667) replicas: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1003 (hd2.bigdata:6667),1001 (hd1.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,28] is  partition 28 leader: 1001 (hd1.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1001 (hd1.bigdata:6667) isr: 1001 (hd1.bigdata:6667),1002 (hd3.bigdata:6667) isUnderReplicated: false
Metadata for partition [albert,29] is  partition 29 leader: 1002 (hd3.bigdata:6667) replicas: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isr: 1002 (hd3.bigdata:6667),1003 (hd2.bigdata:6667) isUnderReplicated: false}),0)

 

七、由六可以知道：hd1.bigdata 对应1001    hd2.bigdata --1003    hd3.bigdata---1002   故albert0 ~albert29 是对应partition0 ~ partition29.