Kafka动态增加Topic的副本(Replication)

 

1. 查看topic的原来的副本分布

[hadoop@sdf-nimbus-perf ~]$ le-kafka-topics.sh --describe --topic http_zhixin_line1
Topic:http_zhixin_line1 PartitionCount:3    ReplicationFactor:1 Configs:
    Topic: http_zhixin_line1    Partition: 0    Leader: 7   Replicas: 7 Isr: 7
    Topic: http_zhixin_line1    Partition: 1    Leader: 8   Replicas: 8 Isr: 8
    Topic: http_zhixin_line1    Partition: 2    Leader: 9   Replicas: 9 Isr: 9

2. 增加Topic的副本的json文件的编写

vi addReplicas.json

{
    "version": 1,
    "partitions": [
        {
            "topic": "http_zhixin_line1",
            "partition": 0,
            "replicas": [
                7,
                1,
                2
            ]
        },
        {
            "topic": "http_zhixin_line1",
            "partition": 1,
            "replicas": [
                8,
                2,
                3
            ]
        },
        {
            "topic": "http_zhixin_line1",
            "partition": 2,
            "replicas": [
                9,
                3,
                4
            ]
        }
    ]
}

3. 执行topic增加副本操作

kafka-reassign-partitions.sh --zookeeper $ZK_CONNECT --reassignment-json-file addReplicas.json --execute

[hadoop@sdf-nimbus-perf topic_reassgin]$ kafka-reassign-partitions.sh --zookeeper $ZK_CONNECT --reassignment-json-file addReplicas.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"http_zhixin_line1","partition":2,"replicas":[9]},{"topic":"http_zhixin_line1","partition":1,"replicas":[8]},{"topic":"http_zhixin_line1","partition":0,"replicas":[7]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"http_zhixin_line1","partition":0,"replicas":[7,1,2]},{"topic":"http_zhixin_line1","partition":1,"replicas":[8,2,3]},{"topic":"http_zhixin_line1","partition":2,"replicas":[9,3,4]}]}

4. 查看执行的状态

kafka-reassign-partitions.sh --zookeeper  $ZK_CONNECT --reassignment-json-file addReplicas.json --verify

[hadoop@sdf-nimbus-perf topic_reassgin]$ kafka-reassign-partitions.sh --zookeeper $ZK_CONNECT --reassignment-json-file addReplicas.json --verify
Status of partition reassignment:
Reassignment of partition [http_zhixin_line1,0] completed successfully
Reassignment of partition [http_zhixin_line1,1] completed successfully
Reassignment of partition [http_zhixin_line1,2] completed successfully

5. 观察日志目录的数据同步情况，生产者，消费者的影响。

观察partitions-0的数据同步情况，由于partitions-0 增加的两个副本为1,2；
登陆 broker 1 和 broker 2所在的服务器；

cd 副本数据所在的目录，观察日志增长情况：

ls /data/hadoop/data*/kafka/log
borker1 和 broker2 所在日志目录都有日志文件生成。borker1 和 borker2的副本数据是同时同步完成的。