kafka修改topic分区的位置

转自：http://www.cnblogs.com/honeybee/p/5671347.html

1. 查看分区topic的分区分布

$ le-kafka-topics.sh --describe --topic http_zhixin_line1

结果如下：

[hadoop@sdf-nimbus-perf project]$ le-kafka-topics.sh --describe --topic http_zhixin_line1
Topic:http_zhixin_line1 PartitionCount:3    ReplicationFactor:1 Configs:
    Topic: http_zhixin_line1    Partition: 0    Leader: 4   Replicas: 4 Isr: 4
    Topic: http_zhixin_line1    Partition: 1    Leader: 5   Replicas: 5 Isr: 5
    Topic: http_zhixin_line1    Partition: 2    Leader: 6   Replicas: 6 Isr: 6

2. 准备分区分配的JSON配置文件

可以使用kafka的kafka-reassign-partitions.sh工具来分配topic的分区位置，操作之前得准备一个Json文件，里面详细地记录着每个分区的副本所在机器等信息，具体如下：

{
    "version": 1,
    "partitions": [
        {
            "topic": "http_zhixin_line1",
            "partition": 0,
            "replicas": [
                7
            ]
        },
        {
            "topic": "http_zhixin_line1",
            "partition": 1,
            "replicas": [
                8
            ]
        },
        {
            "topic": "http_zhixin_line1",
            "partition": 3,
            "replicas": [
                9
            ]
        }
    ]
}

3. 执行topic的分区位置修改操作：

kafka-reassign-partitions.sh --zookeeper $ZK_CONNECT --reassignment-json-file movePartitions.json --execute
执行日志如下：

[hadoop@sdf-nimbus-perf topic_reassgin]$ kafka-reassign-partitions.sh --zookeeper $ZK_CONNECT --reassignment-json-file movePartitions.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"http_zhixin_line1","partition":2,"replicas":[6]},{"topic":"http_zhixin_line1","partition":1,"replicas":[5]},{"topic":"http_zhixin_line1","partition":0,"replicas":[4]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"http_zhixin_line1","partition":0,"replicas":[7]},{"topic":"http_zhixin_line1","partition":1,"replicas":[8]},{"topic":"http_zhixin_line1","partition":2,"replicas":[9]}]}

4. 检查执行的结果

kafka-reassign-partitions.sh --zookeeper $ZK_CONNECT --reassignment-json-file movePartitions.json --verify
执行日志如下：

[hadoop@sdf-nimbus-perf topic_reassgin]$ kafka-reassign-partitions.sh --zookeeper $ZK_CONNECT --reassignment-json-file movePartitions.json --verify
Status of partition reassignment:
Reassignment of partition [http_zhixin_line1,0] completed successfully
Reassignment of partition [http_zhixin_line1,1] completed successfully
Reassignment of partition [http_zhixin_line1,2] completed successfully

5. 检查kafka的对应的topic的日志目录

原目录下的数据已经迁移到新的的分区所对应的服务器下了。

分配状态的变化：分配的状态变化：“is still in progress” --》 “completed successfully”
如果分配脚本有错，或者遇到异常，分配状态会变成“failed” （具体的fail原因：如果是json配置文件的原因，命令行就会打印出error日志；kafka内部错误可以查看kafka的server.log 日志文件等）。

重新分配过程中数据日志文件的变化：
先将原来的分区的数据，copy到对应重新分配后的分区节点下，拷贝成功后，kakfa会将原来的分区目录删除掉。 copy 前后的数据量大小是一致的。