kafka重新分配partition

摘要

当kafka集群扩容时，新加入的节点可以很平滑的加入到集群中。但是原来创建的topic并不会将partition数据均匀的分配到这个节点上，只有新创建的topic才会分配到新的节点上。这时就可以用kafka的kafka-reassign-partitions.sh工具来进行手动平均分配。

This command moves topic partitions between replicas.
Option                                 Description
------                                 -----------
--broker-list <brokerlist>             The list of brokers to which the
                                         partitions need to be reassigned in
                                         the form "0,1,2". This is required
                                         if --topics-to-move-json-file is
                                         used to generate reassignment
                                         configuration
--disable-rack-aware                   Disable rack aware replica assignment
--execute                              Kick off the reassignment as specified
                                         by the --reassignment-json-file
                                         option.
--generate                             Generate a candidate partition
                                         reassignment configuration. Note
                                         that this only generates a candidate
                                         assignment, it does not execute it.
--reassignment-json-file <manual       The JSON file with the partition
  assignment json file path>             reassignment configurationThe format
                                         to use is -
                                       {"partitions":
                                       	[{"topic": "foo",
                                       	  "partition": 1,
                                       	  "replicas": [1,2,3] }],
                                       "version":1
                                       }
--throttle <Long: throttle>            The movement of partitions will be
                                         throttled to this value (bytes/sec).
                                         Rerunning with this option, whilst a
                                         rebalance is in progress, will alter
                                         the throttle value. The throttle
                                         rate should be at least 1 KB/s.
                                         (default: -1)
--topics-to-move-json-file <topics to  Generate a reassignment configuration
  reassign json file path>               to move the partitions of the
                                         specified topics to the list of
                                         brokers specified by the --broker-
                                         list option. The format to use is -
                                       {"topics":
                                       	[{"topic": "foo"},{"topic": "foo1"}],
                                       "version":1
                                       }
--verify                               Verify if the reassignment completed
                                         as specified by the --reassignment-
                                         json-file option. If there is a
                                         throttle engaged for the replicas
                                         specified, and the rebalance has
                                         completed, the throttle will be
                                         removed
--zookeeper <urls>                     REQUIRED: The connection string for
                                         the zookeeper connection in the form
                                         host:port. Multiple URLS can be
                                         given to allow fail-over.

1.配置文件

使用kafka-reassign-partitions.sh工具前，必须先手动写一个配置文件，告诉工具你要分配哪些topic数据。

{
    "topics": [
        {"topic": "test"}
    ],
    "version": 1
}

如果要移动多个topic数据可以输入多个。

2.生成分配计划

./kafka-reassign-partitions.sh --zookeeper druid1:2181 --topics-to-move-json-file topic.json  --broker-list  "0,1,2,4"  --generate

broker-list参数为brokerid，并不是broker的服务地址，执行完毕之后会生成类似下面的数据。

[root@druid1 bin]# ./kafka-reassign-partitions.sh --zookeeper druid1:2181 --topics-to-move-json-file topic.json  --broker-list  "0,1,2,4"  --generate
Current partition replica assignment

{"version":1,"partitions":[{"topic":"test","partition":11,"replicas":[0,1,2]},{"topic":"test","partition":4,"replicas":[1,0,2]},{"topic":"test","partition":13,"replicas":[2,0,1]},{"topic":"test","partition":1,"replicas":[2,0,1]},{"topic":"test","partition":8,"replicas":[1,2,4,0]},{"topic":"test","partition":10,"replicas":[4,0,1,2]},{"topic":"test","partition":5,"replicas":[2,1,4,0]},{"topic":"test","partition":2,"replicas":[4,1,2,0]},{"topic":"test","partition":15,"replicas":[0,2,4,1]},{"topic":"test","partition":9,"replicas":[2,4,0,1]},{"topic":"test","partition":14,"replicas":[4,1,2,0]},{"topic":"test","partition":7,"replicas":[0,4,1,2]},{"topic":"test","partition":12,"replicas":[1,4,0,2]},{"topic":"test","partition":6,"replicas":[4,2,0,1]},{"topic":"test","partition":0,"replicas":[1,4,0,2]},{"topic":"test","partition":3,"replicas":[0,2,4,1]}]}
Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"test","partition":11,"replicas":[2,0,1]},{"topic":"test","partition":4,"replicas":[1,2,0]},{"topic":"test","partition":13,"replicas":[1,0,2]},{"topic":"test","partition":1,"replicas":[1,0,2]},{"topic":"test","partition":8,"replicas":[2,1,0]},{"topic":"test","partition":10,"replicas":[1,2,0]},{"topic":"test","partition":5,"replicas":[2,0,1]},{"topic":"test","partition":2,"replicas":[2,1,0]},{"topic":"test","partition":15,"replicas":[0,1,2]},{"topic":"test","partition":9,"replicas":[0,1,2]},{"topic":"test","partition":14,"replicas":[2,1,0]},{"topic":"test","partition":7,"replicas":[1,0,2]},{"topic":"test","partition":12,"replicas":[0,2,1]},{"topic":"test","partition":6,"replicas":[0,2,1]},{"topic":"test","partition":0,"replicas":[0,2,1]},{"topic":"test","partition":3,"replicas":[0,1,2]}]}

上面一行是topic当前partition的分配规则，下面一行是按照上面命令执行后程序自动分配的分配计划，可以根据机器实际资源情况手动调整，将确认后的分配规则写到一个文件里面，我这里命名为move.json

3.执行分配

./kafka-reassign-partitions.sh --zookeeper druid1:2181 --reassignment-json-file move.json --throttle 31457280 --execute

–throttle 为了避免迁移过程中影响kafka正常使用，这个参数可以设置迁移的流量速度，单位是byte。

4.验证/查看进度

./kafka-reassign-partitions.sh --zookeeper druid1:2181 --reassignment-json-file move.json --verify

可以查看迁移进度

其他