kafka Replication tools

最新推荐文章于 2023-06-26 15:14:06 发布

IT-LeoChen

最新推荐文章于 2023-06-26 15:14:06 发布

阅读量1.4k

点赞数

分类专栏： Kafka

Kafka 专栏收录该内容

7 篇文章 0 订阅

订阅专栏

Pages
Index

Replication tools

Skip to end of metadata

Added by Sriram Subramanian, last edited by Neha Narkhede on Sep 18, 2013 (view change)
show comment

Go to start of metadata

6. Reassign Partitions Tool

What does the tool do?

The goal of this tool is similar to the Referred Replica Leader Election Tool as to achieve load balance across brokers. But instead of only electing a new leader from the assigned replicas of a partition, this tool allows to change the assigned replicas of partitions – remember that followers also need to fetch from leaders in order to keep in sync, hence sometime only balance the leadership load is not enough.

A summary of the steps that the tool does is shown below -

1. The tool updates the zookeeper path "/admin/reassign_partitions" with the list of topic partitions and (if specified in the Json file) the list of their new assigned replicas.
2. The controller listens to the path above. When a data change update is triggered, the controller reads the list of topic partitions and their assigned replicas from zookeeper.
3. For each topic partition, the controller does the following:
3.1. Start new replicas in RAR - AR (RAR = Reassigned Replicas, AR = original list of Assigned Replicas)
3.2. Wait until new replicas are in sync with the leader
3.3. If the leader is not in RAR, elect a new leader from RAR
3.4 4. Stop old replicas AR - RAR
3.5. Write new AR
3.6. Remove partition from the /admin/reassign_partitions path

Note that the tool only updates the zookeeper path and exits. The controller reassign the replicas for the partitions asynchronously.

This tool is only available in the 0.8 branch for now.

How to use the tool?

 
         bin/kafka-reassign-partitions.sh 
        
         Option                                  Description 
        
         ------                                  ----------- 
        
         --broker-list <brokerlist>              The list of brokers to which the 
        
         partitions need to be reassigned in 
        
         the form  
         "0,1,2" 
         . This is required 
        
         for  
         automatic topic reassignment. 
        
         --execute [execute]                     This option does the actual 
        
         reassignment. By  
         default 
         , the tool 
        
         does a dry run 
        
         --manual-assignment-json-file <manual   The JSON file with the list of manual 
        
         assignment json file path>              reassignmentsThis option or topics- 
        
         to-move-json-file needs to be 
        
         specified. The format to use is - 
        
         { 
         "partitions" 
         : 
        
         [{ 
         "topic" 
         :  
         "foo" 
         , 
        
         "partition" 
         :  
         1 
         , 
        
         "replicas" 
         : [ 
         1 
         , 
         2 
         , 
         3 
         ] }], 
        
         "version" 
         : 
         1 
        
         } 
        
         --topics-to-move-json-file <topics to   The JSON file with the list of topics 
        
         reassign json file path>                to reassign.This option or manual- 
        
         assignment-json-file needs to be 
        
         specified. The format to use is - 
        
         { 
         "topics" 
         : 
        
         [{ 
         "topic" 
         :  
         "foo" 
         },{ 
         "topic" 
         :  
         "foo1" 
         }], 
        
         "version" 
         : 
         1 
        
         } 
        
         --zookeeper <urls>                      REQUIRED: The connection string  
         for 
        
         the zookeeper connection in the form 
        
         host:port. Multiple URLS can be 
        
         given to allow fail-over.

Please note that by default the script runs in a dry-run mode and does not initiate the partition movement. Only when the --execute option is specified, the tool proceeds to start the partition movement

Cluster Expansion

The partition reassignment tool can be used to expand an existing 0.8 cluster. Cluster expansion involves including brokers with new broker ids in a Kafka 08 cluster. Typically, when you add new brokers to a cluster, they will not receive any data from existing topics until this tool is run to assign existing topics/partitions to the new brokers. The tool allows 2 options to make it easier to move some topics in bulk to the new brokers. These 2 options are a) topics to move b) list of newly added brokers. Using these 2 options, the tool automatically figures out the placements of partitions for the topics on the new brokers.

The following example moves 2 topics (foo1, foo2) to newly added brokers in a cluster (5,6,7)

 
         nnarkhed$ ./bin/kafka-reassign-partitions.sh --topics-to-move-json-file topics-to-move.json --broker-list  
         "5,6,7"  
         --execute 
        
         nnarkhed$ cat topics-to-move.json 
        
         { 
         "topics" 
         : 
        
         [{ 
         "topic" 
         :  
         "foo1" 
         },{ 
         "topic" 
         :  
         "foo2" 
         }], 
        
         "version" 
         : 
         1 
        
         }

Selectively moving some partitions to a broker

The partition movement tool can also be moved to selectively move some replicas for certain partitions over to a particular broker. Typically, if you end up with an unbalanced cluster, you can use the tool in this mode to selectively move partitions around. In this mode, the tool takes a single file which has a list of partitions to move and the replicas that each of those partitions should be assigned to.

The following example moves 1 partition (foo-1) from replicas 1,2,3 to 1,2,4

 
         nnarkhed$ ./bin/kafka-reassign-partitions.sh --manual-assignment-json-file partitions-to-move.json --execute 
        
         nnarkhed$ cat partitions-to-move.json 
        
         { 
         "partitions" 
         : 
        
         [{ 
         "topic" 
         :  
         "foo" 
         , 
        
         "partition" 
         :  
         1 
         , 
        
         "replicas" 
         : [ 
         1 
         , 
         2 
         , 
         4 
         ] }],               
        
         }], 
        
         "version" 
         : 
         1 
        
         }