Kafka 常用命令

• kafka常用命令一般是关于kafka topic的一些操作，比如create, delete, list, describe等一些操作

• 进入$KAFKA_HOME/bin目录下可以看到kafka-topics.sh这个命令，通过- -help 查看该命名帮助

  1. [root@hadoop001 kafka_2.11-0.10.0.1]# kafka-topics.sh --help
  2. Command must include exactly one action: --list, --describe, --create, --alter or --delete
  3. Option                                   Description                            
  4. ------                                   -----------                            
  5. --alter                                  Alter the number of partitions,        
                                             replica assignment, and/or           
                                             configuration for the topic.         
  6. --config <name=value>                    A topic configuration override for the 
                                             topic being created or altered.The   
                                             following is a list of valid         
                                             configurations:                      
                                                  cleanup.policy                        
                                                  compression.type                      
                                                  delete.retention.ms                   
                                                  file.delete.delay.ms                  
                                                  flush.messages                        
                                                  flush.ms                              
                                                  index.interval.bytes                  
                                                  max.message.bytes                     
                                                  message.format.version                
                                                  message.timestamp.difference.max.ms   
                                                  message.timestamp.type                
                                                  min.cleanable.dirty.ratio             
                                                  min.insync.replicas                   
                                                  preallocate                           
                                                  retention.bytes                       
                                                  retention.ms                          
                                                  segment.bytes                         
                                                  segment.index.bytes                   
                                                  segment.jitter.ms                     
                                                  segment.ms                            
                                                  unclean.leader.election.enable        
                                           See the Kafka documentation for full   
                                             details on the topic configs.        
  7. --create                                 Create a new topic.                    
  8. --delete                                 Delete a topic                         
  9. --delete-config <name>                   A topic configuration override to be   
                                             removed for an existing topic (see   
                                             the list of configurations under the 
                                             --config option).                    
  10. --describe                               List details for the given topics.     
  11. --disable-rack-aware                     Disable rack aware replica assignment  
  12. --help                                   Print usage information.               
  13. --if-exists                              if set when altering or deleting       
                                             topics, the action will only execute 
                                             if the topic exists                  
  14. --if-not-exists                          if set when creating topics, the       
                                             action will only execute if the      
                                             topic does not already exist         
  15. --list                                   List all available topics.             
  16. --partitions <Integer: # of partitions>  The number of partitions for the topic 
                                             being created or altered (WARNING:   
                                             If partitions are increased for a    
                                             topic that has a key, the partition  
                                             logic or ordering of the messages    
                                             will be affected                     
  17. --replica-assignment                     A list of manual partition-to-broker   
    <broker_id_for_part1_replica1 :          assignments for the topic being      
    broker_id_for_part1_replica2 ,           created or altered.                  
    broker_id_for_part2_replica1 :                                                
    broker_id_for_part2_replica2 , ...>                                           
  18. --replication-factor <Integer:           The replication factor for each        
    replication factor>                      partition in the topic being created.
  19. --topic <topic>                          The topic to be create, alter or       
                                             describe. Can also accept a regular  
                                             expression except for --create option
  20. --topics-with-overrides                  if set when describing topics, only    
                                             show topics that have overridden     
                                             configs                              
  21. --unavailable-partitions                 if set when describing topics, only    
                                             show partitions whose leader is not  
                                             available                            
  22. --under-replicated-partitions            if set when describing topics, only    
                                             show under replicated partitions     
  23. --zookeeper <urls>                       REQUIRED: The connection string for    
                                             the zookeeper connection in the form 
                                             host:port. Multiple URLS can be      
                                             given to allow fail-over. 
  

• 创建topic –create

  1. bin/kafka-topics.sh --create \
  2. --zookeeper hadoop001:2181,hadoop002:2181,hadoop003:2181/kafka \
  3. --replication-factor 3 --partitions 3 --topic test
  

• 查看创建的topic –list

  1. bin/kafka-topics.sh --list \
  2. --zookeeper hadoop001:2181,hadoop002:2181,hadoop003:2181/kafka
  

• 查看topic的信息 –describe

  1. bin/kafka-topics.sh --describe \
  2. --zookeeper hadoop001:2181,hadoop002:2181,hadoop003:2181/kafka \
  3. --topic test
  4. Topic:test      PartitionCount:3        ReplicationFactor:3     Configs:
  5.  Topic: test     Partition: 0    Leader: 1       Replicas: 1,3,2 Isr: 1,3,2
  6.  Topic: test     Partition: 1    Leader: 2       Replicas: 2,1,3 Isr: 2,1,3
  7.  Topic: test     Partition: 2    Leader: 3       Replicas: 3,2,1 Isr: 3,2,1
  

  信息解释：
  Partition： topic 的分区，下标从0开始
  Leader ： 该分区负责读写的kafka实例节点
  Replicas ： 该分区和副本所在节点列表
  Isr ： “in-sync” replicas，实时说明该分区正在写和副本所在的节点列表，若与分区列表不一致则可能表示kafka集群有问题，比如某一节点的kafka进程over

• 删除topic –delete

  1. bin/kafka-topics.sh  --delete \
  2. --zookeeper hadoop001:2181,hadoop002:2181,hadoop003:2181/kafka \
  3. --topic test
  4. Topic test is marked for deletion.
  5. Note: This will have no impact if delete.topic.enable is not set to true.
  

  使用 - -delete删除topic时，即使所提示的参数delete.topic.enable 设置为true也不能将topic真正删除
  彻底删除topic方法：
  1）进入zookeeper cli
  zkCli.sh
  2)删除相应topic元数据信息
  rmr /kafka/config/topics/test
  rmr /kafka/brokers/topics/test
  rmr /kafka/admin/delete_topics/test
  3)删除磁盘topic文件
  rm -rf $KAFKA_HOME/logs/test-*

• 修改topic –alter

  1. bin/kafka-topics.sh --alter
  2. --zookeeper hadoop001:2181,hadoop002:2181,hadoop003:2181/kafka \
  3. --topic test --partitions 4
  

  修改topic信息原则上是可以修改所有信息；分区数只能增加不能减少