Kafka深入学习(4)

主题管理

创建主题

如果broker端参数auto.create.topics.enable设置为true，则生产者或者消费者先未知topic进行操作时将自动创建这个topic(建议将这个参数设置为true)
创建topic的kafka命令

kafka-topics.sh --zookeeper localhost:2181/kafka --topic topic-create --partitions 4 --replication-factor 2
# 返回结果：Created topic "topic-create".

根据上面操作所设置的参数，现在topic-create这个topic一共有4的分区，每个分区有2个副本，也就是说一共有8个副本。那么在我们的集群机器上将会有8个文件夹，每个文件夹代表一个副本。
文件夹的名称一般都是<topic>-<paritition>，其路径应该是在Kafka的log.dir或者log.dirs下面的。
当创建后topic后，我们可以在zookeeper里面查看到在相关的元数据：
在/kafka/brokers/topics这个目录下将会创建一个topic-create的子节点

zkCli.sh
get /kafka/brokers/topics/topic-create
# 以json格式返回了分区的情况，"2":[1,2],分区id为2的分区在brokerId为1和2的机器上
{"version","partitions":{"2":[1,2],"1":[0,1],"3":[2,1],"0":[2,0]}}

如果想自己分配分区以及副本到具体的机器上，可以使用参数eplica-assignment

kafka-topics.sh --zookeeper localhost:2181/kafka --create --topic topic-create-same --replica-assignment 2:0,0:1,1:2,2:1
# 不再需要指定partitions和replication-factor这两个参数，2:0,0:1,1:2,2:1代表0分区在2和0机器上，1分区在0和1机器上。：分割相同分区的不同副本所在机器的brokerId，分割不同分区且分区按顺序排

在创建主题的时候，我们也可以手动修改一些config使用参数–config k=v的方式，如果是多个就设置多个–config

kafka-topics.sh --zookeeper localhost:2181/kafka --create --topic topic-config ---partitions 1 --replication-factor 1 --config cleanup.policy=compact --config max.message.bytes=10000

当设置了–config后，我们可以在zookeeper里面看到相关信息

zkCli.sh
get /kafka/config/topics/topic-config
{"version","config":{"max.message.bytes":"10000","cleanup.policy":"compact"}}

在创建topic时，要避免topic的名称相同，可以加上–if-not-exists这个参数。如果没有这topic就创建，如果有就不做任何执行

如果集群对机架敏感的话，需要配置broker.rack=#{机架名称}。副本创建也是所机架的影响，如果想忽略机架的影响，可以在创建topic的时候加上–disable-rack-aware这个参数(默认情况下都是不受机架影响的，只有你标注了机架信息后)

分区副本的分配

分区副本的分配方式分为有机架和无机架，可以从源码core/src/main/scala/kafka/admin/AdminUtils.scala这个类里面查看

/**
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package kafka.admin

import java.util.Random

import kafka.utils.Logging
import org.apache.kafka.common.errors.{InvalidPartitionsException, InvalidReplicationFactorException}

import collection.{Map, mutable, _}

object AdminUtils extends Logging {
  val rand = new Random
  val AdminClientId = "__admin_client"

  /**
   * There are 3 goals of replica assignment:
   *
   * <ol>
   * <li> Spread the replicas evenly among brokers.</li>
   * <li> For partitions assigned to a particular broker, their other replicas are spread over the other brokers.</li>
   * <li> If all brokers have rack information, assign the replicas for each partition to different racks if possible</li>
   * </ol>
   *
   * To achieve this goal for replica assignment without considering racks, we:
   * <ol>
   * <li> Assign the first replica of each partition by round-robin, starting from a random position in the broker list.</li>
   * <li> Assign the remaining replicas of each partition with an increasing shift.</li>
   * </ol>
   *
   * Here is an example of assigning
   * <table cellpadding="2" cellspacing="2">
   * <tr><th>broker-0</th><th>broker-1</th><th>broker-2</th><th>broker-3</th><th>broker-4</th><th>&nbsp;</th></tr>
   * <tr><td>p0      </td><td>p1      </td><td>p2      </td><td>p3      </td><td>p4      </td><td>(1st replica)</td></tr>
   * <tr><td>p5      </td><td>p6      </td><td>p7      </td><td>p8      </td><td>p9      </td><td>(1st replica)</td></tr>
   * <tr><td>p4      </td><td>p0      </td><td>p1      </td><td>p2      </td><td>p3      </td><td>(2nd replica)</td></tr>
   * <tr><td>p8      </td><td>p9      </td><td>p5      </td><td>p6      </td><td>p7      </td><td>(2nd replica)</td></tr>
   * <tr><td>p3      </td><td>p4      </td><td>p0      </td><td>p1      </td><td>p2      </td><td>(3nd replica)</td></tr>
   * <tr><td>p7      </td><td>p8      </td><td>p9      </td><td>p5      </td><td>p6      </td><td>(3nd replica)</td></tr>
   * </table>
   *
   * <p>
   * To create rack aware assignment, this API will first create a rack alternated broker list. For example,
   * from this brokerID -> rack mapping:</p>
   * 0 -> "rack1", 1 -> "rack3", 2 -> "rack3", 3 -> "rack2", 4 -> "rack2", 5 -> "rack1"
   * <br><br>
   * <p>
   * The rack alternated list will be:
   * </p>
   * 0, 3, 1, 5, 4, 2
   * <br><br>
   * <p>
   * Then an easy round-robin assignment can be applied. Assume 6 partitions with replication factor of 3, the assignment
   * will be:
   * </p>
   * 0 -> 0,3,1 <br>
   * 1 -> 3,1,5 <br>
   * 2 -> 1,5,4 <br>
   * 3 -> 5,4,2 <br>
   * 4 -> 4,2,0 <br>
   * 5 -> 2,0,3 <br>
   * <br>
   * <p>
   * Once it has completed the first round-robin, if there are more partitions to assign, the algorithm will start
   * shifting the followers. This is to ensure we will not always get the same set of sequences.
   * In this case, if there is another partition to assign (partition #6), the assignment will be:
   * </p>
   * 6 -> 0,4,2 (instead of repeating 0,3,1 as partition 0)
   * <br><br>
   * <p>
   * The rack aware assignment always chooses the 1st replica of the partition using round robin on the rack alternated
   * broker list. For rest of the replicas, it will be biased towards brokers on racks that do not have
   * any replica assignment, until every rack has a replica. Then the assignment will go back to round-robin on
   * the broker list.
   * </p>
   * <br>
   * <p>
   * As the result, if the number of replicas is equal to or greater than the number of racks, it will ensure that
   * each rack will get at least one replica. Otherwise, each rack will get at most one replica. In a perfect
   * situation where the number of replicas is the same as the number of racks and each rack has the same number of
   * brokers, it guarantees that the replica distribution is even across brokers and racks.
   * </p>
   * @return a Map from partition id to replica ids
   * @throws AdminOperationException If rack information is supplied but it is incomplete, or if it is not possible to
   *                                 assign each replica to a unique rack.
   *
   */
  def assignReplicasToBrokers(brokerMetadatas: Seq[BrokerMetadata],
                              nPartitions: Int,
                              replicationFactor: Int,
                              fixedStartIndex: Int = -1,
                              startPartitionId: Int = -1): Map[Int, Seq[Int]] = {
    if (nPartitions <= 0)
      throw new InvalidPartitionsException("Number of partitions must be larger than 0.")
    if (replicationFactor <= 0)
      throw new InvalidReplicationFactorException("Replication factor must be larger than 0.")
    if (replicationFactor > brokerMetadatas.size)
      throw new InvalidReplicationFactorException(s"Replication factor: $replicationFactor larger than available brokers: ${brokerMetadatas.size}.")
    if (brokerMetadatas.forall(_.rack.isEmpty))
      assignReplicasToBrokersRackUnaware(nPartitions, replicationFactor, brokerMetadatas.map(_.id), fixedStartIndex,
        startPartitionId)
    else {
      if (brokerMetadatas.exists(_.rack.isEmpty))
        throw new AdminOperationException("Not all brokers have rack information for replica rack aware assignment.")
      assignReplicasToBrokersRackAware(nPartitions, replicationFactor, brokerMetadatas, fixedStartIndex,
        startPartitionId)
    }
  }

  private def assignReplicasToBrokersRackUnaware(nPartitions: Int,
                                                 replicationFactor: Int,
                                                 brokerList: Seq[Int],
                                                 fixedStartIndex: Int,
                                                 startPartitionId: Int): Map[Int, Seq[Int]] = {
    val ret = mutable.Map[Int, Seq[Int]]()
    val brokerArray = brokerList.toArray
    val startIndex = if (fixedStartIndex >= 0) fixedStartIndex else rand.nextInt(brokerArray.length)
    var currentPartitionId = math.max(0, startPartitionId)
    var nextReplicaShift = if (fixedStartIndex >= 0) fixedStartIndex else rand.nextInt(brokerArray.length)
    for (_ <- 0 until nPartitions) {
      if (currentPartitionId > 0 && (currentPartitionId % brokerArray.length == 0))
        nextReplicaShift += 1
      val firstReplicaIndex = (currentPartitionId + startIndex) % brokerArray.length
      val replicaBuffer = mutable.ArrayBuffer(brokerArray(firstReplicaIndex))
      for (j <- 0 until replicationFactor - 1)
        replicaBuffer += brokerArray(replicaIndex(firstReplicaIndex, nextReplicaShift, j, brokerArray.length))
      ret.put(currentPartitionId, replicaBuffer)
      currentPartitionId += 1
    }
    ret
  }

  private def assignReplicasToBrokersRackAware(nPartitions: Int,
                                               replicationFactor: Int,
                                               brokerMetadatas: Seq[BrokerMetadata],
                                               fixedStartIndex: Int,
                                               startPartitionId: Int): Map[Int, Seq[Int]] = {
    val brokerRackMap = brokerMetadatas.collect { case BrokerMetadata(id, Some(rack)) =>
      id -> rack
    }.toMap
    val numRacks = brokerRackMap.values.toSet.size
    val arrangedBrokerList = getRackAlternatedBrokerList(brokerRackMap)
    val numBrokers = arrangedBrokerList.size
    val ret = mutable.Map[Int, Seq[Int]]()
    val startIndex = if (fixedStartIndex >= 0) fixedStartIndex else rand.nextInt(arrangedBrokerList.size)
    var currentPartitionId = math.max(0, startPartitionId)
    var nextReplicaShift = if (fixedStartIndex >= 0) fixedStartIndex else rand.nextInt(arrangedBrokerList.size)
    for (_ <- 0 until nPartitions) {
      if (currentPartitionId > 0 && (currentPartitionId % arrangedBrokerList.size == 0))
        nextReplicaShift += 1
      val firstReplicaIndex = (currentPartitionId + startIndex) % arrangedBrokerList.size
      val leader = arrangedBrokerList(firstReplicaIndex)
      val replicaBuffer = mutable.ArrayBuffer(leader)
      val racksWithReplicas = mutable.Set(brokerRackMap(leader))
      val brokersWithReplicas = mutable.Set(leader)
      var k = 0
      for (_ <- 0 until replicationFactor - 1) {
        var done = false
        while (!done) {
          val broker = arrangedBrokerList(replicaIndex(firstReplicaIndex, nextReplicaShift * numRacks, k, arrangedBrokerList.size))
          val rack = brokerRackMap(broker)
          // Skip this broker if
          // 1. there is already a broker in the same rack that has assigned a replica AND there is one or more racks
          //    that do not have any replica, or
          // 2. the broker has already assigned a replica AND there is one or more brokers that do not have replica assigned
          if ((!racksWithReplicas.contains(rack) || racksWithReplicas.size == numRacks)
              && (!brokersWithReplicas.contains(broker) || brokersWithReplicas.size == numBrokers)) {
            replicaBuffer += broker
            racksWithReplicas += rack
            brokersWithReplicas += broker
            done = true
          }
          k += 1
        }
      }
      ret.put(currentPartitionId, replicaBuffer)
      currentPartitionId += 1
    }
    ret
  }

  /**
    * Given broker and rack information, returns a list of brokers alternated by the rack. Assume
    * this is the rack and its brokers:
    *
    * rack1: 0, 1, 2
    * rack2: 3, 4, 5
    * rack3: 6, 7, 8
    *
    * This API would return the list of 0, 3, 6, 1, 4, 7, 2, 5, 8
    *
    * This is essential to make sure that the assignReplicasToBrokers API can use such list and
    * assign replicas to brokers in a simple round-robin fashion, while ensuring an even
    * distribution of leader and replica counts on each broker and that replicas are
    * distributed to all racks.
    */
  private[admin] def getRackAlternatedBrokerList(brokerRackMap: Map[Int, String]): IndexedSeq[Int] = {
    val brokersIteratorByRack = getInverseMap(brokerRackMap).map { case (rack, brokers) =>
      (rack, brokers.iterator)
    }
    val racks = brokersIteratorByRack.keys.toArray.sorted
    val result = new mutable.ArrayBuffer[Int]
    var rackIndex = 0
    while (result.size < brokerRackMap.size) {
      val rackIterator = brokersIteratorByRack(racks(rackIndex))
      if (rackIterator.hasNext)
        result += rackIterator.next()
      rackIndex = (rackIndex + 1) % racks.length
    }
    result
  }

  private[admin] def getInverseMap(brokerRackMap: Map[Int, String]): Map[String, Seq[Int]] = {
    brokerRackMap.toSeq.map { case (id, rack) => (rack, id) }
      .groupBy { case (rack, _) => rack }
      .map { case (rack, rackAndIdList) => (rack, rackAndIdList.map { case (_, id) => id }.sorted) }
  }

  private def replicaIndex(firstReplicaIndex: Int, secondReplicaShift: Int, replicaIndex: Int, nBrokers: Int): Int = {
    val shift = 1 + (secondReplicaShift + replicaIndex) % (nBrokers - 1)
    (firstReplicaIndex + shift) % nBrokers
  }

}

无机架的我们主要看assignReplicasToBrokersRackUnaware()这个方法就可以了

查看主题

先来看下查看具体topic信息的命令

# 多个topic名称有逗号隔开
kafka-topics.sh --zookeeper localhost:2181/kafka --describe --topic topic-create,topic-demo

使用参数–under-replicated-partitions可以显示未完成同步的副本，就是ISR集合将会少于AR集合。
使用参数–unavailable-partitions额可以查看主题中没有leader副本的分区

修改主题

在kafka-topics.sh中使用–alter这个参数来修改分区副本以及一些相应的参数，但是现在kafka不支持减少分区只支持增加分区，并且一般不推荐对分区和副本就行修改，这样很可能会影响下游业务的进行。

配置管理

修改配置可以使用专门的脚本kafka-configs.sh，使用这个脚本修改配置可以动态地修改就是在kafka还在运行的过程中修改

kafka-configs.sh --zookeeper localhost:2181/kafka --describe --entry-type topics --entry-name topic-config

这里的entry-type参数用于指定执行的对象，可以是topics也可以是brokers或者clients甚至是users

entry-tyoe解释	entry-name解释
topics 主题类型	指定主题名称，如上面topic-config
brokers kafka实例	指定brokerId
客户端类型	生产者或者消费者中设定的client.id编号
用户类型，users	指定用户名

添加配置

kafka-configs.sh --zookeeper localhost:2181/kafka --alter --entry-type topics --entry-name topic-config --add-config cleanup.policy=compact,max.message.bytes=10000

删除配置就用–delete-config这个参数，该操作将会把指定的参数恢复为默认值
使用kafka-configs.sh脚本来变更配置时，会在Zookeeper中创建一个命名形式为/config/<entry-type>/<entry-name>的节点，并将更改的配置写入这个节点，比如对也主题topic-config而言，对应的节点为/kafka/config/topics/topic-config
导出内容为：{“version”:1,“config”:{"<property-name>":"<property-value>"}}

删除主题

使用参数–delete，如果不知道主题是否存在添加参数–if-exists参数
使用kafka-topics.sh脚本删除主题的行为本质上只是在Zookeeper的中/kafka/admin/delete_topics路径下创建一个与待删除主题的同名的节点，以此标记这个主题为待删除

手动删除主题的方式

1. 删除Zookeeper中的节点/kafka/config/topics/topic-delete
2. 删除Zookeeper中的节点/kafka/brokers/topics/topic-delete
3. 删除集群中所有与主题topic-delete有关的文件，就是这个topic的副本文件