Kafka——AdminClient API详解

最新推荐文章于 2025-02-28 10:53:08 发布

吴声子夜歌

最新推荐文章于 2025-02-28 10:53:08 发布

阅读量2.1k

点赞数 2

分类专栏： kafka 文章标签： kafka 分布式 AdminClient

本文链接：https://blog.csdn.net/cold___play/article/details/132324514

版权

kafka 专栏收录该内容

29 篇文章

订阅专栏

自0.11.0.0版本起，Kafka社区推出客户端AdminClient统一集群管理API。本文介绍其使用方法，包括创建AdminClient、获取集群信息、创建和删除topic、查询topic和配置信息、修改配置及partition数量等，还提及新旧版本修改配置的不同方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

AdminClient

自0.ll.0.0版本起，Kafka社区推出了AdminClient和KafkaAdminClient,意在统一所有的集群管理API。使用0.11.0.0及以后版本的用户应该始终使用这个类来管理集群。

虽然和原先服务器端的AdminClient类同名，但这个工具是属于客户端的，因此只需要在管理程序项目中添加kafka clients依赖即可:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.3.0</version>
</dependency>

方法名称	作用
`createTopics`	创建一个或多个Topic
`listTopics`	查询Topic列表
`deleteTopics`	删除一个或多个Topic
`describeTopics`	查询Topic的描述信息
`describeConfigs`	查询Topic、Broker等的所有配置项信息
~~`alterConfigs`~~	用于修改Topic、Broker等的配置项信息（该方法在新版本中被标记为已过期）
`incrementalAlterConfigs`	同样也是用于修改Topic、Broker等的配置项信息，但功能更多、更灵活，用于代替`alterConfigs`
`createPartitions`	用于调整Topic的Partition数量，只能增加不能减少或删除，也就是说新设置的Partition数量必须大于等于之前的Partition数量

1、创建AdminClient

public static void main(String[] args) {
    Properties properties = new Properties();
    properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    AdminClient adminClient = AdminClient.create(properties);
    adminClient.close();
}

2、获取集群信息

/**
 * 获取Kafka集群信息
 * @param adminClient
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void describeCluster(AdminClient adminClient) throws ExecutionException, InterruptedException {
    DescribeClusterResult ret = adminClient.describeCluster();
    System.out.println(String.format("Cluster id: %s, controller: %s",
            ret.clusterId().get(), ret.controller().get()));
    System.out.println("Current cluster nodes info: ");
    for (Node node : ret.nodes().get()) {
        System.out.println(node);
    }
}

Cluster id: z-FbYLVTSqK-6TCv0osH3Q, controller: localhost:9092 (id: 0 rack: null)
Current cluster nodes info: 
localhost:9092 (id: 0 rack: null)

3、创建topic

/**
 * 创建一个分区数量为3，副本数量为1的主题
 * @param adminClient
 * @param topicName
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void createTopics(AdminClient adminClient, String topicName) throws ExecutionException, InterruptedException {
    NewTopic newTopic = new NewTopic(topicName, 3, (short)1);
    CreateTopicsResult ret = adminClient.createTopics(Arrays.asList(newTopic));
    ret.all().get();
}

4、查看topic列表

/**
 * 列出所有topic
 * @param adminClient
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void listAllTopics(AdminClient adminClient) throws ExecutionException, InterruptedException {
    //可配置选项
    ListTopicsOptions options = new ListTopicsOptions();
    //是否列出内部使用的topics
    options.listInternal(true);
    ListTopicsResult topicsResult = adminClient.listTopics(options);
    //打印topics名称
    Set<String> topicNames = topicsResult.names().get();
    System.out.println(topicNames);
    //打印topics信息
    Collection<TopicListing> topicListings = topicsResult.listings().get();
    System.out.println(topicListings);
}

[test, testTopic, test-create-topic, my-topic, __consumer_offsets]
[(name=test, internal=false), (name=testTopic, internal=false), (name=test-create-topic, internal=false), (name=my-topic, internal=false), (name=__consumer_offsets, internal=true)]

5、删除topic

/**
 * 删除给定topic
 * @param adminClient
 * @param topicName
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void deleteTopics(AdminClient adminClient, String topicName) throws ExecutionException, InterruptedException {
    DeleteTopicsResult ret = adminClient.deleteTopics(Arrays.asList(topicName));
    ret.all().get();
}

6、查询topic信息

一个Topic会有自身的描述信息，例如：partition的数量，副本集的数量，是否为internal等等。AdminClient中提供了describeTopics方法来查询这些描述信息。代码示例：

/**
 * 查看topic详细信息
 * @param adminClient
 * @param topicName
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void describeTopics(AdminClient adminClient, String topicName) throws ExecutionException, InterruptedException {
    DescribeTopicsResult ret = adminClient.describeTopics(Arrays.asList(topicName, "__consumer_offsets"));
    //等待返回结果完成
    Map<String, TopicDescription> topics = ret.all().get();
    for (Map.Entry<String, TopicDescription> entry : topics.entrySet()) {
        System.out.println(entry.getKey() + "---->" + entry.getValue());
    }

}

testTopic---->(name=testTopic, internal=false, partitions=(partition=0, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)), authorizedOperations=null)
__consumer_offsets---->(name=__consumer_offsets, internal=true, partitions=(partition=0, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=1, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=2, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=3, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=4, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=5, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=6, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=7, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=8, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=9, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=10, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=11, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=12, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=13, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=14, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=15, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=16, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=17, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=18, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=19, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=20, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=21, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=22, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=23, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=24, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=25, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=26, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=27, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=28, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=29, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=30, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=31, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=32, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=33, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=34, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=35, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=36, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=37, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=38, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=39, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=40, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=41, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=42, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=43, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=44, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=45, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=46, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=47, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=48, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)),(partition=49, leader=localhost:9092 (id: 0 rack: null), replicas=localhost:9092 (id: 0 rack: null), isr=localhost:9092 (id: 0 rack: null)), authorizedOperations=null)

7、查询配置信息

/**
 * 获取指定topic的topic级别参数配置信息
 * @param adminClient
 * @param topicName
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void describeConfig(AdminClient adminClient, String topicName) throws ExecutionException, InterruptedException {
    DescribeConfigsResult ret = adminClient.describeConfigs(Collections.singleton(new ConfigResource(ConfigResource.Type.TOPIC, topicName)));
    Map<ConfigResource, Config> configMap = ret.all().get();
    for (Map.Entry<ConfigResource, Config> entry : configMap.entrySet()) {
        ConfigResource key = entry.getKey();
        Config value = entry.getValue();
        System.out.println(String.format("Resource type: %s, resource name: %s", key.type(), key.name()));
        Collection<ConfigEntry> configEntries = value.entries();
        for (ConfigEntry each : configEntries) {
            System.out.println(each.name() + " = " + each.value());
        }
    }
}

Resource type: TOPIC, resource name: testTopic
compression.type = producer
leader.replication.throttled.replicas = 
message.downconversion.enable = true
min.insync.replicas = 1
segment.jitter.ms = 0
cleanup.policy = delete
flush.ms = 9223372036854775807
follower.replication.throttled.replicas = 
segment.bytes = 1073741824
retention.ms = 604800000
flush.messages = 9223372036854775807
message.format.version = 3.0-IV1
file.delete.delay.ms = 60000
max.compaction.lag.ms = 9223372036854775807
max.message.bytes = 1048588
min.compaction.lag.ms = 0
message.timestamp.type = CreateTime
preallocate = false
min.cleanable.dirty.ratio = 0.5
index.interval.bytes = 4096
unclean.leader.election.enable = false
retention.bytes = -1
delete.retention.ms = 86400000
segment.ms = 604800000
message.timestamp.difference.max.ms = 9223372036854775807
segment.index.bytes = 10485760

8、修改配置信息

8.1、alterConfigs

在早期版本中，使用alterConfigs方法来修改配置项。代码示例：

/**
 * 修改指定topic的topic级别参数
 * @param adminClient
 * @param topicName
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void alterConfigs(AdminClient adminClient, String topicName) throws ExecutionException, InterruptedException {
    // 指定ConfigResource的类型及名称
    ConfigResource configResource = new ConfigResource(
            ConfigResource.Type.TOPIC, topicName
    );
    // 配置项以ConfigEntry形式存在
    Config config = new Config(Arrays.asList(
            new ConfigEntry("cleanup.policy", "compact")
    ));
    //构建
    Map<ConfigResource, Config> configMap = new HashMap<>();
    configMap.put(configResource, config);
    //执行
    AlterConfigsResult ret = adminClient.alterConfigs(configMap);
    ret.all().get();

    //查看
    describeConfig(adminClient, topicName);
}

Resource type: TOPIC, resource name: my-topic
compression.type = producer
leader.replication.throttled.replicas = 
message.downconversion.enable = true
min.insync.replicas = 1
segment.jitter.ms = 0
cleanup.policy = compact
flush.ms = 9223372036854775807
follower.replication.throttled.replicas = 
segment.bytes = 1073741824
retention.ms = 604800000
flush.messages = 9223372036854775807
message.format.version = 3.0-IV1
file.delete.delay.ms = 60000
max.compaction.lag.ms = 9223372036854775807
max.message.bytes = 1048588
min.compaction.lag.ms = 0
message.timestamp.type = CreateTime
preallocate = false
min.cleanable.dirty.ratio = 0.5
index.interval.bytes = 4096
unclean.leader.election.enable = false
retention.bytes = -1
delete.retention.ms = 86400000
segment.ms = 604800000
message.timestamp.difference.max.ms = 9223372036854775807
segment.index.bytes = 10485760

8.2、incrementalAlterConfigs

在新版本中则是使用incrementalAlterConfigs方法来修改Topic的配置项，该方法使用起来相对于alterConfigs要略微复杂一些，但因此功能更多、更灵活。代码示例：

/**
 * 修改指定topic的topic级别配置
 * @param adminClient
 * @param topicName
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void incrementalAlterConfigs(AdminClient adminClient, String topicName) throws ExecutionException, InterruptedException {
    // 指定ConfigResource的类型及名称
    ConfigResource configResource = new ConfigResource(
            ConfigResource.Type.TOPIC, topicName
    );
    // 配置项同样以ConfigEntry形式存在，只不过增加了操作类型
    // 以及能够支持操作多个配置项，相对来说功能更多、更灵活
    Collection<AlterConfigOp> configs = Arrays.asList(
            new AlterConfigOp(
                    new ConfigEntry("preallocate", "false"),
                    AlterConfigOp.OpType.SET
            )
    );
    Map<ConfigResource, Collection<AlterConfigOp>> configMaps = new HashMap<>();
    configMaps.put(configResource, configs);
    AlterConfigsResult result = adminClient.incrementalAlterConfigs(configMaps);
    result.all().get();

    //查看
    describeConfig(adminClient, topicName);
}

Resource type: TOPIC, resource name: my-topic
compression.type = producer
leader.replication.throttled.replicas = 
message.downconversion.enable = true
min.insync.replicas = 1
segment.jitter.ms = 0
cleanup.policy = compact
flush.ms = 9223372036854775807
follower.replication.throttled.replicas = 
segment.bytes = 1073741824
retention.ms = 604800000
flush.messages = 9223372036854775807
message.format.version = 3.0-IV1
file.delete.delay.ms = 60000
max.compaction.lag.ms = 9223372036854775807
max.message.bytes = 1048588
min.compaction.lag.ms = 0
message.timestamp.type = CreateTime
preallocate = false
min.cleanable.dirty.ratio = 0.5
index.interval.bytes = 4096
unclean.leader.election.enable = false
retention.bytes = -1
delete.retention.ms = 86400000
segment.ms = 604800000
message.timestamp.difference.max.ms = 9223372036854775807
segment.index.bytes = 10485760

9、修改partition数量

在创建Topic时我们需要设定Partition的数量，但如果觉得初始设置的Partition数量太少了，那么就可以使用createPartitions方法来调整Topic的Partition数量，但是需要注意在Kafka中Partition只能增加不能减少。代码示例：

/**
 * 增加指定Topic的Partition数量
 * @param adminClient
 * @param topicName
 * @throws ExecutionException
 * @throws InterruptedException
 */
public static void incrPartitions(AdminClient adminClient, String topicName) throws ExecutionException, InterruptedException {
    Map<String, NewPartitions> newPartitions = new HashMap<>();
    //将Partition数量调整为2
    newPartitions.put(topicName, NewPartitions.increaseTo(2));
    CreatePartitionsResult ret = adminClient.createPartitions(newPartitions);
    ret.all().get();
}