参考文章:
https://www.bbsmax.com/A/n2d9bqDvzD/
对应的 kafka版本 : kafka_2.12-2.0.0.jar ,即 scala 2.12 ,kafka 2.0.0 版本。
注意 kafka 版本,建议选择 1.0.0 + 的版本。
我们查询所有消费的 group.id 一般都采用如下的方式进行查询 :
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
[root@master ~]# kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
console-consumer-464
group
但是,此种方式查询到的是所有 topic 的所有 group.id , 而没有根据 topic 查询 group.id 的方式。
要完成这样的功能 ,我们可以借助于 kafka-java 实现。
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>kafka-test</groupId>
<artifactId>kafka-ext-api</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.12</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>
</plugins>
</build>
</project>
工具类:
package com.test.kafka.util;
import kafka.admin.AdminClient;
import kafka.coordinator.group.GroupOverview;
import org.apache.kafka.common.TopicPartition;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* Created by szh on 2018/10/23.
*
* @author szh
*/
public class KafkaUtil {
public static Set<String> getAllGroupsForTopic(String brokerListUrl, String topic) {
AdminClient client = AdminClient.createSimplePlaintext(brokerListUrl);
try {
List<GroupOverview> allGroups = scala.collection.JavaConversions.seqAsJavaList(client.listAllGroupsFlattened().toSeq());
Set<String> groups = new HashSet<>();
for (GroupOverview overview : allGroups) {
String groupID = overview.groupId();
// 版本不匹配
// org.apache.kafka.common.errors.UnsupportedVersionException: The broker only supports OffsetFetchRequest v1, but we need v2 or newer to request all topic partitions.
// Map<TopicPartition, Object> offsets = scala.collection.JavaConversions.mapAsJavaMap(client.listGroupOffsets(groupID));
Map<TopicPartition, Object> offsets = scala.collection.JavaConversions.mapAsJavaMap(client.listGroupOffsets(groupID));
Set<TopicPartition> partitions = offsets.keySet();
for (TopicPartition tp : partitions) {
if (tp.topic().equals(topic)) {
groups.add(groupID);
}
}
}
return groups;
} finally {
client.close();
}
}
}
实现思路:
Step1:
查询到所有的 group.id
Step2:
遍历所有的group.id , 根据 group.id 查询到 各个partition 的消费情况 ,与 topic name
Step3 :
将 TopicPartition 中的 topic name 提取出来,如果 topic name 与指定的 topic name相同,则将 group id 加入 set 中
Step4 :
将 set <String> 返回
测试类:
package com.test.kafka.util;
import org.junit.Test;
import java.util.Set;
/**
* Created by szh on 2018/10/23.
*
* @author szh
*/
public class KafkaUtilTest {
@Test
public void testGetAllGroupsForTopic() {
// // 170 集群
// // topic : topic_advert_impress
// Set<String> consumerSet = KafkaUtil.getAllGroupsForTopic("10.170.0.6:9092,10.170.0.7:9092,10.170.0.8:9092", "topic_advert_impress");
Set<String> consumerSet = KafkaUtil.getAllGroupsForTopic("192.168.75.128:9092", "test_find");
for (String tmp : consumerSet) {
System.out.println(tmp);
}
}
}
注意 kafka 版本,若版本过低 ,如 0.10 ,会出现如下报错
// 版本不匹配 org.apache.kafka.common.errors.UnsupportedVersionException: The broker only supports OffsetFetchRequest v1, but we need v2 or newer to request all topic partitions.