对于kafka的consumer接口，提供两种版本，
high-level
一种high-level版本，比较简单不用关心offset, 会自动的读zookeeper中该Consumer group的last offset
参考，https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
不过要注意一些注意事项，对于多个partition和多个consumer
1. 如果consumer比partition多，是浪费，因为kafka的设计是在一个partition上是不允许并发的，所以consumer数不要大于partition数
2. 如果consumer比partition少，一个consumer会对应于多个partitions，这里主要合理分配consumer数和partition数，否则会导致partition里面的数据被取的不均匀
最好partiton数目是consumer数目的整数倍，所以partition数目很重要，比如取24，就很容易设定consumer数目
3. 如果consumer从多个partition读到数据，不保证数据间的顺序性，kafka只保证在一个partition上数据是有序的，但多个partition，根据你读的顺序会有不同
4. 增减consumer，broker，partition会导致rebalance，所以rebalance后consumer对应的partition会发生变化
5. High-level接口中获取不到数据的时候是会block的
简单版，
简单的坑，如果测试流程是，先produce一些数据，然后再用consumer读的话，记得加上第一句设置
因为初始的offset默认是非法的，然后这个设置的意思是，当offset非法时，如何修正offset，默认是largest，即最新，所以不加这个配置，你是读不到你之前produce的数据的，而且这个时候你再加上smallest配置也没用了，因为此时offset是合法的，不会再被修正了，需要手工或用工具改重置offset
Properties props =new Properties();
props.put("auto.offset.reset", "smallest"); //必须要加，如果要读旧数据
props.put("zookeeper.connect", "localhost:2181");
props.put("group.id", "pv");
props.put("zookeeper.session.timeout.ms","400");
props.put("zookeeper.sync.time.ms","200");
props.put("auto.commit.interval.ms", "1000");
ConsumerConfig conf = newConsumerConfig(props);
ConsumerConnector consumer =kafka.consumer.Consumer.createJavaConsumerConnector(conf);
String topic = "page_visits";
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(topic, new Integer(1));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap =consumer.createMessageStreams(topicCountMap);
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
KafkaStream<byte[], byte[]> stream = streams.get(0);
ConsumerIterator<byte[], byte[]> it =stream.iterator();
while(it.hasNext()){
System.out.println("message: " + new String(it.next().message()));
}
if(consumer != null) consumer.shutdown(); //其实执行不到，因为上面的hasNext会block
在用high-level的consumer时，两个给力的工具，
1.bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group pv
可以看到当前group offset的状况，比如这里看pv的状况，3个partition
Group Topic Pid Offset logSize Lag Owner
pv page_visits 0 21 21 0 none
pv page_visits 1 19 19 0 none
pv page_visits 2 20 20 0 none
关键就是offset，logSize和Lag
这里以前读完了，所以offset=logSize，并且Lag=0
2.bin/kafka-run-class.sh kafka.tools.UpdateOffsetsInZK earliestconfig/consumer.properties page_visits
3个参数，
[earliest | latest]，表示将offset置到哪里
consumer.properties ，这里是配置文件的路径
topic，topic名，这里是page_visits
我们对上面的pv group执行完这个操作后，再去checkgroup offset状况，结果如下，
Group Topic Pid Offset logSize Lag Owner
pv page_visits 0 0 21 21 none
pv page_visits 1 0 19 19 none
pv page_visits 2 0 20 20 none
可以看到offset已经被清0，Lag=logSize
底下给出原文中多线程consumer的完整代码
import kafka.consumer.ConsumerConfig;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public classConsumerGroupExample {
private finalConsumerConnector consumer;
private finalString topic;
private ExecutorService executor;
publicConsumerGroupExample(String a_zookeeper, String a_groupId, String a_topic){
consumer = kafka.consumer.Consumer.createJavaConsumerConnector( // 创建Connector，注意下面对conf的配置
createConsumerConfig(a_zookeeper, a_groupId));
this.topic= a_topic;
}
public void shutdown() {
if(consumer != null) consumer.shutdown();
if(executor != null) executor.shutdown();
}
public void run(int a_numThreads) { // 创建并发的consumers
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(topic, new Integer(a_numThreads)); // 描述读取哪个topic，需要几个线程读
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap =consumer.createMessageStreams(topicCountMap); // 创建Streams
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic); // 每个线程对应于一个KafkaStream
// now launch all the threads
//
executor =Executors.newFixedThreadPool(a_numThreads);
// now create an object to consume the messages
//
intthreadNumber = 0;
for (finalKafkaStream stream : streams) {
executor.submit(new ConsumerTest(stream, threadNumber)); // 启动consumerthread
threadNumber++;
}
}
private static ConsumerConfig createConsumerConfig(String a_zookeeper,String a_groupId) {
Properties props = newProperties();
props.put("zookeeper.connect", a_zookeeper);
props.put("group.id", a_groupId);
props.put("zookeeper.session.timeout.ms", "400");
props.put("zookeeper.sync.time.ms", "200");
props.put("auto.commit.interval.ms", "1000");
return new ConsumerConfig(props);
}
public static void main(String[] args) {
String zooKeeper = args[0];
String groupId = args[1];
String topic = args[2];
intthreads = Integer.parseInt(args[3]);
ConsumerGroupExample example = newConsumerGroupExample(zooKeeper, groupId, topic);
example.run(threads);
try{
Thread.sleep(10000);
} catch(InterruptedException ie) {
}
example.shutdown();
}
}
LOW LEVEL
SimpleConsumer
另一种是SimpleConsumer，名字起的，以为是简单的接口，其实是low-level consumer，更复杂的接口
参考，https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
什么时候用这个接口?
- Read a message multiple times （多次）
- Consume only a subset of the partitions in a topic in a process （子集）
- Manage transactions to make sure a message is processed once and only once （事务）
当然用这个接口是有代价的，即partition,broker,offset对你不再透明，需要自己去管理这些，并且还要handle broker leader的切换，很麻烦
所以不是一定要用，最好别用
- You must keep track of the offsets in your application to know where you left off consuming. （记录）
- You must figure out which Broker is the lead Broker for a topic and partition （计算出）
- You must handle Broker leader changes
使用SimpleConsumer的步骤：
- Find an active Broker and find out which Broker is the leader for your topic and partition
- Determine who the replica Brokers are for your topic and partition
- Build the request defining what data you are interested in
- Fetch the data
- Identify and recover from leader changes
首先，你必须知道读哪个topic的哪个partition
然后，找到负责该partition的brokerleader，从而找到存有该partition副本的那个broker
再者，自己去写request并fetch数据
最终，还要注意需要识别和处理broker leader的改变
逐步来看，
Finding the Lead Broker for a Topic andPartition
思路就是，遍历每个broker，取出该topic的metadata，然后再遍历其中的每个partitionmetadata，如果找到我们要找的partition就返回
根据返回的PartitionMetadata.leader().host()找到leaderbroker
privatePartitionMetadata findLeader(List<String> a_seedBrokers, int a_port, String a_topic, int a_partition){
PartitionMetadata returnMetaData = null;
loop:
for(String seed : a_seedBrokers) { //遍历每个broker
SimpleConsumer consumer = null;
try {
//创建SimpleConsumer，
//classSimpleConsumer(val host: String,val port: Int,val soTimeout: Int
// ,val bufferSize: Int,valclientId: String)
consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024,"leaderLookup");
List<String> topics =Collections.singletonList(a_topic);
TopicMetadataRequest req = newTopicMetadataRequest(topics); //
kafka.javaapi.TopicMetadataResponse resp = consumer.send(req); //发送TopicMetadata Request请求
List<TopicMetadata> metaData = resp.topicsMetadata(); //取到Topic的Metadata
for (TopicMetadata item : metaData) {
for (PartitionMetadata part : item.partitionsMetadata()) {//遍历每个partition的metadata
if (part.partitionId() == a_partition) { //确认是否是我们要找的partition
returnMetaData = part;
break loop; //找到就返回
}
}
}
} catch (Exception e) {
System.out.println("Error communicating with Broker[" + seed + "] to find Leader for [" + a_topic
+ ", " + a_partition +"] Reason: " +e);
} finally {
if (consumer != null) consumer.close();
}
}
returnreturnMetaData;
}
Finding Starting Offset forReads
request主要的信息就是Map<TopicAndPartition, PartitionOffsetRequestInfo>
TopicAndPartition就是对topic和partition信息的封装
PartitionOffsetRequestInfo的定义
caseclass PartitionOffsetRequestInfo(time: Long, maxNumOffsets: Int)
其中参数time，表示where tostart reading data，两个取值
kafka.api.OffsetRequest.EarliestTime()，the beginning of the data inthe logs
kafka.api.OffsetRequest.LatestTime()，will only stream new messages
不要认为起始的offset一定是0，因为messages会过期，被删除
另外一个参数不清楚什么含义，代码中取的是1
public static long getLastOffset(SimpleConsumer consumer,String topic, int partition,
long whichTime, String clientName) {
TopicAndPartitiontopicAndPartition = new TopicAndPartition(topic, partition);
Map<TopicAndPartition, PartitionOffsetRequestInfo>requestInfo = new HashMap<TopicAndPartition,PartitionOffsetRequestInfo>();
requestInfo.put(topicAndPartition, newPartitionOffsetRequestInfo(whichTime, 1)); //buildoffset fetch request info
kafka.javaapi.OffsetRequestrequest = new kafka.javaapi.OffsetRequest(requestInfo,
kafka.api.OffsetRequest.CurrentVersion(),clientName);
OffsetResponse response = consumer.getOffsetsBefore(request); //取到offsets
if(response.hasError()) {
System.out.println("Error fetching data Offset Datathe Broker. Reason: " +response.errorCode(topic, partition) );
return 0;
}
long[]offsets = response.offsets(topic, partition); //取到的一组offset
returnoffsets[0]; //取第一个开始读
}
Reading the Data
首先在FetchRequest上加上Fetch，指明topic，partition，开始的offset，读取的大小
如果producer在写入很大的message时，也许这里指定的1000000是不够的，会返回anempty message set，这时需要增加这个值，直到得到一个非空的message set。
// When calling FetchRequestBuilder, it's important NOT to call.replicaId(), which is meant for internal use only.
// Setting the replicaId incorrectly will cause the brokers to behaveincorrectly.
FetchRequest req = new FetchRequestBuilder()
.clientId(clientName)
.addFetch(a_topic, a_partition,readOffset, 100000) // 1000000bytes
.build();
FetchResponse fetchResponse = consumer.fetch(req);
if (fetchResponse.hasError()) {
// See Error Handling
}
numErrors = 0;
long numRead = 0;
for (MessageAndOffset messageAndOffset :fetchResponse.messageSet(a_topic, a_partition)) {
longcurrentOffset =messageAndOffset.offset();
if(currentOffset <readOffset) { // 必要判断，因为对于compressed message，会返回整个block，所以可能包含old的message
System.out.println("Found an old offset: " +currentOffset + " Expecting: " + readOffset);
continue;
}
readOffset = messageAndOffset.nextOffset(); // 获取下一个readOffset
ByteBuffer payload = messageAndOffset.message().payload();
byte[] bytes =new byte[payload.limit()];
payload.get(bytes);
System.out.println(String.valueOf(messageAndOffset.offset()) + ": " + new String(bytes, "UTF-8"));
numRead++;
}
if (numRead == 0) {
try {
Thread.sleep(1000);
} catch(InterruptedException ie) {
}
}
Error Handling
if(fetchResponse.hasError()) {
numErrors++;
// Something went wrong!
shortcode =fetchResponse.errorCode(a_topic, a_partition);
System.out.println("Error fetching data from theBroker:" + leadBroker + " Reason: " + code);
if(numErrors > 5) break;
if (code==ErrorMapping.OffsetOutOfRangeCode()) { // 处理offset非法的问题，用最新的offset
// We asked for an invalid offset. For simple case ask forthe last element to reset
readOffset = getLastOffset(consumer,a_topic,a_partition, kafka.api.OffsetRequest.LatestTime(), clientName);
continue;
}
consumer.close();
consumer = null;
leadBroker = findNewLeader(leadBroker, a_topic, a_partition, a_port); // 更新leader broker
continue;
}
没有特别的逻辑，只是重新调用findLeader获取leaderbroker
并且防止在切换过程中，取不到leader信息，加上sleep逻辑
private StringfindNewLeader(String a_oldLeader, String a_topic, int a_partition, int a_port) throws Exception{
for (int i = 0; i < 3; i++){
boolean goToSleep = false;
PartitionMetadata metadata =findLeader(m_replicaBrokers, a_port, a_topic, a_partition);
if (metadata == null) {
goToSleep = true;
} else if (metadata.leader() == null) {
goToSleep = true;
} else if(a_oldLeader.equalsIgnoreCase(metadata.leader().host()) && i == 0) {
// first time through if the leader hasn't changed giveZooKeeper a second to recover
// second time, assume thebroker did recover before failover, or it was a non-Broker issue
//
goToSleep = true;
} else {
returnmetadata.leader().host();
}
if (goToSleep) {
try {
Thread.sleep(1000);
} catch (InterruptedExceptionie) {
}
}
}
System.out.println("Unable to find new leader after Broker failure.Exiting");
throw new Exception("Unable to find new leader after Brokerfailure. Exiting");
}
Full Source Code
packagecom.test.simple;
importkafka.api.FetchRequest;
importkafka.api.FetchRequestBuilder;
importkafka.api.PartitionOffsetRequestInfo;
importkafka.common.ErrorMapping;
importkafka.common.TopicAndPartition;
importkafka.javaapi.*;
importkafka.javaapi.consumer.SimpleConsumer;
importkafka.message.MessageAndOffset;
importjava.nio.ByteBuffer;
importjava.util.ArrayList;
importjava.util.Collections;
importjava.util.HashMap;
importjava.util.List;
importjava.util.Map;
public class SimpleExample{
public static void main(String args[]) {
SimpleExample example = new SimpleExample();
long maxReads = Long.parseLong(args[0]);
String topic = args[1];
int partition = Integer.parseInt(args[2]);
List<String> seeds = new ArrayList<String>();
seeds.add(args[3]);
int port = Integer.parseInt(args[4]);
try {
example.run(maxReads, topic,partition, seeds, port);
} catch (Exception e) {
System.out.println("Oops:" +e);
e.printStackTrace();
}
}
private List<String> m_replicaBrokers = newArrayList<String>();
public SimpleExample() {
m_replicaBrokers = new ArrayList<String>();
}
public void run(long a_maxReads, String a_topic, int a_partition,List<String> a_seedBrokers, int a_port) throws Exception {
//find the meta data about the topic and partition we are interestedin
//
PartitionMetadata metadata =findLeader(a_seedBrokers, a_port, a_topic, a_partition);
if (metadata == null) {
System.out.println("Can't find metadata for Topic and Partition.Exiting");
return;
}
if (metadata.leader() == null) {
System.out.println("Can't find Leader for Topic and Partition.Exiting");
return;
}
String leadBroker =metadata.leader().host();
String clientName = "Client_" + a_topic + "_" + a_partition;
SimpleConsumer consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
long readOffset =getLastOffset(consumer,a_topic, a_partition,kafka.api.OffsetRequest.EarliestTime(), clientName);
int numErrors = 0;
while (a_maxReads > 0){
if (consumer == null) {
consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
}
FetchRequest req = new FetchRequestBuilder()
.clientId(clientName)
.addFetch(a_topic,a_partition, readOffset, 100000) // Note: this fetchSize of 100000 might need to beincreased if large batches are written to Kafka
.build();
FetchResponse fetchResponse =consumer.fetch(req);
if(fetchResponse.hasError()) {
numErrors++;
// Something went wrong!
short code = fetchResponse.errorCode(a_topic, a_partition);
System.out.println("Error fetching data from the Broker:" +leadBroker + " Reason: " +code);
if (numErrors > 5) break;
if (code == ErrorMapping.OffsetOutOfRangeCode()) {
// We asked for an invalid offset. For simple case ask forthe last element to reset
readOffset = getLastOffset(consumer,a_topic, a_partition,kafka.api.OffsetRequest.LatestTime(), clientName);
continue;
}
consumer.close();
consumer = null;
leadBroker =findNewLeader(leadBroker, a_topic, a_partition, a_port);
continue;
}
numErrors = 0;
long numRead = 0;
for (MessageAndOffsetmessageAndOffset : fetchResponse.messageSet(a_topic, a_partition)) {
long currentOffset = messageAndOffset.offset();
if (currentOffset < readOffset) {
System.out.println("Found an old offset: " + currentOffset +" Expecting: " +readOffset);
continue;
}
readOffset =messageAndOffset.nextOffset();
ByteBuffer payload =messageAndOffset.message().payload();
byte[] bytes = new byte[payload.limit()];
payload.get(bytes);
System.out.println(String.valueOf(messageAndOffset.offset()) + ": " + new String(bytes, "UTF-8"));
numRead++;
a_maxReads--;
}
if (numRead == 0) {
try {
Thread.sleep(1000);
} catch (InterruptedExceptionie) {
}
}
}
if (consumer != null) consumer.close();
}
public static long getLastOffset(SimpleConsumer consumer, String topic, int partition,
long whichTime, StringclientName) {
TopicAndPartitiontopicAndPartition = newTopicAndPartition(topic, partition);
Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo = newHashMap<TopicAndPartition, PartitionOffsetRequestInfo>();
requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(whichTime, 1));
kafka.javaapi.OffsetRequestrequest = new kafka.javaapi.OffsetRequest(
requestInfo,kafka.api.OffsetRequest.CurrentVersion(), clientName);
OffsetResponse response =consumer.getOffsetsBefore(request);
if (response.hasError()) {
System.out.println("Error fetching data Offset Data the Broker. Reason:" + response.errorCode(topic, partition));
return 0;
}
long[] offsets =response.offsets(topic, partition);
return offsets[0];
}
private String findNewLeader(String a_oldLeader, String a_topic,inta_partition, inta_port) throwsException {
for (int i = 0; i < 3; i++){
boolean goToSleep = false;
PartitionMetadata metadata =findLeader(m_replicaBrokers, a_port, a_topic, a_partition);
if (metadata == null) {
goToSleep = true;
} else if (metadata.leader() == null) {
goToSleep = true;
} else if(a_oldLeader.equalsIgnoreCase(metadata.leader().host()) && i == 0) {
// first time through if the leader hasn't changed giveZooKeeper a second to recover
// second time, assumethe broker did recover before failover, or it was a non-Broker issue
//
goToSleep = true;
} else {
returnmetadata.leader().host();
}
if (goToSleep) {
try {
Thread.sleep(1000);
} catch (InterruptedExceptionie) {
}
}
}
System.out.println("Unable to find new leader after Broker failure.Exiting");
throw new Exception("Unable to find new leader after Brokerfailure. Exiting");
}
private PartitionMetadata findLeader(List<String>a_seedBrokers, int a_port, String a_topic, int a_partition) {
PartitionMetadata returnMetaData = null;
loop:
for (String seed : a_seedBrokers) {
SimpleConsumer consumer = null;
try {
consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024,"leaderLookup");
List<String> topics =Collections.singletonList(a_topic);
TopicMetadataRequest req = new TopicMetadataRequest(topics);
kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);
List<TopicMetadata> metaData = resp.topicsMetadata();
for (TopicMetadata item :metaData) {
for (PartitionMetadata part: item.partitionsMetadata()) {
if (part.partitionId() == a_partition) {
returnMetaData = part;
break loop;
}
}
}
} catch (Exception e){
System.out.println("Error communicating with Broker [" + seed +"] to find Leader for [" +a_topic
+ ", " + a_partition + "] Reason: " + e);
} finally {
if (consumer != null)consumer.close();
}
}
if (returnMetaData != null) {
m_replicaBrokers.clear();
for (kafka.cluster.Brokerreplica : returnMetaData.replicas()) {
m_replicaBrokers.add(replica.host());
}
}
return returnMetaData;
}
}
源文档 <http://blog.csdn.net/strawbingo/article/details/45366061>
源文档 <http://blog.csdn.net/strawbingo/article/details/45366061>
扫一扫
4万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
