我这里的Kafka Consumer代码是拷贝网上的,就是开一个线程监听kafka topic,一有消息就处理。开始的代码是这样的:
public void kafkaStart() {
final String topic = hipchatAction.properties.getProperty("kafka.hipchat.topic");
final int partitionNum = Integer.valueOf(hipchatAction.properties.getProperty("kafka.hipchat.topic.partitionNum"));
log.debug("Comes to kafkaStart() with topic : " + topic + ", partitionNum : " + partitionNum);
Map<String, Integer> topicCountMap = new HashMap<>();
topicCountMap.put(topic, partitionNum);
Map<String, List<KafkaStream<byte[], byte[]>>> streams = connector.createMessageStreams(topicCountMap);
List<KafkaStream<byte[], byte[]>> partitions = streams.get(topic);
ExecutorService executor = Executors.newFixedThreadPool(partitionNum * 2);
for (KafkaStream<byte[], byte[]> partition : partitions) {
/**
* Here also it is an unknown issue, if use anonymous inner class, then the thread seems died, must have
* a named inner class!
*/
executor.execute(new MessageRunner(partition));
}
}
class MessageRunner implements Runnable {
private KafkaStream<byte[], byte[]> partition;
MessageRunner(KafkaStream<byte[], byte[]> partition) {
this.partition = partition;
}
public void run() {
ConsumerIterator<byte[], byte[]> it = partition.iterator();
while (it.hasNext()) {
MessageAndMetadata<byte[], byte[]> msgMeta = it.next();
/**
* In order to consume Chinese message, here should use charset.
*/
String jsonStr = new String(msgMeta.message(), StandardCharsets.UTF_8);
log.debug("********* Message to be consumed in UTF-8 is :: " + jsonStr);
KafkaMsgVO msg = new Gson().fromJson(jsonStr, KafkaMsgVO.class);
hipchatAction.sendMessageToRoom(msg.getRoomName(), msg.getToken(), msg.getMsgText());
}
}
}
方法kafkaStart()会在Spring容器启动的时候执行。
对于 KafkaMsgVO msg = new Gson().fromJson(jsonStr, KafkaMsgVO.class); 这行,因为我的程序对消息要求是一个对象名为 KafkaMsgVO 的JSON格式数据,但是测试的时候测试人员就随便发了一条消息,没有进行对象的属性赋值并组装成 JSON 数据,所以抛出 JSONSyntaxException。问题就出在这,因为一旦抛出异常,这个线程就被破坏了,没办法进行后续消息的消费,尽管 kafka consumer 仍然能感知到 topic 里面有新的消息到来。
解决办法:
为了不让异常破坏 consumer 线程,我进行了出错代码的位置转移,将得到的消息直接通过 Apache AKKA 进行转发,然后由 AKKA 的 onReceive() 对消息进行处理,也就是出错代码移到 onReceive() 里面,这样就保证了 consumer 线程的健壮性。代码如下:
class MessageRunner implements Runnable {
private KafkaStream<byte[], byte[]> partition;
MessageRunner(KafkaStream<byte[], byte[]> partition) {
this.partition = partition;
}
public void run() {
ConsumerIterator<byte[], byte[]> it = partition.iterator();
while (it.hasNext()) {
MessageAndMetadata<byte[], byte[]> msgMeta = it.next();
/**
* In order to consume Chinese message, here should use charset.
*/
String jsonStr = new String(msgMeta.message(), StandardCharsets.UTF_8);
log.debug("********* Message to be consumed in UTF-8 is :: " + jsonStr);
ActorRef sender = akkaSystem.getMsgConductor();
sender.tell(new AkkaAdaptor(jsonStr, hipchatAction), sender);
}
}
}
一旦消息被 AKKA 转发后,对消息的处理代码就放在了AKKA 的 Actor 里面了:
} else if (message instanceof AkkaAdaptor) {
AkkaAdaptor akkaAdaptor = (AkkaAdaptor)message;
String textMessage = akkaAdaptor.getTextMessage();
KafkaMsgVO msg = null;
try {
msg = new Gson().fromJson(textMessage, KafkaMsgVO.class);
} catch (Exception e) {
log.debug(textMessage + " is malformed, it may miss some important property (value).");
return;
}
HipchatAction hipchatAction = akkaAdaptor.getHipchatAction();
log.debug("Kafka message sent by AKKA is :: " + msg.getMsgText());
hipchatAction.sendMessageToRoom(msg.getRoomName(), msg.getToken(), msg.getMsgText());
}