java consumer kafka,Kafka Java Consumer已经关闭

I have just started using Kafka. I am facing a small issue with the consumer. I have written a consumer in Java.

I get this exception - IllegalStateException This consumer has already been closed.

I get exception on the following line :

ConsumerRecords consumerRecords = consumer.poll(1000);

This started happening after my consumer crashed with some exception and when I tried running it again it gave me this exception.

Here is the complete code :

package StreamApplicationsTest;

import org.apache.kafka.clients.consumer.*;

import org.apache.kafka.common.serialization.StringDeserializer;

import java.util.*;

public class StreamAppConsumer {

public static void main(String[] args){

int i = 0;

//List topics = new ArrayList<>();

List topics = Collections.singletonList("test_topic");

//topics.add("test_topic");

Properties consumerConfigurations = new Properties();

consumerConfigurations.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092");

consumerConfigurations.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

consumerConfigurations.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class.getName());

consumerConfigurations.put(ConsumerConfig.GROUP_ID_CONFIG,"TestId");

Consumer consumer = new KafkaConsumer<>(consumerConfigurations);

consumer.subscribe(topics);

while(true){

ConsumerRecords consumerRecords = consumer.poll(1000);

Iterator> iterator = consumerRecords.iterator();

while(iterator.hasNext()){

i++;

ConsumerRecord consumerRecord = iterator.next();

String key = consumerRecord.key();

String value = consumerRecord.value();

if(key=="exit" || value=="exit")

break;

System.out.println("Key="+key+"\tValue="+value);

}

System.out.println("Messages processed = "+Integer.toString(i));

consumer.close();

}

}

}

I am just stuck with this issue any sort of help will be useful.

解决方案

This is happening because you are closing the consumer at the end of your infinite loop so when it polls a second time the consumer has been closed. To handle the immediate problem I'd wrap the entire while(true) loop in a try-catch and handle the consumer close in the catch or finally block.

However if different shutdown signals aren't handled carefully with a Kafka consumer you run the risk of losing data. I'd recommend looking at Confluent's example for graceful consumer shutdown here. In your case since you're running in the main thread it'd look something like this ...

public static void main(String[] args) {

int i = 0;

//List topics = new ArrayList<>();

List topics = Collections.singletonList("test_topic");

//topics.add("test_topic");

Properties consumerConfigurations = new Properties();

consumerConfigurations.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

consumerConfigurations.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

consumerConfigurations.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

consumerConfigurations.put(ConsumerConfig.GROUP_ID_CONFIG, "TestId");

Consumer consumer = new KafkaConsumer<>(consumerConfigurations);

consumer.subscribe(topics);

Runtime.getRuntime().addShutdownHook(new Thread()

{

public void run() {

consumer.wakeup();

}

});

try {

while (true) {

ConsumerRecords consumerRecords = consumer.poll(1000);

Iterator> iterator = consumerRecords.iterator();

while (iterator.hasNext()) {

i++;

ConsumerRecord consumerRecord = iterator.next();

String key = consumerRecord.key();

String value = consumerRecord.value();

if (key == "exit" || value == "exit")

break;

System.out.println("Key=" + key + "\tValue=" + value);

}

System.out.println("Messages processed = " + Integer.toString(i));

}

} catch (WakeupExection e) {

// Do Nothing

} finally {

consumer.close();

}

}

}

basically running consumer.wakeup() is the only threadsafe method in the consumer so it's the only one than can be ran inside of Java's shutdown hook. Since the consumer is not asleep when wakeup is called it trips the wakeupexection which falls through to gracefully shutting down the consumer.