kafka的生产与消费

因为项目的业务于kafka有关，自己也对kafka有了一定的了解，这里全当总结，有解释的错误的地方，还请指认！   

kafka的介绍不是本文的重点，若有对kafka还不是很了解的同学可以先去了解一下。

可以参考下：http://www.orchome.com/kafka/index

下面回到正题。

kafka是依赖于zookeeper使用的，zookeeper上会存kafka的broker信息、topic信息、conusergroup信息(对应的消费量)等等，其实还有其他的很多信息，如配置信息、Controller_epoch等这里不做说明。

kafka的生产信息会同步到zookeeper的，但是消费信息不一定。kafka的消费分为高级消费与低级消费，将消费信息同步到zookeeper的为高级消费，仅仅将消费信息存在kafka上的为低级消费(这里的分类可能不太严谨，比较直白，这个是公司大牛总结出来的)。

---

这里我做了两套以javaapi的方式去实现的kafka的生产与消费，一套高级生产与消费，一套低级生产与消费。

高级生产与消费:

生产消息：

package kafkaModel;

import java.util.Properties;

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

/**
*  @ClassName kafkaProducerTest3
*  @author chenb
*  @date 创建时间：2018年9月6日 下午4:15:32
**/
public class kafkaProducerTest3 {
	private final Producer<String, String> producer;
    public final static String TOPIC = "chenbTest0906";
 
    private kafkaProducerTest3(){
    	
        Properties props = new Properties();
        
        // 此处配置的是kafka的broker地址:端口列表
        props.put("metadata.broker.list", "192.168.5.56:9092,192.168.5.56:9093");
 
        //配置value的序列化类
        props.put("serializer.class", "kafka.serializer.StringEncoder");
        
        //配置key的序列化类
        props.put("key.serializer.class", "kafka.serializer.StringEncoder");
 
        //request.required.acks
        //0, which means that the producer never waits for an acknowledgement from the broker (the same behavior as 0.7). This option provides the lowest latency but the weakest durability guarantees (some data will be lost when a server fails).
        //1, which means that the producer gets an acknowledgement after the leader replica has received the data. This option provides better durability as the client waits until the server acknowledges the request as successful (only messages that were written to the now-dead leader but not yet replicated will be lost).
        //-1, which means that the producer gets an acknowledgement after all in-sync replicas have received the data. This option provides the best durability, we guarantee that no messages will be lost as long as at least one in sync replica remains.
        props.put("request.required.acks","-1");
 
        producer = new Producer<String, String>(new ProducerConfig(props));
    }
 
    void produce() {
        int messageNo = 1;
        final int COUNT = 10;
 
        int messageCount = 0;
        while (messageNo <= COUNT) {
            String key = String.valueOf(messageNo);
            String data = "Hello kafka message :" + key;
            producer.send(new KeyedMessage<String, String>(TOPIC, key ,data));
            System.out.println(data);
            messageNo ++;
            messageCount++;
        }
        
        System.out.println("Producer端一共产生了" + messageCount + "条消息！");
    }
 
    public static void main( String[] args )
    {
        new kafkaProducerTest3().produce();
    }

}

上面是生产消息的代码，生产10条消息成功。

使用KafkaOffsetMonitor监控，可以看到topicList里已经新加了一个名为chenbTest0906的topic。

不了解KafkaOffsetMonitor工具的可以看https://blog.csdn.net/catshitone/article/details/50462462

消费消息：

package kafkaModel;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;

import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
import kafka.serializer.StringDecoder;
import kafka.utils.VerifiableProperties;

/**
*  @ClassName KafkaConsumerTest3
*  @author chenb
*  @date 创建时间：2018年9月6日 下午4:15:51
**/
public class KafkaConsumerTest3 {
	private final ConsumerConnector consumer;
	 
    private KafkaConsumerTest3() {
        Properties props = new Properties();
        
        // zookeeper 配置
        props.put("zookeeper.connect", "192.168.5.56:2181");
 
        // 消费者所在组
        //props.put("group.id", "group0830");
        props.put("group.id", "test-group0906");
 
        // zk连接超时
        props.put("zookeeper.session.timeout.ms", "4000");
        props.put("zookeeper.sync.time.ms", "200");
        props.put("auto.commit.interval.ms", "1000");
        props.put("auto.offset.reset", "smallest");
        
        // 序列化类
        props.put("serializer.class", "kafka.serializer.StringEncoder");
 
        ConsumerConfig config = new ConsumerConfig(props);
 
        consumer = kafka.consumer.Consumer.createJavaConsumerConnector(config);
    }
 
    void consume() {
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put("chenbTest0906", new Integer(1));
 
        StringDecoder keyDecoder = new StringDecoder(new VerifiableProperties());
        StringDecoder valueDecoder = new StringDecoder(new VerifiableProperties());
 
        Map<String, List<KafkaStream<String, String>>> consumerMap = 
                consumer.createMessageStreams(topicCountMap,keyDecoder,valueDecoder);
        KafkaStream<String, String> stream = consumerMap.get("chenbTest0906").get(0);
        ConsumerIterator<String, String> it = stream.iterator();
        
        int messageCount = 0;
        while (it.hasNext()){
        	System.out.println(it.next().message());
        	messageCount++;
        }
        System.out.println("Consumer端一共消费了" + messageCount + "条消息！");
    }
 
    public static void main(String[] args) {
        new KafkaConsumerTest3().consume();
    }

}

消费消息成功。

在高级消费中，需要注明zookeeper的信息，将消费情况同步到zk中，而KafkaOffsetMonitor工具用于监控zk中的变化，所以这里可以看到消费。

---

低级生产与消费:

package kafkaModel;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

/**
*  @ClassName KafkaProducerTest
*  @author chenb
*  @date 创建时间：2018年9月06日 下午3:33:01
**/
public class KafkaProducerTest implements Runnable {
	
    public static void main(String args[]) {
        KafkaProducerTest test = new KafkaProducerTest("chenbTest0906");
        Thread thread = new Thread(test);
        thread.start();
    }
	
	private final KafkaProducer<String, String> producer;
	private final String topic;
	
    public KafkaProducerTest(String topicName) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "192.168.5.56:9092,192.168.5.56:9093");
        props.put("acks", "all");
        props.put("retries", 0);
        props.put("batch.size", 16384);
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        this.producer = new KafkaProducer<String, String>(props);
        this.topic = topicName;
    }
    
	public void run() {
	   int messageNo = 1;
        try {
        	for (int i = 0; i < 5; i++) {
                String messageStr="你好，这是第"+messageNo+"条数据";
                producer.send(new ProducerRecord<String, String>(topic, "Message", messageStr));
                System.out.println("发送的信息:" + messageStr);
                messageNo++;
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            producer.close();
        }
	}
	
}

生产消息：

消息成功生产5条。

在KafkaOffsetMonitor中也可以看出多了5条消息。

消费消息：

package kafkaModel;

import java.util.Arrays;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

/**
*  @ClassName KafkaConsumerTest
*  @author chenb
*  @date 创建时间：2018年9月06日 下午3:41:41
**/
public class KafkaConsumerTest implements Runnable {
    public static void main(String args[]) {
        KafkaConsumerTest test1 = new KafkaConsumerTest("chenbTest0906");
        Thread thread1 = new Thread(test1);
        thread1.start();
    }
	   
	private final KafkaConsumer<String, String> consumer;
    private ConsumerRecords<String, String> msgList;
    private final String topic;
    private static final String GROUPID = "test-group0906";

    public KafkaConsumerTest(String topicName) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "192.168.5.56:9092,192.168.5.56:9093");
        props.put("group.id", GROUPID);
        //props.put("group.id", "test-group1");
        //props.put("enable.auto.commit", "true");
        props.put("auto.commit.interval.ms", "1000");
        props.put("session.timeout.ms", "30000");
        props.put("auto.offset.reset", "earliest");
        //props.put("key.deserializer", StringDeserializer.class.getName());
        //props.put("value.deserializer", StringDeserializer.class.getName());
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        this.consumer = new KafkaConsumer<String, String>(props);
        this.topic = topicName;
        this.consumer.subscribe(Arrays.asList(topic));
    }

    @Override
    public void run() {
        int messageNo = 1;
        System.out.println("---------开始消费---------");
        try {
        	System.out.println(123123);
        	for (int i = 0; i < 5; i++) {
                    msgList = consumer.poll(100);
                    System.out.println("msgList.count():"+msgList.count());
                    if(null!=msgList&&msgList.count()>0){
                    for (ConsumerRecord<String, String> record : msgList) {
                        System.out.println(messageNo+"=======receive: key = " + record.key() + ", value = " + record.value()+" offset==="+record.offset());
                        messageNo++;
                    }
                }else{  
                    Thread.sleep(1000);
                }
            }       
        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            consumer.close();
        }
    }  
}

可以看出低级消费这里不是消费的5条，而是消费的15条数据，为什么会这样呢？因为低级消费的offset存在kafka中，而高级消费的offset存在zk中，kafka中offset为0，所以从头开始消费。

下面我再生产5条信息，这里再消费。

可以看到这次消费就是接着上次的15开始消费了。

但是看下图

图中生产的消息logsize可以同步过来，消费的消息offset，低级消费的并没有同步过来，这是因为kafka低级消费不往zk里存数据。只有高级消费才会存入数据到zk的缘故。

下面补充一下导入的jar包

    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka -->
	<dependency>
	    <groupId>org.apache.kafka</groupId>
	    <artifactId>kafka_2.12</artifactId>
	    <version>1.1.0</version>
	</dependency>	
	<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
	<dependency>
	    <groupId>org.apache.kafka</groupId>
	    <artifactId>kafka-clients</artifactId>
	    <version>1.1.0</version>
	</dependency>

注意：低级消费者的api只支持1.8以上的jdk。