Kafka实战

        从搭建Windows下的Kafka运行环境和搭建Linux下的Kafka运行环境可以看出，环境一旦搭建完成，执行以下逻辑程序是固定的顺序，与环境无关：

启动Zookeeper
启动Kafka
创建Topic
启动生产者Producer发消息
启动消费者Consumer消费消息

        因此，这里以Windows下的运行环境为例（windows环境搭建可以参见笔者之前的博客），展示Kafka在程序中的应用：

第一步：打开cmd，运行zookeeper

zkserver

第二步：进入kafka目录，启动kafka

.\bin\windows\kafka-server-start.bat .\config\server.properties  

===========================================================

【实现方案一：在程序中创建Topic】

（创建Maven工程，添加依赖）

 

<dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.10</artifactId>
            <version>0.8.2.2</version>
        </dependency>
 </dependencies>

（2）创建消息生产者Producer

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

import java.util.Properties;

public class MsgProducer {

    private static Producer<String, String> producer;
    private final Properties properties = new Properties();

    public MsgProducer(){
        //配置连接的Broker List
        properties.put("metadata.broker.list", "127.0.0.1:9092");
        //序列化
        properties.put("serializer.class", "kafka.serializer.StringEncoder");
        producer = new Producer<String, String>(new ProducerConfig(properties));
    }

    public static void main(String[] args){
        MsgProducer msgProducer = new MsgProducer();
        //定义topic
        String topic = "testKafka";
        //定义要发送的消息
        String msg = "2017.11.03，kafka测试";
        //构建消息对象
        KeyedMessage<String, String> data = new KeyedMessage<String, String>(topic, msg);
        //发送消息
        producer.send(data);
        producer.close();
    }

}

（3）创建消息消费者Consumer

 

import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;

public class MsgConsumer {

    private ConsumerConnector consumer;
    private String topic;

    public MsgConsumer(String zookeeper, String groupId, String topic){
        Properties properties = new Properties();
        //配置zookeeper信息
        properties.put("zookeeper.connect", zookeeper);
        //配置消费群组
        properties.put("group.id", groupId);
        //配置超时连接
        properties.put("zookeeper.session.timeout.ms", "500");
        //配置重连接时间间隔
        properties.put("auto.commit.interval.ms", "1000");
        consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(properties));
        this.topic = topic;
    }

    public void testConsumer(){
        Map<String, Integer> topicCount = new HashMap<String, Integer>();
        //定义订阅topic数量
        topicCount.put(topic, new Integer(1));
        //返回的是所有topic的Map
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerStreams = consumer.createMessageStreams(topicCount);
        //取出我们需要的topic中的消息流
        List<KafkaStream<byte[], byte[]>> streams = consumerStreams.get(topic);
        for (final KafkaStream stream : streams) {
            ConsumerIterator<byte[], byte[]> consumerIte = stream.iterator();
            while (consumerIte.hasNext()) {
                System.out.println(new String(consumerIte.next().message()));
            }
        }
        if (consumer != null) {
            consumer.shutdown();
        }
    }

    public static void main(String[] args){
        String topic = "testKafka";
        MsgConsumer msgConsumer = new MsgConsumer("127.0.0.1:2181", "Msg", topic);
        msgConsumer.testConsumer();
    }
    
}

启动消费者、然后启动消息生产者，程序运行结果如下：

Consumer

=============================================================

 

【实现方案二：转下面“第三步”，全部写到配置文件中，并启动多线程执行程序】

第三步：进入kafka文件目录D:\kafka_2.12-0.11.0.0\bin\windows，创建kafka的消息topics

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 5 --topic test20171103

 

第四步：程序实现Producer、Consumer，实现消息的生产和消费

 

（1）创建Maven工程，添加如下依赖：

<dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.10</artifactId>
            <version>0.8.2.2</version>
        </dependency>
 </dependencies>

（2）将相关的配置信息单独写在一个文件中KafkaConf.java中，便于维护和修改

public interface KafkaConf {
    String zookeeperConnect = "127.0.0.1:2181";
    String groupId = "group";
    String topic1 = "test20171103";
    String brokerList ="127.0.0.1:9092";
    String zkSessionTimeout = "20000";
    String zkSyncTime = "200";
    String reconnectIntervel = "1000";
}

（3）实现消息生产者Producer

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

import java.util.Properties;

public class KafkaProducer extends Thread{
    private Producer<Integer, String> producer;
    private String topic;

    public KafkaProducer(String topic){
        Properties properties = new Properties();
        properties.put("serializer.class", "kafka.serializer.StringEncoder");
        properties.put("metadata.broker.list", KafkaConf.brokerList);
        producer = new Producer<>(new ProducerConfig(properties));
        this.topic = topic;
    }

    @Override
    public void run(){
        int messageNo = 1;
        while(true){
            String message = new String("Message_" + messageNo);
            System.out.println("Send: " + message);
            producer.send(new KeyedMessage<>(topic, message));
            messageNo++;
            try{
                sleep(3000);
            }catch (InterruptedException e){
                e.printStackTrace();
            }
        }
    }
}

（4）实现消息消费者Consumer

import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;

public class KafkaConsumer extends Thread{
    private ConsumerConnector consumer;
    private String topic;

    public KafkaConsumer(String topic){
        Properties properties = new Properties();
        properties.put("zookeeper.connect", KafkaConf.zookeeperConnect);
        properties.put("group.id", KafkaConf.groupId);
        properties.put("zookeeper.session.timeout.ms", KafkaConf.zkSessionTimeout);
        properties.put("zookeeper.sync.time.ms", KafkaConf.zkSyncTime);
        properties.put("auto.commit.interval.ms", KafkaConf.reconnectIntervel);
        consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(properties));
        this.topic = topic;
    }

    @Override
    public void run(){
        Map<String, Integer> topicCountMap = new HashMap<>();
        topicCountMap.put(topic, new Integer(1));
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
        KafkaStream<byte[], byte[]> stream = consumerMap.get(topic).get(0);
        ConsumerIterator<byte[], byte[]> it = stream.iterator();
        while(it.hasNext()){
            System.out.println("Receive: " + new String(it.next().message()));
            try{
                sleep(1000);
            }catch (InterruptedException e){
                e.printStackTrace();
            }
        }
    }
}

（5）启动客户端程序，实现消费的生产与消费

public class Main {
    public static void main(String[] args){
        KafkaProducer producerThread = new KafkaProducer(KafkaConf.topic1);
        producerThread.start();

        KafkaConsumer consumerThread = new KafkaConsumer(KafkaConf.topic1);
        consumerThread.start();
    }
}

程序运行结果如下：

消息生产者消费一条“Message+编号”，消息消费者消费该条消息

参考资料：

1、https://cwiki.apache.org/confluence/display/KAFKA/Index

2、http://www.nohup.cc/article/195/

3、http://blog.csdn.net/honglei915/article/details/37563647