Spring-Kafka（二）—— 快速入门

创建工程

一

二

三

这里需要注意一下，我们导入的Spring-Kafka为2.1.8版本，SpringBoot为2.0.4的正式版，请保持版本一致、

好了，已经三秒了，真男人，你可以关闭屏幕冷静一下了，停止你那颤抖的身体。

---

编写第一个Demo

实现顺序

1. 创建消费者和生产者的Map配置
2. 根据Map配置创建对应的消费者工厂(consumerFactory)和生产者工厂(producerFactory)
3. 根据consumerFactory创建监听器的监听器工厂
4. 根据producerFactory创建KafkaTemplate(Kafka操作类)
5. 创建监听容器

先给你们瞄一眼项目结构，记得把Kafka 启动...

项目结构

创建KafkaConfiguration配置类

都是一些配置参数，具体的作用也在代码中写明了，值得注意的是，KafkaTemplate的类型为<Integer,String>，我们可以找kafkaTemplate的send方法，有多个重载方法，其中有个方法如下，key和data参数都为泛型，这其实就是对应着KafkaTemplate<Integer,String>。那具体有什么用呢，还记得我们的Topic中可以包含多个Partition(分区)吗，那我们如果不想手动指定发送到哪个分区，我们则可以利用key去实现。这里我们的key是Integer类型，template会根据 key 路由到对应的partition中，如果key存在对应的partitionID则发送到该partition中，否则由算法选择发送到哪个partition。

    public ListenableFuture<SendResult<K, V>> send(String topic, K key, V data) {
        ProducerRecord<K, V> producerRecord = new ProducerRecord(topic, key, data);
        return this.doSend(producerRecord);
    }

@Configuration
@EnableKafka
public class KafkaConfiguration {
<span class="hljs-comment">//ConcurrentKafkaListenerContainerFactory为创建Kafka监听器的工程类，这里只配置了消费者</span>
<span class="hljs-meta">@Bean</span>
<span class="hljs-function"><span class="hljs-keyword">public</span> ConcurrentKafkaListenerContainerFactory&lt;Integer, String&gt; <span class="hljs-title">kafkaListenerContainerFactory</span><span class="hljs-params">()</span> </span>{
    ConcurrentKafkaListenerContainerFactory&lt;Integer, String&gt; factory = <span class="hljs-keyword">new</span> ConcurrentKafkaListenerContainerFactory&lt;&gt;();
    factory.setConsumerFactory(consumerFactory());
    <span class="hljs-keyword">return</span> factory;
}

<span class="hljs-comment">//根据consumerProps填写的参数创建消费者工厂</span>
<span class="hljs-meta">@Bean</span>
<span class="hljs-function"><span class="hljs-keyword">public</span> ConsumerFactory&lt;Integer, String&gt; <span class="hljs-title">consumerFactory</span><span class="hljs-params">()</span> </span>{
    <span class="hljs-keyword">return</span> <span class="hljs-keyword">new</span> DefaultKafkaConsumerFactory&lt;&gt;(consumerProps());
}

<span class="hljs-comment">//根据senderProps填写的参数创建生产者工厂</span>
<span class="hljs-meta">@Bean</span>
<span class="hljs-function"><span class="hljs-keyword">public</span> ProducerFactory&lt;Integer, String&gt; <span class="hljs-title">producerFactory</span><span class="hljs-params">()</span> </span>{
    <span class="hljs-keyword">return</span> <span class="hljs-keyword">new</span> DefaultKafkaProducerFactory&lt;&gt;(senderProps());
}

<span class="hljs-comment">//kafkaTemplate实现了Kafka发送接收等功能</span>
<span class="hljs-meta">@Bean</span>
<span class="hljs-function"><span class="hljs-keyword">public</span> KafkaTemplate&lt;Integer, String&gt; <span class="hljs-title">kafkaTemplate</span><span class="hljs-params">()</span> </span>{
    KafkaTemplate template = <span class="hljs-keyword">new</span> KafkaTemplate&lt;Integer, String&gt;(producerFactory());
    <span class="hljs-keyword">return</span> template;
}

<span class="hljs-comment">//消费者配置参数</span>
<span class="hljs-function"><span class="hljs-keyword">private</span> Map&lt;String, Object&gt; <span class="hljs-title">consumerProps</span><span class="hljs-params">()</span> </span>{
    Map&lt;String, Object&gt; props = <span class="hljs-keyword">new</span> HashMap&lt;&gt;();
    <span class="hljs-comment">//连接地址</span>
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, <span class="hljs-string">"localhost:9092"</span>);
    <span class="hljs-comment">//GroupID</span>
    props.put(ConsumerConfig.GROUP_ID_CONFIG, <span class="hljs-string">"bootKafka"</span>);
    <span class="hljs-comment">//是否自动提交</span>
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, <span class="hljs-keyword">true</span>);
    <span class="hljs-comment">//自动提交的频率</span>
    props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, <span class="hljs-string">"100"</span>);
    <span class="hljs-comment">//Session超时设置</span>
    props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, <span class="hljs-string">"15000"</span>);
    <span class="hljs-comment">//键的反序列化方式</span>
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
    <span class="hljs-comment">//值的反序列化方式</span>
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    <span class="hljs-keyword">return</span> props;
}

<span class="hljs-comment">//生产者配置</span>
<span class="hljs-function"><span class="hljs-keyword">private</span> Map&lt;String, Object&gt; <span class="hljs-title">senderProps</span> <span class="hljs-params">()</span></span>{
    Map&lt;String, Object&gt; props = <span class="hljs-keyword">new</span> HashMap&lt;&gt;();
    <span class="hljs-comment">//连接地址</span>
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, <span class="hljs-string">"localhost:9092"</span>);
    <span class="hljs-comment">//重试，0为不启用重试机制</span>
    props.put(ProducerConfig.RETRIES_CONFIG, <span class="hljs-number">1</span>);
    <span class="hljs-comment">//控制批处理大小，单位为字节</span>
    props.put(ProducerConfig.BATCH_SIZE_CONFIG, <span class="hljs-number">16384</span>);
    <span class="hljs-comment">//批量发送，延迟为1毫秒，启用该功能能有效减少生产者发送消息次数，从而提高并发量</span>
    props.put(ProducerConfig.LINGER_MS_CONFIG, <span class="hljs-number">1</span>);
    <span class="hljs-comment">//生产者可以使用的总内存字节来缓冲等待发送到服务器的记录</span>
    props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, <span class="hljs-number">1024000</span>);
    <span class="hljs-comment">//键的序列化方式</span>
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
    <span class="hljs-comment">//值的序列化方式</span>
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
    <span class="hljs-keyword">return</span> props;
}

}

创建DemoListener消费者

这里的消费者其实就是一个监听类，指定监听名为topic.quick.demo的Topic，consumerID为demo。

@Component
public class DemoListener {
<span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">final</span> Logger log= LoggerFactory.getLogger(DemoListener.class);

<span class="hljs-comment">//声明consumerID为demo，监听topicName为topic.quick.demo的Topic</span>
<span class="hljs-meta">@KafkaListener</span>(id = <span class="hljs-string">"demo"</span>, topics = <span class="hljs-string">"topic.quick.demo"</span>)
<span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">listen</span><span class="hljs-params">(String msgData)</span> </span>{
    log.info(<span class="hljs-string">"demo receive : "</span>+msgData);
}

}

创建测试类

这里的send方法第一参数为TopicName，第二个参数则是发送的数据

@SpringBootTest
@RunWith(SpringRunner.class)
public class DemoTest {
<span class="hljs-meta">@Autowired</span>
<span class="hljs-keyword">private</span> KafkaTemplate kafkaTemplate;

<span class="hljs-meta">@Test</span>
<span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">testDemo</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> InterruptedException </span>{
    kafkaTemplate.send(<span class="hljs-string">"topic.quick.demo"</span>, <span class="hljs-string">"this is my first demo"</span>);
    <span class="hljs-comment">//休眠5秒，为了使监听器有足够的时间监听到topic的数据</span>
    Thread.sleep(<span class="hljs-number">5000</span>);
}

}

接下来直接运行这个测试方法，我们可以看到日志中输出了我们发送的消息，这就代表我们成功的消费了测试方法中发送的消息。

2018-09-06 17:26:20.850  INFO 6232 --- [     demo-0-C-1] com.viu.kafka.listen.DemoListener        : demo receive : this is my first demo

启动项目

看清楚了是启动项目，不是测试类,我们来观察一下控制台的输出日志

首先这个是KafkaConsumer的配置信息，每个消费者都会输出该配置信息，配置太多就不做讲解了

2018-09-06 17:40:15.258  INFO 9944 --- [           main] o.a.k.clients.consumer.ConsumerConfig    : ConsumerConfig values: 
    auto.commit.interval.ms = 100
    auto.offset.reset = latest
    bootstrap.servers = [localhost:9092]
    check.crcs = true
    client.id = 
    connections.max.idle.ms = 540000
    enable.auto.commit = true
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = demo
    heartbeat.interval.ms = 3000
    interceptor.classes = null
    internal.leave.group.on.close = true
    isolation.level = read_uncommitted
    key.deserializer = class org.apache.kafka.common.serialization.IntegerDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 500
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 305000
    retry.backoff.ms = 100
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.mechanism = GSSAPI
    security.protocol = PLAINTEXT
    send.buffer.bytes = 131072
    session.timeout.ms = 15000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

2018-09-06 17:40:15.274 INFO 9944 — [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version : 1.0.2
2018-09-06 17:40:15.274 INFO 9944 — [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId : 2a121f7b1d402825

这些日志就代表我们成功的创建了Consumer，由于没有做并发配置，所以现在为单个消费者模式，系统会做一个分配Partition的操作，也就是将某个Partition指定给某个消费者消费。 这里有个地方需要注意一下，
看到日志中有输出[Consumer clientId=consumer-1, groupId=demo]，我们之前在监听中@KafkaListener注解中配置的id=demo，怎么就变成了groupId=demo，这是因为@KafkaListener注解如果没有指定groupId这个属性的值，则会默认把id作为groupId。

2018-09-06 17:40:15.287  INFO 9944 --- [     demo-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=demo] Discovered group coordinator admin-PC:9092 (id: 2147483647 rack: null)
2018-09-06 17:40:15.290  INFO 9944 --- [     demo-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, groupId=demo] Revoking previously assigned partitions []
2018-09-06 17:40:15.290  INFO 9944 --- [     demo-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: []
2018-09-06 17:40:15.290  INFO 9944 --- [     demo-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=demo] (Re-)joining group
2018-09-06 17:40:15.301  INFO 9944 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2018-09-06 17:40:15.302  INFO 9944 --- [     demo-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=demo] Successfully joined group with generation 33
2018-09-06 17:40:15.303  INFO 9944 --- [     demo-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, groupId=demo] Setting newly assigned partitions [topic.quick.demo-0]

---

结束

SpringBoot2.0已经提供了Kafka的自动配置，可以在application.properties文件中配置，别问我为什么要写一堆代码来创建这些工厂，相对于properties方式我更喜欢java Config方法创建这些配置，因为很直观，虽然是有点麻烦。