kafka 小栗子:
topic:
public static void main(String[] args) {
//创建topic
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.180.128:9092");
AdminClient adminClient = AdminClient.create(props);
ArrayList<NewTopic> topics = new ArrayList<NewTopic>();
NewTopic newTopic = new NewTopic("topic-test", 1, (short) 1);
topics.add(newTopic);
CreateTopicsResult result = adminClient.createTopics(topics);
try {
result.all().get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
生产者:
public static void main(String[] args){
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.180.128:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<String, String>(props);
for (int i = 0; i < 100; i++)
producer.send(new ProducerRecord<String, String>("topic-test", Integer.toString(i), Integer.toString(i)));
producer.close();
}
消费者:
public static void main(String[] args){
Properties props = new Properties();
props.put("bootstrap.servers", "192.168.12.65:9092");
props.put("group.id", "test");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
final KafkaConsumer<String, String> consumer = new KafkaConsumer<String,String>(props);
consumer.subscribe(Arrays.asList("topic-test"),new ConsumerRebalanceListener() {
public void onPartitionsRevoked(Collection<TopicPartition> collection) {
}
public void onPartitionsAssigned(Collection<TopicPartition> collection) {
//将偏移设置到最开始
consumer.seekToBeginning(collection);
}
});
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
}
作为一个MQ做基本的功能自然就是消息的生产和消费,本章以XML配置的方式实现消息的生产和消费。
生产者配置:
spring-kafka 提供了org.springframework.kafka.core.KafkaTemplate
xml配置如下,producerProperties中的具体配置暂时不用在意,后面有一章专门讲xml配置:
1 <?xml version="1.0" encoding="UTF-8"?>
2 <beans xmlns="http://www.springframework.org/schema/beans"
3 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
4 xsi:schemaLocation="http://www.springframework.org/schema/beans
5 http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
6 http://www.springframework.org/schema/context
7 http://www.springframework.org/schema/context/spring-context.xsd">
8 <context:property-placeholder location="classpath*:config/application.properties" />
9 <!-- 定义producer的参数 -->
10 <bean id="producerProperties" class="java.util.HashMap">
11 <constructor-arg>
12 <map>
13 <entry key="bootstrap.servers" value="${bootstrap.servers}" />
14 <entry key="group.id" value="${group.id}" />
15 <entry key="retries" value="${retries}" />
16 <entry key="batch.size" value="${batch.size}" />
17 <entry key="linger.ms" value="${linger.ms}" />
18 <entry key="buffer.memory" value="${buffer.memory}" />
20 <entry key="acks" value="${acks}" />
22 <entry key="key.serializer"
23 value="org.apache.kafka.common.serialization.StringSerializer" />
24 <entry key="value.serializer"
25 value="org.apache.kafka.common.serialization.StringSerializer" />
26 </map>
27 </constructor-arg>
28 </bean>
29
30 <!-- 创建kafkatemplate需要使用的producerfactory bean -->
31 <bean id="producerFactory"
32 class="org.springframework.kafka.core.DefaultKafkaProducerFactory">
33 <constructor-arg>
34 <ref bean="producerProperties" />
35 </constructor-arg>
36 </bean>
37
38 <!-- 创建kafkatemplate bean,使用的时候,只需要注入这个bean,即可使用template的send消息方法 -->
39 <bean id="kafkaTemplate" class="org.springframework.kafka.core.KafkaTemplate">
40 <constructor-arg ref="producerFactory" />
41 <constructor-arg name="autoFlush" value="true" />
42 <property name="defaultTopic" value="default" />
43 </bean>
46 </beans>
map也可以这样给具有集合性质的属性赋值
<!-- Map<String,Integer> -->
<property name="mymap">
<map>
<entry key="weight" value="80" /> <!-- key-value -->
<entry key="height" value="180" />
</map>
</property>
如上图,xml主要配置了KafkaTemplate的构造参数producerFactory和autoFlush,对应了一个KafkaTemplate源码中的2参构造函数。
producerProperties:设置生产者公产需要的配置
producerFactory:定义了生产者工厂构造方法
kafkaTemplate:定义了使用producerFactory和是否自动刷新,2个参数来构造kafka生产者模板类。
发送kafka消息:
1.根据topic、partition、key发送数据data。
2.接收ListenableFuture添加成功、失败回调函数。
ListenableFuture<SendResult<String, String>> listenableFuture = kafkaTemplate.send("topic", "partition","key","data");
//发送成功回调
SuccessCallback<SendResult<String, String>> successCallback = new SuccessCallback<SendResult<String, String>>() {
@Override
public void onSuccess(SendResult<String, String> result) {
//成功业务逻辑
}
}
//发送失败回调
FailureCallback failureCallback = new FailureCallback() {
@Override
public void onFailure(Throwable ex) {
//失败业务逻辑
}
}
listenableFuture.addCallback(successCallback, failureCallback);
配置消费者:
consumerProperties中的具体配置暂时不用在意,后面有一章专门讲xml配置
1.consumerProperties-》consumerFactory 载入配置构造消费者工厂
2.messageListener-》containerProperties 载入容器配置(topics)
3.consumerFactory+containerProperties-》messageListenerContainer 容器配置(topics)+消息监听器,构造一个并发消息监听容器,并执行初始化方法doStart
<!-- 1.定义consumer的参数 -->
<bean id="consumerProperties" class="java.util.HashMap">
<constructor-arg>
<map>
<entry key="bootstrap.servers" value="${bootstrap.servers}" />
<entry key="group.id" value="${group.id}" />
<entry key="enable.auto.commit" value="${enable.auto.commit}" />
<entry key="session.timeout.ms" value="${session.timeout.ms}" />
<entry key="key.deserializer"
value="org.apache.kafka.common.serialization.StringDeserializer" />
<entry key="value.deserializer"
value="org.apache.kafka.common.serialization.StringDeserializer" />
</map>
</constructor-arg>
</bean>
<!-- 2.创建consumerFactory bean -->
<bean id="consumerFactory"
class="org.springframework.kafka.core.DefaultKafkaConsumerFactory" >
<constructor-arg>
<ref bean="consumerProperties" />
</constructor-arg>
</bean>
<!-- 3.定义消费实现类 -->
<bean id="kafkaConsumerService" class="xxx.service.impl.KafkaConsumerSerivceImpl" />
<!-- 4.消费者容器配置信息 -->
<bean id="containerProperties" class="org.springframework.kafka.listener.config.ContainerProperties">
<!-- topic -->
<constructor-arg name="topics">
<list>
<value>${kafka.consumer.topic.credit.for.lease}</value>
<value>${loan.application.feedback.topic}</value>
<value>${templar.agreement.feedback.topic}</value>
<value>${templar.aggrement.active.feedback.topic}</value>
<value>${templar.aggrement.agreementRepaid.topic}</value>
<value>${templar.aggrement.agreementWithhold.topic}</value>
<value>${templar.aggrement.agreementRepayRemind.topic}</value>
</list>
</constructor-arg>
<property name="messageListener" ref="kafkaConsumerService" />
</bean>
<!-- 5.消费者并发消息监听容器,执行doStart()方法 -->
<bean id="messageListenerContainer" class="org.springframework.kafka.listener.ConcurrentMessageListenerContainer" init-method="doStart" >
<constructor-arg ref="consumerFactory" />
<constructor-arg ref="containerProperties" />
<property name="concurrency" value="${concurrency}" />
</bean>
消费kfaka消息:
方法1:
直接实现MessageListener接口,复写onMessage方法,实现自定义消费业务逻辑。
public class KafkaConsumerSerivceImpl implements MessageListener<String, String> {
@Override
public void onMessage(ConsumerRecord<String, String> data) {
//根据不同主题,消费
if("主题1".equals(data.topic())){
//逻辑1
}else if("主题2".equals(data.topic())){
//逻辑2
}
}
}
方法2:
使用@KafkaListener注解,并设置topic,支持SPEL表达式。这样方便拆分多个不同topic处理不同业务逻辑。(特别是有自己的事务的时候,尤其方便)
import org.springframework.kafka.annotation.KafkaListener;
public class KafkaConsumerSerivceImpl {
@KafkaListener(topics = "${templar.aggrement.agreementWithhold.topic}")
void templarAgreementNoticewithhold(ConsumerRecord<String, String> data){
//消费业务逻辑
}
}
三、总结
本章我们实现了一个简单的kafka生产、消费消息的实践。到这里我们已经会基本使用kafka了。是不是很简单…
下一章,我们从源码角度来深入分析spring-kafka。