linux服务器上部署的kafka版本是:kafka_2.11-0.9.0.1.tgz
kafka的安装请自行百度。
在此讲述的是spring整合后的集成,并非原生api操作的集成
pom依赖
<!-- kafka start -->
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>1.1.1.RELEASE</version>
</dependency>
<!-- kafka end -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.2</version>
</dependency>
springboot配置文件
##### kafka配置start #####
# 指定kafka 代理地址,可以多个
spring.kafka.bootstrap-servers=192.168.106.239:9092,192.168.106.240:9092,192.168.106.241:9092
#发送失败重试次数
spring.kafka.producer.retries=0
# 每次批量发送消息的数量
spring.kafka.producer.batch-size=16384
#32MB的批处理缓冲区
spring.kafka.producer.buffer-memory=33554432
#消费者群组ID,发布-订阅模式,即如果一个生产者,多个消费者都要消费,那么需要定义自己的群组,同一群组内的消费者只有一个能消费到消息
spring.kafka.consumer.group-id=test-consumer-group
#关闭自动提交offset,spring会手工提交offset
spring.kafka.consumer.enable-auto-commit=false
#key-value序列化反序列化
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
##### kafka配置end #####
熟悉kafka的同学都知道,kafka在消费数据时,需要维护offset(偏移量)
一种是kafka自动提交,可以在配置中自行配置,在这里不讲述
一种是手动提交,就是每消费一条或一批数据,人为的去提交一次,在spring没集成kafka之前,我们使用commitSync()方法提交,集成之后,只需要将自动提交配置设置为false即可。
#关闭自动提交offset,spring会手工提交offset
spring.kafka.consumer.enable-auto-commit=false
User类:
package com.ireport.demo.entity;
import lombok.Data;
import lombok.experimental.Accessors;
import java.io.Serializable;
@Data
@Accessors(chain = true)
public class User implements Serializable {
private int id;
private String username;
private int age;
private int status;
}
Producer类:
package com.ireport.demo.kafka;
import com.google.gson.Gson;
import com.ireport.demo.entity.User;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;
import java.text.SimpleDateFormat;
import java.util.Date;
@Component
@Slf4j
public class Producer {
@Autowired
private KafkaTemplate kafkaTemplate;
public void sendMsg(){
while(true){
SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmsss");
Gson gson = new Gson();
int x=(int)(Math.random()*100);
User user = new User();
user.setId(x);
user.setAge(12);
user.setUsername("张三_"+sdf.format(new Date()));
user.setStatus(1);
String msg = gson.toJson(user);
kafkaTemplate.send("test", msg);
log.info("----------------- sendMsg:" + msg);
try {
Thread.sleep(1000);
}catch (Exception e){
e.printStackTrace();
}
}
}
public static void main(String[] args) {
new Producer().sendMsg();
}
}
其中,send方法为生产者向kafka集群发送数据,第一个参数是topic,第二个参数是数据,topic可以不存在,会自动创建
Consumer类:
package com.ireport.demo.kafka;
import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
@Slf4j
public class Consumer {
//注解监听
@KafkaListener(topics = {"test"})
public void consumerMsg(ConsumerRecord<?, ?> record){
log.info("【消费{}主题, 第{}分区, 数据:{}, 偏移量:{}】", record.topic(), record.partition(), record.value(), record.offset());
}
}
@KafkaListener(topics = {"test"})这个注解是kafka监听topic入口,topic可以是多个。此注解在项目启动时会自动监听