1、依赖和配置(基于springboot2.1.0)
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
#kafka producer and comsumer 基本配置,这些配置会注入给KafkaProperties这个配置bean中,供#spring自动配置kafkaTemplate这个对象时使用
spring.kafka.producer.bootstrap-servers=192.168.199.128:9092,192.168.199.128:9093,192.168.199.128:9094
spring.kafka.consumer.bootstrap-servers=192.168.199.128:9092,192.168.199.128:9093,192.168.199.128:9094
spring.kafka.consumer.group-id=test-group
2、生产者and消费者
@Component
public class Producer {
@Autowired
private KafkaTemplate kafkaTemplate;
private AtomicInteger counter= new AtomicInteger();
@Scheduled(cron = "0/3 * * * * ? ")
public void send() {
ListenableFuture future = kafkaTemplate.send("test", "第" + counter.getAndIncrement() + "条消息");
//ListenableFuture future = kafkaTemplate.send("test","xxx", "第" + counter.getAndIncrement() + "条消息");
//ListenableFuture future = kafkaTemplate.send("test", 2, "xxx", "第" + counter.getAndIncrement() + "条消息");
future.addCallback(o -> {}, err -> System.out.println("发送失败,"+err.getCause()));
}
}
@Component
public class Consumer {
@KafkaListener(topics = "test")
public void getMsg(String msg){
System.out.println(msg);
}
}
3.消息分区路由源码
ListenableFuture future = kafkaTemplate.send("test", "第" + a + "条消息");---->最终进入默认分区器DefaultPartitioner的int partion(..)方法。如果没有传消息的key则进入图中标签A
标签B:如果没有消息的key,会去根据topic取该一个ConcurrentHashMap类型的计数器counter(key为topic,value为一个AtomoicInteger)并自增返回1。然后标签C处根据该值与可用分区数(这里是3,3个borker默认0、1、2分区 )取余决定将消息发往指定topic的哪个分区。这种算法会沦陷往3个分区发送消息,负载比较均衡。
如果传入的发送消息传入了消息的key(ListenableFuture future = kafkaTemplate.send("test","xxx", "第" + counter.getAndIncrement() + "条消息"),则根据该key计算发往哪个分区,始终会往所有可用分区中的同一个分区发送。如果ListenableFuture future = kafkaTemplate.send("test", 2, "xxx", "第" + counter.getAndIncrement() + "条消息")指定了发往哪个分区,则不会进入分区器计算分区路由直接发送到该topic的指定分区2。