SpringBoot笔记：SpringBoot2.3集成Kafka组件配置

最新推荐文章于 2024-07-04 18:16:36 发布

leo825...

最新推荐文章于 2024-07-04 18:16:36 发布

阅读量4.2k

点赞数 3

分类专栏： Kafka笔记 SpringBoot

本文链接：https://blog.csdn.net/u011047968/article/details/112411847

版权

SpringBoot 同时被 2 个专栏收录

19 篇文章 4 订阅

订阅专栏

Kafka笔记

3 篇文章 0 订阅

订阅专栏

文章目录

说明
Springboot集成Kafka
注意事项

说明

本文是接《Kafka学习：CentOS7下Kafka集群搭建》后续，搭建完成kafka集群之后进行项目的集成。

Springboot集成Kafka

依赖配置

首先，在springboot中引入kafka的依赖jar包

		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka</artifactId>
		</dependency>

配置文件yml配置

配置文件使用的是yml文件，其中配置如下所示：

# 端口，项目上下文
server:
  port: 8080
  servlet:
    context-path: /kafka-demo

# kafka集群配置
spring:
  kafka:
    bootstrap-servers: hadoop-slave1:9092,hadoop-slave2:9092,hadoop-slave3:9092
    producer:
      # 发生错误后，消息重发的次数。
      retries: 0
      #当有多个消息需要被发送到同一个分区时，生产者会把它们放在同一个批次里。该参数指定了一个批次可以使用的内存大小，按照字节数计算。
      batch-size: 16384
      # 设置生产者内存缓冲区的大小。
      buffer-memory: 33554432
      # 键的序列化方式
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      # 值的序列化方式
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      # acks=0 ： 生产者在成功写入消息之前不会等待任何来自服务器的响应。
      # acks=1 ： 只要集群的首领节点收到消息，生产者就会收到一个来自服务器成功响应。
      # acks=all ：只有当所有参与复制的节点全部收到消息时，生产者才会收到一个来自服务器的成功响应。
      acks: 1
    consumer:
      # 设置kafka的消费者组
      group-id: mygroup1
      # 自动提交的时间间隔 在spring boot 2.X 版本中这里采用的是值的类型为Duration 需要符合特定的格式，如1S,1M,2H,5D
      auto-commit-interval: 1S
      # 该属性指定了消费者在读取一个没有偏移量的分区或者偏移量无效的情况下该作何处理：
      # latest（默认值）在偏移量无效的情况下，消费者将从最新的记录开始读取数据（在消费者启动之后生成的记录）
      # earliest ：在偏移量无效的情况下，消费者将从起始位置读取分区的记录
      auto-offset-reset: earliest
      # 是否自动提交偏移量，默认值是true,为了避免出现重复数据和数据丢失，可以把它设置为false,然后手动提交偏移量
      enable-auto-commit: false
      # 键的反序列化方式
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      # 值的反序列化方式
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

    listener:
      # 在侦听器容器中运行的线程数。
      concurrency: 5
      #listner负责ack，每调用一次，就立即commit
      ack-mode: manual_immediate
      # 如果Broker上不存在至少一个配置的主题（topic），则容器是否无法启动，
      # 该设置项结合Broker设置项allow.auto.create.topics=true，如果为false，则会自动创建不存在的topic
      missing-topics-fatal: false

# 日志输出配置
logging:
  level:
    root: INFO
    org:
      springframework:
        security: WARN
        web: ERROR
  file:
    path: ./logs
    name: './logs/kafka-demo.log'
  pattern:
    file: '%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50}:%L - %msg%n'
    console: '%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50}:%L - %msg%n'

Producer生产者

生产者代码如下所示：

package com.demo.kafka;

import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;
import org.springframework.stereotype.Component;
import org.springframework.util.concurrent.ListenableFuture;
import org.springframework.util.concurrent.ListenableFutureCallback;

/**
 * 模拟kafka生产者
 */
@Component
@Slf4j
public class KafkaProducer {
    @Autowired
    private KafkaTemplate<String, Object> kafkaTemplate;
    //自定义topic
    public static final String TOPIC_TEST = "mytopic";
    //消费者组mygroup1
    public static final String TOPIC_GROUP1 = "mygroup1";
    //消费者组mygroup2
    public static final String TOPIC_GROUP2 = "mygroup2";

    public void send(Object obj) {
        String obj2String = JSON.toJSONString(obj);
        log.info("准备发送消息为：{}", obj2String);
        //发送消息
        ListenableFuture<SendResult<String, Object>> future = kafkaTemplate.send(TOPIC_TEST, obj);
        future.addCallback(new ListenableFutureCallback<SendResult<String, Object>>() {
            @Override
            public void onFailure(Throwable throwable) {
                //发送失败的处理
                log.info(TOPIC_TEST + " - 生产者 发送消息失败：" + throwable.getMessage());
            }

            @Override
            public void onSuccess(SendResult<String, Object> stringObjectSendResult) {
                //成功的处理
                log.info(TOPIC_TEST + " - 生产者 发送消息成功：" + stringObjectSendResult.toString());
            }
        });
    }
}

Consumer消费者

消费者代码如下：

package com.demo.kafka;

import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Component;

import java.util.Optional;

/**
 * 模拟Consumer消费消息
 */
@Component
@Slf4j
public class KafkaConsumer {
    @KafkaListener(topics = KafkaProducer.TOPIC_TEST, groupId = KafkaProducer.TOPIC_GROUP1)
    public void topicTest(ConsumerRecord<?, ?> record, Acknowledgment ack, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
        Optional<Object> message = Optional.ofNullable(record.value());
        if (message.isPresent()) {
            Object msg = message.get();
            log.info("topic_test 消费了： Topic:" + topic + ",Message:" + msg);
            ack.acknowledge();
        }
    }

    @KafkaListener(topics = KafkaProducer.TOPIC_TEST, groupId = KafkaProducer.TOPIC_GROUP2)
    public void topicTest1(ConsumerRecord<?, ?> record, Acknowledgment ack, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
        Optional<Object> message = Optional.ofNullable(record.value());
        if (message.isPresent()) {
            Object msg = message.get();
            log.info("topic_test1 消费了： Topic:" + topic + ",Message:" + msg);
            ack.acknowledge();
        }
    }
}

测试代码

使用一个简单controller测试kafka发送和接收消息。

package com.demo.controller;

import com.demo.kafka.KafkaProducer;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * 测试kafka发送和接收消息
 */
@Slf4j
@RestController
@RequestMapping("/kafka")
public class KafkaController {

    @Autowired
    private KafkaProducer kafkaProducer;

    @GetMapping("/send")
    @Transactional(rollbackFor = Exception.class)
    public void send() {
        kafkaProducer.send("this is a test kafka topic message");
    }
}

测试结果如下：

2021-01-09 22:39:50.341 [http-nio-8080-exec-1] INFO  o.a.c.c.C.[Tomcat].[localhost].[/kafka-demo]:173 - Initializing Spring DispatcherServlet 'dispatcherServlet'
2021-01-09 22:39:50.395 [http-nio-8080-exec-1] INFO  com.demo.kafka.KafkaProducer:30 - 准备发送消息为："this is a test kafka topic message"
2021-01-09 22:39:50.399 [http-nio-8080-exec-1] INFO  org.apache.kafka.clients.producer.ProducerConfig:347 - ProducerConfig values: 
	acks = 1
	batch.size = 16384
	bootstrap.servers = [hadoop-slave1:9092, hadoop-slave2:9092, hadoop-slave3:9092]
	buffer.memory = 33554432
	client.dns.lookup = default
	client.id = producer-1
	compression.type = none
	connections.max.idle.ms = 540000
	delivery.timeout.ms = 120000
	enable.idempotence = false
	interceptor.classes = []
	key.serializer = class org.apache.kafka.common.serialization.StringSerializer
	linger.ms = 0
	max.block.ms = 60000
	max.in.flight.requests.per.connection = 5
	max.request.size = 1048576
	metadata.max.age.ms = 300000
	metadata.max.idle.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
	receive.buffer.bytes = 32768
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retries = 0
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	security.providers = null
	send.buffer.bytes = 131072
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.2
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	transaction.timeout.ms = 60000
	transactional.id = null
	value.serializer = class org.apache.kafka.common.serialization.StringSerializer

2021-01-09 22:39:50.409 [http-nio-8080-exec-1] INFO  org.apache.kafka.common.utils.AppInfoParser:117 - Kafka version: 2.5.0
2021-01-09 22:39:50.410 [http-nio-8080-exec-1] INFO  org.apache.kafka.common.utils.AppInfoParser:118 - Kafka commitId: 66563e712b0b9f84
2021-01-09 22:39:50.410 [http-nio-8080-exec-1] INFO  org.apache.kafka.common.utils.AppInfoParser:119 - Kafka startTimeMs: 1610203190409
2021-01-09 22:39:50.418 [kafka-producer-network-thread | producer-1] INFO  org.apache.kafka.clients.Metadata:280 - [Producer clientId=producer-1] Cluster ID: DmchEHdzTi2ARIhv5RBBVQ
2021-01-09 22:39:50.443 [kafka-producer-network-thread | producer-1] INFO  com.demo.kafka.KafkaProducer:43 - mytopic - 生产者 发送消息成功：SendResult [producerRecord=ProducerRecord(topic=mytopic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value=this is a test kafka topic message, timestamp=null), recordMetadata=mytopic-1@6]
2021-01-09 22:39:50.454 [org.springframework.kafka.KafkaListenerEndpointContainer#1-0-C-1] INFO  com.demo.kafka.KafkaConsumer:24 - topic_test 消费了： Topic:mytopic,Message:this is a test kafka topic message
2021-01-09 22:39:50.454 [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] INFO  com.demo.kafka.KafkaConsumer:34 - topic_test1 消费了： Topic:mytopic,Message:this is a test kafka topic message

注意事项

需要注意的是，本文虚拟机kafka集群使用的都是域名，如果同样是本地进行测试需要在host文件中添加域名和ip的映射。
hosts文件地址：

C:\Windows\System32\drivers\etc

例如，我这里添加的域名和ip的映射关系如下：

192.168.223.128 hadoop-slave1
192.168.223.129 hadoop-slave2
192.168.223.130 hadoop-slave3
192.168.223.131 hadoop-master

leo825...

关注

3
点赞
踩
6

收藏

觉得还不错? 一键收藏
打赏
0
评论
SpringBoot笔记：SpringBoot2.3集成Kafka组件配置

文章目录说明Springboot集成Kafka依赖配置配置文件yml配置Producer生产者Consumer消费者测试代码说明本文是接《Kafka学习：CentOS7下Kafka集群搭建》后续，搭建完成kafka集群之后进行项目的集成。Springboot集成Kafka依赖配置首先，在springboot中引入kafka的依赖jar包 <dependency> <groupId>org.springframework.kafka</groupId>
复制链接

扫一扫