1. 常用配置
生产者
配置 | 描述 | 类型 | 默认值 |
---|---|---|---|
bootstrap.servers | 用于建立与kafka集群的连接,仅影响用于初始化的hosts,来发现全部的servers。 格式:host1:port1,host2:port2,…,数量尽量不止一个,以防其中一个down了。 | list | |
acks | 生产者要求Leader在确认请求完成之前已收到的确认数。 acks = 0:如果设置为零,那么生产者将完全不等待服务器的任何确认。 该记录将立即添加到套接字缓冲区中并视为已发送。 在这种情况下,不能保证服务器已收到记录,并且重试配置不会生效(因为客户端通常不会知道任何故障)。 为每个记录提供的偏移量将始终设置为-1。 acks = 1:这将意味着Leader会将记录写入其本地日志,但会在不等待所有Followers的完全确认的情况下做出响应。 在这种情况下,如果领导者在确认记录后立即失败,但在追随者复制记录之前失败,则记录将丢失。 acks=all 或 acks=-1:这意味着Leader将等待完整的同步副本集确认记录。 这保证了只要至少一个同步副本仍处于活动状态,记录就不会丢失。 这是最有力的保证。 | string | 1 |
request.timeout.ms | 客户端等待请求响应的最长时间。如果超时之前仍未收到响应,则客户端将在必要时重新发送请求,如果重试已用尽,则会使请求失败。 | int | 30000 |
retries | 请求失败时重试次数。 | int | 2147483647 |
batch.size | 批次大小,批次就是一组消息,这些消息属于同一个主题和分区,把消息分成批次传输可以减少网络开销。 | int | 163834 |
linger.ms | 发送时延,即发送消息时,生产者等待给定的延迟,才允许发送,以便可以聚合更多的消息。 | long | 0 |
buffer.memory | 生产者可以用来缓冲等待发送到服务器的记录的总内存字节,即生产者所管理的最大内存。 如果记录的发送速度超过了将记录发送到服务器的速度,则生产者将阻塞max.block.ms,此后它将引发异常。 | long | 33554432 |
key.serializer | 消息key序列化方式 | class | |
value.serializer | 消息本身的序列化方式 | class |
消费者
配置 | 描述 | 类型 | 默认值 |
---|---|---|---|
bootstrap.servers | 用于建立与kafka集群的连接,仅影响用于初始化的hosts,来发现全部的servers。 格式:host1:port1,host2:port2,…,数量尽量不止一个,以防其中一个down了。 | list | |
group.id | 消费者所属消费组的唯一标识 | string | null |
enable.auto.commit | 如果为true,则消费者的offset将在后台定期提交。 | boolean | true |
auto.commit.interval.ms | 如果将enable.auto.commit设置为true,则消费者offset自动提交给Kafka的频率(以毫秒为单位)。 | int | 5000 |
session.timeout.ms | 检测消费者是否失效的超时时间,该值必须在group.min.session.timeout.ms和group.max.session.timeout.ms允许范围内。 | int | 10000 |
group.min.session.timeout.ms | 检测消费者是否失效的超时最小时间。 | int | 6000 |
group.max.session.timeout.ms | 检测消费者是否失效的超时最大时间。 | int | 1800000 |
key.deserializer | 消息key反序列化方式 | class | |
value.deserializer | 消息value反序列化方式 | class |
2. kafka-clients
添加依赖
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>2.5.0</version>
</dependency>
2.1 生产者
同步发送消息
KafkaProducer是线程安全的,可以在多个线程之间共享生产者实例。
@Test
public void syncProducer() {
Properties properties = new Properties();
properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
properties.put("acks", "all");
properties.put("retries", 0);
properties.put("batch.size", 16384);
properties.put("buffer.memory", 33554432);
properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);
// 同步发送消息,发送10条
for (int i = 1; i <= 10; i++) {
kafkaProducer.send(new ProducerRecord<>("java-client-test", "test,test,test " + i));
}
try{
// 同步发送消息,可获取RecordMetadata,服务器已确认记录的元数据
Future<RecordMetadata> future = kafkaProducer.send(new ProducerRecord<>("java-client-test", "test,test,test"));
RecordMetadata recordMetadata = future.get();
System.out.printf("RecordMetadata : topic = %s, partition = %d, offset = %d, toString() = %s\n",
recordMetadata.topic(),recordMetadata.partition(),recordMetadata.offset(),recordMetadata.toString());
} catch(Exception e) {
// 连接错误、No Leader错误都可以通过重试解决
// 消息太大这类错误kafkaProducer不会进行任何重试,直接抛出异常
e.printStackTrace();
}
kafkaProducer.close();
}
异步发送消息
发送消息时,传递一个回调对象
@Test
public void producer() {
Properties properties = new Properties();
properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
properties.put("acks", "all");
properties.put("retries", 0);
properties.put("batch.size", 16384);
properties.put("buffer.memory", 33554432);
properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);
// 异步发送消息,发送10条
for (int i = 1; i <= 10; i++) {
// 发送消息时,传递一个回调对象
kafkaProducer.send(new ProducerRecord<>("java-client-test", "test,test,test " + i), new Callback() {
@Override
public void onCompletion(RecordMetadata recordMetadata, Exception e) {
System.out.printf("ProducerCallback RecordMetadata : topic = %s, partition = %d, offset = %d, toString() = %s\n",
recordMetadata.topic(),recordMetadata.partition(),recordMetadata.offset(),recordMetadata.toString());
// 如果Kafka返回一个错误,onCompletion方法抛出一个non null异常。
if (e != null) {
// 对异常进行一些处理,这里只是简单打印出来
e.printStackTrace();
}
}
});
}
kafkaProducer.close();
}
2.2 消费者
KafkaConsumer 是线程不安全的。
@Test
public void consumer() {
Properties properties = new Properties();
properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
properties.put("group.id", "test");
properties.put("enable.auto.commit", "true");
properties.put("auto.commit.interval.ms", "1000");
properties.put("session.timeout.ms", "30000");
properties.put("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
properties.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);
consumer.subscribe(Arrays.asList("java-client-test"));
try{
while (true) {
// 间隔100ms去拉取数据
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records){
System.out.printf("消费 : topic = %s, partition = %d, offset = %d, value = %s\n",
record.topic(),record.partition(),record.offset(), record.value());
}
}
}finally {
consumer.close();
}
}
2.3 多线程
生产者
将生产者定义为一个工具类
public class BaseProducer {
private static KafkaProducer<String, String> kafkaProducer;
static {
Properties properties = new Properties();
properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
properties.put("acks", "all");
properties.put("retries", 0);
properties.put("batch.size", 16384);
properties.put("buffer.memory", 33554432);
properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
kafkaProducer = new KafkaProducer<>(properties);
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
BaseProducer.kafkaProducer.close();
}
});
}
public static void send(String content, String... topics){
if (topics != null && topics.length > 0) {
for (int i = 0;i < topics.length; i++) {
System.out.println("send " + topics[i] + " message " + content);
kafkaProducer.send(new ProducerRecord<>(topics[i], content));
}
}
}
}
多线程消费者
实现Runnable,定义一个抽象的消费者,每个线程里面都有一个KafkaConsumer。
public abstract class AbstractConsumer implements Runnable {
protected KafkaConsumer<String, String> consumer;
protected String topic;
public AbstractConsumer(String topic) {
this.topic = topic;
}
/**
* 创建消费者,订阅topic
*/
private void connect() {
Properties properties = new Properties();
properties.put("bootstrap.servers", "192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094");
properties.put("group.id", "test-user");
properties.put("enable.auto.commit", "false");
properties.put("session.timeout.ms", "30000");
properties.put("key.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
properties.put("value.deserializer","org.apache.kafka.common.serialization.StringDeserializer");
consumer = new KafkaConsumer<>(properties);
consumer.subscribe(Arrays.asList("topic-user"));
}
@Override
public void run() {
this.connect();
while(true) {
try {
ConsumerRecords<String, String> consumerRecords = this.consumer.poll(Duration.ofMillis(500));
Iterator iterator = consumerRecords.iterator();
while(iterator.hasNext()) {
ConsumerRecord<String, String> item = (ConsumerRecord)iterator.next();
this.handler(item.value());
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
public void handler(String message) {
this.consumer.commitAsync();
}
}
继承AbstractConsumer,实现handler()方法,在里面进行消息的消费。
public class UserConsumer extends AbstractConsumer {
public UserConsumer(String topic) {
super(topic);
}
@Override
public void handler(String message) {
try {
// 消费数据
List<User> userList = JSON.parseObject(message, new TypeReference<List<User>>(){});
if (!userList.isEmpty()) {
for (User user : userList) {
System.out.println(user);
}
}
this.consumer.commitAsync();
}catch (Exception e) {
this.consumer.commitAsync();
e.printStackTrace();
}
}
}
消费者组,定义线程池
public class UserConsumerGroup {
private List<Runnable> consumerThreadList = new ArrayList<>();
public UserConsumerGroup() {
consumerThreadList.add(new UserConsumer("topic-user"));
}
public void start() {
ThreadFactory factory = r -> new Thread(r, "ThreadPool thread: UserConsumerGroup");
// 核心线程池大小;最大线程池大小;线程最大空闲时间;时间单位;线程等待队列;线程创建工厂
ThreadPoolExecutor executor = new ThreadPoolExecutor(3, 6, 200, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(3), factory);
for (Runnable item : consumerThreadList) {
executor.execute(item);
}
executor.shutdown();
}
}
3. spring-kafka
SpringBoot 2.3.0,spring-kafka 2.5.0
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
application.yml
spring:
kafka:
bootstrap-servers: 192.168.110.40:9092,192.168.110.40:9093,192.168.110.40:9094
producer:
acks: all
retries: 0
batch-size: 16384
buffer-memory: 33554432
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
consumer:
group-id: spring-test
enable-auto-commit: true
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
properties:
auto.commit.interval.ms: 1000
session.timeout.ms: 30000
生产者
@Component
public class Producer {
@Autowired
private KafkaTemplate<String, String> kafkaTemplate;
/**
* 同步
* @param topic
*/
public void syncSend(String topic){
for (int i = 1; i <= 10; i++) {
ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send(topic, "test,test,test " + i);
try {
SendResult<String, String> sendResult = future.get();
ProducerRecord<String, String> producerRecord = sendResult.getProducerRecord();
System.out.printf("Sync ProducerRecord : topic = %s, partition = %d, toString() = %s\n",
producerRecord.topic(),producerRecord.partition(),producerRecord.toString());
RecordMetadata recordMetadata = sendResult.getRecordMetadata();
System.out.printf("Sync RecordMetadata : topic = %s, partition = %d, offset = %d, toString() = %s\n",
recordMetadata.topic(),recordMetadata.partition(),recordMetadata.offset(),recordMetadata.toString());
}catch (Exception e) {
e.printStackTrace();
}
}
}
/**
* 异步
* @param topic
*/
public void asyncSend(String topic){
for (int i = 1; i <= 10; i++) {
ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send(topic, "test,test,test " + i);
future.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
@Override
public void onFailure(Throwable ex) {
ex.printStackTrace();
}
@Override
public void onSuccess(SendResult<String, String> result) {
ProducerRecord<String, String> producerRecord = result.getProducerRecord();
System.out.printf("Async ProducerRecord : topic = %s, partition = %d, toString() = %s\n",
producerRecord.topic(),producerRecord.partition(),producerRecord.toString());
RecordMetadata recordMetadata = result.getRecordMetadata();
System.out.printf("Async RecordMetadata : topic = %s, partition = %d, offset = %d, toString() = %s\n",
recordMetadata.topic(),recordMetadata.partition(),recordMetadata.offset(),recordMetadata.toString());
}
});
}
}
}
消费者,这个并发3,启动3个线程
@Component
public class Consumer {
@KafkaListener(topics = "spring-topic", concurrency = "3")
public void listen (ConsumerRecord<?, ?> record) {
System.out.printf("消费 : topic = %s, partition = %d, offset = %d, value = %s\n",
record.topic(),record.partition(),record.offset(),record.value());
}
}
参考:
Kafka 官方文档
kafka-clients介绍
kafka-clients API文档 KafkaConsumer
创建多线程Kafka消费者
spring-kafka 官方文档