kafka 并发
springboot setConcurrency 决定kafka 并发,kafka 的分区决定了并发上限 ,concurrency<=分区数 ,如果大于分区数并发会等于分区
每个分区都是一个有序的、不可变的记录序列,这些记录不断附加到结构化提交日志。分区中的记录被分配给一个顺序 ID 号,称为唯一标识分区内每个记录的偏移量。
KafkaBootstrapConfiguration
@SuppressWarnings(“rawtypes”)
@Bean(name = KafkaListenerConfigUtils.KAFKA_LISTENER_ANNOTATION_PROCESSOR_BEAN_NAME)
@Role(BeanDefinition.ROLE_INFRASTRUCTURE)
public KafkaListenerAnnotationBeanPostProcessor kafkaListenerAnnotationProcessor() {
return new KafkaListenerAnnotationBeanPostProcessor();
}
@Bean(name = KafkaListenerConfigUtils.KAFKA_LISTENER_ENDPOINT_REGISTRY_BEAN_NAME)
public KafkaListenerEndpointRegistry defaultKafkaListenerEndpointRegistry() {
return new KafkaListenerEndpointRegistry();
}
ConcurrentKafkaListenerContainerFactory
ConcurrentMessageListenerContainer 会进行创建 3 个数 KafkaMessageListenerContainer(多个)
KafkaMessageListenerContainer(多个)
for (int i = 0; i < this.concurrency; i++) {
KafkaMessageListenerContainer<K, V> container;
if (topicPartitions == null) {
container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties);
}
else {
container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties,
partitionSubset(containerProperties, i));
}
if (getBeanName() != null) {
container.setBeanName(getBeanName() + "-" + i);
}
if (getApplicationEventPublisher() != null) {
container.setApplicationEventPublisher(getApplicationEventPublisher());
}
container.setClientIdSuffix("-" + i);
container.start();
this.containers.add(container);
}
KafkaMessageListenerContainer(多个)中包含 ListenerConsumer
SimpleAsyncTaskExecutor( CustomizableThreadCreator AsyncListenableTaskExecutor TaskExecutor) 用于启动执行consumer
如何动态设置并发
@GetMapping("/stop")
public String stop(){
KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry=
context.getBean(KafkaListenerConfigUtils.KAFKA_LISTENER_ENDPOINT_REGISTRY_BEAN_NAME,
KafkaListenerEndpointRegistry.class);
Collection<MessageListenerContainer> containers= kafkaListenerEndpointRegistry.getListenerContainers();
ConcurrentMessageListenerContainer concurrentMessageListenerContainer
=(ConcurrentMessageListenerContainer)containers.iterator().next();
concurrentMessageListenerContainer.stop();
return "";
}
@GetMapping("/start")
public String start(int num){
KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry=
context.getBean(KafkaListenerConfigUtils.KAFKA_LISTENER_ENDPOINT_REGISTRY_BEAN_NAME,
KafkaListenerEndpointRegistry.class);
Collection<MessageListenerContainer> containers= kafkaListenerEndpointRegistry.getListenerContainers();
ConcurrentMessageListenerContainer concurrentMessageListenerContainer
=(ConcurrentMessageListenerContainer)containers.iterator().next();
concurrentMessageListenerContainer.setConcurrency(num);
concurrentMessageListenerContainer.start();
return "";
}
如上,通过获取到KafkaListenerEndpointRegistry 然后可以获取到对应的管理的 containers ,通过重启ConcurrentMessageListenerContainer,重新创建MessageListenerContainer数量达到动态修改业务并发,但是注意同一组的topic将发生重平衡。