spring-kafka消费线程加载过程分析

因为最近遇到spring-kafka消费线程中断消费的问题，所以看了它消费线程的启动过程，来总结一下。

Lifecycle和SmartLifecycle

要想知道它的加载过程，首先要先了解这两个接口：
Lifecycle是Spring中最基础的生命周期接口，该接口定义了容器启动和停止的方法。

SmartLifecycle是对Lifecycle的一个扩展接口，相比Lifecycle增加以下功能：

1. 无需容器显示调用start()方法，就可以回调SmartLifecycle接口的start()
2. 容器中如果有多个SmartLifecycle实例，可以方便控制调用顺序。

spring-kafka版本

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>1.3.5.RELEASE</version>
</dependency>

启动过程分析

spring加载完所有bean(实例化+初始化)后，要刷新容器org.springframework.context.support.AbstractApplicationContext#finishRefresh

protected void finishRefresh() {
		// Initialize lifecycle processor for this context.
		initLifecycleProcessor();

		// Propagate refresh to lifecycle processor first.
		getLifecycleProcessor().onRefresh();

		// Publish the final event.
		publishEvent(new ContextRefreshedEvent(this));

		// Participate in LiveBeansView MBean, if active.
		LiveBeansView.registerApplicationContext(this);
	}

首先是默认的Lifecycle处理器DefaultLifecycleProcessor
第二步：getLifecycleProcessor().onRefresh();

所有实现了Lifecycle的bean都被start了，即调用以下方法
org.springframework.context.support.DefaultLifecycleProcessor#startBeans

private void startBeans(boolean autoStartupOnly) {
		Map<String, Lifecycle> lifecycleBeans = getLifecycleBeans();
		Map<Integer, LifecycleGroup> phases = new HashMap<Integer, LifecycleGroup>();
		for (Map.Entry<String, ? extends Lifecycle> entry : lifecycleBeans.entrySet()) {
			Lifecycle bean = entry.getValue();
			if (!autoStartupOnly || (bean instanceof SmartLifecycle && ((SmartLifecycle) bean).isAutoStartup())) {
				int phase = getPhase(bean);
				LifecycleGroup group = phases.get(phase);
				if (group == null) {
					group = new LifecycleGroup(phase, this.timeoutPerShutdownPhase, lifecycleBeans, autoStartupOnly);
					phases.put(phase, group);
				}
				group.add(entry.getKey(), bean);
			}
		}
		if (!phases.isEmpty()) {
			List<Integer> keys = new ArrayList<Integer>(phases.keySet());
			Collections.sort(keys);
			for (Integer key : keys) {
				phases.get(key).start();
			}
		}
	}

按顺序phase排序，值越小越靠前，最后遍历调start方法：

		public void start() {
			if (this.members.isEmpty()) {
				return;
			}
			if (logger.isInfoEnabled()) {
				logger.info("Starting beans in phase " + this.phase);
			}
			Collections.sort(this.members);
			for (LifecycleGroupMember member : this.members) {
				if (this.lifecycleBeans.containsKey(member.name)) {
					doStart(this.lifecycleBeans, member.name, this.autoStartupOnly);
				}
			}
		}

最终会进到

private void doStart(Map<String, ? extends Lifecycle> lifecycleBeans, String beanName, boolean autoStartupOnly) {
		Lifecycle bean = lifecycleBeans.remove(beanName);
		if (bean != null && bean != this) {
			String[] dependenciesForBean = this.beanFactory.getDependenciesForBean(beanName);
			for (String dependency : dependenciesForBean) {
				doStart(lifecycleBeans, dependency, autoStartupOnly);
			}
			if (!bean.isRunning() &&
					(!autoStartupOnly || !(bean instanceof SmartLifecycle) || ((SmartLifecycle) bean).isAutoStartup())) {
				if (logger.isDebugEnabled()) {
					logger.debug("Starting bean '" + beanName + "' of type [" + bean.getClass() + "]");
				}
				try {
					bean.start();
				}
				catch (Throwable ex) {
					throw new ApplicationContextException("Failed to start bean '" + beanName + "'", ex);
				}
				if (logger.isDebugEnabled()) {
					logger.debug("Successfully started bean '" + beanName + "'");
				}
			}
		}
	}

就看bean.start()这句，其中spring-kafka中一个关键类：KafkaListenerEndpointRegistry就实现了SmartLifecycle，所以会进到它里面去，好，我们进到它的start方法：

@Override
	public void start() {
		for (MessageListenerContainer listenerContainer : getListenerContainers()) {
			startIfNecessary(listenerContainer);
		}
	}

getListenerContainers()这个是啥东西，我这里打印出来了

它是ConcurrentMessageListenerContainer这个对象，像是启动消费者的类（先透露一下，这里对应我们代码中使用**@KafkaListener**注解的次数，也就是说某方法如果使用此注解，就给你生成一个这样的对象），那它是什么时候放进来的，带着这个疑问，先看是哪里对它赋值的，找到赋值给它的代码位置：

public void registerListenerContainer(KafkaListenerEndpoint endpoint, KafkaListenerContainerFactory<?> factory,
			boolean startImmediately) {
		Assert.notNull(endpoint, "Endpoint must not be null");
		Assert.notNull(factory, "Factory must not be null");

		String id = endpoint.getId();
		Assert.hasText(id, "Endpoint id must not be empty");
		synchronized (this.listenerContainers) {
			Assert.state(!this.listenerContainers.containsKey(id),
					"Another endpoint is already registered with id '" + id + "'");
			MessageListenerContainer container = createListenerContainer(endpoint, factory);
			this.listenerContainers.put(id, container);
	// 省略部分代码

好，那又是哪里调了这个registerListenerContainer方法呢，继续往前推导，发现在类KafkaListenerEndpointRegistrar中分别有registerEndpoint方法和registerAllEndpoints方法都调到了它，那究竟是哪个调了呢？我们先看第一个registerEndpoint方法

public void registerEndpoint(KafkaListenerEndpoint endpoint, KafkaListenerContainerFactory<?> factory) {
		Assert.notNull(endpoint, "Endpoint must be set");
		Assert.hasText(endpoint.getId(), "Endpoint id must be set");
		// Factory may be null, we defer the resolution right before actually creating the container
		KafkaListenerEndpointDescriptor descriptor = new KafkaListenerEndpointDescriptor(endpoint, factory);
		synchronized (this.endpointDescriptors) {
			if (this.startImmediately) { // Register and start immediately
				this.endpointRegistry.registerListenerContainer(descriptor.endpoint,
						resolveContainerFactory(descriptor), true);
			}
			else {
				this.endpointDescriptors.add(descriptor);
			}
		}
	}

startImmediately这个刚刚开始是false的，所以应该不是它，那么就剩下registerAllEndpoints方法了。我们看到registerAllEndpoints里面需要拿this.endpointDescriptors这个集合，所以肯定要事先写入它，

而KafkaListenerEndpointRegistrar#registerEndpoint正好就是干了这件事，因为if (this.startImmediately) 不满足，所以先写到了this.endpointDescriptors.add(descriptor);
当调用registerAllEndpoints方法就能拿的到。

那么又是谁调了
KafkaListenerEndpointRegistrar#registerEndpoint ？
发现其中有一个类KafkaListenerAnnotationBeanPostProcessor，看这个名字就知道它是一个BeanPostProcessor，也就是Bean后处理器，它是在bean初始化阶段调用的，我们进去它里面的方法看看

public Object postProcessAfterInitialization(final Object bean, final String beanName) throws BeansException {
		if (!this.nonAnnotatedClasses.contains(bean.getClass())) {
			Class<?> targetClass = AopUtils.getTargetClass(bean);
			Collection<KafkaListener> classLevelListeners = findListenerAnnotations(targetClass);
			final boolean hasClassLevelListeners = classLevelListeners.size() > 0;
			final List<Method> multiMethods = new ArrayList<Method>();
			Map<Method, Set<KafkaListener>> annotatedMethods = MethodIntrospector.selectMethods(targetClass,
					new MethodIntrospector.MetadataLookup<Set<KafkaListener>>() {

						@Override
						public Set<KafkaListener> inspect(Method method) {
							Set<KafkaListener> listenerMethods = findListenerAnnotations(method);
							return (!listenerMethods.isEmpty() ? listenerMethods : null);
						}

					});
			if (hasClassLevelListeners) {
				Set<Method> methodsWithHandler = MethodIntrospector.selectMethods(targetClass,
						new MethodFilter() {

							@Override
							public boolean matches(Method method) {
								return AnnotationUtils.findAnnotation(method, KafkaHandler.class) != null;
							}

						});
				multiMethods.addAll(methodsWithHandler);
			}
			if (annotatedMethods.isEmpty()) {
				this.nonAnnotatedClasses.add(bean.getClass());
				if (this.logger.isTraceEnabled()) {
					this.logger.trace("No @KafkaListener annotations found on bean type: " + bean.getClass());
				}
			}
			else {
				// Non-empty set of methods
				for (Map.Entry<Method, Set<KafkaListener>> entry : annotatedMethods.entrySet()) {
					Method method = entry.getKey();
					for (KafkaListener listener : entry.getValue()) {
						processKafkaListener(listener, method, bean, beanName);
					}
				}
				if (this.logger.isDebugEnabled()) {
					this.logger.debug(annotatedMethods.size() + " @KafkaListener methods processed on bean '"
							+ beanName + "': " + annotatedMethods);
				}
			}
			if (hasClassLevelListeners) {
				processMultiMethodListeners(classLevelListeners, multiMethods, bean, beanName);
			}
		}
		return bean;
	}

这里有点长，在这里看到了熟悉的注解@KafkaListener，没错，我们平时在方法上加@KafkaListener注解要发挥作用也就是靠它，这段逻辑就是解析当前bean加了这个注解的所有方法，接进去这句：
processKafkaListener(listener, method, bean, beanName);

protected void processListener(MethodKafkaListenerEndpoint<?, ?> endpoint, KafkaListener kafkaListener, Object bean,
			Object adminTarget, String beanName) {
		endpoint.setBean(bean);
		endpoint.setMessageHandlerMethodFactory(this.messageHandlerMethodFactory);
		endpoint.setId(getEndpointId(kafkaListener));
		endpoint.setGroupId(getEndpointGroupId(kafkaListener, endpoint.getId()));
		endpoint.setTopicPartitions(resolveTopicPartitions(kafkaListener));
		endpoint.setTopics(resolveTopics(kafkaListener));
		endpoint.setTopicPattern(resolvePattern(kafkaListener));
		String group = kafkaListener.containerGroup();
		// 省略部分代码

		this.registrar.registerEndpoint(endpoint, factory);
	}

先解析一遍注解方法的属性，生成对象，最后就调到了上面留下来的疑问，谁调了this.registrar.registerEndpoint(endpoint, factory)方法，好，到此整条链路就串起来了。

我们接着最开始的位置继续分析，也就是：KafkaListenerEndpointRegistry#start

@Override
	public void start() {
		for (MessageListenerContainer listenerContainer : getListenerContainers()) {
			startIfNecessary(listenerContainer);
		}
	}

经过上面一轮分析后，知道listenerContainer它是ConcurrentMessageListenerContainer，所以最终会进到它里面的start（）：

@Override
	public final void start() {
		synchronized (this.lifecycleMonitor) {
			if (!isRunning()) {
				Assert.isTrue(
						this.containerProperties.getMessageListener() instanceof KafkaDataListener,
						"A " + GenericMessageListener.class.getName() + " implementation must be provided");
				doStart();
			}
		}
	}

doStart()发现它是一个抽象方法，ConcurrentMessageListenerContainer实现了它，进去看

@Override
	protected void doStart() {
		if (!isRunning()) {
			ContainerProperties containerProperties = getContainerProperties();
			TopicPartitionInitialOffset[] topicPartitions = containerProperties.getTopicPartitions();
			if (topicPartitions != null
					&& this.concurrency > topicPartitions.length) {
				this.logger.warn("When specific partitions are provided, the concurrency must be less than or "
						+ "equal to the number of partitions; reduced from " + this.concurrency + " to "
						+ topicPartitions.length);
				this.concurrency = topicPartitions.length;
			}
			setRunning(true);

			for (int i = 0; i < this.concurrency; i++) {
				KafkaMessageListenerContainer<K, V> container;
				if (topicPartitions == null) {
					container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties);
				}
				else {
					container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties,
							partitionSubset(containerProperties, i));
				}
				String beanName = getBeanName();
				container.setBeanName((beanName != null ? beanName : "consumer") + "-" + i);
				if (getApplicationEventPublisher() != null) {
					container.setApplicationEventPublisher(getApplicationEventPublisher());
				}
				container.setClientIdSuffix("-" + i);
				container.start();
				this.containers.add(container);
			}
		}
	}

很简单，就是按并发度concurrency创建KafkaMessageListenerContainer对象，这个跟ConcurrentMessageListenerContainer一样也是实现了MessageListenerContainer接口，
看到container.start()没有，又是调用了start方法，这回是KafkaMessageListenerContainer的start，跟进去，最终会进入到doStart方法，如下

@Override
	protected void doStart() {
		if (isRunning()) {
			return;
		}
		ContainerProperties containerProperties = getContainerProperties();
		if (!this.consumerFactory.isAutoCommit()) {
			AckMode ackMode = containerProperties.getAckMode();
			if (ackMode.equals(AckMode.COUNT) || ackMode.equals(AckMode.COUNT_TIME)) {
				Assert.state(containerProperties.getAckCount() > 0, "'ackCount' must be > 0");
			}
			if ((ackMode.equals(AckMode.TIME) || ackMode.equals(AckMode.COUNT_TIME))
					&& containerProperties.getAckTime() == 0) {
				containerProperties.setAckTime(5000);
			}
		}

		Object messageListener = containerProperties.getMessageListener();
		Assert.state(messageListener != null, "A MessageListener is required");
		if (messageListener instanceof GenericAcknowledgingMessageListener) {
			this.acknowledgingMessageListener = (GenericAcknowledgingMessageListener<?>) messageListener;
		}
		else if (messageListener instanceof GenericMessageListener) {
			this.listener = (GenericMessageListener<?>) messageListener;
		}
		else {
			throw new IllegalStateException("messageListener must be 'MessageListener' "
					+ "or 'AcknowledgingMessageListener', not " + messageListener.getClass().getName());
		}
		if (containerProperties.getConsumerTaskExecutor() == null) {
			SimpleAsyncTaskExecutor consumerExecutor = new SimpleAsyncTaskExecutor(
					(getBeanName() == null ? "" : getBeanName()) + "-C-");
			containerProperties.setConsumerTaskExecutor(consumerExecutor);
		}
		this.listenerConsumer = new ListenerConsumer(this.listener, this.acknowledgingMessageListener);
		setRunning(true);
		this.listenerConsumerFuture = containerProperties
				.getConsumerTaskExecutor()
				.submitListenable(this.listenerConsumer);
	}

创建了ListenerConsumer对象，这个正是监听消费线程对象，并且它是一个Runnable， 接着
this.listenerConsumerFuture = containerProperties
.getConsumerTaskExecutor()
.submitListenable(this.listenerConsumer);
}
这里submitListenable提交任务，getConsumerTaskExecutor默认是SimpleAsyncTaskExecutor将ListenerConsumer包装成ListenableFutureTask，它是FutureTask，这点很重要啊！

上面这段简单的理解就是提交Task任务，然后开启一个线程去处理它，最终到run方法，因为是FutureTask所以就会进入FutureTask的run方法。

因为ListenableFutureTask包装进去的Runnable是ListenerConsumer，所以最后执行的run方法肯定就是它里面的，也就是会走到org.springframework.kafka.listener.KafkaMessageListenerContainer.ListenerConsumer#run

@Override
		public void run() {
			this.consumerThread = Thread.currentThread();
			if (this.theListener instanceof ConsumerSeekAware) {
				((ConsumerSeekAware) this.theListener).registerSeekCallback(this);
			}
			if (this.transactionManager != null) {
				ProducerFactoryUtils.setConsumerGroupId(this.consumerGroupId);
			}
			this.count = 0;
			this.last = System.currentTimeMillis();
			if (isRunning() && this.definedPartitions != null) {
				initPartitionsIfNeeded();
			}
			long lastReceive = System.currentTimeMillis();
			long lastAlertAt = lastReceive;
			while (isRunning()) {
				try {
					if (!this.autoCommit && !this.isRecordAck) {
						processCommits();
					}
					processSeeks();
					ConsumerRecords<K, V> records = this.consumer.poll(this.containerProperties.getPollTimeout());
					this.lastPoll = System.currentTimeMillis();

					if (records != null && this.logger.isDebugEnabled()) {
						this.logger.debug("Received: " + records.count() + " records");
					}
					if (records != null && records.count() > 0) {
		           // 省略部分代码

ConsumerRecords<K, V> records = this.consumer.poll(this.containerProperties.getPollTimeout());
这句就是去kafka服务端拉取消息，并且发现它是开了while(1)这样方式保证线程不退出，循环去拉取消息。

总结如下：

refresh方法加载完所有bean后，开始完成刷新（finishRefresh），Lifecycle的后处理器DefaultLifecycleProcessor对所有Lifecycle实现类，按phase顺序（越小越优先）执行其start方法，开始进入spring-kafka模块的KafkaListenerEndpointRegistry#start，遍历listenerContainers集合（集合里面是一系列ConcurrentMessageListenerContainer对象），添加到集合的时机是KafkaListenerAnnotationBeanPostProcessor在初始化阶段执行postProcessAfterInitialization方法完成的，为什么会调此方法，熟悉bean创建过程的应该知道，在bean初始化完成后会调BeanPostProcessor的postProcessAfterInitialization方法。

遍历listenerContainers集合，跟着会进入ConcurrentMessageListenerContainer#doStart，在这里创建了KafkaMessageListenerContainer，平时注解@KafkaListener配置的concurrency并发度，也是在这用到，接着会调KafkaMessageListenerContainer#doStart，生成真正干活的线程任务ListenerConsumer，它实现了Runnable，接着将其包装成ListenableFutureTask(FutureTask)，最后提交任务，创建线程Thread，最终执行ListenerConsumer的run方法，循环去kafka服务拉取消息回来消费，while(true)保证线程不退出。