文章标题

Spring RabbitListener 监听 RabbitMQ异常问题排查

问题现象：
1、程序启动后可以消费堆积的消息。
2、运行业务逻辑后，出现如下错误日志，同时rabbitMQ监控界面 consume退出。
先贴上异常日志

2017-03-23 13:13:11,345:INFO SimpleAsyncTaskExecutor-1 (PublishMsgServiceHandler.java:55) - publishMsgServiceHandler handle:publish
2017-03-23 13:13:11,345:INFO SimpleAsyncTaskExecutor-1 (CoreMessageCenterService.java:94) - publish message
2017-03-23 13:13:11,347:INFO SimpleAsyncTaskExecutor-1 (CoreMessageCenterService.java:124) - mc publish message
2017-03-23 13:13:11,348:ERROR SimpleAsyncTaskExecutor-1 (SimpleMessageListenerContainer.java:1470) - Consumer thread error, thread abort.
java.lang.NoSuchMethodError: com.unisound.iot.busi.msg.dubbo.facade.PushClientFacade.pushClients(Ljava/lang/String;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;Ljava/lang/Integer;JLjava/lang/String;)Lcom/unisound/iot/busi/msg/dubbo/model/ReturnResultVO;
    at audioPlatform.unisound.iot.mc.adapter.service.CoreMessageCenterService.pushClient(CoreMessageCenterService.java:322) ~[classes/:?]
    at audioPlatform.unisound.iot.mc.adapter.service.CoreMessageCenterService.doOperation(CoreMessageCenterService.java:148) ~[classes/:?]
    at audioPlatform.unisound.logicArchitecture.core.impl.sh.PublishMsgServiceHandler.processServiceMessage(PublishMsgServiceHandler.java:58) ~[classes/:?]
    at audioPlatform.unisound.logicArchitecture.core.impl.Dispatcher.innerDispatchLogicSyn(Dispatcher.java:242) ~[classes/:?]
    at audioPlatform.unisound.logicArchitecture.core.impl.Dispatcher.dispatchSyn(Dispatcher.java:277) ~[classes/:?]
    at audioPlatform.unisound.routerClient.impl.DispatcherRouteClient.callHandlerSyn(DispatcherRouteClient.java:156) ~[classes/:?]
    at audioPlatform.unisound.util.CommonUtil.sendMCMessage(CommonUtil.java:122) ~[classes/:?]
    at audioPlatform.unisound.util.CommonUtil.sendMCMessageUsingUserId(CommonUtil.java:139) ~[classes/:?]
    at audioPlatform.unisound.service.athenaService.DeviceOnlineRelatedActionsService.appOnlineCheck(DeviceOnlineRelatedActionsService.java:132) ~[classes/:?]
    at audioPlatform.unisound.service.athenaService.DeviceOnlineRelatedActionsService.doRelatedAction(DeviceOnlineRelatedActionsService.java:149) ~[classes/:?]
    at audioPlatform.unisound.athenaService.logicArchitecture.core.impl.longConnMessageHandler.OnLineReportServiceHandler.processServiceMessage(OnLineReportServiceHandler.java:73) ~[classes/:?]
    at audioPlatform.unisound.athenaService.logicArchitecture.core.impl.LongConnControllerServiceHandler.processServiceMessage(LongConnControllerServiceHandler.java:61) ~[classes/:?]
    at audioPlatform.unisound.logicArchitecture.core.impl.Dispatcher.innerDispatchLogicSyn(Dispatcher.java:242) ~[classes/:?]
    at audioPlatform.unisound.logicArchitecture.core.impl.Dispatcher.dispatchSyn(Dispatcher.java:277) ~[classes/:?]
    at audioPlatform.unisound.routerClient.impl.DispatcherRouteClient.callHandlerSyn(DispatcherRouteClient.java:156) ~[classes/:?]
    at audioPlatform.unisound.iot.mc.adapter.service.CoreMessageCenterService.dealWithOnline(CoreMessageCenterService.java:223) ~[classes/:?]
    at audioPlatform.unisound.iot.mc.adapter.mq.service.RabbitMQListenerService.handleEvent(RabbitMQListenerService.java:68) ~[classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-messaging-4.3.0.RELEASE.jar:4.3.0.RELEASE]
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:115) ~[spring-messaging-4.3.0.RELEASE.jar:4.3.0.RELEASE]
    at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:779) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:702) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:95) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:186) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1227) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:683) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1181) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1165) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:95) ~[spring-rabbit-1.7.0.RELEASE.jar:?]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1367) [spring-rabbit-1.7.0.RELEASE.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
2017-03-23 13:13:11,351:ERROR SimpleAsyncTaskExecutor-1 (SimpleMessageListenerContainer.java:1502) - Stopping container from aborted consumer
2017-03-23 13:13:11,352:INFO SimpleAsyncTaskExecutor-1 (SimpleMessageListenerContainer.java:855) - Waiting for workers to finish.
2017-03-23 13:13:11,352:INFO SimpleAsyncTaskExecutor-1 (SimpleMessageListenerContainer.java:858) - Successfully waited for workers to finish.

问题分析：
1、抛出的异常是NoSuchMethodError，出现错误的位置是调用dubbo服务的时候出现的。
2、看到 SimpleMessageListenerContainer Consumer thread error, thread abort.

解决办法：
1、先查看rabbitMQ server的日志，并未看到连接断开。
2、多次测试后，每次都是因为dubbo接口调用错误才出现监听异常。
3、观察错误日志发现MQ监听线程 和 业务处理线程在同一个 SimpleAsyncTaskExecutor中。
4、怀疑是由于SimpleAsyncTaskExecutor中出现未捕获异常导致监听退出。
5、为SimpleMessageListenerContainer单独指定TaskExecutor

@Bean
    public ThreadPoolTaskExecutor taskExecutor(){
        ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
        taskExecutor.setCorePoolSize(5);
        taskExecutor.setMaxPoolSize(10);
        taskExecutor.setKeepAliveSeconds(30000);
        taskExecutor.initialize();
        return taskExecutor;
    }
    @Bean
    public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory() {
        SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
        factory.setConnectionFactory(connectionFactory());
        factory.setTaskExecutor(taskExecutor());
        return factory;
    }

问题解决！