记一次MQ发送失败的错误

症状：某个MQ发送失败

错误日志：

15:00:00.888 [HSFBizProcessor-DEFAULT-6-thread-23] ERROR com.bosideng.bsdyun.msg.service.MqServiceImpl - com.aliyun.openservices.ons.api.exception.ONSClientException: Connect broker failed, Topic=VIP_JOIN_ACTIVITY_PRD, msgId=null

15:00:07.265 [HSFBizProcessor-DEFAULT-6-thread-50] WARN io.netty.channel.AbstractChannel - Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xd2b4893d]
java.util.concurrent.RejectedExecutionException: event executor terminated

排查 /home/admin/logs/rocketmq_client.log 发现出现过oom：

可以使用 jstat -gcutils pid 1000 查看gc情况，pid需换成业务程序的进程id

查看gc.lg 出现多次full gc

 定位应用日志发现在14:11出现oom

重点排查业务系统的内存溢出问题，排查中... 

Exception in thread "RebalanceService" java.lang.OutOfMemoryError: Exception From Host:[10.101.3.101]GC overhead limit exceeded14:11:25.197 [HSFBizProcessor-DEFAULT-6-thread-49] ERROR com.bosideng.bsdyun.msg.service.MsInterfaceMessageServiceImpl - org.springframework.jdbc.UncategorizedSQLException: Error attempting to get column 'xml_content' from result set.  Cause: java.sql.SQLException: Error
; uncategorized SQLException for SQL []; SQL state [null]; error code [0]; Error; nested exception is java.sql.SQLException: Error