故障描述
业务线程经常超时,排查时mysql、CPU和GC均无异常。
- 排查结果:
项目采用异步方式输出日志:
上图标记,dbAppender是将日志存储到mysql。项目在高并发请求下,产生大量日志,但是dbAppender是写到mysql,所以日志输出较慢。虽然采用异步输出日志,但是日志输出慢,导致任务积压,队列满了,默认策略是产生日志业务线程同步等待队列释放后,放入日志任务,所以导致业务线程耗时在输出日志处,出现超时现象。
源码分析
AsyncLoggerConfigDisruptor
@Override
public synchronized void start() {
if (disruptor != null) {
LOGGER.trace("AsyncLoggerConfigDisruptor not starting new disruptor for this configuration, "
+ "using existing object.");
return;
}
LOGGER.trace("AsyncLoggerConfigDisruptor creating new disruptor for this configuration.");
ringBufferSize = DisruptorUtil.calculateRingBufferSize("AsyncLoggerConfig.RingBufferSize");
final WaitStrategy waitStrategy = DisruptorUtil.createWaitStrategy("AsyncLoggerConfig.WaitStrategy");
final ThreadFactory threadFactory = new Log4jThreadFactory("AsyncLoggerConfig", true, Thread.NORM_PRIORITY) {
@Override
public Thread newThread(final Runnable r) {
final Thread result = super.newThread(r);
backgroundThreadId = result.getId();
return result;
}
};
//创建任务队列满的策略
asyncQueueFullPolicy = AsyncQueueFullPolicyFactory.create();
translator = mutable ? MUTABLE_TRANSLATOR : TRANSLATOR;
factory = mutable ? MUTABLE_FACTORY : FACTORY;
disruptor = new Disruptor<>(factory, ringBufferSize, threadFactory, ProducerType.MULTI, waitStrategy);
final ExceptionHandler<Log4jEventWrapper> errorHandler = DisruptorUtil.getAsyncLoggerConfigExceptionHandler();
disruptor.setDefaultExceptionHandler(errorHandler);
final Log4jEventWrapperHandler[] handlers = {new Log4jEventWrapperHandler()};
disruptor.handleEventsWith(handlers);
LOGGER.debug("Starting AsyncLoggerConfig disruptor for this configuration with ringbufferSize={}, "
+ "waitStrategy={}, exceptionHandler={}...", disruptor.getRingBuffer().getBufferSize(), waitStrategy
.getClass().getSimpleName(), errorHandler);
disruptor.start();
super.start();
}
查看asyncQueueFullPolicy = AsyncQueueFullPolicyFactory.create();这行代码:
进入DefaultAsyncQueueFullPolicy:
可以看到是同步策略,业务线程会在此等待。
解决方式
1.提升日志输出速度,或者减少日志量。我这里将mysql替换成es存储了。
2.修改拒绝策略,改为丢弃。采用该方式,会导致日志丢失情况出现,我未采用。如下图:
如何配置: