springboot打包成war，由于log4j2 starter使用不当而引发的memory leak排查解决

1.背景

公司有个springboot项目，需要打包成war发布到tomcat，无意间看了一验tomcat日志，发现在shutdown过程中有一些异常信息，如下：

27-Jun-2018 08:46:42.332 WARNING [mainApp.com-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Log4j2-TF-7-Scheduled-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
27-Jun-2018 08:46:42.334 WARNING [mainApp.com-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Log4j2-TF-4-AsyncLoggerConfig-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
 com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
 com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
 com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)

这个日志描述的还是比较清楚的，可以发现tomcat警告有两个log4j的线程stop失败了，他说似乎是内存泄露。

2.解决

看下项目的依赖：

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-tomcat</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- log4j -->

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-log4j2</artifactId>
        </dependency>

这边是排除了springboot的默认日志框架，依赖spring-boot-starter-log4j2使用log4j2，乍一看没啥问题。

参照下springboot官方文档的配置，确实是没啥问题的。

我们再看下log4j2的doc：

有一段很显眼的英文：

太多了，而且英文不是很好，我借助一下百度翻译：

当使用JavaEE Web应用程序中的Log4J或任何其他日志记录框架时，必须特别小心。当容器关闭或Web应用程序卸载时，对日志资源进行适当清理（数据库连接关闭、文件关闭等）是很重要的。由于Web应用程序中类装载器的性质，无法通过正常方式清理Log4J资源。当Web应用程序部署和“关闭”Web应用程序未卸载时，Log4J必须“启动”。这取决于应用程序是Servlet 3还是更新的或servlet 2.5 Web应用程序。

在这两种情况下，您需要将Log4J Web模块添加到部署中，如Maven、Ivy和Gradle工件手册页中详细说明的那样。

也就是说我们如果是一个web程序，则需要借助log4j-web这个依赖，这个jar中有一个ServletContainerInitializer的实现类：Log4jServletContainerInitializer.java。

先看下ServletContainerInitializer.java：

public interface ServletContainerInitializer {

    /**
     * Receives notification during startup of a web application of the classes
     * within the web application that matched the criteria defined via the
     * {@link javax.servlet.annotation.HandlesTypes} annotation.
     *
     * @param c     The (possibly null) set of classes that met the specified
     *              criteria
     * @param ctx   The ServletContext of the web application in which the
     *              classes were discovered
     *
     * @throws ServletException If an error occurs
     */
    void onStartup(Set<Class<?>> c, ServletContext ctx) throws ServletException;
}

这是Servlet3.0的新特性，servlet容器会在启动时扫描：

META-INF/services/javax.servlet.ServletContainerInitializer这个文件中指定的类(实现ServletContainerInitializer)，这个类将收到容器的onStartup事件。

然后我们看下这个文件中的内容：

打开Log4jServletContainerIntializer.java，

public class Log4jServletContainerInitializer implements ServletContainerInitializer {
    private static final Logger LOGGER = StatusLogger.getLogger();

    public Log4jServletContainerInitializer() {
    }

    public void onStartup(Set<Class<?>> classes, ServletContext servletContext) throws ServletException {
        if (servletContext.getMajorVersion() > 2 && servletContext.getEffectiveMajorVersion() > 2 && !"true".equalsIgnoreCase(servletContext.getInitParameter("isLog4jAutoInitializationDisabled"))) {
            LOGGER.debug("Log4jServletContainerInitializer starting up Log4j in Servlet 3.0+ environment.");
            Dynamic filter = servletContext.addFilter("log4jServletFilter", Log4jServletFilter.class);
            if (filter == null) {
                LOGGER.warn("WARNING: In a Servlet 3.0+ application, you should not define a log4jServletFilter in web.xml. Log4j 2 normally does this for you automatically. Log4j 2 web auto-initialization has been canceled.");
                return;
            }

            Log4jWebLifeCycle initializer = WebLoggerContextUtils.getWebLifeCycle(servletContext);
            initializer.start();
            initializer.setLoggerContext();
            servletContext.addListener(new Log4jServletContextListener());
            filter.setAsyncSupported(true);
            filter.addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), false, new String[]{"/*"});
        }

    }
}

这个onStartup中添加了一个监听器，再看下这个监听器：

代码比较多，就不贴出来了，这个监听器实现了ServletContextListener接口，然后再contextDestroyed方法中清理了log4j的相关资源，这也就解释了，网上说引用了log-web依赖后内存泄露可以解决。

最后，加入依赖-打包-调试-查看日志，tomcat（我这边用的是tomcat8）日志干净了，没有出现memory leak。

3.总结

记录了一次简单的内存泄漏排查和解决过程。

重点在于，解决问题的时候要稍微想一想工作原理。