Tomcat无法shutdown进程问题解决办法

1 问题现象

执行shutdown.sh之后，Web已经无法访问：

Tomcat关联java进程仍然存活：

#/home/apache-tomcat-7.0.34/bin/shutdown.sh
Using CATALINA_BASE:   /home/apache-tomcat-7.0.34
Using CATALINA_HOME:   /home/apache-tomcat-7.0.34
Using CATALINA_TMPDIR: /home/apache-tomcat-7.0.34/temp
Using JRE_HOME:        /home/jdk1.6.0_34
Using CLASSPATH:       /home/apache-tomcat-7.0.34/bin/bootstrap.jar:/home/apache-tomcat-7.0.34/bin/tomcat-juli.jar
# ps aux | grep 'java.*tomcat' | grep -v grep
root     22465 15.3  8.6 1294300 346264 pts/0  Sl   14:55   1:08 /home/jdk1.6.0_34/bin/java -Djava.util.logging.config.file=/home/apache-tomcat-7.0.34/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/home/apache-tomcat-7.0.34/endorsed -classpath /home/apache-tomcat-7.0.34/bin/bootstrap.jar:/home/apache-tomcat-7.0.34/bin/tomcat-juli.jar -Dcatalina.base=/home/apache-tomcat-7.0.34 -Dcatalina.home=/home/apache-tomcat-7.0.34 -Djava.io.tmpdir=/home/apache-tomcat-7.0.34/temp org.apache.catalina.startup.Bootstrap start

2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesJdbccatalina.out报内存泄露警告：

严重: The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Thread-2] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [thread-snatch-picture] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-0] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-1] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-2] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-3] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-4] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-5] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-6] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Xmemcached-Reactor-7] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Heal-Session-Thread] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [pool-2-thread-2] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [pool-2-thread-3] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [pool-2-thread-4] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
严重: The web application [] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@1d97383]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@9a827]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@12e71f1]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@855793]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
2013-1-8 15:02:53 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [com.opensymphony.xwork2.inject.ContainerImpl$10] (value [com.opensymphony.xwork2.inject.ContainerImpl$10@b6c562]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@5164df]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

2 问题分析

这个在windows下没有碰到过，因为此前跑Tomcat都是以服务而不是命令脚本的形式跑的，而且已经换了一个项目，所以暂时不考察windows下是否存在该问题。

难道是Tomcat版本问题？或者用带内存泄漏保护的Tomcat 7可以解决该问题？尝试将web应用跑在apache-tomcat-6.0.18、apache-tomcat-6.0.35、apache-tomcat-7.0.34，发现均存在无法shutdown.sh进程问题。

难道在CentOS（Linux）下，Tomcat无法用shutdown.sh停掉进程？显然不可能的。于是又把web应用从webapps中移走，换回tomcat自带的ROOT，果然启动后再shutdown.sh，查找不到原来的tomcat进程了，也就证明了是自己web应用的问题了。

3 解决方案

现在已经确定是web应用的问题了，所以可以提解决方案了。

3.1 Kill进程

忽略日志中的严重警告，因为这是关闭tomcat时候引起的，正常情况下不会发生这种内存泄露情况，而且Tomcat6.18以上版本的Tomcat已经做了内存泄露保护，交给Tomcat完成吧，我们只需要在shutdown.sh之后，补上一个kill -9 pid即可。要是嫌这样太麻烦了，可以如下这样改：

==============================bin/shutdown.sh 
exec "$PRGDIR"/"$EXECUTABLE" stop  -force "$@"  #加上 -force  
 
==============================bin/catalina.sh  
在PRGDIR=`dirname "$PRG"`后面加上
if [ -z "$CATALINA_PID" ]; then
      CATALINA_PID=$PRGDIR/CATALINA_PID
      cat $CATALINA_PID
fi

3.2 web应用中关闭后台线程

shutdown.sh后存在tomcat僵尸进程的原因是shutdown.sh后web应用没有去关闭其启用的后台线程，所以处理的办法是监听到tomcat关闭事件时，关闭掉其启用的后台线程。

Pstree  -p pid打印进程的线程树，查看shutdown.sh后未关闭的后台线程：

通过Java VisualVM查看shutdown.sh后未关闭的活动线程

在web应用中，如果有后台线程，这时用shutdown.sh是关闭不掉Tomcat的。我们需要重写相关的ServletContextListener的contextDestroyed(ServletContextEventarg0)方法，在它里面关闭后台线程。

但是，现在很多web应用都是由spring来管理bean的，实际情况下我们可能需要注入很多第三方jar中的bean。比如在applicationContext.xml中配置如下bean：

<bean id ="dataSource" class ="org.apache.commons.dbcp.BasicDataSource"destroy-method="close">

则在org.apache.commons.dbcp.BasicDataSource需要做好销毁数据库线程池的逻辑。

当然我们也可以自己设置销毁逻辑。在web.xml文件中增加监听器。

<listener>
    <listener-class>com.mysite.MySpecialListener</listener-class>
</listener>

MySpecialListener实现如下：

public class MySpecialListener extends ApplicationContextListener {
    @Override
    public void contextInitialized(ServletContextEvent sce) {
        // On Application Startup, please…
        // Usually I'll make a singleton in here, set up my pool, etc.
    }
    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        // On Application Shutdown, please…
        // 1. Go fetch that DataSource
        Context initContext = new InitialContext();
        Context envContext  = (Context)initContext.lookup("java:/comp/env");
        DataSource datasource = (DataSource)envContext.lookup("jdbc/database");
        // 2. Call the close() method on it
        datasource.close();
        // 3. Deregister Driver
        try {
            java.sql.Driver mySqlDriver = DriverManager.getDriver("jdbc:mysql://localhost:3306/");
            DriverManager.deregisterDriver(mySqlDriver);
        } catch (SQLException ex) {
            logger.info("Could not deregister driver:".concat(ex.getMessage()));
        } 
        dataSource = null;
    }
}