调用shutdown脚本关闭tomcat时，某些进程关闭不掉（一）

首先从Tomcat的启动过程出发：

Tomcat启动的入口类：org.apache.catalina.startup.Bootstrap#main

main方法是整个tomcat启动时的入口。在main方法中，使用bootstrap.init()来初始化类加载器和创建Catalina实例，然后再启动Catalina线程。

bootstrap.init()方法，用于初始化容器相关，首先创建类加载器，然后通过反射创建org.apache.catalina.startup.Catalina实例：

   public void init()
        throws Exception
    {

        // Set Catalina path
        setCatalinaHome();
        setCatalinaBase();

        initClassLoaders();

        Thread.currentThread().setContextClassLoader(catalinaLoader);

        SecurityClassLoad.securityClassLoad(catalinaLoader);
。。。。。。
}

之后Bootstrap的demon.start()方法就会调用Catalina的start方法。

Catalina实例执行start方法。这里有两个点，一个是load()加载server.xml配置、初始化Server的过程，一个是getServer().start()开启服务、初始化并开启一系列组件、子容器的过程。

public void start() {

    if (getServer() == null) {
        load();
    }

    if (getServer() == null) {
        log.fatal("Cannot start server. Server instance is not configured.");
        return;
    }

    long t1 = System.nanoTime();

    // Start the new server
    try {
        getServer().start();
    } catch (LifecycleException e) {
        log.fatal(sm.getString("catalina.serverStartFail"), e);
        try {
            getServer().destroy();
        } catch (LifecycleException e1) {
            log.debug("destroy() failed for failed Server ", e1);
        }
        return;
    }

    long t2 = System.nanoTime();
    if(log.isInfoEnabled()) {
        log.info("Server startup in " + ((t2 - t1) / 1000000) + " ms");
    }

    // Register shutdown hook
    //①
    if (useShutdownHook) {
        if (shutdownHook == null) {
            shutdownHook = new CatalinaShutdownHook();
        }
        Runtime.getRuntime().addShutdownHook(shutdownHook);
        。。。。。。
    }

    //②
    if (await) {
        await();
        stop();
    }
}

初始化过程暂时不关心，重点在shutdown的部分

在代码①处，用到了Jvm的shutdownHook机制。shutdownHook是一个已经初始化但是还没有启动的线程，当JVM关闭的时候，它会启动并且并发的运行所有已经注册过的shutdown hooks。

接下来，看下CatalinaShutdownHook 线程做了什么，它是Catalina的一个内部类，源码如下：org.apache.catalina.startup.Catalina.CatalinaShutdownHook

protected class CatalinaShutdownHook extends Thread {

    @Override
    public void run() {
        try {
            if (getServer() != null) {
                Catalina.this.stop();
            }
        } catch (Throwable ex) {
            ExceptionUtils.handleThrowable(ex);
            log.error(sm.getString("catalina.shutdownHookFail"), ex);
        } finally {
            // If JULI is used, shut JULI down *after* the server shuts down
            // so log messages aren't lost
            LogManager logManager = LogManager.getLogManager();
            if (logManager instanceof ClassLoaderLogManager) {
                ((ClassLoaderLogManager) logManager).shutdown();
            }
        }
    }
}

通过上面的代码，可以看到调用了Catalina的stop方法。而Catalina#stop方法最终又是调用了StandardServer#stop方法和destroy方法。StandardServer的这两个方法会一层一层的关闭之前开启的一系列服务、组件、和子容器。暂时不关心这部分，先略过。

从这里可以知道Tomcat利用了shutdown hook机制来在JVM关闭的时候关闭各个组件。

接下来，需要知道JVM是何时退出的，根据②处的代码，首先判断了await属性是否为true,如果为true就调用await()，调用完以后，再调用stop方法，接下来看await()方法,而catalina的await方法又调用了StandardServer#await方法，它的代码如下：org.apache.catalina.core.StandardServer#await

public void await() {

    。。。。。。
    // Set up a server socket to wait on
    try {
        awaitSocket = new ServerSocket(port, 1,
                InetAddress.getByName(address));
    } catch (IOException e) {
        log.error("StandardServer.await: create[" + address
                           + ":" + port
                           + "]: ", e);
        return;
    }

    try {
        awaitThread = Thread.currentThread();

        // Loop waiting for a connection and a valid command
        while (!stopAwait) {
            ServerSocket serverSocket = awaitSocket;
            if (serverSocket == null) {
                break;
            }

            // Wait for the next connection
            Socket socket = null;
            StringBuilder command = new StringBuilder();
            try {
                InputStream stream;
                long acceptStartTime = System.currentTimeMillis();
                try {
                    socket = serverSocket.accept();
                    socket.setSoTimeout(10 * 1000);  // Ten seconds
                    stream = socket.getInputStream();
                } catch (SocketTimeoutException ste) {
                    // This should never happen but bug 56684 suggests that
                    // it does.
                    log.warn(sm.getString("standardServer.accept.timeout",
                            Long.valueOf(System.currentTimeMillis() - acceptStartTime)), ste);
                    continue;
                } catch (AccessControlException ace) {
                    log.warn("StandardServer.accept security exception: "
                            + ace.getMessage(), ace);
                    continue;
                } catch (IOException e) {
                    if (stopAwait) {
                        // Wait was aborted with socket.close()
                        break;
                    }
                    log.error("StandardServer.await: accept: ", e);
                    break;
                }

                // Read a set of characters from the socket
                int expected = 1024; // Cut off to avoid DoS attack
                while (expected < shutdown.length()) {
                    if (random == null)
                        random = new Random();
                    expected += (random.nextInt() % 1024);
                }
                while (expected > 0) {
                    int ch = -1;
                    try {
                        ch = stream.read();
                    } catch (IOException e) {
                        log.warn("StandardServer.await: read: ", e);
                        ch = -1;
                    }
                    // Control character or EOF (-1) terminates loop
                    if (ch < 32 || ch == 127) {
                        break;
                    }
                    command.append((char) ch);
                    expected--;
                }
            } finally {
                // Close the socket now that we are done with it
                try {
                    if (socket != null) {
                        socket.close();
                    }
                } catch (IOException e) {
                    // Ignore
                }
            }

            // Match against our command string
            boolean match = command.toString().equals(shutdown);
            if (match) {
                log.info(sm.getString("standardServer.shutdownViaPort"));
                break;
            } else
                log.warn("StandardServer.await: Invalid command '"
                        + command.toString() + "' received");
        }
    } finally {
        ServerSocket serverSocket = awaitSocket;
        awaitThread = null;
        awaitSocket = null;

        // Close the server socket and return
        if (serverSocket != null) {
            try {
                serverSocket.close();
            } catch (IOException e) {
                // Ignore
            }
        }
    }
}

通过上面的代码可以看出在配置的端口上通过ServerSocket来监听一个请求的到来，如果请求的字符串和配置的字符串相同的话即跳出循环，这样的话就会运行stop方法，运行完了以后，main线程就退出了。

这里ServerSocket监听的端口，以及对比的字符串都是在conf/server.xml中配置的，默认情况下，配置如下：

<Server port="8005" shutdown="SHUTDOWN">

从这里可以看出监听端口为8005,关闭请求发送的字符串为SHUTDOWN。

当运行shutdown.sh时，其实最终也是向server发送了一个SHUTDOWN字符串。

查看shutdown.sh最终是调用了 catalina.sh，并传递了stop参数。

PRGDIR=`dirname "$PRG"`
EXECUTABLE=catalina.sh
exec "$PRGDIR"/"$EXECUTABLE" stop "$@"

查看catalina.sh脚本，最终其实是调用了 org.apache.catalina.startup.Bootstrap#main ,并传递参数stop。

elif [ "$1" = "start" ] ; then

# 此处省略参数校验的脚本

  eval "\"$_RUNJAVA\"" $LOGGING_MANAGER $JAVA_OPTS \
    -D$ENDORSED_PROP="\"$JAVA_ENDORSED_DIRS\"" \
    -classpath "\"$CLASSPATH\"" \
    -Dcatalina.base="\"$CATALINA_BASE\"" \
    -Dcatalina.home="\"$CATALINA_HOME\"" \
    -Djava.io.tmpdir="\"$CATALINA_TMPDIR\"" \
    org.apache.catalina.startup.Bootstrap "$@" stop

  # stop failed. Shutdown port disabled? Try a normal kill.
  if [ $? != 0 ]; then
    if [ ! -z "$CATALINA_PID" ]; then
      echo "The stop command failed. Attempting to signal the process to stop through OS signal."
      kill -15 `cat "$CATALINA_PID"` >/dev/null 2>&1
    fi
  fi

  if [ ! -z "$CATALINA_PID" ]; then
    if [ -f "$CATALINA_PID" ]; then
      while [ $SLEEP -ge 0 ]; do
        kill -0 `cat "$CATALINA_PID"` >/dev/null 2>&1
        if [ $? -gt 0 ]; then
          rm -f "$CATALINA_PID" >/dev/null 2>&1
          if [ $? != 0 ]; then
            if [ -w "$CATALINA_PID" ]; then
              cat /dev/null > "$CATALINA_PID"
              # If Tomcat has stopped don't try and force a stop with an empty PID file
              FORCE=0
            else
              echo "The PID file could not be removed or cleared."
            fi
          fi
          echo "Tomcat stopped."
          break
        fi
        if [ $SLEEP -gt 0 ]; then
          sleep 1
        fi
        if [ $SLEEP -eq 0 ]; then
          echo "Tomcat did not stop in time."
          if [ $FORCE -eq 0 ]; then
            echo "PID file was not removed."
          fi
          echo "To aid diagnostics a thread dump has been written to standard out."
          kill -3 `cat "$CATALINA_PID"`
        fi
        SLEEP=`expr $SLEEP - 1 `
      done
    fi
  fi

查看Bootstrap#main方法，发现会调用 org.apache.catalina.startup.Bootstrap#stopServer ,而Bootstrap#stopServer通过反射调用了 org.apache.catalina.startup.Catalina#stopServer ,代码如下：org.apache.catalina.startup.Catalina#stopServer

public void stopServer(String[] arguments) {

    if (arguments != null) {
        arguments(arguments);
    }

    Server s = getServer();
    //①
    if( s == null ) {
        // Create and execute our Digester
        Digester digester = createStopDigester();
        File file = configFile();
        FileInputStream fis = null;
        try {
            InputSource is =
                new InputSource(file.toURI().toURL().toString());
            fis = new FileInputStream(file);
            is.setByteStream(fis);
            digester.push(this);
            digester.parse(is);
        } catch (Exception e) {
            log.error("Catalina.stop: ", e);
            System.exit(1);
        } finally {
            if (fis != null) {
                try {
                    fis.close();
                } catch (IOException e) {
                    // Ignore
                }
            }
        }
    } else {
        // Server object already present. Must be running as a service
        try {
            s.stop();
            s.destroy();
        } catch (LifecycleException e) {
            log.error("Catalina.stop: ", e);
        }
        return;
    }

    // Stop the existing server
    s = getServer();
    //②
    if (s.getPort()>0) {
        Socket socket = null;
        OutputStream stream = null;
        try {
            socket = new Socket(s.getAddress(), s.getPort());
            stream = socket.getOutputStream();
            String shutdown = s.getShutdown();
            for (int i = 0; i < shutdown.length(); i++) {
                stream.write(shutdown.charAt(i));
            }
            stream.flush();
        } catch (ConnectException ce) {
            log.error(sm.getString("catalina.stopServer.connectException",
                                   s.getAddress(),
                                   String.valueOf(s.getPort())));
            log.error("Catalina.stop: ", ce);
            System.exit(1);
        } catch (IOException e) {
            log.error("Catalina.stop: ", e);
            System.exit(1);
        } finally {
            if (stream != null) {
                try {
                    stream.close();
                } catch (IOException e) {
                    // Ignore
                }
            }
            if (socket != null) {
                try {
                    socket.close();
                } catch (IOException e) {
                    // Ignore
                }
            }
        }
    } else {
        log.error(sm.getString("catalina.stopServer"));
        System.exit(1);
    }
}

标注1的代码，此时因为是新开了一个进程,并且conf/server.xml还没有解析，因此s是NULL，通过Digester解析conf/server.xml，最终生成了未初始化的StandardServer对象。
标注2的代码，向standardServer.getPort返回的端口（其实这里面返回即是conf/server.xml中Server根节点配置的port和shutdown属性）发送了standardServer.getShutdown()返回的字符串，而默认情况下这个字符串就是SHUTDOWN.

总结一下：

Tomcat启动的时候的主线程会在8005端口（默认配置，可以更改）上建立socket监听，当关闭的时候，最终其实就是新起了一个进程然后向Tomcat主线程监听的8005端口发送了一个SHUTDOWN字符串，这样主线程就会结束了，主线程结束了以后，因为其它的线程都是dameon线程，这样依赖JVM就会退出了。

因此，如果说此时还有其他非dameon线程的线程存在，那么JVM并不会退出，导致进程仍然存在。只能采取强制杀死进程或者把其他线程设为demeon线程之类的方式处理。

归结起来，还是在业务代码层面上出的问题，比如使用了线程池并未正确关闭，代码中使用了shutdownhook之类的原因。