JAVA优雅停机

JAVA优雅停机

项目需要对SpringCLoud做一些优雅停机的改造，所以研究了下linux和java的停机方法，以及业界其他框架的做法。

Kill 结束进程

在 Linux上，kill 命令发送指定的信号到相应进程，不指定信号则默认发送 SIGTERM(15) 终止指定进程。如果无法终止，可以发送 SIGKILL(9) 来强制结束进程。kill 命令信号共有64个信号值，其中常用的是：

• 2 (SIGINT：中断，Ctrl+C)。

• 15 (SIGTERM：终止，默认值)。

• 9 (SIGKILL：强制终止)。

这里我们重点说一下 15 和 9 的情况。

kill PID/kill -15 PID 命令系统发送 SIGTERM 进程信号给响应的应用程序，当应用程序接收到 SIGTERM 信号，可以进行释放相应资源后再停止，此时程序可能仍然继续运行。

而kill -9 PID 命令没有给进程遗留善后处理的条件。应用程序将会被直接终止。

java的关闭钩子（Shutdown Hook）

Runtime.getRuntime().addShutdownHook(shutdownHook);

这个方法的意思就是在jvm中增加一个关闭的钩子，当jvm关闭的时候，会执行系统中已经设置的所有通过方法addShutdownHook添加的钩子，当系统执行完这些钩子后，jvm才会关闭。所以这些钩子可以在jvm关闭的时候进行内存清理、对象销毁等操作。

下面先了解一下JDK的ShutdownHook函数会在哪些时候生效

• 程序正常退出
• 程序中使用System.exit()退出JVM
• 系统发生OutofMemory异常
• 使用kill pid干掉JVM进程的时候（kill -9时候是不能触发ShutdownHook生效的）

一个服务最好只注册一个Shutdown Hook，可能引起死锁，多个shutdown hook同步执行，无法保证顺序。

Dubbo中的优雅停机

Dubbo的优雅停机是依赖于JDK的ShutdownHook函数。

Dubbo优雅停机代码解读

dubbo的优雅停机代码入口就在于AbstractConfig的静态代码块中：

static {
    DubboShutdownHook.getDubboShutdownHook().register();
}

public class DubboShutdownHook extends Thread {
    @Override
    public void run() {
        if (logger.isInfoEnabled()) {
            logger.info("Run shutdown hook now.");
        }
        doDestroy();
    }
    /**
     * Destroy all the resources, including registries and protocols.
     */
    public void doDestroy() {
        if (!destroyed.compareAndSet(false, true)) {
            return;
        }
        // destroy all the registries
        AbstractRegistryFactory.destroyAll();
        // destroy all the protocols
        destroyProtocols();
    }

    /**
     * Destroy all the protocols.
     * Destroy protocol: <br>
     * 1. Cancel all services this protocol exports and refers <br>
     * 2. Release all occupied resources, for example: connection, port, etc. <br>
     * 3. Protocol can continue to export and refer new service even after it's destroyed.
     */
    private void destroyProtocols() {
        ...
            protocol.destroy();
        ...
}

public void destroy() {
    for (String key : new ArrayList<>(serverMap.keySet())) {
        ExchangeServer server = serverMap.remove(key);

        if (server == null) {
            continue;
        }

        try {
            if (logger.isInfoEnabled()) {
                logger.info("Close dubbo server: " + server.getLocalAddress());
            }
						//关停所有的Server，作为provider将不再接收新的请求
            server.close(ConfigurationUtils.getServerShutdownTimeout());

        } catch (Throwable t) {
            logger.warn(t.getMessage(), t);
        }
    }

    for (String key : new ArrayList<>(referenceClientMap.keySet())) {
        List<ReferenceCountExchangeClient> clients = referenceClientMap.remove(key);

        if (CollectionUtils.isEmpty(clients)) {
            continue;
        }
        //关停所有的Client，作为consumer将不再发送新的请求
        for (ReferenceCountExchangeClient client : clients) {
            closeReferenceCountExchangeClient(client);
        }
    }
  
  //每个Server和Client都有自己的线程池
  public abstract class AbstractServer extends AbstractEndpoint implements Server {
   ...
    ExecutorService executor;
   ...
  }
  
  public abstract class AbstractClient extends AbstractEndpoint implements Client {
		...
    protected volatile ExecutorService executor;
    ...
  }

//ExecutorUtil
public static void gracefulShutdown(Executor executor, int timeout) {
        if (!(executor instanceof ExecutorService) || isTerminated(executor)) {
            return;
        }
        final ExecutorService es = (ExecutorService) executor;
        try {
            // Disable new tasks from being submitted
            es.shutdown();
        } catch (SecurityException ex2) {
            return;
        } catch (NullPointerException ex2) {
            return;
        }
        try {
            // Wait a while for existing tasks to terminate
            if (!es.awaitTermination(timeout, TimeUnit.MILLISECONDS)) {
                es.shutdownNow();
            }
        } catch (InterruptedException ex) {
            es.shutdownNow();
            Thread.currentThread().interrupt();
        }
        if (!isTerminated(es)) {
            newThreadToCloseExecutor(es);
        }
    }

//AbstractRegistryFactory.destroyAll()
public static void destroyAll() {
    if (LOGGER.isInfoEnabled()) {
        LOGGER.info("Close all registries " + getRegistries());
    }
    // Lock up the registry shutdown process
    LOCK.lock();
    try {
        for (Registry registry : getRegistries()) {
            try {
                registry.destroy();
            } catch (Throwable e) {
                LOGGER.error(e.getMessage(), e);
            }
        }
        REGISTRIES.clear();
    } finally {
        // Release the lock
        LOCK.unlock();
    }
}

//AbstractRegistry.destory()
//处理通用的destory逻辑
public void destroy() {
    if (logger.isInfoEnabled()) {
        logger.info("Destroy registry:" + getUrl());
    }
    //作为provider，取消所有的服务注册
    Set<URL> destroyRegistered = new HashSet<>(getRegistered());
    if (!destroyRegistered.isEmpty()) {
        for (URL url : new HashSet<>(getRegistered())) {
            if (url.getParameter(DYNAMIC_KEY, true)) {
                try {
                    unregister(url);
                    if (logger.isInfoEnabled()) {
                        logger.info("Destroy unregister url " + url);
                    }
                } catch (Throwable t) {
                    logger.warn("Failed to unregister url " + url + " to registry " + getUrl() + " on destroy, cause: " + t.getMessage(), t);
                }
            }
        }
    }
  //作为consumer，取消所有的订阅关系
    Map<URL, Set<NotifyListener>> destroySubscribed = new HashMap<>(getSubscribed());
    if (!destroySubscribed.isEmpty()) {
        for (Map.Entry<URL, Set<NotifyListener>> entry : destroySubscribed.entrySet()) {
            URL url = entry.getKey();
            for (NotifyListener listener : entry.getValue()) {
                try {
                   //将listener从订阅者对应的listener集合中移除（监听的服务变更将不再进行通知）
                    unsubscribe(url, listener);
                    if (logger.isInfoEnabled()) {
                        logger.info("Destroy unsubscribe url " + url);
                    }
                } catch (Throwable t) {
                    logger.warn("Failed to unsubscribe url " + url + " to registry " + getUrl() + " on destroy, cause: " + t.getMessage(), t);
                }
            }
        }
    }
}

总结，注册jvm停机钩子，执行停机功能，主要分为注册中心下线和协议关闭。

Spring中的停机

同样的，注册jvm停机钩子，方法执行中，首先发布ContextClosedEvent，之后关闭清理bean，关闭beanfactory，执行后续操作。web容器在最后一步关闭，会导致许多失败的请求。

//SpringApplication


/**
 * Sets if the created {@link ApplicationContext} should have a shutdown hook
 * registered. Defaults to {@code true} to ensure that JVM shutdowns are handled
 * gracefully.
 * @param registerShutdownHook if the shutdown hook should be registered
 */
public void setRegisterShutdownHook(boolean registerShutdownHook) {
   this.registerShutdownHook = registerShutdownHook;
}

//ApplicationContext

/**
 * Register a shutdown hook with the JVM runtime, closing this context
 * on JVM shutdown unless it has already been closed at that time.
 * <p>Delegates to {@code doClose()} for the actual closing procedure.
 * @see Runtime#addShutdownHook
 * @see #close()
 * @see #doClose()
 */
@Override
public void registerShutdownHook() {
   if (this.shutdownHook == null) {
      // No shutdown hook registered yet.
      this.shutdownHook = new Thread() {
         @Override
         public void run() {
            synchronized (startupShutdownMonitor) {
               doClose();
            }
         }
      };
      Runtime.getRuntime().addShutdownHook(this.shutdownHook);
   }
}

/**
	 * Actually performs context closing: publishes a ContextClosedEvent and
	 * destroys the singletons in the bean factory of this application context.
	 * <p>Called by both {@code close()} and a JVM shutdown hook, if any.
	 * @see org.springframework.context.event.ContextClosedEvent
	 * @see #destroyBeans()
	 * @see #close()
	 * @see #registerShutdownHook()
	 */
	protected void doClose() {
		if (this.active.get() && this.closed.compareAndSet(false, true)) {
			if (logger.isInfoEnabled()) {
				logger.info("Closing " + this);
			}
			//JMX
			LiveBeansView.unregisterApplicationContext(this);

			try {
				// Publish shutdown event.
				publishEvent(new ContextClosedEvent(this));
			}
			catch (Throwable ex) {
				logger.warn("Exception thrown from ApplicationListener handling ContextClosedEvent", ex);
			}

			// Stop all Lifecycle beans, to avoid delays during individual destruction.
			if (this.lifecycleProcessor != null) {
				try {
					this.lifecycleProcessor.onClose();
				}
				catch (Throwable ex) {
					logger.warn("Exception thrown from LifecycleProcessor on context close", ex);
				}
			}

			// Destroy all cached singletons in the context's BeanFactory.
			destroyBeans();

			// Close the state of this context itself.
			closeBeanFactory();

			// Let subclasses do some final clean-up if they wish...
			onClose();

			this.active.set(false);
		}
	}

//ReactiveWebServerApplicationContext

protected void onClose() {
   super.onClose();
   stopAndReleaseReactiveWebServer();
}

private void stopAndReleaseReactiveWebServer() {
   ServerManager serverManager = this.serverManager;
   try {
      ServerManager.stop(serverManager);
   }
   finally {
      this.serverManager = null;
   }
}

SOFA-Boot对Spring-Boot的扩展

/**
 * Spring上下文监听器.负责关闭SOFABoot RPC 的资源。
 *
 * @author <a href="mailto:lw111072@antfin.com">LiWei</a>
 */
public class ApplicationContextClosedListener implements ApplicationListener {
    private final ProviderConfigContainer providerConfigContainer;
    private final ServerConfigContainer   serverConfigContainer;

    public ApplicationContextClosedListener(ProviderConfigContainer providerConfigContainer,
                                            ServerConfigContainer serverConfigContainer) {
        this.providerConfigContainer = providerConfigContainer;
        this.serverConfigContainer = serverConfigContainer;
    }

    @Override
    public void onApplicationEvent(ApplicationEvent event) {
        if ((event instanceof ContextClosedEvent) || (event instanceof ContextStoppedEvent)) {
            providerConfigContainer.unExportAllProviderConfig();
            serverConfigContainer.closeAllServer();
        }
    }
}

通过监听ContextClosedEvent和ContextStoppedEvent来执行关闭功能，关闭过程类似于dubbo。

分析

在spring boot环境中应用优雅停机可以通过注册JVM钩子和监听spring上下文事件，分析两者优缺点：

• jvm钩子可以控制整个关停过程，保证各个组件的关闭顺序（比如注册中心下线），需要在启动类里注册钩子
• 监听spring上下文事件，不需要修改现有代码，只需添加监听器，但事件发布时异步事件，不能保证监听器中逻辑的正常执行。

所以选择jvm钩子来进行优雅停机。参考dubbo和sofa-boot的实现，主要要注册中心下线，服务挡板（不在接受新请求），和资源释放（考虑资源释放的顺序）几个过程。根据平台现状需要关闭的主要资源有：undertow容器，Hikari数据库连接池、redis等。

经过实验，已经初步实现了eureka下线，undertow容器优雅关闭、Hikari数据库连接池优雅关闭。