线上问题-关于@Async

问题日志

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
#   JVM is running with Unscaled Compressed Oops mode in which the Java heap is
#     placed in the first 4GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 4GB virtual address.
# This output file may be truncated or incomplete.
#
---------------  T H R E A D  ---------------

Current thread (0x00007f1480287800):  JavaThread "Keep-Alive-Timer" daemon [_thread_new, id=955887, stack(0x00007f12221cd000,0x00007f122220e000)]

Stack: [0x00007f12221cd000,0x00007f122220e000],  sp=0x00007f122220cb40,  free space=254k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xb5718d]
V  [libjvm.so+0x50675a]
V  [libjvm.so+0x93cfc0]
V  [libjvm.so+0x9343be]
V  [libjvm.so+0x93e557]
V  [libjvm.so+0xafa7d6]
V  [libjvm.so+0xafe97c]
V  [libjvm.so+0x93a382]
C  [libpthread.so.0+0x7ea5]  start_thread+0xc5


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
=>0x00007f1480287800 JavaThread "Keep-Alive-Timer" daemon [_thread_new, id=955887, stack(0x00007f12221cd000,0x00007f122220e000)]
  0x00007f1292a14000 JavaThread "SimpleAsyncTaskExecutor-122011" [_thread_blocked, id=955832, stack(0x00007f1222290000,0x00007f12222d1000)]
  0x00007f1292a16000 JavaThread "SimpleAsyncTaskExecutor-122009" [_thread_blocked, id=955829, stack(0x00007f12222d1000,0x00007f1222312000)]
  0x00007f1292a9a800 JavaThread "SimpleAsyncTaskExecutor-122007" [_thread_blocked, id=955823, stack(0x00007f122220e000,0x00007f122224f000)]
.....

翻译：

Native memory allocation ，本地内存分配失败。

可能的解决办法：

1. 减少系统内存使用

2. 增加物理内存或缓冲区

3. 减少 JVM堆大小（-Xmx/-Xms）

4. 减少线程数量

5. 减少线程栈大小(-Xss)

   6.设置大的code cache（-XX：ReservedCodeCacheSize=）

分析问题：

线程数统计：32278 个SimpleAsyncTaskExecutor，开了好多线程啊！ 应该是线程太多

SimpleAsyncTaskExecutor是什么？

这个org.springframework.core.task.SimpleAsyncTaskExecutor 类注释如下：

/**
 * {@link TaskExecutor} implementation that fires up a new Thread for each task,
 * executing it asynchronously.
 *
 * <p>Supports limiting concurrent threads through the "concurrencyLimit"
 * bean property. By default, the number of concurrent threads is unlimited.
 *
 * <p><b>NOTE: This implementation does not reuse threads!</b> Consider a
 * thread-pooling TaskExecutor implementation instead, in particular for
 * executing a large number of short-lived tasks.
 *
 * @author Juergen Hoeller
 * @since 2.0
 * @see #setConcurrencyLimit
 * @see SyncTaskExecutor
 * @see org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor
 * @see org.springframework.scheduling.commonj.WorkManagerTaskExecutor
 */
@SuppressWarnings("serial")

SimpleAsyncTaskExecutor 不复用线程，每次都新启动一个线程.与问题表现一致，但是代码中没有显示使用这个类啊！谁在用呢？？？

SimpleAsyncTaskExecutor谁在用？

查看系统中所有多线程的部分，初步怀疑是@Async注解使用了这个线程池,看源码验证。

 * @see AnnotationAsyncExecutionInterceptor
 * @see AsyncAnnotationAdvisor
 */
@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Async {
  String value() default "";
}

上边特意说明，查看AnnotationAsyncExecutionInterceptor 类（异步注解拦截器），AsyncAnnotationAdvisor（异步注解通知和切入点定义类），显然要做默认线程池需要查看AnnotationAsyncExecutionInterceptor 类（异步注解拦截器）。

getDefaultExecutor方法如下：

	@Override
	protected Executor getDefaultExecutor(BeanFactory beanFactory) {
		Executor defaultExecutor = super.getDefaultExecutor(beanFactory);
		return (defaultExecutor != null ? defaultExecutor : new     SimpleAsyncTaskExecutor());
	}

目前可以确定，@Async 在不设置线程池的情况下，默认使用SimpleAsyncTaskExecutor、

溢出原因

由于@Async标注的方法处理较慢，请求较多，造成创建大量的线程且没有执行完毕，线程回收不掉，占用本地内存过多，内存分配失败。

解决办法

1. 修改业务逻辑，可以采用消息队列削峰填谷。

2. Spring Boot项目设置线程池，@Async注解复用线程，方法如下：

   @Configuration
   @EnableAsync
   public class BeanConfig {
   
       @Bean
       public TaskExecutor taskExecutor() {
           ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
           // 设置核心线程数
           executor.setCorePoolSize(5);
           // 设置最大线程数
           executor.setMaxPoolSize(10);
           // 设置队列容量
           executor.setQueueCapacity(2000);
           // 设置线程活跃时间（秒）
           executor.setKeepAliveSeconds(60);
           // 设置默认线程名称
           executor.setThreadNamePrefix("hello-");
           // 设置拒绝策略
           executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
           // 等待所有任务结束后再关闭线程池
           executor.setWaitForTasksToCompleteOnShutdown(true);
           return executor;
       }
   }