解决办法:增加了服务器的cpu数量和内存扩容,当然如果你是集群部署,可以考虑的是加服务器,降低目前的服务器的压力。
解决过程如下:
通过一顿top,
top -Hp pid,
print “%x\n” tid,
jstack pid |grep tid -A 50
的操作看见了如下异常堆栈,堆栈有所不同,但是终都是挂阻塞在Unsafe.park()的有限时长等待这里,这是一个本地方法,有限时长等待,就算它因为等待不到它要的条件或者资源,那么到时间了自然也会返回,但是奇怪的是,它们永远停在了这里
因为只有测试环境才频繁出现这个问题,正式环境一直运行良好,它们的差别在于cpu和内存的差异,测试环境使1核2G的,我不得不说一句:太抠了。。。。。
因为内存过小,所以考虑是不是因为内存太小,导致了大量的Gc从而使这些等待得不到cpu,因为确实然后就这样挂了,因为确实发现了频繁的GC,Full GC所以对内存扩容到了8G,但事后还是发生了这个事情,并且挂得更快了,那就加了cpu到2核,到目前已经一个多月过去了,没有再出现这个问题
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000af662cf0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
TID: 2310 STATE: TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:898)
org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1256)
org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1231)
org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:730)
com.changjia.commodity.task.es.impl.EsSearchServiceImpl.search(EsSearchServiceImpl.java:31)
```cpp
TID: 118 STATE: TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
com.aliyun.openservices.shade.com.alibaba.rocketmq.common.CountDownLatch2.await(CountDownLatch2.java:114)
com.aliyun.openservices.shade.com.alibaba.rocketmq.common.ServiceThread.waitForRunning(ServiceThread.java:116)
com.aliyun.openservices.shade.com.alibaba.rocketmq.client.impl.consumer.RebalanceService.run(RebalanceService.java:40)
java.lang.Thread.run(Thread.java:748)