故障背景
线上批量发服务下线的告警邮件,偶发nacos连接超时。采用了spring boot admin(以下称sba)进行服务监控。
原因分析
因为sba服务是基于nacos对其它服务进行监控,所以遇到这个问题,第一怀疑对象是nacos发生问题,但不清楚具体是什么问题。由于服务过一段事件会恢复,所以nacos肯定是没有挂掉的,那么排查方向应该是针对nacos的配置,或者是服务器性能。
排查过程
首先查看nacos的堆情况,使用命令jmap -heap PID,得到如下信息:
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 2147483648 (2048.0MB)
NewSize = 1073741824 (1024.0MB)
MaxNewSize = 1073741824 (1024.0MB)
OldSize = 1073741824 (1024.0MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 134217728 (128.0MB)
CompressedClassSpaceSize = 327155712 (312.0MB)
MaxMetaspaceSize = 335544320 (320.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 1058013184 (1009.0MB)
used = 660154960 (629.5728302001953MB)
free = 397858224 (379.4271697998047MB)
62.39572152628298% used
From Space:
capacity = 7864320 (7.5MB)
used = 6914048 (6.59375MB)
free = 950272 (0.90625MB)
87.91666666666667% used
To Space:
capacity = 7864320 (7.5MB)
used = 0 (0.0MB)
free = 7864320 (7.5MB)
0.0% used
PS Old Gener