java资源争夺_JVM调优实战

对项目 i..net 网站进行了接口的压测，压测过程中发现各微服务的 heap 内存设置过小(-Xms64m –Xmx128m)，导致压测过程中服务假死(频繁fullgc导致服务假死)和发生OOM。

1、xx接口的压测情况如下，150并发进行压测，发现响应时间在小于1s一段时间后，响应时间陡升到3s。

2、 TPS 也是从420 陡降到 60 左右。

3、trace 调用链跟踪发现 xxxxsevice中的 xxxxWorktime方法调用耗时3S

4、继续跟踪该方法的调用链，发现调用 attend-xxxx项目中的接口耗时3S

5、查看后台日志发现：atten-xxxx 服务发生了 OOM；通过telnet命令连接xxxx的端口，发现无法连接端口，服务处于假死状态。

6、查看该服务的heap使用情况，基本上被耗尽

7、jstat 发现 发生了 频繁的 FullGC

8、压测过程中发现在进行资源争夺时，有时还把mongo 给拖死：

ERROR c.y.a.sign.exception.BaseControllerException - attend-xxxx统一异常捕获...

org.springframework.dao.DataAcce***esourceFailureException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is

Caused by: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=xx.xx.0.40:47020, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, {address=xx.xx.0.40:47020, type=REPLICA_SET_ARBITER, roundTripTime=0.6 ms, state=CONNECTED}, {address=xx.xx.0.40:47020, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]

at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:377)

at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:104)

at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)

at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)

at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)

at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:411)

at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:133)

at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:60)

at com.mongodb.Mongo.execute(Mongo.java:845)

at com.mongodb.Mongo$2.execute(Mongo.java:828)

at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:342)

at com.mongodb.DBCollection.insert(DBCollection.java:337)

at com.mongodb.DBCollection.insert(DBCollection.java:328)

at com.mongodb.DBCollection.insert(DBCollection.java:298)

at com.mongodb.DBCollection.insert(DBCollection.java:264)

at com.mongodb.DBCollection.insert(DBCollection.java:186)

at com.mongodb.DBCollection.save(DBCollection.java:407)

at com.mongodb.DBCollection.save(DBCollection.java:382)

9、数据库在压测过程中被压死：

综上所述，调优建议如下：

1、 微服务的heap 设置太小了，需要扩容内存，建议设置为(-Xms512m –Xmx1024m)

2、 数据库服务服务器建议单独部署物理机，且数据库内存建议设置64G以上；建议采用多主从或分片部署。