flink oom问题排查

最新推荐文章于 2024-08-07 09:00:48 发布

wppwpp1

最新推荐文章于 2024-08-07 09:00:48 发布

阅读量502

点赞数

分类专栏： flink 文章标签： flink html5 sql

本文链接：https://blog.csdn.net/wppwpp1/article/details/115098320

版权

flink 专栏收录该内容

69 篇文章 4 订阅

订阅专栏

系统提示任务有延迟，自动调优显示内存不足。当前配置为tm内存4GB，建议提升到6GB以解决PT3M的400ms延迟问题。通过VVP添加了GC日志打印，日志显示频繁的ParNew GC，可能是新生代大小设置不适当导致。已调整新生代大小，设置为-XX:NewRatio=2，以期望改善内存分配问题并减少垃圾回收频率。

摘要由CSDN通过智能技术生成

1、查看作业，提示某个task有延迟

2、点击 "自动调优"菜单，提示 pt3m的是400ms，内存不够，需要把tm的内存从4g变成6g

3、vvp上加入gc日志打印信息

env.java.opts: >-
  -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/flink/log/gc.log
  -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=50M

4、相关日志查看位置如下：

5、部分日志如下；

2021-03-22T21:59:25.471+0800: 510.043: [CMS-concurrent-mark-start]
2021-03-22T21:59:25.642+0800: 510.214: [GC (Allocation Failure) 2021-03-22T21:59:25.642+0800: 510.214: [ParNew: 76671K->8512K(76672K), 0.1891483 secs] 3073442K->3034303K(3530432K), 0.1894421 secs] [Times: user=0.18 sys=0.01, real=0.19 secs] 
2021-03-22T21:59:25.977+0800: 510.549: [GC (Allocation Failure) 2021-03-22T21:59:25.977+0800: 510.549: [ParNew: 76672K->8511K(76672K), 0.1130852 secs] 3102463K->3066355K(3530432K), 0.1132797 secs] [Times: user=0.11 sys=0.00, real=0.11 secs] 
2021-03-22T21:59:26.240+0800: 510.812: [GC (Allocation Failure) 2021-03-22T21:59:26.240+0800: 510.812: [ParNew: 76671K->8512K(76672K), 0.1753545 secs] 3134515K->3102785K(3530432K), 0.1755990 secs] [Times: user=0.18 sys=0.00, real=0.18 secs] 
2021-03-22T21:59:26.566+0800: 511.138: [GC (Allocation Failure) 2021-03-22T21:59:26.567+0800: 511.138: [ParNew: 76672K->8510K(76672K), 0.1328703 secs] 3170945K->3129630K(3530432K), 0.1330545 secs] [Times: user=0.13 sys=0.00, real=0.13 secs] 
2021-03-22T21:59:26.851+0800: 511.423: [GC (Allocation Failure) 2021-03-22T21:59:26.851+0800: 511.423: [ParNew: 76670K->8512K(76672K), 0.1237257 secs] 3197790K->3164357K(3530432K), 0.1239200 secs] [Times: user=0.12 sys=0.00, real=0.12 secs] 
2021-03-22T21:59:27.112+0800: 511.684: [GC (Allocation Failure) 2021-03-22T21:59:27.113+0800: 511.685: [ParNew: 76672K->8511K(76672K), 0.1748845 secs] 3232517K->3200508K(3530432K), 0.1750872 secs] [Times: user=0.17 sys=0.00, real=0.18 secs] 
2021-03-22T21:59:27.436+0800: 512.008: [GC (Allocation Failure) 2021-03-22T21:59:27.436+0800: 512.008: [ParNew: 76367K->8511K(76672K), 0.1376000 secs] 3268364K->3228034K(3530432K), 0.1378985 secs] [Times: user=0.14 sys=0.00, real=0.13 secs] 
2021-03-22T21:59:27.695+0800: 512.267: [GC (Allocation Failure) 2021-03-22T21:59:27.695+0800: 512.267: [ParNew: 76671K->8510K(76672K), 0.1280840 secs] 3296194K->3266303K(3530432K), 0.1282691 secs] [Times: user=0.13 sys=0.00, real=0.13 secs] 
2021-03-22T21:59:27.969+0800: 512.541: [GC (Allocation Failure) 2021-03-22T21:59:27.969+0800: 512.541: [ParNew: 71715K->8512K(76672K), 0.1583017 secs] 3329507K->3289236K(3530432K), 0.1585163 secs] [Times: user=0.16 sys=0.00, real=0.16 secs] 
2021-03-22T21:59:28.266+0800: 512.838: [GC (Allocation Failure) 2021-03-22T21:59:28.266+0800: 512.838: [ParNew: 76672K->8510K(76672K), 0.1369435 secs] 3357396K->3325727K(3530432K), 0.1372206 secs] [Times: user=0.13 sys=0.00, real=0.13 secs] 
2021-03-22T21:59:28.535+0800: 513.107: [GC (Allocation Failure) 2021-03-22T21:59:28.535+0800: 513.107: [ParNew: 76670K->8511K(76672K), 0.1299317 secs] 3393887K->3363637K(3530432K), 0.1301524 secs] [Times: user=0.13 sys=0.01, real=0.13 secs] 
2021-03-22T21:59:28.824+0800: 513.396: [GC (Allocation Failure) 2021-03-22T21:59:28.824+0800: 513.396: [ParNew: 75409K->8511K(76672K), 0.1839333 secs] 3430534K->3391714K(3530432K), 0.1841673 secs] [Times: user=0.18 sys=0.00, real=0.19 secs] 
2021-03-22T21:59:29.122+0800: 513.694: [GC (Allocation Failure) 2021-03-22T21:59:29.122+0800: 513.694: [ParNew: 76671K->8510K(76672K), 0.1163922 secs] 3459874K->3424803K(3530432K), 0.1166100 secs] [Times: user=0.12 sys=0.00, real=0.12 secs] 
2021-03-22T21:59:29.375+0800: 513.947: [GC (Allocation Failure) 2021-03-22T21:59:29.375+0800: 513.947: [ParNew: 72328K->72328K(76672K), 0.0000370 secs]2021-03-22T21:59:29.375+0800: 513.947: [CMS2021-03-22T21:59:31.379+0800: 515.951: [CMS-concurrent-mark: 3.895/5.908 secs] [Times: user=5.78 sys=0.11, real=5.91 secs] 
 (concurrent mode failure): 3416293K->2756564K(3453760K), 6.5734869 secs] 3488621K->2756564K(3530432K), [Metaspace: 64768K->64768K(1107968K)], 6.5737765 secs] [Times: user=6.54 sys=0.01, real=6.58 secs] 
2021-03-22T21:59:36.124+0800: 520.696: [GC (Allocation Failure) 2021-03-22T21:59:36.125+0800: 520.697: [ParNew: 68160K->8511K(76672K), 0.1110947 secs] 2824724K->2789531K(3530432K), 0.1113143 secs] [Times: user=0.11 sys=0.01, real=0.12 secs] 
2021-03-22T21:59:36.368+0800: 520.940: [GC (Allocation Failure) 2021-03-22T21:59:36.368+0800: 520.940: [ParNew: 76671K->8510K(76672K), 0.1163960 secs] 2857691K->2825279K(3530432K), 0.1166403 secs] [Times: user=0.12 sys=0.00, real=0.11 secs] 
2021-03-22T21:59:36.605+0800: 521.177: [GC (Allocation Failure) 2021-03-22T21:59:36.605+0800: 521.177: [ParNew: 67523K->8512K(76672K), 0.1501337 secs] 2884292K->2850444K(3530432K), 0.1503500 secs] [Times: user=0.15 sys=0.00, real=0.16 secs] 
2021-03-22T21:59:36.872+0800: 521.444: [GC (Allocation Failure) 2021-03-22T21:59:36.872+0800: 521.444: [ParNew: 76672K->8510K(76672K), 0.1120007 secs] 2918604K->2883605K(3530432K), 0.1122097 secs] [Times: user=0.11 sys=0.00, real=0.11 secs] 
2021-03-22T21:59:37.099+0800: 521.671: [GC (Allocation Failure) 2021-03-22T21:59:37.099+0800: 521.671: [ParNew: 76670K->8512K(76672K), 0.1341794 secs] 2951765K->2926565K(3530432K), 0.1344005 secs] [Times: user=0.13 sys=0.00, real=0.13 secs] 
2021-03-22T21:59:37.366+0800: 521.938: [GC (Allocation Failure) 2021-03-22T21:59:37.366+0800: 521.938: [ParNew: 76672K->8511K(76672K), 0.1626722 secs] 2994725K->2952436K(3530432K), 0.1629048 secs] [Times: user=0.16 sys=0.00, real=0.16 secs] 
2021-03-22T21:59:37.647+0800: 522.219: [GC (Allocation Failure) 2021-03-22T21:59:37.647+0800: 522.219: [ParNew: 76671K->8510K(76672K), 0.1100316 secs] 3020596K->2985065K(3530432K), 0.1102117 secs] [Times: user=0.11 sys=0.00, real=0.11 secs] 
2021-03-22T21:59:37.874+0800: 522.446: [GC (Allocation Failure) 2021-03-22T21:59:37.874+0800: 522.446: [ParNew: 76670K->8510K(76672K), 0.1623085 secs] 3053225K->3020756K(3530432K), 0.1625360 secs] [Times: user=0.16 sys=0.00, real=0.17 secs] 
2021-03-22T21:59:38.043+0800: 522.615: [GC (CMS Initial Mark) [1 CMS-initial-mark: 3012245K(3453760K)] 3025231K(3530432K), 0.0184047 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
2021-03-22T21:59:38.069+0800: 522.641: [CMS-concurrent-mark-start]
2021-03-22T21:59:38.213+0800: 522.785: [GC (Allocation Failure) 2021-03-22T21:59:38.213+0800: 522.785: [ParNew: 76670K->8510K(76672K), 0.1455792 secs] 3088916K->3049692K(3530432K), 0.1458183 secs] [Times: user=0.15 sys=0.00, real=0.15 secs] 
2021-03-22T21:59:38.490+0800: 523.062: [GC (Allocation Failure) 2021-03-22T21:59:38.490+0800: 523.062: [ParNew: 76670K->8510K(76672K), 0.1292822 secs] 3117852K->3084788K(3530432K), 0.1295316 secs] [Times: user=0.12 sys=0.00, real=0.13 secs] 
2021-03-22T21:59:38.743+0800: 523.314: [GC (Allocation Failure) 2021-03-22T21:59:38.743+0800: 523.315: [ParNew: 73672K->8511K(76672K), 0.1818953 secs] 3149950K->3122442K(3530432K), 0.1820931 secs] [Times: user=0.19 sys=0.00, real=0.18 secs] 
2021-03-22T21:59:39.071+0800: 523.643: [GC (Allocation Failure) 2021-03-22T21:59:39.072+0800: 523.644: [ParNew: 76671K->8510K(76672K), 0.1722696 secs] 3190602K->3155601K(3530432K), 0.1724842 secs] [Times: user=0.17 sys=0.00, real=0.17 secs] 
2021-03-22T21:59:39.402+0800: 523.974: [GC (Allocation Failure) 2021-03-22T21:59:39.403+0800: 523.975: [ParNew: 76670K->8511K(76672K), 0.1255075 secs] 3223761K->3188842K(3530432K), 0.1257325 secs] [Times: user=0.12 sys=0.00, real=0.13 secs] 
2021-03-22T21:59:39.710+0800: 524.282: [GC (Allocation Failure) 2021-03-22T21:59:39.710+0800: 524.282: [ParNew: 75921K->8511K(76672K), 0.1720593 secs] 3256251K->3213718K(3530432K), 0.1722503 secs] [Times: user=0.17 sys=0.00, real=0.17 secs] 
2021-03-22T21:59:39.999+0800: 524.571: [GC (Allocation Failure) 2021-03-22T21:59:39.999+0800: 524.571: [ParNew: 76671K->8512K(76672K), 0.0879451 secs]

发现新生代一直失败，调整一下新生代的大小,参数如下，在重启任务