flink on yarn ——调节堆内存

报错:OutOfMemoryError: GC overhead limit exceeded

java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOfRange(Arrays.java:3664)
	at java.lang.String.<init>(String.java:207)
	at java.lang.StringBuilder.toString(StringBuilder.java:407)
	at org.apache.hadoop.conf.Configuration$Parser.handleEndElement(Configuration.java:3246)
	at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3331)
	at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3114)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3007)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2968)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:211)
	at com.gtcom.governance.until.HdfsUtils.getFileSystem(HdfsUtils.java:65)
	at com.gtcom.governance.source.zip.ScanZipExtractData.invoke(ScanZipExtractData.java:224)
	at com.gtcom.governance.source.zip.ScanZipExtractData$1.lambda$visitFile$1(ScanZipExtractData.java:159)
	at com.gtcom.governance.source.zip.ScanZipExtractData$1$$Lambda$637/1084422546.accept(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at com.gtcom.governance.source.zip.ScanZipExtractData$1.visitFile(ScanZipExtractData.java:157)
	at com.gtcom.governance.source.zip.ScanZipExtractData$1.visitFile(ScanZipExtractData.java:88)
	at java.nio.file.Files.walkFileTree(Files.java:2670)
	at java.nio.file.Files.walkFileTree(Files.java:2742)
	at com.gtcom.governance.source.zip.ScanZipExtractData.scan(ScanZipExtractData.java:88)
	at com.gtcom.governance.source.zip.ScanZipExtractData.run(ScanZipExtractData.java:78)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
2022-03-21 16:41:38.961 [Source: Custom Source (1/1)] INFO  com.gtcom.governance.source.zip.ScanZipExtractData - cancelling.......

增加-ytm :

$FLINK_HOME/bin/flink run -d -m yarn-cluster -yjm 2048m -ytm 15360m -ynm yarn_monitorZip_news_second -yD env.java.opts="-Duser.timezone=GMT+8"  -ys 1 $FLINK_HOME/jobs/gtcom-governance-service-jk-2.3.0.jar --active pro --driver MonitorZip --topicName MinitorZip_wait_second --sinkPara 3 --sourcePara 1  --opertatorPara 3 --processorType MonitorZip  --scanPath /home/data/packageAll/secondPackageZiptmp/   --targetPath /home/data/packageAll/secondTargetPath/ --passWd 12345 --backupPath /home/data/packageAll/secondBackupPath/

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值