spark mysql 数据异常,Spark 异常总结及解决办法

最新推荐文章于 2023-10-06 21:24:01 发布

weixin_39950470

最新推荐文章于 2023-10-06 21:24:01 发布

阅读量686

点赞数

文章标签： spark mysql 数据异常

前言

总结Spark开发中遇到的异常及解决办法，之前也写过几篇，之所以不再一个异常写一篇博客，是因为现在Spark用的比较熟悉了一些，觉得没必要把异常信息写那么详细了，所以就把异常总结在一篇博客里了，这样既能备忘也方便查找。

1、之前的几篇

2、 spark.executor.memoryOverhead

堆外内存(默认是executor内存的10%)，当数据量比较大的时候，如果按默认的就会有下面的异常，导致程序崩溃

异常

Container killed by YARN for exceeding memory limits. 1.8 GB of 1.8 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

解决

具体值根据实际情况配置

新版

--conf spark.executor.memoryOverhead=2048

旧版

--conf spark.yarn.executor.memoryOverhead=2048

新版如果用旧版，会:

WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead.

3、No more replicas available for rdd_

异常

19/01/08 12:36:46 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_3250_73 !

19/01/08 12:36:46 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_12_38 !

19/01/08 12:36:46 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_3250_38 !

19/01/08 12:36:46 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_3250_148 !

19/01/08 12:36:46 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_3250_6 !

19/01/08 12:36:46 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_3250_112 !

19/01/08 12:36:46 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_12_100 !

解决

增大executor的内存

--executor-memory 4G

4、Failed to allocate a page

异常

19/01/09 09:12:39 WARN TaskMemoryManager: Failed to allocate a page (1048576 bytes), try again.

19/01/09 09:12:41 WARN TaskMemoryManager: Failed to allocate a page (1048576 bytes), try again.

19/01/09 09:12:41 WARN NioEventLoop: Unexpected exception in the selector loop.

java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.lang.Integer.valueOf(Integer.java:832)

at sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:120)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:98)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)

at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

at java.lang.Thread.run(Thread.java:748)

19/01/09 09:12:46 WARN TransportChannelHandler: Exception in connection from /172.16.29.236:47012

java.lang.OutOfMemoryError: GC overhead limit exceeded

19/01/09 09:12:44 WARN AbstractChannelHandlerContext: An exception 'java.lang.OutOfMemoryError: GC overhead limit exceeded' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:

java.lang.OutOfMemoryError: GC overhead limit exceeded

19/01/09 09:12:42 WARN TaskMemoryManager: Failed to allocate a page (1048576 bytes), try again.

Exception in thread "dispatcher-event-loop-11" java.lang.OutOfMemoryError: GC overhead limit exceeded

19/01/09 09:12:51 WARN TaskMemoryManager: Failed to allocate a page (1048576 bytes), try again.

19/01/09 09:12:53 WARN TransportChannelHandler: Exception in connection from /172.16.29.233:34226

java.lang.OutOfMemoryError: GC overhead limit exceeded

解决

增大driver的内存

--driver-memory 6G

参考

5、Uncaught exception in thread task-result-getter-3

异常

19/01/10 09:31:50 ERROR Utils: Uncaught exception in thread task-result-getter-3

java.lang.OutOfMemoryError: Java heap space

at java.lang.reflect.Array.newArray(Native Method)

at java.lang.reflect.Array.newInstance(Array.java:75)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1938)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1566)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2286)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2210)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2068)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1974)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1566)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)

at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)

at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:108)

at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:88)

at org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:94)

at org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply(TaskResultGetter.scala:63)

at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1991)

at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:62)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Exception in thread "task-result-getter-3" java.lang.OutOfMemoryError: Java heap space