小记-bug解决:Exception thrown in awaitResult/Connection reset by peer

最新推荐文章于 2024-06-18 19:33:35 发布

xiaolin_xinji

最新推荐文章于 2024-06-18 19:33:35 发布

阅读量9.1k

点赞数 1

分类专栏： Spark BUG

本文链接：https://blog.csdn.net/weixin_44131414/article/details/107099578

版权

Spark 同时被 2 个专栏收录

25 篇文章 0 订阅

订阅专栏

BUG

14 篇文章 0 订阅

订阅专栏

org.apache.spark.SparkException: Exception thrown in awaitResult

"time":"2020-07-02 11:10:15","logtype":"ERROR","loginfo":"Executor self-exiting due to : Driver masterba17fc94-f00c-477a-b1d4-4c165c7893d9cupid-11-194-251-15:45755 disassociated! Shutting down."}
{"time":"2020-07-02 11:10:15","logtype":"WARN","loginfo":"Issue communicating with driver in heartbeater"}
org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
	at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
	at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:92)
	at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:785)
	at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply$mcV$sp(Executor.scala:814)
	at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply(Executor.scala:814)
	at org.apache.spark.executor.Executor$$anon$2$$anonfun$run$1.apply(Executor.scala:814)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988)
	at org.apache.spark.executor.Executor$$anon$2.run(Executor.scala:814)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

解决:

#调整参数大小:
	spark.sql.broadcastTimeout=1200 默认大小300

java.io.IOException: Connection reset by peer

java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:745)

解决:

调大参数: 
 spark.network.timeout 默认大小 120 s 
 spark.executor.heartbeatInterval 默认大小10s
 #注:spark.network.timeout的参数要大于 spark.executor.heartbeatInterval 心跳参数
 	Interval between each executor's heartbeats to the driver. 
 	Heartbeats let the driver know that the executor is still alive 
 	and update it with metrics for in-progress tasks. 
 	spark.executor.heartbeatInterval should be significantly less than spark.network.timeout