ClosedChannelException

最新推荐文章于 2024-06-22 16:00:38 发布

蜡笔小新hyp

最新推荐文章于 2024-06-22 16:00:38 发布

阅读量2.5k

点赞数 1

分类专栏：遗留问题大数据 spark 用户画像文章标签： spark

本文链接：https://blog.csdn.net/hyp1006346386/article/details/72845710

版权

大数据同时被 3 个专栏收录

31 篇文章 0 订阅

订阅专栏

spark

15 篇文章 0 订阅

订阅专栏

用户画像

5 篇文章 0 订阅

订阅专栏

遗留问题：在secureCRT上跑spark代码时偶尔会报错：ClosedChannelException
错误信息：

17/05/26 17:34:09 INFO YarnClientSchedulerBackend: Stopped
17/05/26 17:34:09 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/05/26 17:34:09 INFO BlockManagerInfo: Removed broadcast_10_piece0 on 192.168.16.100:36241 in memory (size: 598.0 KB, free: 12.6 GB)
17/05/26 17:34:09 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(driver, 192.168.16.100, 36241),broadcast_10_piece0,StorageLevel(1 replicas),0,0))
17/05/26 17:34:09 ERROR TransportClient: Failed to send RPC 7502960340689747796 to /192.168.16.108:35336: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
17/05/26 17:34:09 ERROR TransportClient: Failed to send RPC 6241621360882774485 to /192.168.16.107:52929: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
17/05/26 17:34:09 WARN BlockManagerMaster: Failed to remove broadcast 10 with removeFromMaster = true - Failed to send RPC 7502960340689747796 to /192.168.16.108:35336: java.nio.channels.ClosedChannelException
java.io.IOException: Failed to send RPC 7502960340689747796 to /192.168.16.108:35336: java.nio.channels.ClosedChannelException
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
        at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
        at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
17/05/26 17:34:09 INFO MemoryStore: MemoryStore cleared
17/05/26 17:34:09 INFO BlockManager: BlockManager stopped
17/05/26 17:34:09 INFO BlockManagerMaster: BlockManagerMaster stopped
17/05/26 17:34:09 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/05/26 17:34:09 INFO SparkContext: Successfully stopped SparkContext
17/05/26 17:34:09 INFO ShutdownHookManager: Shutdown hook called
17/05/26 17:34:09 INFO ShutdownHookManager: Deleting directory /tmp/spark-26f7a564-81b5-4df0-8228-f4cc7f4a96bf