遗留问题:在secureCRT上跑spark代码时偶尔会报错:ClosedChannelException
错误信息:
17/05/26 17:34:09 INFO YarnClientSchedulerBackend: Stopped
17/05/26 17:34:09 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/05/26 17:34:09 INFO BlockManagerInfo: Removed broadcast_10_piece0 on 192.168.16.100:36241 in memory (size: 598.0 KB, free: 12.6 GB)
17/05/26 17:34:09 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(driver, 192.168.16.100, 36241),broadcast_10_piece0,StorageLevel(1 replicas),0,0))
17/05/26 17:34:09 ERROR TransportClient: Failed to send RPC 7502960340689747796 to /192.168.16.108:35336: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
17/05/26 17:34:09 ERROR TransportClient: Failed to send RPC 6241621360882774485 to /192.168.16.107:52929: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
17/05/26 17:34:09 WARN BlockManagerMaster: Failed to remove broadcast 10 with removeFromMaster = true - Failed to send RPC 7502960340689747796 to /192.168.16.108:35336: java.nio.channels.ClosedChannelException
java.io.IOException: Failed to send RPC 7502960340689747796 to /192.168.16.108:35336: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801)
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
17/05/26 17:34:09 INFO MemoryStore: MemoryStore cleared
17/05/26 17:34:09 INFO BlockManager: BlockManager stopped
17/05/26 17:34:09 INFO BlockManagerMaster: BlockManagerMaster stopped
17/05/26 17:34:09 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/05/26 17:34:09 INFO SparkContext: Successfully stopped SparkContext
17/05/26 17:34:09 INFO ShutdownHookManager: Shutdown hook called
17/05/26 17:34:09 INFO ShutdownHookManager: Deleting directory /tmp/spark-26f7a564-81b5-4df0-8228-f4cc7f4a96bf