在运行SparkStreaming程序时遇到频繁的网络错误，包括节点连接关闭、Slave丢失、executor丢失和RPC通信失败。尝试调整Spark参数如spark.core.connection.ack.wait.timeout等以增加网络等待时间，但未能完全解决问题。报错的根本原因可能是集群网络状况不佳。尽管如此，问题依然存在，导致NotLeaderForPartitionException。这通常是由客户端元数据过时导致，日志中出现的错误级别可能过高，建议调整为WARN。
提交了一个SparkStreaming程序，经常出现如下的错误：
ERROR TransportResponseHandler: Still have 5 requests outstanding when connection from hadoop006/172.33.2.6:55862 is closed
WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to get executor loss reason for executor id 12 at RPC address hadoop008:54388, but got no response. Marking as slave lost.
java.io.IOException: Connection from hadoop006/172.33.2.6:55862 closed
ERROR YarnScheduler: Lost executor 12 on hadoop008: Slave lost
ERROR TransportClient: Failed to send RPC 8179126089530426088 to hadoop006/172.33.2.6:55862: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to get executor loss reason for executor id 2 at RPC address hadoop004:49980, but got no response. Marking as slave lost.
java.io.IOException: Failed to send RPC 8179126089530426088 to hadoop006/172.33.2.6:55862: java.nio.channels.ClosedChannel
最低0.47元/天 解锁文章
1596
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
