pyspark java.net.SocketException: Connection reset by peer

最新推荐文章于 2022-02-11 15:27:28 发布

markix

最新推荐文章于 2022-02-11 15:27:28 发布

阅读量3.2k

点赞数 1

分类专栏： BigData 异常小记文章标签： pyspark connection reset by peer

本文为博主原创文章，未经博主允许不得转载。

本文链接：https://blog.csdn.net/qq_31772441/article/details/86560437

版权

异常小记同时被 2 个专栏收录

39 篇文章 1 订阅

订阅专栏

BigData

5 篇文章 0 订阅

订阅专栏

在window、运行pyspark训练模型，报错
Caused by: java.net.SocketException: Connection reset by peer: socket write error

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.net.SocketException: Connection reset by peer: socket write error
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
	...
Caused by: java.net.SocketException: Connection reset by peer: socket write error
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
	...

解决：修改python环境下的pyspark包中的worker. py 中的process函数

（注：python环境下的pyspark包是从spark目录拷贝过来的）
eg: E:\Anaconda3\envs\py27\Lib\site-packages\pyspark\worker. py
定位到 process 函数：

        def process():
            iterator = deserializer.load_stream(infile)
            serializer.dump_stream(func(split_index, iterator), outfile)
            # 添加以下内容
            for obj in interator:
                pass

refer:
https://issues.apache.org/jira/browse/SPARK-12261
https://stackoverflow.com/questions/26954566/apache-spark-pyspark-crash-for-large-dataset

end