windows上pyspark glom()或map之后collect报错

最新推荐文章于 2023-12-31 15:07:24 发布

ThisIsNicole

最新推荐文章于 2023-12-31 15:07:24 发布

阅读量1k

点赞数 3

文章标签： spark scala big data

本文链接：https://blog.csdn.net/NICOLENICOLE_/article/details/125236487

版权

在Windows环境中，使用pyspark进行大数据处理时遇到错误，尝试通过findspark初始化和降低pyspark版本至2.3.2解决了问题。问题起因可能是pyspark 3.x版本与某些接口不兼容导致。

摘要由CSDN通过智能技术生成

from pyspark import SparkConf,SparkContext
conf=SparkConf().setAppName('sparkApp1').setMaster("local")
sc=SparkContext.getOrCreate(conf)

rdd01=sc.parallelize([1,2,3,4,5,6],7)
print(rdd01.collect())
print(rdd01.glom().collect())

出现以下错误
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 2.0 failed 1 times, most recent failure: Lost task 5.0 in stage 2.0 (TID 19) : org.apache.spark.SparkException: Python worker failed to connect back.
	at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:188)
	at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:108)
	at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:121)
	at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:162)