No module named pyspark解决方式

最新推荐文章于 2024-04-12 02:48:17 发布

klzgr19870401_2017

最新推荐文章于 2024-04-12 02:48:17 发布

阅读量1.2w

点赞数

本文链接：https://blog.csdn.net/klzgr19870401_2017/article/details/92742405

版权

当遇到`No module named pyspark`的问题时，原因是缺少必要的Spark Python库。解决方法是在Spark配置中指定依赖文件路径，例如：设置`spark.yarn.dist.files`为 pyspark 和 py4j 的路径，并通过`PYTHONPATH`环境变量确保Executor能找到这些库。

摘要由CSDN通过智能技术生成

1.问题现象：

Caused by: org.apache.spark.SparkException:
Error from python worker:
/usr/bin/python: No module named pyspark
PYTHONPATH was:
/root/hadoop/tmp/nm-local-dir/usercache/root/filecache/22/__spark_libs__297119794367585949.zip/spark-core_2.11-2.4.0.jar
org.apache.spark.SparkException: No port number in pyspark.daemon's stdout
   at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:204)
   at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:122)
   at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:95)
   at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)
   at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:108)
   at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
   at org.apache.spark.rdd.RDD.computeOrRead