Python in worker has different version 3.7 than that in driver 3.6, PySpark cannot run with differe

最新推荐文章于 2024-03-05 09:36:51 发布

有风微冷

最新推荐文章于 2024-03-05 09:36:51 发布

阅读量3k

点赞数 1

分类专栏：大数据

本文链接：https://blog.csdn.net/qq_36770189/article/details/98647372

版权

博客讲述了在使用PySpark时遇到的Python版本不一致问题，即worker使用的是Python3.7，而driver使用的是Python3.6。作者列举了两种解决方案：一是针对特定工程进行配置；二是修改环境变量设置全局生效。

摘要由CSDN通过智能技术生成

错误如下：使用Anaconda，默认是python37，下载python36，idea中更换为python3.6,报了如下错误。

Exception: Python in worker has different version 3.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298)
	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:438)
	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:421)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:1126)
	at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:1132)
	at scala.collection.Iterator$$anon$11.hasNext(Itera