PYSPARK_PYTHON 环境问题

datadev_sh

于 2020-04-17 14:31:22 发布

阅读量1.1k

点赞数

分类专栏： DataScience

本文链接：https://blog.csdn.net/datadev_sh/article/details/105579866

版权

DataScience 专栏收录该内容

19 篇文章

订阅专栏

报错

Exception: Python in worker has different version 2.7 than that in driver 3.7, 
PySpark cannot run with different minor versions.
Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

找到对应的python环境

(base) xin:~ xin$ which python
/Users/xin/anaconda3/bin/python

设置一下

os.environ["PYSPARK_PYTHON"]="/Users/xin/anaconda3/bin/python"
os.environ["PYSPARK_DRIVER_PYTHON"]="/Users/xin/anaconda3/bin/python"