Exception: Python in worker has different version 2.7 than that in driver 3.8, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PY
PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PY)
原因是机器装着不同版本的python,pyspark不知道用哪个,所以代码需要配置要使用的版本即可,可通过设置环境变量来实现
import os
# 此处为自己环境中的python路径
os.environ['PYSPARK_PYTHON'] = '/usr/local/bin/python3'