pyspark修改python版本

最新推荐文章于 2024-06-30 07:15:00 发布

会spark的篮球少年

最新推荐文章于 2024-06-30 07:15:00 发布

阅读量5.2k

点赞数 1

分类专栏： python

本文链接：https://blog.csdn.net/qq_15230053/article/details/82021253

版权

python 专栏收录该内容

33 篇文章 0 订阅

订阅专栏

ubuntu自带的python 版本是2.7，

我们要把pyspark默认改成anaconda python 3.6

down vot

You can specify the version of Python for the driver by setting the appropriate environment variables in the ./conf/spark-env.sh file. If it doesn't already exist, you can use the spark-env.sh.templatefile provided which also includes lots of other variables.

Here is a simple example of a spark-env.sh file to set the relevant Python environment variables:

#!/usr/bin/env bash

# This file is sourced when running various Spark programs.

export PYSPARK_PYTHON=/usr/bin/python3

export PYSPARK_DRIVER_PYTHON=/usr/bin/ipython

In this case it sets the version of Python used by the workers/executors to Python3 and the driver version of Python to iPython for a nicer shell to work in.

意思就是把spark文件夹下的./conf/spark-env.sh.tempalte 重命名成spark-env.sh

然后添加如下内容：

# This file is sourced when running various Spark programs.

export PYSPARK_PYTHON=/usr/bin/python3

export PYSPARK_DRIVER_PYTHON=/usr/bin/ipython

重启spark 即可