[Dynamic Language] pyspark Python3.7环境设置 及py4j.protocol.Py4JJavaError: An error occurred while calli...

pyspark Python3.7环境设置 及py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe解决!

环境设置

1. JDK: java version "1.8.0_66"
2. Python 3.7
3. spark-2.3.1-bin-hadoop2.7.tgz
4. 环境变量
   • export PYSPARK_PYTHON=python3
   • export PYSPARK_DRIVER_PYTHON=ipython3

mac-abeen:spark-2.3.1-bin-hadoop2.7 abeen$ ./bin/pyspark 
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 20:42:06) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.
1Using Python version 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018 20:42:06)
SparkSession available as 'spark'.

In [1]: sc
Out[1]: <SparkContext master=local[*] appName=PySparkShell>

In [2]: lines = sc.textFile("README.md")

In [3]: lines.count()
Out[3]: 103                                                                     

In [4]: lines.first()
Out[4]: '# Apache Spark'

Py4JJavaError PythonRDD.collectAndServe解决!

注