Exception in thread "main" java.util.NoSuchElementException 错误解决

spark-2.4.0-bin-hadoop2.7 和 spark-2.3.0-bin-hadoop2.7是有区别的。

当时pyspark测试程序的运行环境

 

C:\Users\test>java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

C:\Users\pingan>python -V
Python 3.5.4

spark目录下的 ~\python\lib的两个文件 py4j-0.10.6-src.zip 和 pyspark.zip 已释放到 python目录下的 ~\Lib 或是 ~\Lib\site-packages，可以正常导入pyspark

C:\Users\pingan>python
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyspark import SparkConf,SparkContext

在运行测试代码时

import os,sys
from pyspark import SparkConf,SparkContext
os.environ['SPARK_HOME'] = "F:\spark"
sc = SparkContext(appName="Spark WordCount test")
datatest = ["test spark","test spark sql","spark python"]
rdd = sc.parallelize(datatest,2)
print(rdd.first())
print(rdd.count())
sc.stop()
print("hello")

当在 “F:\spark” 目录下为spark-2.4.0-bin-hadoop2.7的文件 时，报错Exception in thread "main" java.util.NoSuchElementException，当 在"F:\spark"目录下为 spark-2.3.0-bin-hadoop2.7，错误消失，由此看来，两个spark版本对python3兼容性是有区别的。