老卫带你学---pyspark运行报错

最近老卫在玩spark，安装完pyspark之后（pyspark=2.3.2；python=3.7.0）
可是在运行代码的时候出现了如下报错：

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/I:/python_project/spark/venv/Lib/site-packages/pyspark/jars/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2019-11-13 21:55:42 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2019-11-13 21:55:45 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: '<pyspark.conf.SparkConf object at 0x000001F7A200D240>'
	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2744)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:834)
Traceback (most recent call last):
2019-11-13 21:55:45 WARN  MetricsSystem:66 - Stopping a MetricsSystem that is not running
  File "I:/python_project/spark/spark3.py", line 9, in <module>
    sc=SparkContext(conf)
  File "I:\python_project\spark\venv\lib\site-packages\pyspark\context.py", line 118, in __init__
    conf, jsc, profiler_cls)
  File "I:\python_project\spark\venv\lib\site-packages\pyspark\context.py", line 180, in _do_init
    self._jsc = jsc or self._initialize_context(self._conf._jconf)
  File "I:\python_project\spark\venv\lib\site-packages\pyspark\context.py", line 290, in _initialize_context
    return self._jvm.JavaSparkContext(jconf)
  File "I:\python_project\spark\venv\lib\site-packages\py4j\java_gateway.py", line 1525, in __call__
    answer, self._gateway_client, None, self._fqn)
  File "I:\python_project\spark\venv\lib\site-packages\py4j\protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: org.apache.spark.SparkException: Could not parse Master URL: '<pyspark.conf.SparkConf object at 0x000001F7A200D240>'
	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2744)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:834)

原来是pyspark2.3不支持高版本的jdk（之前是jdk-11），所以我换了jdk-8，程序成功运行。

下面附上程序代码及结果：

from pyspark import SparkContext
import os
os.environ['JAVA_HOME']='D:\jdk-8'
sc=SparkContext('local')
old=sc.parallelize([1,2,3,4,5],2)
newMap=old.map(lambda x:(x,x**2))
newReduce=old.reduce(lambda a,b:a+b)
print(newMap.glom().collect())
print(newReduce)