IDEA2020中测试PySpark的运行出错

项目环境：

Windows： Idea2020

JDK: java version "1.8.0_231"

Python： 3.8.3

spark：spark-3.2.1-bin-hadoop2.7.tgz

 以前有hadoop2.7的环境，故使用的Spark3是基于hadoop2.7的。

---

问题描述

IDEA中测试PySpark的Map算子，出错Message

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times,
most recent failure: Lost task 1.0 in stage 0.0 (TID 1) (XXXX.mshome.net executor driver):
java.io.IOException: Cannot run program "python3": CreateProcess error=2, 系统找不到指定的文件

 详细错误信息： 

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.	
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1) (E490.mshome.net executor driver): 
java.io.IOException: Cannot run program "python3": CreateProcess error=2, 系统找不到指定的文件。	
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:166)
	at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:108)
	at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:121)
	at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:162)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: CreateProcess error=2, 系统找不到指定的文件。	
	at java.lang.ProcessImpl.create(Native Method)
	at java.lang.ProcessImpl.<init>(ProcessImpl.java:444)
	at java.lang.ProcessImpl.start(ProcessImpl.java:140)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 15 more
	

对策

Python的安装目录里面复制python.exe，并改名为python3.exe 

 Spark的Map算子运行成功。