I configured eclipse with pyspark
i am using latest version of SPARK and PYTHON.
when i try to code something and run. i get below error.
java.io.IOException: Cannot run program "python": CreateProcess error=2, The system cannot find the file specified
code i have written is below
'''
Created on 23-Dec-2017
@author: lenovo
'''
from pyspark import SparkContext,SparkConf
from builtins import int
#from org.spark.com.PySparkDemo import data
from pyspark.sql import Row
from pyspark.sql.context import SQLContext
conf = SparkConf().setAppName("FileSystem").setMaster("local")
sc=SparkContext(conf=conf)
sqlContext=SQLContext(sc)
a = sc.textFile("C:/Users/lenovo/Desktop/file.txt")
b = a.map(lambda x:x.split(",")).map(lambda x:Row(id=int(x[0]),name=x[1],marks=int(x[2])))
c = sqlContext.createDataFrame(b)
c.show()
please suggest
解决方案
Assuming you have installed pydev
under Windows > preferences > Pydev > interpreters > python interpreters > go to environment
under environment you need to give path of pyhton.exe file, Variable name as PYSPARK_PYTHON