Spark环境搭建 (Python)

1. 安装lib

材料:

spark : http://spark.apache.org/downloads.html

hadoop : http://hadoop.apache.org/releases.html

jdk: http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html

hadoop-commin : 

https://github.com/SweetInk/hadoop-common-2.7.1-bin

https://github.com/SweetInk/hadoop-common-2.7.1-bin

(for windows7)

DirectX修复工具: http://www.downxia.com/downinfo/21233.html, http://js.downxia.com/down/DirectXRepairzqb.rar

 

需要下载对应的版本

步骤：

a. 安装jdk，默认步骤即可

b. 解压spark  （D:\spark-2.0.0-bin-hadoop2.7）

c. 解压hadoop  (D:\hadoop2.7)

d. 解压hadoop-commin （for w7）

e. copy hadoop-commin/bin to hadoop/bin （for w7）

f. 运行 hadoop/bin 下的winutils.exe 如果没有报错则正常，如果提示0x0....等错误编码则需要DirectX修复

 

2. 环境变量设置

SPARK_HOME = D:\spark-2.0.0-bin-hadoop2.7

HADOOP_HOME = D:\hadoop2.7

PATH append = D:\spark-2.0.0-bin-hadoop2.7\bin;%HADOOP_HOME%\bin;%HADOOP_HOME%\sbin

 

3. Python lib设置

a. copy D:\spark-2.0.0-bin-hadoop2.7\python\pyspark to [Your-Python-Home]\Lib\site-packages

b. pip install py4j

c. pip install psutil

   (for windows: http://www.lfd.uci.edu/~gohlke/pythonlibs/#psutil)

 

4. Testing

cmd -> pyspark 不报错并且有相应的cmd