1、下载
http://spark.apache.org/downloads.html
选择下载源码
2、源码编译
1)解压
tar -zxvf spark-1.4.1.tgz
2、编译
进入根目录下,采用make-distribution.sh进行编译。
cd spark-1.4.1
sudo ./make-distribution.sh --tgz --skip-java-test -Pyarn -Phadoop-2.2-Dhadoop.version=2.2.0 -Phive -Phive-thriftserver -DskipTests clean package
如果中间有报错,请重新跑,多试几次,一般都能成功。
编译成功后,其安装文件在根目录下:
spark-1.4.1-bin-2.2.0.tgz
3、安装
省略,和之前版本一样,就不写了。
4、报错问题
1)问题1 : worek节点不能启动
localhost:starting org.apache.spark.deploy.worker.Worker, logging to/home/lib/spark-1.4.1/sbin/../logs/org.apache.spark.deploy.worker.Worker-1-is xxxx.out
localhost:failed to launch org.apache.spark.deploy.worker.Worker:
localhost: at org.apache.spark.launcher.SparkClassCommandBuilder.buildCommand(SparkClassCommandBuilder.java:98)
localhost: atorg.apache.spark.launcher.Main.main(Main.java:74)
localhost:full log in/home/lib/spark-1.4.1/sbin/../logs/org.apache.spark.deploy.worker.Worker-1-is xxxx.out
localhost:Connection to localhost closed.
原因是系统自带java问题
rpm -qa | grep java
gcc-java-4.4.7-4.el6.x86_64
java_cup-0.10k-5.el6.x86_64
java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64卸载
rpm -e --nodeps java_cup-0.10k-5.el6.x86_64
rpm -e --nodepsjava-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
2)问题2 :JAVA_HOME is not set
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/lib/spark-1.4.1/sbin/../logs/spark-org.apache.spark.deploy.worker.Worker-1-is xxxx.out
localhost: failed to launch org.apache.spark.deploy.worker.Worker:
localhost: JAVA_HOME is not set
localhost: full log in /lib/spark-1.4.1/sbin/../logs/org.apache.spark.deploy.worker.Worker-1-isxxxx.out
localhost: Connection to localhost closed.
找到出错的shell文件,加入export JAVA_HOME=... 即可
spark-env.sh,加入export JAVA_HOME=... 即可
启动成功后的界面:
转载请注明出处: