Spark的安装和测试

部署是环境为3台服务器 ，里面安装了相关的hadoop集群，安装的spark1.6.3

总共分为如下安装方式

1，单机模式测试安装

2，Standalone集群模式

3，Standalone集群HA模式

4，Yarn集群模式

 

一，单机模式测试安装

1，解压安装包

[root@hadoop01 bigdata]# pwd
/home/tools/bigdata
[root@hadoop01 bigdata]# tar -zxvf spark-1.6.3-bin-hadoop2.4.tgz -C /home/bigdata/

2，配置环境变量

vi /etc/profile
export SPARK_HOME=/home/bigdata/spark-1.6.3-bin-hadoop2.4
export PATH=$SPARK_HOME/bin:$PATH

source /etc/profile

 

3，测试

 

root@hadoop01 lib]# cd /home/bigdata/spark-1.6.3-bin-hadoop2.4/lib/
[root@hadoop01 lib]# spark-submit --class org.apache.spark.examples.SparkPi --master local[*] spark-examples-1.6.3-hadoop2.4.0.jar 100

测试结果

18/07/07 15:36:52 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 3.541017 s
Pi is roughly 3.1416247141624716
18/07/07 15:36:52 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}

二，Standalone集群模式

1，配置slaves

[root@hadoop01 conf]# cd /home/bigdata/spark-1.6.3-bin-hadoop2.4/conf/
[root@hadoop01 conf]# cp slaves.template slaves
[root@hadoop01 conf]# vi slaves
hadoop01
hadoop02
hadoop03

2，配置spark-env.sh

[root@hadoop01 conf]# cp spark-env.sh.template spark-env.sh
[root@hadoop01 conf]# vi spark-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_79
export SPARK_MASTER_IP=hadoop01
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=4
export SPARK_WORKER_INSTANCES=4
export SPARK_WOR