Spark完全分布式安装

2019独角兽企业重金招聘Python工程师标准>>>

安装Spark前需要安装Hadoop，已在VM上安装了三台虚拟机

一、前置条件

1、Hadoop集群，作者已经在VM上安装了三台虚拟机，里面已安装好Hadoop集群

二、所需软件

1、 Scala:2.10

2、Spark:2.20

三、安装Scala(集群上三台机器都需要安装，下面已一台为例)

1、下载scala-2.10.0,并解压到/usr/local/目录，作者下载的scala目录在/home/hadoop/tools

hadoop@Worker1:~$ sudo mv /home/hadoop/tools/scala-2.11.0.tgz /usr/local/

hadoop@Worker1:/usr/local$ sudo tar -zxvf scala-2.11.0.tgz

2、修改~/.bashrc文件，增加SCALA_HOME

hadoop@Worker1:/usr/local$ vim ~/.bashrc

export SCALA_HOME=/usr/local/scala-2.11.0
export JAVA_HOME=/usr/local/bin/jdk1.8.0_131
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
export PATH=${SCALA_HOME}/bin:$PATH
export HADOOP_HOME=/usr/local/hadoop-2.7.3
export PATH=$PATH:${HADOOP_HOME}/bin
export SCALA_HOME=/usr/local/scala-2.11.0

3、使修改生效

hadoop@Worker1:/usr/local$ source ~/.bashrc 

4、验证Scala是否安装成功

hadoop@Worker1:/usr/local$ scala -version

如果出现：Scala code runner version 2.11.0 -- Copyright 2002-2013, LAMP/EPFL，则表示安装成功

5、在其它机器上重复上面的步骤安装成功scala

四、安装Spark集群：Spark集群支持三种模式，分别是Mesos、YARN、Standalone模式，现在作者配置Standalone模式

1、下载spark-2.2.0-bin-hadoop2.7.tgz,并解压

hadoop@Master:/usr/local$ sudo mv /home/hadoop/tools/spark-2.2.0-bin-hadoop2.7.tgz /usr/local/

hadoop@Master:/usr/local$ sudo tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz

2、授权

hadoop@Master:/usr/local$ sudo chown -R hadoop:root ./spark-2.2.0-bin-hadoop2.7

3、bashrc配置spark环境

hadoop@Master:/usr/local$ vim ~/.bashrc 

export SCALA_HOME=/usr/local/scala-2.11.0
export JAVA_HOME=/usr/local/bin/jdk1.8.0_131
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
export PATH=${SCALA_HOME}/bin:$PATH
export HADOOP_HOME=/usr/local/hadoop-2.7.3
export PATH=$PATH:${HADOOP_HOME}/bin
export SCALA_HOME=/usr/local/scala-2.11.0
export SPARK_HOME=/usr/local/spark-2.2.0-bin-hadoop2.7
export PATH=${SPARK_HOME}/bin:${SPARK_HOME}/sbin/sbin:$PATH

4、使配置生效

hadoop@Master:/usr/local$ source ~/.bashrc 

5、进入Spark的conf目录，配置spark-env.sh

hadoop@Master:/usr/local/spark-2.2.0-bin-hadoop2.7/conf$ cp spark-env.sh.template spark-env.sh

hadoop@Master:/usr/local/spark-2.2.0-bin-hadoop2.7/conf$ vim spark-env.sh

export JAVA_HOME=/usr/local/bin/jdk1.8.0_131
export SCALA_HOME=/usr/local/scala-2.11.0
export HADOOP_HOME=/usr/local/hadoop-2.7.3
export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.3/etc/hadoop
export SPARK_MASTER_IP=Master
export SPARK_WORKER_CORES=2
export SPARK_DRIVER_MEMORY=1G
export SPARK_WORKER_MEMORY=1g
export SPARK_EXECUTOR_MEMORY=1g

6、配置slaves

hadoop@Master:/usr/local/spark-2.2.0-bin-hadoop2.7/conf$ cp slaves.template slaves

hadoop@Master:/usr/local/spark-2.2.0-bin-hadoop2.7/conf$ vim slaves

Worker1
Worker2

7、配置spark-defaults.conf

spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://Master:9000/historyserverforSpark
spark.yarn.historyServer.address  Master:18080
spark.history.fs.logDirectory  hdfs:Master:9000/historyserverforSpark

其中historyserverforSpark目录需要手动创建，否则在启动Spark-shell的时候会报错。

8、启动

hadoop@Master:/usr/local/spark-2.2.0-bin-hadoop2.7/sbin$ ./start-all.sh 

 

转载于:https://my.oschina.net/u/729917/blog/1556871