spark部署

安装

官网地址：http://spark.apache.org/downloads.html 、 https://www.scala-lang.org/download/
下载：举例

【SPARK】
Download Apache Spark™
    Choose a Spark release: 
	Choose a package type:
Download Spark: spark-2.4.1-bin-hadoop2.7.tgz
Verify this release using the 2.4.1 signatures, checksums and project release KEYS.
Spark下载后解压即可，master配置完毕，slave节点直接scp同步即可。

【SCALA】
Other ways to install Scala
	Download the Scala binaries for [UNIX](此处翻墙后才可看到UNIX字样，并可点击下载二进制包)
Scala下载后解压，配置环境变量即可，检查是否可用执行命令：scala。

配置

环境变量

[root@config]# vi ~/.bashrc
## SPARK ##
export SPARK_HOME=/usr/local/spark-2.3.0
export PATH=$SPARK_HOME/bin:$PATH

## SCALA ##
export PATH=/usr/local/scala/bin:$PATH

spark-env.sh

[root@config]# vi conf/spark-env.sh
export JAVA_HOME=/home/java
export SCALA_HOME=/usr/local/scala
export HADOOP_HOME=/home/hadoop
export HADOOP_CONF_DIR=/home/hadoop/etc/hadoop
export SPARK_MASTER_IP=test215
export SPARK_WORKER_WEBUI_PORT=18081
export SPARK_LOCAL_HOSTNAME=`hostname`
export SPARK_WORKER_MEMORY=10g
export SPARK_DAEMON_MEMORY=1024M
export SPARK_WORKER_CORES=32
export SPARK_WORKER_INSTANCES=1

spark-defaults.conf

spark.default.parallelism       64  #并行度

slaves

# 节点列表

启动

# ${SPARK_HOME}/sbin/start-all.sh
# ${SPARK_HOME}/sbin/stop-all.sh