最近需要搭建单机版sprak环境,分享一下:
1.首先准备软件包
下载地址:https://www.apache.org/dyn/closer.lua/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz
2.解压到对应目录后,修改配置文件(前提Java已经安装过了)
解压的Linux命令:tar -zxvf spark-2.4.3-bin-hadoop2.7.tgz
3.配置环境变量
a.进入到解压的目录,找到conf文件夹 slaves.template 去掉templat后缀,为slaves,进入slaves末尾添加localhost保存
或者copy一份也是可以的 cp slaves.template slaves
修改内容为:
b. spark-env.sh.template 去掉template后缀为 spark-env.sh进入spark-env.sh ,
export JAVA_HOME=/home/software/soft/jdk1.8.0_11 (改为自己本机地址)
#export SCALA_HOME=/usr/local/scala/scala-2.11.8
export HADOOP_HOME=/home/software/soft/hadoop-2.7.0
export HADOOP_CONF_DIR=/home/software/soft/hadoop-2.7.0/etc/hadoop
export YARN_CONF_DIR=/home/software/soft/hadoop-2.7.0/etc/hadoop
export SPARK_MASTER_IP=hadoop2
export SPARK_LOCAL_IP=192.168.183.162
export SPARK_WORKER_MEMORY=4g
export SPARK_HOME=/home/software/soft/spark-2.4.3-bin-hadoop2.7
export SPARK_LOCAL_DIRS=/home/software/soft/data/spark
copy一份spark-env.sh:cp spark-env.sh.template spark-env.sh
c.修改 /etc/hosts vi /etc/hosts添加127.0.0.1 bogon localhost (bogon根据主机名更改)
至此spark单机已经安装Ok了,接下来启动一下,检测一下:
sbin/start-all.sh
访问 测试:
Ok,至此轻松搞定
如果需要集群的话,和hadoop类似,把文件copy过去,在slaves加入对应的worker就可以了