【一】基于Ubuntu16.04环境编译Spark源码及安装
先确定已经编译好了源码,并且安装好了local模式。(每台机器都要安装)
node1 | Master |
node2 | worker |
node3 | worker |
node4 | worker |
cd /app/spark/spark-2.2.0-bin-2.9.0/conf
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
SPARK_MASTER_HOST=node1
SPARK_MASTER_PORT=7077
JAVA_HOME=/app/java/jdk1.8.0_101
cp slaves.template slaves
vi slaves
这里是指明worker机器所在节点的IP
node2
node3
node4
启动(只用在master上运行)
cd /app/spark/spark-2.2.0-bin-2.9.0/sbin
./start-all.sh
jps查看
node1
node2
node3
node4
UI
standalone模式使用
cd /app/spark/spark-2.2.0-bin-2.9.0/bin
./spark-shell --master spark://node1:7077