Spark Standalone安装

Spark Standalone安装

0.基本信息

	CentOS7
	jdk-8u181
	spark-2.4.4-bin-hadoop2.7.tgz
	主机
		192.168.1.101
		192.168.1.102

1.免密码登录配置

	0.登录到主节点
	1.ssh-keygen -t rsa
	2.cd ~/.ssh/
	3.cat id_rsa.pub >> authorized_keys
	
	4.authorized_keys拷贝到子节点
	
	5.登录子节点 cd ~/.ssh/
	6.cat id_rsa.pub >> authorized_keys
	7.authorized_keys拷贝到主机点

2.Spark配置spark-2.4.4-bin-hadoop2.7

conf/spark-env.sh
	SPARK_MASTER_HOST=192.168.1.102
	SPARK_MASTER_PORT=7077
conf/slaves
	192.168.1.101
	192.168.1.102

3.同步文件

spark-2.4.4-bin-hadoop2.7同步到从节点172.17.0.2

4.启动

主节点/opt/spark/spark-2.4.4/sbin/start-all.sh

5.验证

bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://192.168.1.102:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
./examples/jars/spark-examples_2.11-2.4.4.jar \
100

6.停止

主节点/opt/spark/spark-2.4.4/sbin/start-all.sh