SPARK安装（HADOOP配置 ubuntu18.04-live-server）

SPARK 安装配置

eg: 版本3.1.1
HADOOP配置HA集群

配置$SPARK_HOME/conf/spark-env.sh文件在其中新增内容

1. 复制模板文件

   cp spark-env.sh.template spark-env.sh
   

2. 添加内容到spark-env.sh中

   # <<<<<< java start
   export JAVA_HOME=/home/hadoop/jdk1.8.0_291
   # >>>>>> java end
   # <<<<<< hadoop start
   export HADOOP_INSTALL=/home/hadoop/hadoop-3.2.2
   export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop
   export YARN_CONFIG_DIR=$HADOOP_INSTALL/etc/hadoop
   # >>>>>> hadoop end
   # <<<<<< spark start
   export SPARK_MASTER_HOST=master
   export SPARK_LOCAL_DIRS=/home/hadoop/spark-3.1.1-bin-hadoop3.2
   export SPARK_DRIVER_MEMORY=4G
   # >>>>>> spark end
   

配置集群节点

1. 复制模板文件

   cp workers.template workers
   

2. 添加内容到workers中,一行为一个节点主机

   master
   node1
   node2
   

复制文件至从节点中

scp -r /home/hadoop/spark-3.1.1-bin-hadoop3.2 hadoop@node1:/home/hadoop/
scp -r /home/hadoop/spark-3.1.1-bin-hadoop3.2 hadoop@node2:/home/hadoop/