Spark Local模式安装及测试

1、解压文件

解压spark压缩文件到/export/server/目录下(我的是:spark-3.2.0-bin-hadoop3.2.tgz)

(pyspark) [root@node1 export]# tar -zxvf spark-3.2.0-bin-hadoop3.2.tgz  -C /export/server/

2、配置软连接(也可以不配置，配置是为了输入这个文件夹是比较方便)

(pyspark) [root@node1 server]# ln -s /export/server/spark-3.2.0-bin-hadoop3.2 /export/server/spark

 3、配置系统环境变量

(pyspark) [root@node1 server]# vim /etc/profile

插入以下内容

export JAVA_HOME=/export/server/jdk1.8.0_241

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export HADOOP_HOME=/export/server/hadoop-3.3.0

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export SPARK_HOME=/export/server/spark

export PYSPARK_PYTHON=/export/server/anaconda3/envs/pyspark/bin/python3.8

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin

  4、配置bashrc环境变量

(pyspark) [root@node1 server]# vim ~/.bashrc

插入以下内容:

export JAVA_HOME=/export/server/jdk1.8.0_241

export PYSPARK_PYTHON=/export/server/anaconda3/envs/pyspark/bin/python3.8

5、启动测试Pyspark(/export/server/spark/bin目录下执行)

(pyspark) [root@node1 bin]# ./pyspark

 出现以上界面说明启动成功

页面中出现的网址代表监控页面的链接:http://node1:4040