CentOs7 单节点spark测试环境配置

一. 准备工作

• CentOs7 单节点配置spark2.4.7测试环境

• 需要首先安装 jdk 和 Scala

• 参考jdk安装与scala安装

做好准备工作后开始spark安装配置

二.spark安装

1.spark下载

使用wget下载spark源码 可以使用国内镜像源 hadoop版本使用2.6

wget https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.6.tgz

下载完成后解压到指定目录

tar -zxvf spark-2.4.7-bin-hadoop2.6.tgz -C /usr/local/

修改文件夹名

mv spark-2.4.7-bin-hadoop2.6/ spark

2. 配置环境变量

vi /etc/profile

export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin

3. 验证成功

输入spark-shell,如下显示则配置成功

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://192.168.0.102:4040
Spark context available as 'sc' (master = local[*], app id = local-1600786747472).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.7
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_261)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

通过虚拟机ip和端口4040访问管理界面