spark单机版本搭建

spark单机版本搭建

服务器配置

CPU	内存	磁盘	操作系统	内核版本
2c	2g	40G	CentOS 7	3.10.0

1.JDK 下载安装

# 列出版本
yum -y list java*
# 安装
yum -y install java-1.8.0-openjdk   java-1.8.0-openjdk-devel
# 检查
java -version

2.spark下载

spark下载地址
本文下载：spark-3.1.3-bin-hadoop3.2.tgz

# 方式一：下载到本地后，上传到服务器/opt/soft/spark目录下
# 方式二：找到对应版本，复制下载链接
cd /opt/soft/spark
wget https://archive.apache.org/dist/spark/spark-3.1.3/spark-3.1.3-bin-hadoop3.2.tgz

# 解压
cd /opt/soft/spark
tar -zxvf spark-3.1.3-bin-hadoop3.2.tgz -C ./

# 配置环境变量
vi /etc/profile
#添加
export SPARK_HOME=/opt/soft/spark/spark-3.1.3-bin-hadoop3.2
export PATH=$SPARK_HOME/bin:$PATH

输入:wq保存退出
# 生效环境变量
source /etc/profile

# 验证：输入spark- 按Tab键，出现其他命令说明配置正确
root@localhost:~# spark-
spark-class       spark-class2.cmd  spark-shell       spark-sql         spark-submit 

3. 使用spark-shell

执行spark-shell命令

root@localhost:~# spark-shell

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://iZuf63p0fp2fo7bc8x8w8oZ:4040
Spark context available as 'sc' (master = local[*], app id = local-1739020529368).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.3
      /_/
         
Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_442)
Type in expressions to have them evaluated.
Type :help for more information.

scala> 

4.使用spark sql

执行spark-sql命令

root@localhost:~# spark-sql

spark-sql> create table test(col1 STRING, col2 STRING) USING parquet;
Time taken: 0.071 seconds
spark-sql> insert into test(col1, col2) values('ss','hh');
Time taken: 0.224 seconds
spark-sql> select * from test;
ss      hh
Time taken: 0.375 seconds, Fetched 1 row(s)
spark-sql> quit;