Spark安装与基本操作

Spark安装与基本操作

• spark的下载通过访问spark官网下载自己Hadoop对应的版本吧，
• spark安装包：https://dlcdn.apache.org/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3-scala2.13.tgz
• 然后通过上传工具上传到虚拟里的自己的 目录里，

spark安装

1.进入到 你上传的目录里，解压压缩包

tar -zvxf spark-2.4.0-bin-without-hadoop.tgz        ##spark-2.4.0-bin-without-hadoop.tgz为你自己在官网下载的压缩包

2.将解压的压缩包重命名为spark

 mv spark-2.4.0-bin-without-hadoop spark

3.修改权限

chown -R root:root ./spark

4.修改spark-env.sh.template文件

cd /usr/java/spark/conf    ##进入spark安装路径下的conf文件夹

cp spark-env.sh.template spark-env.sh

5.配置环境变量，进入 /etc/profile ,在文件末尾添加Spark的PATH路径

vim /etc/profile
export SPARK_HOME=/uar/java/spark   ##spark 的安装路径
export PATH=$SPARK_HOME/bin:$PATH

6.运行下面的命令使配置生效

source /etc/profile

7.启动

cd /usr/local/spark/bin/
./run-example SparkPi    #计算pi值
./spark-shell            #启动spark 
scala>:quit              # 退出spark             

运行spark示例

cd /usr/java/spark/bin/
./run-example SparkPi
./run-example SparkPi SparkPi 2>&1 |grep "PI is roughly"

交互分析

./spark-shell
 
 val textFile = sc.textFile("file:///usr/java/spark/README.md")

textFile.count()

textFile.first()

val linesWithSpark = textFile.filter(line =>line.contains("Spark"))

linesWithSpark.count()

textFile.filter(line =>line.contains("Spark")).count()

RDD

textFile.map(line=>line.split(" ").size).reduce((a,b)=>if(a>b) a else b)

import java.lang.Math
textFile.map(line=>line.split(" ").size).reduce((a,b)=>Math.max(a,b))

val wordCounts =textFile.flatMap(line=>line.split(" ")).map(word=>(word,1))reduceByKey((a,b)=>a+b)
wordCounts.collect()

缓存

val sqlContext = spark.sqlContext

val df= sqlContext.read.json("file:///usr/java/spark/examples/src/main/resources/people.json")

df.show()
df.select("name").show()
df.select(df("name"),df("age")+1).show()
df.filter(df("age")>21).show()
df.groupBy("age").count().show()

df.registerTempTable("people")
val result = sqlContext.sql("SELECT name,age FROM people WHERE age>=13 AND age<=19")
result.show()