目录
RDD、DF、DS三者之间的转化
转换成RDD .rdd
转换成DF .toDF()
转换成DS
RDD->DS .toDS()
DF->DS .as[Person]
Spark SQL自定义函数
spark.udf.register("toUpperAdd123",(str:String)=>{
//根据业务逻辑对数据进行处理
//str.toUpperCase()+ " 123"
//str.length*10
str.length*10/2/2.toDouble
})
开窗函数的作用
既显示聚集前/排序前的原始的数据,又显示聚集后/排序后的名次 的数据。
开窗函数的分类
-
聚和开窗函数
聚合函数(列) OVER(选项),这里的选项可以是PARTITION BY 子句
-
排序聚和函数
排序函数(列) OVER(选项),这里的选项可以是ORDER BY 子句,也可以是 OVER(PARTITION BY 子句 ORDER BY 子句)
聚和开窗函数
select name, class, score, count(name) over() name_count from scores
select name, class, score, count(name) over(PARTITION BY class) name_count from scores
排序聚和函数
select name, class, score, row_number() over(order by score) rank from scores
select name, class, score, row_number() over(partition by class order by score) rank from scores
RANK跳跃排序
select name, class, score, rank() over(order by score) rank from scores
select name, class, score, rank() over(partition by class order by score) rank from scores
DENSE_RANK连续排序
select name, class, score, dense_rank() over(order by score) rank from scores
NTILE分组排名
select name, class, score, ntile(6) over(order by score) rank from scores