/**
* DataFrame中的操作操作
*/
object DataFrameCase {
def main(args: Array[String]) {
val spark = SparkSession.builder().appName("DataFrameRDDApp").master("local[2]").getOrCreate()
// RDD ==> DataFrame
val rdd = spark.sparkContext.textFile("file:///Users/rocky/data/student.data")
//注意:需要导入隐式转换
import spark.implicits._
val studentDF = rdd.map(_.split("\\|")).map(line => Student(line(0).toInt, line(1), line(2), line(3))).toDF()
//show默认只显示前20条
studentDF.show
studentDF.show(30)
studentDF.show(30, false)
studentDF.take(10)
studentDF.first()
studentDF.head(3)
studentDF.select("email").show(30,false)
studentDF.filter("name=''").show
studentDF.filter("name='' OR name='NULL'").show
//name以M开头的人
studentDF.filter("SUBSTR(name,0,1)='M'").show
studentDF.sort(studentDF("name")).show
studentDF.sort(studentDF("name").desc).show
studentDF.sort("name","id").show
studentDF.sort(studentDF("name").asc, studentDF("id").desc).show
studentDF.select(studentDF("name").as("student_name")).show
val studentDF2 = rdd.map(_.split("\\|")).map(line => Student(line(0).toInt, line(1), line(2), line(3))).toDF()
studentDF.join(studentDF2, studentDF.col("id") === studentDF2.col("id")).show
spark.stop()
}
case class Student(id: Int, name: String, phone: String, email: String)
}
DataFrame它不是Spark SQL提出的,而是早起在R、Pandas语言就已经有了的。
A Dataset is a distributed collection of data:分布式的数据集
A DataFrame is a Dataset organized into named columns.
以列(列名、列的类型、列值)的形式构成的分布式数据集,按照列赋予不同的名称
student
id:int
name:string
city:string
It is conceptually equivalent to a table in a relational database
or a data frame in R/Python
RDD:
java/scala ==> jvm
python ==> python runtime
DataFrame:
java/scala/python ==> Logic Plan
DataFrame和RDD互操作的两种方式:
1)反射:case class 前提:事先需要知道你的字段、字段类型
2)编程:Row 如果第一种情况不能满足你的要求(事先不知道列)
3) 选型:优先考虑第一种
val rdd = spark.sparkContext.textFile("file:///home/hadoop/data/student.data")
DataFrame = Dataset[Row]
Dataset:强类型 typed case class
DataFrame:弱类型 Row
SQL:
seletc name from person; compile ok, result no
DF:
df.select("name") compile no
df.select("nname") compile ok
DS:
ds.map(line => line.itemid) compile no