spark DataFrame 相关操作简记

创建Dataset和Dataframe
最简单的方式就是使用spark.range方法来创建一个Dataset
val DS = spark.range(5, 100, 5)

创建DataFrame
val DF = spark.createDataFrame(List(("Scala", 35), ("Python", 30), ("R", 15), ("Java", 20)))  //创建
val DF1 = DF.withColumnRenamed("_1", "language").withColumnRenamed("_2", "percent")  //列重命名
或者
val DF = spark.createDataFrame(List(("330903198111185759", 179, 0, 10), ("330902196010220222", 0, 1, 20), ("330902196011060646", 0, 1, 0), ("32102319460121285X", 0, 0, 0),

                              ("33090219601028762X", 0, 1, 0), ("330902196010210235", 0, 1, 0), ("330901195608122917", 1, 0, 0))).toDF("id", "vec1", "vec2", "vec3")

通过rdd创建

val df = spark.createDataFrame(rdd).toDF("attr1_name","attr2_name")

读取csv文件
val DF = spark.read.option("header","true").csv("/usr/wc.csv")
读取json文件
val DF = spark.read.json("/usr/wc.json")
创建临时表并使用spark sql来操作
scala> DF.createOrReplaceTempView("zips_table")    
scala> DF.cache()  
scala> val resultsDF = spark.sql("SELECT * FROM zips_table")