spark sql算子数据类型

spark sql算子返回值数据类型介绍

初始变量类型DataFrame

• 加入persist()固化存储，运行速度更快

val online_profile_score_df: DataFrame = profile_df.join(score_df, usingColumns =Seq("serv_number","statis_month"))
      .persist()   //线上发展用户+app信息

加入select算子后返回DataFrame

    val a: DataFrame = online_profile_score_df.select("cur_product", "prod_id", "prod_name","statis_month","serv_number")

filter算子后返回Dataset[Row]

    val a: Dataset[Row] = online_profile_score_df.select("cur_product", "prod_id", "prod_name","statis_month","serv_number")
      .filter(s"product_type <> '1'")

所以采用toDF()函数来：Dataset[Row]->DataFrame

    val a: DataFrame = online_profile_score_df.select("cur_product", "prod_id", "prod_name","statis_month","serv_number")
      .filter(s"product_type <> '1'")
      .toDF()

存成hive表后，返回Unit

• $存成hive表为DataFrame的子方法，故统计结果最后务必转成DataFrame$

    val a: Unit = online_profile_score_df.select("cur_product", "prod_id", "prod_name","statis_month","serv_number")
      .filter(s"product_type <> '1'")
      .toDF()
      .write.mode(SaveMode.Overwrite).saveAsTable(tableName = s"online_non_2i2c_app_${this_month}_m")

groupBy算子后返回RelationalGroupedDataset

    val a: RelationalGroupedDataset = online_profile_score_df.select("cur_product", "prod_id", "prod_name","statis_month","serv_number")
      .filter(s"product_type <> '1'")
      .groupBy("cur_product", "prod_id","prod_name","statis_month","serv_number")

$注意:$

• $采用groupBy算子后，后面需要加入聚合函数。才能转换为DataFrame。$

使用agg聚合函数后，正常返回DataFrame

     val a: DataFrame = online_profile_score_df.select("cur_product", "prod_id", "prod_name","statis_month","serv_number")
      .filter(s"product_type <> '1'")
      .groupBy("cur_product", "prod_id","prod_name","statis_month")
        .agg(count("serv_number").alias("p_sum"))

各种聚合算子spark sql完成降序排序，输出排名.

正常返回DataFrame，可存成hive表

     val a: Unit = online_profile_score_df.select("cur_product", "prod_id", "prod_name","statis_month","serv_number")
      .filter(s"product_type <> '1'")
      .groupBy("cur_product", "prod_id","prod_name","statis_month")
        .agg(count("serv_number").alias("p_sum"))
        .sort($"p_sum".desc)
      .withColumn("a_rank", monotonically_increasing_id+1)
        .limit(10)
       .write.mode(SaveMode.Overwrite).saveAsTable(tableName = s"online_non_2i2c_app_${this_month}_m")

输出结果(为保证数据安全，数据为模拟数据)

cur_product	prod_id	prod_name	statis_month	p_sum	a_rank
套餐1	C54185	某app1	201903	999999	1
套餐2	C32158	某app2	201903	77777	2