Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSch

最新推荐文章于 2024-07-25 18:31:24 发布

zkeqing

最新推荐文章于 2024-07-25 18:31:24 发布

阅读量1.7k

点赞数

分类专栏： scala 文章标签： scala GenericRowWithSchema cannot be

本文链接：https://blog.csdn.net/zkeqing/article/details/83419263

版权

scala 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

这个功能实现非常简单：

主要代码如下，具体讲的重点，请参看文末。

case class GeoDistrictsWithCnt(geohash_code: String, county_id: Int, cnt:Int)
val makeGeoDistricts = udf[GeoDistrictsWithCnt, String, Int, Int]((geohash_code, county_id, cnt) => new GeoDistrictsWithCnt(geohash_code, county_id, cnt))

val mostFreqValue = udf[GeoDistrictsWithCnt, Seq[GeoDistrictsWithCnt]] {
  r=>
    val m = r.maxBy(_.cnt)
    new GeoDistrictsWithCnt(m.geohash_code, m.county_id, m.cnt)
  }
}


val a = Seq(
  new GeoDistrictsWithCnt("a", 1,2),
  new GeoDistrictsWithCnt("a", 7,22),
  new GeoDistrictsWithCnt("b", 11,12),
  new GeoDistrictsWithCnt("cv", 1,62),
  new GeoDistrictsWithCnt("cv", 5,2),
  new GeoDistrictsWithCnt("a", 11,23),
  new GeoDistrictsWithCnt("va", 1,2),
  new GeoDistrictsWithCnt("cv", 1,62),
  new GeoDistrictsWithCnt("cv", 5,277),
  new GeoDistrictsWithCnt("a", 11,23),
  new GeoDistrictsWithCnt("va", 1,42)
).toDF


val b = a.groupBy("geohash_code", "county_id").agg(sum("cnt").as("cnt")).withColumn("new_fd", makeGeoDistricts(col("geohash_code"), col("county_id"), col("cnt")))
b.show()
+------------+---------+---+----------+
|geohash_code|county_id|cnt|    new_fd|
+------------+---------+---+----------+
|           a|        1|  2|   [a,1,2]|
|           a|        7| 22|  [a,7,22]|
|          cv|        1|124|[cv,1,124]|
|          cv|        5|279|[cv,5,279]|
|           a|       11| 46| [a,11,46]|
|           b|       11| 12| [b,11,12]|
|          va|        1| 44| [va,1,44]|
+------------+---------+---+----------+

val c = b.groupBy("geohash_code").agg(collect_list("new_fd").as("new_fd"))
c.show
+------------+--------------------+
|geohash_code|              new_fd|
+------------+--------------------+
|          cv|[[cv,1,124], [cv,...|
|          va|         [[va,1,44]]|
|           b|         [[b,11,12]]|
|           a|[[a,1,2], [a,7,22...|
+------------+--------------------+


val d = b.groupBy("geohash_code").agg(mostFreqValue(collect_list("new_fd")).as("new_fd"))
d.show
+------------+----------+
|geohash_code|    new_fd|
+------------+----------+
|          cv|[cv,5,279]|
|          va| [va,1,44]|
|           b| [b,11,12]|
|           a| [a,11,46]|
+------------+----------+

代码看起来很正常，写起来也感觉挺流畅的，但是执行会发现，问题来啦。类型转换失败！！！

Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to GeoDistrictsWithCnt
at $anonfun$1$$anonfun$2.apply(<console>:27)
at scala.collection.TraversableOnce$$anonfun$maxBy$1.apply(TraversableOnce.scala:243)
at scala.collection.TraversableOnce$$anonfun$maxBy$1.apply(TraversableOnce.scala:242)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableOnce$class.maxBy(TraversableOnce.scala:242)
at scala.collection.AbstractTraversable.maxBy(Traversable.scala:104)
at $anonfun$1.apply(<console>:27)
at $anonfun$1.apply(<console>:26)

将上述代码段中的 mostFreqValue方法替换为以下部分，即可。

val mostFreqValue = udf[GeoDistrictsWithCnt, Seq[Row]] {
  r => {
    val rows = r.map {
      row =>
        new GeoDistrictsWithCnt(row.getString(0), row.getInt(1), row.getInt(2))
    }.toSeq
    val m = rows.maxBy(_.cnt)
    new GeoDistrictsWithCnt(m.geohash_code, m.county_id, m.cnt)
  }
}