code
val linkPairSum = F.udf(
(list:List[Map[Long,Int]]) => {
var map = Map[Long,Int]()
for(m <- list){
if(m != null){
println("----")
println(m)
map = map ++ m.map(t => t._1 -> (t._2 + map.getOrElse(t._1, 0)))
println("====")
println(map)
}
}
map
}
)
val sum = all.groupBy("window", "mapVersion", "linkId")
.agg(F.sum("passthrough").alias("passthrough"),F.sum("resident").alias("resident"),F.first("driverId").alias("driverId"),
linkPairSum(F.collect_list("inLink")).alias("inLink"),linkPairSum(F.collect_list("outLink")).alias("outLink")
).as[PassThroughFeature]
报错
Caused by: java.lang.ClassCastException: scala.collection.mutable.WrappedArray$ofRef cannot be cast to [Lscala.collection.immutable.Map;
原因
So it looks like the ArrayType on Dataframe "idDF" is really a WrappedArray and not an Array - So the function call to "filterMapKeysWithSet" failed as it expected an Array but got a WrappedArray/ Seq instead (which doesn't implicitly convert to Array in Scala 2.8 and above).
修改
val linkPairSum = F.udf(
(list:List[Map[Long,Int]]) => {
var map = Map[Long,Int]()
for(m <- list){
if(m != null){
println("----")
println(m)
map = map ++ m.map(t => t._1 -> (t._2 + map.getOrElse(t._1, 0)))
println("====")
println(map)
}
}
map
}
)
修改为:
val linkPairSum = F.udf(
(list:Seq[Map[Long,Int]]) => {
var map = Map[Long,Int]()
for(m <- list){
if(m != null){
println("----")
println(m)
map = map ++ m.map(t => t._1 -> (t._2 + map.getOrElse(t._1, 0)))
println("====")
println(map)
}
}
map
}
)