报错如下:
Serialization stack:
- object not serializable (class: java.lang.Object, value: java.lang.Object@1430d9d9)
- field (class: com.yeahmobi.dmp.export.Hive2Es_user_bundle_country_many$$anonfun$run$2, name: nonLocalReturnKey1$1, type: class java.lang.Object)
- object (class com.yeahmobi.dmp.export.Hive2Es_user_bundle_country_many$$anonfun$run$2, <function1>)
- element of array (index: 4)
- array (class [Ljava.lang.Object;, size 8)
- field (class: org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10, name: references$1, type: class [Ljava.lang.Object;)
- object (class org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10, <function2>)
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:342)
... 38 more
我的udf是:
ss.udf.register("getRouteCode",(routeKey:String) => {
if(routeKey.isEmpty){
return 99
}
(routeKey.hashCode() & Integer.MAX_VALUE) % 8
})
修改后的udf是
ss.udf.register("getRouteCode",(routeKey:String) => {
val routeCode: Int =routeKey match {
case null|""=> 99
case _=> {
(routeKey.hashCode() & Integer.MAX_VALUE) % 8
}
}
routeCode
})
错误就是 routeKey.isEmpty 这块的问题,如果要判空,就用模式匹配做,要不就会序列化不了
问题解决