项目中,有个类序列化有问题:
Caused by: java.io.NotSerializableException: entity.Unit
Serialization stack:
- object not serializable (class: entity.Unit, value: Unit(id=1, name=m2))
- writeObject data (class: java.util.ArrayList)
- object (class java.util.ArrayList, [Unit(id=1, name=m2), Unit(id=7, name=m3), Unit(id=8, name=kg), Unit(id=37, name=m)])
- field (class: service.impl.ORToBSMap, name: units, type: interface java.util.List)
- object (class service.impl.ORToBSMap, tz.lion.service.impl.ORToBSMap@74130456)
- field (class: org.apache.spark.sql.execution.MapElementsExec, name: func, type: class java.lang.Object)
- object (class org.apache.spark.sql.execution.MapElementsExec, MapElements tz.lion.service.impl.ORToBSMap@74130456, obj#37: tz.lion.entity.BuildingStruct
+- DeserializeToObject createexternalrow(floor#21.toString, struct#24.toString, StructField(floor,StringType,true), StructField(struct,StringType,true)), obj#36: org.apache.spark.sql.Row
+- HashAggregate(keys=[floor#21, struct#24], functions=[], output=[floor#21, struct#24])
+- Exchange hashpartitioning(floor#21, struct#24, 200)
+- *(1) HashAggregate(keys=[floor#21, struct#24], functions=[], output=[floor#21, struct#24])
+- *(1) Project [floor#21, struct#24]
+- *(1) Sort [id#22L ASC NULLS FIRST], true, 0
+- Exchange rangepartitioning(id#22L ASC NULLS FIRST, 200)
+- LocalTableScan [floor#21, id#22L, struct#24]
SparkConf的配置是正确的:
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.registerKryoClasses(new Class[]{Unit.class})
跟踪kryo日志,注册成功了
00:04 TRACE: [kryo] Register class ID 39: entity.Unit (com.esotericsoftware.kryo.serializers.FieldSerializer)
后来发现,广播数据,就不会报错了
final JavaSparkContext sc = new JavaSparkContext(session.sparkContext());
final Broadcast<List<Unit>> bcUnit = sc.broadcast(unitRepository.loadAll());
跟踪kryo日志,序列化正常:
00:04 DEBUG: [kryo] Write: Unit(id=1, name=m2)
00:04 TRACE: [kryo] FieldSerializer.write fields of class: entity.Unit
00:04 TRACE: [kryo] Write field: id (entity.Unit) pos=25
00:04 TRACE: [kryo] Write: 1
00:04 TRACE: [kryo] Write field: name (entity.Unit) pos=27
00:04 TRACE: [kryo] Write initial object reference 2: m2
00:04 TRACE: [kryo] Write: m2