在提交spark任务的过程中,如果设置了使用kryo进行序列化,即:
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
可能会遇到以下异常:
com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
location_ (knowledge.pub.Properties$AccessPlayControl$RegionControl)
regionControl_ (knowledge.pub.Properties$AccessPlayControl)
accessPlayControl_ (knowledge.pub.Knowledge$Episode)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:127)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:127)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
原因是因为kryo早期的版本中,对immutable list支持的不太好,这个时候需要引入第三方包:
<dependency>
<groupId>de.javakaffee</groupId>
<artifactId>kryo-serializers</artifactId>
<version>0.42</version>
</dependency>
然后自定义注册类:
import com.esotericsoftware.kryo.Kryo; import de.javakaffee.kryoserializers.protobuf.ProtobufSerializer; import org.apache.spark.serializer.KryoRegistrator;
public class MyKryoRegistrator implements KryoRegistrator { @Override public void registerClasses(Kryo kryo) { // Probably should use proto serializer for your proto classes kryo.register( Knowledge.Episode.class, new ProtobufSerializer() ); } }然后在conf中注册这个类:
conf.set("spark.kryo.registrator", "com.iqiyi.lego.job.beehive.spark.registrator.MyKryoRegistrator");可以解决这个问题。