sprak 序列化相关错误

在提交spark任务的过程中,如果设置了使用kryo进行序列化,即:

conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
可能会遇到以下异常:
com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
location_ (knowledge.pub.Properties$AccessPlayControl$RegionControl)
regionControl_ (knowledge.pub.Properties$AccessPlayControl)
accessPlayControl_ (knowledge.pub.Knowledge$Episode)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
	at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:127)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
	at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:127)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
原因是因为kryo早期的版本中,对immutable list支持的不太好,这个时候需要引入第三方包:

<dependency>
    <groupId>de.javakaffee</groupId>
    <artifactId>kryo-serializers</artifactId>
    <version>0.42</version>
</dependency>
然后自定义注册类:
import com.esotericsoftware.kryo.Kryo;
import de.javakaffee.kryoserializers.protobuf.ProtobufSerializer;
import org.apache.spark.serializer.KryoRegistrator;

public class MyKryoRegistrator implements KryoRegistrator {
    @Override
    public void registerClasses(Kryo kryo) {
        // Probably should use proto serializer for your proto classes
        kryo.register( Knowledge.Episode.class, new ProtobufSerializer() );

    }
}
然后在conf中注册这个类:
conf.set("spark.kryo.registrator", "com.iqiyi.lego.job.beehive.spark.registrator.MyKryoRegistrator");
可以解决这个问题。

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值