错误日志
Exception in thread "adaptive-query-stage-0" java.lang.NoSuchMethodError: com.esotericsoftware.kryo.Kryo.setInstantiatorStrategy(Lorg/objenesis/strategy/InstantiatorStrategy;)V
at com.twitter.chill.KryoBase.setInstantiatorStrategy(KryoBase.scala:96)
at com.twitter.chill.EmptyScalaKryoInstantiator.newKryo(ScalaKryoInstantiator.scala:59)
at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:97)
at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:309)
问题分析
- KryoBase的setInstantiatorStrategy 方法参数类型为
org.objenesis.strategy.InstantiatorStrategy
,super调用Kryo.setInstantiatorStrategy() 方法,但是该方法的参数类型为com.esotericsoftware.shaded.org.objenesis.strategy.InstantiatorStrategy
。 - 通过反编译
com.esotericsoftware.kryo.Kryo
发现源码中确实是在引用com.esotericsoftware.shaded.org.objenesis.strategy.InstantiatorStrategy
类,太坑了. - 和官方给的源码都不一致。在官方发布的包中出现引用shaded的类,导致chill_2.11-0.8.4.jar 和Kryo中的方法冲突。看来官方还是推荐使用kryo-shaded包,但是kryo-shaded 中jar的代码被打包时搞乱了,别人也没法用~~~
- 通过对比发现新的spark 使用了kryo-shaded-3.0.3.jar ,所以删除 kryo-2.22.jar 包,问题解决。
$ sm -d com.esotericsoftware.kryo.Kryo setInstantiatorStrategy
declaring-class com.twitter.chill.KryoBase
method-name setInstantiatorStrategy
modifier public
annotation
parameters org.objenesis.strategy.InstantiatorStrategy
return void
exceptions
declaring-class com.esotericsoftware.kryo.Kryo
method-name setInstantiatorStrategy
modifier public
annotation
parameters com.esotericsoftware.shaded.org.objenesis.strategy.InstantiatorStrategy
return void
exceptions
org.objenesis.strategy.InstantiatorStrategy
$ sc -d *InstantiatorStrategy
class-info com.esotericsoftware.shaded.org.objenesis.strategy.InstantiatorStrategy
code-source /usr/lib/spark-2.3.0-bin-ae/jars/kryo-2.22.jar
name com.esotericsoftware.shaded.org.objenesis.strategy.InstantiatorStrategy
isInterface true
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name InstantiatorStrategy
modifier abstract,interface,public
annotation
interfaces
super-class
class-loader +-sun.misc.Launcher$AppClassLoader@3cd1a2f1
+-sun.misc.Launcher$ExtClassLoader@d6da883
classLoaderHash 3cd1a2f1
class-info org.objenesis.strategy.InstantiatorStrategy
code-source /usr/lib/spark-2.3.0-bin-ae/jars/objenesis-2.1.jar
name org.objenesis.strategy.InstantiatorStrategy
isInterface true
isAnnotation false
isEnum false
isAnonymousClass false
isArray false
isLocalClass false
isMemberClass false
isPrimitive false
isSynthetic false
simple-name InstantiatorStrategy
modifier abstract,interface,public
annotation
interfaces
super-class
class-loader +-sun.misc.Launcher$AppClassLoader@3cd1a2f1
+-sun.misc.Launcher$ExtClassLoader@d6da883
classLoaderHash 3cd1a2f1
## JAD 反编译的源码
jad com.esotericsoftware.kryo.Kryo
import com.esotericsoftware.shaded.org.objenesis.instantiator.ObjectInstantiator;
import com.esotericsoftware.shaded.org.objenesis.strategy.InstantiatorStrategy;
[hadoop@hostname jars]$ ll chill*
-rw-r----- 1 hadoop hadoop 223573 7月 5 2018 chill_2.11-0.8.0.jar
-rw-r----- 1 hadoop hadoop 50619 7月 5 2018 chill-java-0.8.0.jar
[hadoop@hostname jars]$ ll kryo-shaded-3.0.3.jar
-rw-r----- 1 hadoop hadoop 358390 7月 5 2018 kryo-shaded-3.0.3.jar
[hadoop@hostname jars]$ ll chill*
-rw-r--r-- 1 hadoop hadoop 224167 1月 16 10:57 chill_2.11-0.8.4.jar
-rw-r--r-- 1 hadoop hadoop 60483 1月 16 10:57 chill-java-0.8.4.jar
[hadoop@hostname jars]$ ll kryo-*
-rw-r--r-- 1 hadoop hadoop 442805 1月 16 10:57 kryo-2.22.jar
-rw-r--r-- 1 hadoop hadoop 358390 1月 16 10:57 kryo-shaded-3.0.3.jar