spark Graphx图计算java.lang.AbstractMethodError
spark.graphx.GraphOps.connectedComponents
项目场景:
用spark Graphx进行图计算
问题描述:
图计算报异常如下
Exception in thread "main" java.lang.AbstractMethodError
at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
at org.apache.spark.graphx.Pregel$.initializeLogIfNecessary(Pregel.scala:56)
at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
at org.apache.spark.graphx.Pregel$.log(Pregel.scala:56)
at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
at org.apache.spark.graphx.Pregel$.logInfo(Pregel.scala:56)
at org.apache.spark.graphx.Pregel$.apply(Pregel.scala:148)
at org.apache.spark.graphx.lib.ConnectedComponents$.run(ConnectedComponents.scala:54)
at org.apache.spark.graphx.lib.ConnectedComponents$.run(ConnectedComponents.scala:73)
at org.apache.spark.graphx.GraphOps.connectedComponents(GraphOps.scala:434)
at com.bjsxt.scalaspark.sparkGraphx.HelloGraphx$.main(HelloGraphx.scala:42)
at com.bjsxt.scalaspark.sparkGraphx.HelloGraphx.main(HelloGraphx.scala)
原因分析:
spark-graphx对应版本和其他组件版本不一致
解决方案:
将版本都统一为2.3.1<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-graphx_2.11</artifactId>
<version>2.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.3.1</version>
</dependency>
<!--SparkStreaming-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.3.1</version>
<!--<scope>provided</scope>-->
</dependency>
测试demo见:https://blog.csdn.net/weixin_43654098/article/details/83994882