Spark Streaming 写kafka报错:kafka.cluster.BrokerEndPoint cannot be cast to kafka.cluster.Broker

 写了一个spark 读写kafka任务，直接在idea 以local模式运行直接报错

报错:Caused by: java.lang.ClassNotFoundException: kafka.cluster.BrokerEndPoint

Exception in thread "main" java.lang.NoClassDefFoundError: kafka/cluster/BrokerEndPoint
	at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$2$$anonfun$3$$anonfun$apply$6$$anonfun$apply$7.apply(KafkaCluster.scala:97)
	at scala.Option.map(Option.scala:146)
	at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$2$$anonfun$3$$anonfun$apply$6.apply(KafkaCluster.scala:97)
	at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$2$$anonfun$3$$anonfun$apply$6.apply(KafkaCluster.scala:94)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:252)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$2$$anonfun$3.apply(KafkaCluster.scala:94)
	at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$2$$anonfun$3.apply(KafkaCluster.scala:93)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252)
	at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:252)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
	at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$2.apply(KafkaCluster.scala:93)
	at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$2.apply(KafkaCluster.scala:92)
	at scala.util.Either$RightProjection.flatMap(Either.scala:522)
	at org.apache.spark.streaming.kafka.KafkaCluster.findLeaders(KafkaCluster.scala:92)
	at org.apache.spark.streaming.kafka.KafkaCluster.getLeaderOffsets(KafkaCluster.scala:186)
	at org.apache.spark.streaming.kafka.KafkaCluster.getLeaderOffsets(KafkaCluster.scala:168)
	at org.apache.spark.streaming.kafka.KafkaCluster.getLatestLeaderOffsets(KafkaCluster.scala:157)
	at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$5.apply(KafkaUtils.scala:215)
	at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$5.apply(KafkaUtils.scala:211)
	at scala.util.Either$RightProjection.flatMap(Either.scala:522)
	at org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:211)
	at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:484)
	at com.hxh.spark.project.ImoocStatStreamingApp$.main(ImoocStatStreamingApp.scala:31)
	at com.hxh.spark.project.ImoocStatStreamingApp.main(ImoocStatStreamingApp.scala)
Caused by: java.lang.ClassNotFoundException: kafka.cluster.BrokerEndPoint
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 31 more

Process finished with exit code 1

后来查看kafka包

 

       发现确实没有BrokerEndPoint，然后百度查看了一下，发现只有0.9版本才有。

       反复修改pom.xml文件，打包运行。错误就仍然照旧同样一条。修改pom.xml文件中，查看了下kafka0.9比0.8多了BrokerEndPoint类，可问题是maven的都是0.8版的。
      后来再到官网去看介绍发现

          http://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html

重点：spark-streaming-kafka-0-8_2.11需要2.4.0版本的，而我采用的是2.2.0版本的。