记录一次消费kafka数据报错
我是用spark消费kafka中的数据时报了如下错误
Exception in thread "streaming-start" java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V
我的scala代码是这样的
package com.study.spark.project
import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe
import org.apache.spark.streaming.kafka010.KafkaUtils
import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent
import org.apache.spark.streaming.{Seconds, StreamingContext}
object StatStreamingApp {
def main(args: Array[String]):Unit = {
val ssc = new StreamingContext("local[*]","StatStreamingApp",Seconds(5))
val kafkaParams = Map[String, Object](
"bootstrap.servers" -> " 192.168.192.128:9092",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer],
"group.id" -> "test",
"auto.offset.reset" -> "latest",
"enable.auto.commit" -> (false: java.lang.Boolean)
)
val topics = Array("topic")
val kafkaStream = KafkaUtils.createDirectStream[String, String](
ssc,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
).map(_.value())
kafkaStream.print()
ssc.start()
ssc.awaitTermination()
}
}
问题解决:
导入的依赖错了
原有依赖:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.0</version>
</dependency>
版本换成0.10.0.0的就好了