Spark streaming整合Kafka之Direct方式

1.spark streaming应用程序编写

import org.apache.spark.SparkConf
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}
import kafka.serializer.StringDecoder

object KafkaDirectWordCount {
  def main(args: Array[String]): Unit = {

        if(args.length != 2) {
          System.err.println("Usage: KafkaDirectWordCount <brokers> <topics>")
          System.exit(1)
        }

        val Array(brokers, topics) = args

        val sparkConf = new SparkConf().setAppName("KafkaDirectWordCount").setMaster("local[2]")

        val ssc = new StreamingContext(sparkConf, Seconds(5))

        val topicsSet = topics.split(",").toSet
        val kafkaParams = Map[String,String]("metadata.broker.list"-> brokers)

        val messages = KafkaUtils.createDirectStream[String,String,StringDecoder,StringDecoder](
        ssc,kafkaParams,topicsSet
        )

        messages.map(_._2).flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).print()

        ssc.start()
        ssc.awaitTermination()
      }
}

2.本地环境测试

启动zookeeper，kafka，kafak producer

输入brokers和topics值，启动应用程序，发现报错

Exception in thread "main" java.lang.ClassCastException: kafka.cluster.BrokerEndPoint cannot be cast to kafka.cluster.Broker

经排查，这是由于版本不兼容造成的，更改pom.xml中的依赖，将kafka版本改为0.8.2.1，问题解决

启动应用程序后，在服务器输入测试数据：a a b b c c c

得到结果

3.服务器测试

用maven将应用程序打包，上传到服务器

启动spark-submit，在kafka producer中输入测试数据：a a b b c c c

得到结果