spark自带的example中就有streaming结合kafka使用的案例:
$SPARK_HOME/examples/src/main/scala/org/apache/spark/examples/streaming/KafkaWordCount.scala
使用方法参见代码描述:
Usage: KafkaWordCount <zkQuorum> <group> <topics> <numThreads> <zkQuorum> is a list of one or more zookeeper servers that make quorum <group> is the name of kafka consumer group <topics> is a list of one or more kafka topics to consume from <numThreads> is the number of threads the kafka consumer should use Example: `$ bin/run-example \ org.apache.spark.examples.streaming.KafkaWordCount zoo01,zoo02,zoo03 \ my-consumer-group topic1,topic2 1`
运行步骤:
1、启动ZK
zkServer.sh start
2、启动KAFKA SERVER
kafka-server-start.sh $KAFKA_HOME/config/server.properties &
3、运行Producer
run-example org.apache.spark.examples.streaming.KafkaWordCountProducer hadoop000:9092 test 3 5
参数描述:
hadoop000:9092表示producer的地址和端口;
test表示topic;
3表示每秒发多少条消息;
5表示每条消息中有几个单词;
4、运行Consumer
run-example org.apache.spark.examples.streaming.KafkaWordCount hadoop000:2181 test-consumer-group test 1
参数描述:
hadoop000:2181表示zookeeper的监听地址;
test-consumer-group表示consumer-group的名称,必须和$KAFKA_HOME/config/consumer.properties中的group.id的配置内容一致;
test表示topic;
1表示线程数;
注意观察consumer控制台的数据输出,类似于下面的输出:
------------------------------------------- Time: 1410483028000 ms ------------------------------------------- (4,467) (8,463) (6,467) (0,475) (2,401) (7,464) (5,447) (9,419) (3,430) (1,452)
注意:
1、运行该案例的时候不需要启动spark;
2、我已经将$KAFKA_HOME/bin和$SPARK_HOME/bin添加到系统环境变量中,故在任意路径均可以执行运行步骤的脚本,如果没配置到环境变量,需要指定路径再执行脚本。
请转载的同学表明转载地址, 谢谢!