flume的Wordcount

最新推荐文章于 2021-08-21 18:00:17 发布

mine_9999

最新推荐文章于 2021-08-21 18:00:17 发布

阅读量104

点赞数

本文链接：https://blog.csdn.net/mine_9999/article/details/103176786

版权

启动flume ：
进入flume里
bin/flume-ng agent -c conf -f conf/spark-flume-conf.properties -n a1 -Dflume.root.logger=INFO,consloe
object FlumeWordCount{
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName(“wc”).setMaster(“local[2]”)
val context = new StreamingContext(conf,Seconds(10));
val value = FlumeUtils.createPollingStream(context,“192.168.56.101”,8888)
val line = value.map(x=>{new String(x.event.getBody.array())})
line.flatMap(.split(" ")).map((,1)).reduceByKey(+).print()
context.start()
context.awaitTermination()
}
}