SparkStreaming读取Hdfs文件完成wordCount

首先完成逻辑代码如下

import org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.DStream
import org.apache.spark.streaming.{Seconds, StreamingContext}

object ReadHdfsFileDemo extends App {
  private val conf = new SparkConf().setAppName("test").setMaster("local[*]")
  private val ssc = new StreamingContext(conf,Seconds(5))
  //写一个hdfs上的文件夹路径即可
  private val dstream: DStream[String] = ssc.textFileStream("hdfs://mypc01/sparkstream")
  private val value: DStream[(String, Int)] = dstream.flatMap((_.split(" "))).map((_, 1)).reduceByKey(_ + _)
  value.print()
  ssc.start()
  ssc.awaitTermination()
}

之后往hdfs中添加文件

 hdfs dfs -put b.txt /sparkstream

执行结果

(everything;they,1)
(enough,1)
(can,1)
(dream;go,1)
(heartaches.,1)
(dreams,1)
(want,6)

总结

• 用put命令往hdfs上上传文件
• hdfs上的文件路径要写正确