首先完成逻辑代码如下
import org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.DStream
import org.apache.spark.streaming.{Seconds, StreamingContext}
object ReadHdfsFileDemo extends App {
private val conf = new SparkConf().setAppName("test").setMaster("local[*]")
private val ssc = new StreamingContext(conf,Seconds(5))
//写一个hdfs上的文件夹路径即可
private val dstream: DStream[String] = ssc.textFileStream("hdfs://mypc01/sparkstream")
private val value: DStream[(String, Int)] = dstream.flatMap((_.split(" "))).map((_, 1)).reduceByKey(_ + _)
value.print()
ssc.start()
ssc.awaitTermination()
}
之后往hdfs中添加文件
hdfs dfs -put b.txt /sparkstream
执行结果
(everything;they,1)
(enough,1)
(can,1)
(dream;go,1)
(heartaches.,1)
(dreams,1)
(want,6)
总结
- 用put命令往hdfs上上传文件
- hdfs上的文件路径要写正确