spark从入门到放弃三: worldcount-scala

最新推荐文章于 2022-01-23 19:12:09 发布

WQ同学

最新推荐文章于 2022-01-23 19:12:09 发布

阅读量3k

点赞数

分类专栏： spark 文章标签： spark scala idea

本文链接：https://blog.csdn.net/u012957549/article/details/79234314

版权

spark 专栏收录该内容

122 篇文章 15 订阅

订阅专栏

文章地址：http://www.haha174.top/article/details/255855
项目源码：https://github.com/haha174/spark.git
1.安装scala
2. 安装idea scala 插件

object SparkOnFile {
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("WorldCount").setMaster("local");
    val sc = new SparkContext(sparkConf);

    //val textFile = sc.textFile("hdfs://spark1:8020/world-count.txt")
    val textFile = sc.textFile("C:\\Users\\haha174\\Desktop\\data\\world-count.txt")
    val counts = textFile.flatMap(line => line.split("-"))
      .map(word => (word, 1))
      .reduceByKey(_ + _)
    counts.foreach(count=>println(count._1+" appeard "+count._2+" times"))
    //counts.saveAsTextFile("hdfs://spark1:8020/world-count-result.txt")
  }
}