scala-wordcount详解

 用scala语言写wordcount，主要也就那么几个流程，和mapreduce的思想步骤还是差不多的。下面我们来详细讲解一下：先看一下代码

import org.apache.spark.{SparkConf, SparkContext}

/**
  * @Date: 2018/12/17 9:03
  */
object wc_scala {

  def main(args: Array[String]): Unit = {
      val conf = new SparkConf().setAppName("wc_scala")/*.setMaster("local")*/;

      val sc = new SparkContext(conf);
      
      val lines = sc.textFile("hdfs://192.168.33.100:9000/spark.txt",1)

      val words = lines.flatMap{line => line.split(" ")};
      val pairs = words.map{word => (word,1)}
      val wordcounts = pairs.reduceByKey(_+_);

      wordcounts.foreach(wordcount => println(wordcount._1+ " : "+wordcount._2+" times"))
  }
}