Spark英文单词分析案例

1、有如下文件testdata.txt（）

At a high level

every Spark application consists of a driver program that runs the user’s main function and executes various parallel operations on a cluster

The main abstraction Spark provides is a resilient distributed dataset (RDD)

which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel

RDDs are created by starting with a file in the Hadoop file system (or any other Hadoop-supported file system)

or an existing Scala collection in the driver program

and transforming it

Users may also ask Spark to persist an RDD in memory

allowing it to be reused efficiently across parallel operations. Finally

RDDs automatically recover from node failures

要求完成如下功能

（1）筛选出包含Spark的行，并统计行数

（2）输出包含单词最多的那一行的单词数

（3）统计数据中包含“a”的行数和包含“b”的行数

import org.apache.spark.{Accumulator, SparkConf, SparkContext} import org.apache.spark.rdd.RDD object work02 { def main(args: Array[String]): Unit = { val conf = new SparkConf().setAppName("localtion").setMaster("local[*]") val sc=new SparkContext(conf) //获取数据 val user1:RDD[String]=sc.textFile("E://aaa//testdata.txt",1) //1.统计Spark行数 val result=user1.map((lines:String)=>if(lines.contains("Spark")) ("Spark",1) else (" ",0)).filter(_.equals(("Spark",1))) val result1=result.reduceByKey(_+_) //result1.foreach(println) //2.单词最多的哪一行的单词数 var i:Int=0 val result2=user1.map((lines:String)=>{ val str=lines.split(" ").toList i=i+1 (str,i) }) val result3=result2.map(temp=>(temp._1.length,temp._2)).sortByKey(false).take(1) //result3.foreach(println) //3.统计数据中包含“a”的行数和包含“b”的行数 val result4=user1.map((lines:String)=>if(lines.contains("a")) ("a",1) else (" ",0)).filter(_.equals(("a",1))) val result5=result4.reduceByKey(_+_) result5.foreach(println) val result6=user1.map((lines:String)=>if(lines.contains("b")) ("b",1) else (" ",0)).filter(_.equals(("b",1))) val result7=result6.reduceByKey(_+_) result7.foreach(println) } }