Spark内核天机解密（2016）第三讲 Scala函数式编程

课程笔记：

变量与函数

>def fun1(name: String) {println(name)}

fun1: (name: String)Unit

>val fun1_v = fun1 _  //注意，是空格和下划线

fun1_v: String => Unit = <function1>

>fun1("Spark")

Spark

>fun1_v("Spark")

Spark

>fun1_v("Scala")

Scala

//匿名函数

>val fun2 = (content: String) => println(content:)

fun2: String => Unit = <function1>

>fun2("Hadoop")

Hadoop

//函数的参数可以是函数

>val hiScala = (content : String) => println(context)

hiScala: String => Unit = <function1>

>def bigData(func: (String) => Unit, context: String)  {func(content)}

bigData:(func: String => Unit, content: String) Unit

>bigData(hiScala,"Spark")

Spark

//循环遍历数组每个值

>val array = Array(1,2,3,4,5,6,7,8,9)

>array.map(item => 2 * item)

 Array[Int] = Array(2,4,6,8,10,12,14,16,18)

>array.map(item => print(item))

//函数作为返回值

>def func_Returned(content : String) = (message : String) => println(message)

func_Returned: (content: String)String => Unit

>func_Returned("Spark")

res7: String => Unit = <function1>

>val returned = func_Returned("Spark")

returned: String => Unit = <function1>

//用变量调用执行

>returned("Scala")

Spark    Scala

>def spark(fun: (String) => Unit, name: String) {func(name)}

>spark((name: String) => println(name),"Scala")

Scala

>spark((name) => println(name),"Scala")

Scala

>spark(name => println(name),"Scala")

Scala

>spark(println(_), "Scala")

Scala

>spark(println,"Scala")

Scala

>array.map(2*_)

res17:Array[Int] = Array(2,4,6,8,10,12,14,16,18)

// print array elements

>array.map(2*_).foreach(println(_))

>array.map(2*_).foreach(println _)     

>array.map(2*_).filter(_ >12).foreach(println)

14

16

18

//闭包：函数的变量超出函数的作用域时，在外部还可以对它进行访问； Scala内部维护了函数的参数（参数作为函数的成员被保存）

>def scala(content : String) = (message : String) => println(content + "   :" + message)

scala: (content: String) String => Unit

>val funcResult = scala("Spark")

funcResult: String => Unit = <function1>

>funcResult("Flink")

Spark    :Flink

//柯里化：如果一个函数有两个参数，那么可以把这个函数转换成两个函数来实现，返回值也是一个函数。

>def sum(x: Int, y: Int) = x+y

>sum(1,2)

res24: Int = 3

>def sum_currying(x : Int) = (y: Int) => x + y

sum_currying: (x: Int) Int => Int

>sum_currying(1)(2)

也可以把两个函数合并起来定义和使用。

>def sum_Currying_Better(x : Int) (y: Int) = x + y

sum_Currying_Better: (x: Int) (y: Int) Int

>sum_Currying_better(1)(3)

res27: Int = 4

//很奇妙的处理方式

>(1 to 100).reduceLeft(_+_)

res28: Int = 5050

集合

>val list = List("Scala","Spark","Flink")

list: List[String] = List(Scala,Spark,Flink)

>list.map("The content is : " + _)

res29: List[String] = List(The content is : Scala,The content is : Spark, The content is : Scalaala

作业：统计一个文件夹下面所有的单词出现的总次数(转)
http://blog.sina.com.cn/s/blog_7dc74d730102vyhv.html

object WordCounter {
  import scala.io.Source
  import java.io._
  

  // store word and counter
  var map = Map.empty[String, Int]
 
  def main(args: Array[String]): Unit = {
    scanDir(new File("D:/workspace"))
    map.foreach(f =>
      println(f)  
    )
  }
 
  def scanDir(dir: File): Unit = {
    dir.listFiles.foreach { file => 
      if(file.isFile()){
        readFile(file)
        println(file)
      }      
    }
  }
 
  def readFile(file: File){
    val f = Source.fromFile(file)
    for (line <- f.getLines()){
      count(line)
    }
  }
 
  def count(line: String) = {
    for(word <- line.split("[,:.!\\s?*\\/-=+]()><")){
      if(map.contains(word))
        map += (word -> (map(word)+1))
      else
        map += (word -> 1)
    }
  }
}

object WordCounter {
  //导入jar包
  import scala.io.Source
  import java.io._
  //存储单词和个数
  var map = Map.empty[String, Int]
 
  def main(args: Array[String]): Unit = {
    scanDir(new File("D:/TMPFOLDER"))
    map.foreach(f =>
      println(f)  
    )
  }
 
  def scanDir(dir: File): Unit = {
    dir.listFiles.foreach { file => 
      if(file.isFile()){
        readFile(file)
        println(file)
      }      
    }
  }
 
  def readFile(file: File){
    val f = Source.fromFile(file)
    for (line <- f.getLines()){
      count(line)
    }
  }
 
  def count(line: String) = {
    for(word <- line.split("[,:.!\\s?*\\/-=+]()><")){
      if(map.contains(word))
        map += (word -> (map(word)+1))
      else
        map += (word -> 1)
    }
  }
}