spark：学习杂记+案例--40

《快学scala》第九章课后习题：

1.编写一小段Scala代码，将某个文件中的行倒转顺序(将最后一行作为第一行,依此类推)

2.编写Scala程序,从一个带有制表符的文件读取内容,将每个制表符替换成一组空格,使得制表符隔开的n列仍然保持纵向对齐,并将结果写入同一个文件

3.编写一小段Scala代码,从一个文件读取内容并把所有字符数大于12的单词打印到控制台。如果你能用单行代码完成会有额外奖励

4.编写Scala程序，从包含浮点数的文本文件读取内容，打印出文件中所有浮点数之和，平均值，最大值和最小值

5.编写Scala程序，向文件中写入2的n次方及其倒数，指数n从0到20。对齐各列:
  1         1
  2         0.5
  4         0.25
...         ...

6.编写正则表达式,匹配Java或C++程序代码中类似"like this,maybe with \" or\\"这样的带引号的字符串。编写Scala程序将某个源文件中所有类似的字符串打印出来

7.编写Scala程序，从文本文件读取内容，并打印出所有的非浮点数的词法单位。要求使用正则表达式

8.编写Scala程序打印出某个网页中所有img标签的src属性。使用正则表达式和分组

9.编写Scala程序，盘点给定目录及其子目录中总共有多少以.class为扩展名的文件

10.扩展那个可序列化的Person类，让它能以一个集合保存某个人的朋友信息。构造出一些Person对象，让他们中的一些人成为朋友，然后将Array[Person]保存到文件。将这个数组从文件中重新读出来，校验朋友关系是否完好,注意,请在main中执行。脚本执行无法序列化。

//

package llf

import java.io.{File, PrintWriter}
import scala.io.Source

/**
 * Created by sendoh on 2015/5/7.
 */
//9
class answer9 {
  //1
  val path = "test.txt"
  val source = Source.fromFile(path)
  val lines = source.getLines.toArray.reverse
  val pri = new PrintWriter(path)
  lines.foreach(pri.println(_))
  pri.close()
  //2
  val path2 = "test2.txt"
  val lines2 = Source.fromFile(path2).getLines.toArray
  val pri2 = new PrintWriter(path2)
  lines2.foreach(line => {
    val replacedLine = line.replaceAll("\\t", " ")
    pri2.println(replacedLine)
  })
  pri2.close()
  //3
  Source.fromFile("test3.txt").mkString.split("\\s+").filter(_.length > 12).foreach(println(_))
  //4
  val source4 = Source.fromFile("test4.txt")
  val content4 = source4.getLines.mkString
  val floatPattern = "\\d+\\.\\d+".r
  val floatValues = floatPattern.findAllIn(content4).toArray.map(_.toDouble)
  val max = floatValues.max
  val min = floatValues.min
  val sum = floatValues.sum
  val avg = sum / floatValues.length
  println(max + min + sum + avg)
  //5
  val pri5 = new PrintWriter("test5.txt")
  for (n <- 0 to 20){
    pri5.println("%-8.0f\t\t%f".format(Math.pow(2,n), 1.0 / Math.pow(2,n.toDouble)))
  }
  pri5.close()
  //6
  val source6 = Source.fromFile("test6.txt").mkString
  val pattern6 = "\\w+\\s+\"".r
  pattern6.findAllIn(source6).mkString.foreach(println)
  //7
  val source7 = Source.fromFile("test7.txt").mkString
  val pattern7 = """[^((\d+\.){0,1}\d+)^\s+]+""".r
  pattern7.findAllIn(source7).foreach(println)
  //8
  val source8 = Source.fromURL("http://xxx.com","UTF-8").mkString
  val pattern8 = """<img[^>]+(src\s*=\s*"[^>^"]+")[^>]*>""".r
  for (pattern8(str) <- pattern8.findAllIn(source8)) println(str)
  //9
  val path = "test9.txt"
  val dir = new File(path)

  def subdirs(dir:File):Iterator[File]={
    val children = dir.listFiles().filter(_.getName.endsWith("class"))
    children.toI

}

//

SparkTachyonHdfsLR:

package llf

import java.util.Random

import scala.math.exp

import breeze.linalg.{Vector, DenseVector}
import org.apache.hadoop.conf.Configuration

import org.apache.spark._
import org.apache.spark.scheduler.InputFormatInfo
import org.apache.spark.storage.StorageLevel

/**
 * Created by sendoh on 2015/5/10.
 */
object SparkTachyonHdfsLR { //Tachyon是一个分布式内存文件系统,可以在集群里以访问内存的速度来访问存在tachyon里的文件
  val D = 10 //num of dimension
  val rand = new Random(42)
  def showWarning(): Unit ={
    System.err.println(
      """WARN: This is a naive implementation of Logistic Regression and is given as an example!
        |Please use either org.apache.spark.mllib.classification.LogisticRegressionWithSGD or
        |org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
        |for more conventional use.
      """.stripMargin)
  }

  case class DataPoint(x: Vector[Double], y: Double)
  def parsePoint(line: String): DataPoint = {
    val tok = new java.util.StringTokenizer(line, " ")
    var y = tok.nextToken.toDouble
    var x = new Array[Double](D)
    var i = 0
    while (i < D){
      x(i) = tok.nextToken.toDouble
      i += 1
    }
    DataPoint(new DenseVector(x), y)
  }
  def main(args: Array[String]): Unit ={
    showWarning()
    val inputPath = args(0)
    val conf = new SparkConf().setAppName("SparkTachyonHdfsLR")
    val sconf = new Configuration()
    val sc = new SparkContext(conf, InputFormatInfo.computePreferredLocations(
      Seq(new InputFormatInfo(sconf, classOf[org.apache.hadoop.mapred.TextInputFormat], inputPath))
    ))
    val lines = sc.textFile(inputPath)
    val points = lines.map(parsePoint _).persist(StorageLevel.OFF_HEAP)
    val ITERATIONS = args(1).toInt
    var w = DenseVector.fill(D){2 * rand.nextDouble - 1}
    println("Initial w: " + w)
    for (i <- 1 to ITERATIONS){
      println("On iteration " + i)
      val gradient = points.map { p =>
        p.x * (1 / (1 + exp(-p.y * (w.dot(p.x)))) - 1) * p.y
      }.reduce(_ + _)
      w -= gradient
    }
    println("Final w: " + w)
    sc.stop()
  }

}