spark（一）之RDD行动和转换

RDD：是弹性分布式数据集，是元素的集合。
RDD操作分为转换（filter、map）和行动（counter、foreach）。
转换具有延迟执行的特点，即只记录下RDD的谱系构建“操作图”，当行动时才会真正执行转换。
map、flatMap、distinct

scala> val lines = sc.textFile("file:///home/student/client-dis.log")  //从本地读取文件前面写file，从hdfs读取文件写hdfs
lines: org.apache.spark.rdd.RDD[String] = file:///home/student/client-dis.log MapPartitionsRDD[1]

scala> lines.count
res0: Long = 4
//文件共四行

scala> val idsStr = lines.map(line => line.split(","))
idsStr: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[2] at map at <console>:25

scala> idsStr.foreach(println)  
[Ljava.lang.String;@495e0bb5
[Ljava.lang.String;@6822600b
[Ljava.lang.String;@785d63f0
[Ljava.lang.String;@411164b9
//文件中的每行读取存为一个数组

scala> idsStr.first
res2: Array[String] = Array(15, 16, 20, 20)
//查看第一个数组

scala> intIds.collect
res7: Array[Int] = Array(15, 16, 20, 20, 77, 80, 94, 94, 98, 16, 31, 31, 15, 20)
//将RDD所有元素收集到该数组中

scala> val ids =lines.flatMap(_.split(","))
ids: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[4] at flatMap at <console>:25
//将所有的数组拍平放到一个数组中

scala> ids.collect
res4: Array[String] = Array(15, 16, 20, 20, 77, 80, 94, 94, 98, 16, 31, 31, 15, 20)

scala> ids.first
res5: String = 15

scala> ids.collect.mkString(";")
res6: String = 15;16;20;20;77;80;94;94;98;16;31;31;15;20
//mkString将RDD中的元素拼接起来以";"连接

scala> val intIds = ids.map(_.toInt)
intIds: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[5] at map at <console>:25
//把String类型转化成Int

scala> intIds.collect
res7: Array[Int] = Array(15, 16, 20, 20, 77, 80, 94, 94, 98, 16, 31, 31, 15, 20)

scala> val uniqueIds = intIds.distinct
uniqueIds: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[8] at distinct at <console>:25
//distinct返回包含删除重复元素的新的RDD

scala> uniqueIds.collect
res2: Array[Int] = Array(16, 80, 98, 20, 94, 15, 77, 31)

scala> val finalCount = uniqueIds.count
finalCount: Long = 8

sample、takeSample、take

• sample：sample是转换操作，从RDD中抽样创建新的RDD，sample(withReplacement,fraction,seed):

withReplacement	是否有放回抽样，false说明下次抽样前不放回
fraction	有放回抽样时，是每个元素被采样的预期次数 ；无放回时，是每个元素被采样的概率
seed	生成随机数的种子，有默认值

//放回抽样
scala> val swr =  uniqueIds.sample(true,0.5)
swr: org.apache.spark.rdd.RDD[Int] = PartitionwiseSampledRDD[13] at sample at <console>:25

scala> swr.count
res6: Long = 9

scala> swr.collect
res7: Array[Int] = Array(16, 80, 94, 15, 15, 15, 77, 31, 31)

//不放回抽样
scala> val s = uniqueIds.sample(false,0.3)
s: org.apache.spark.rdd.RDD[Int] = PartitionwiseSampledRDD[12] at sample at <console>:25

scala> s.count
res3: Long = 2

scala> s.collect
res5: Array[Int] = Array(98, 77)

• takeSample：takeSample(withReplacement,num,seed)):

sample	takeSample
转换操作	动作操作，返回一个数组
第二个参数是一个小数，代表每个元素被抽取的概率	第二个参数是抽取元素的数量

scala> val taken = uniqueIds.takeSample(false,5)
taken: Array[Int] = Array(94, 80, 31, 15, 77)

• take：返回请求的元素数量

scala> uniqueIds.take(3)
res3: Array[Int] = Array(16, 80, 98)

Double RDD函数基础统计

scala> intIds.mean
res4: Double = 44.785714285714285

scala> intIds.sum
res5: Double = 627.0

scala> intIds.variance
res6: Double = 1114.8826530612246

scala> intIds.stdev
res7: Double = 33.38985853610681

scala> intIds.histogram(Array(1.0,50.0,100.0))
res8: Array[Long] = Array(9, 5)

scala> intIds.histogram(3)
res10: (Array[Double], Array[Long]) = (Array(15.0, 42.66666666666667, 70.33333333333334, 98.0),Array(9, 0, 5))