MLlib-Basics (III)

最新推荐文章于 2021-02-13 21:11:43 发布

麦兜NG

最新推荐文章于 2021-02-13 21:11:43 发布

阅读量982

点赞数

文章标签： spark-1.0.0 MLlib 大的稀疏矩阵

MLlib-Basics

4.3CoordinateMatrix

坐标矩阵也是一种RDD存储的分布式矩阵。顾名思义，这里的每一项都是一个(i: Long, j: Long, value: Double)指示行列值的元组tuple。

其中i是行坐标，j是列坐标，value是值。如果矩阵是非常大的而且稀疏，坐标矩阵一定是最好的选择。

坐标矩阵通过RDD[MatrixEntry]实例创建，MatrixEntry是(long,long.Double)形式。坐标矩阵可以转化为IndexedRowMatrix。

importorg.apache.spark.mllib.linalg.distributed.{CoordinateMatrix,MatrixEntry}

valentries:RDD[MatrixEntry]=...// an RDD of matrix entries

// Create a CoordinateMatrix from an RDD[MatrixEntry].

valmat:CoordinateMatrix= new CoordinateMatrix(entries)

// Get its size.

valm= mat.numRows()

valn= mat.numCols()

// Convert it to an IndexRowMatrix whose rows are sparse vectors.

valindexedRowMatrix= mat.toIndexedRowMatrix()

我们仍然用这组数据做实验。

1   1   2
2   3   4
5   6   7

* 先从hdfs上load数据，text中的每行看作是一个String ,是RDD[String]格式。

scala> val textfile=sc.textFile("hdfs://node001:9000/spark/input/data.txt") 14/07/11 05:01:46 INFO MemoryStore: ensureFreeSpace(167504) called with curMem=0, maxMem=309225062 14/07/11 05:01:46 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated size 163.6 KB, free 294.7 MB) textfile: org.apache.spark.rdd.RDD[String] = MappedRDD[1] at textFile at <console>:12

*将每行，分成了一个字符数组.变成RDD[Array[String]]

scala> val middle=textfile.map((arg)=>arg.split("\\t")) middle: org.apache.spark.rdd.RDD[Array[String]] = MappedRDD[2] at map at <console>:14

* 想办法剖离出（long,long,Double)的元组。

scala> val mid=middle.map((arg)=>(arg(0).toLong,arg(1).toLong,arg(2).toDouble)) mid: org.apache.spark.rdd.RDD[(Long, Long, Double)] = MappedRDD[3] at map at <console>:16

*生成MatrixEntry。

scala> val entries=mid.map((arg)=>MatrixEntry(arg._1,arg._2,arg._3)) entries: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.distributed.MatrixEntry] = MappedRDD[4] at map at <console>:19

*生成坐标矩阵

scala> val mat: CoordinateMatrix = new CoordinateMatrix(entries) mat: org.apache.spark.mllib.linalg.distributed.CoordinateMatrix= org.apache.spark.mllib.linalg.distributed.CoordinateMatrix@be5b71c

* 计算它的行数

scala> val m = mat.numRows() 14/07/11 05:15:32 INFO FileInputFormat: Total input paths to process : 1 14/07/11 05:15:32 INFO SparkContext: Starting job: reduce at CoordinateMatrix.scala:99 14/07/11 05:15:32 INFO DAGScheduler: Got job 0 (reduce at CoordinateMatrix.scala:99) with 2 output partitions (allowLocal=false) 14/07/11 05:15:32 INFO DAGScheduler: Final stage: Stage 0(reduce at CoordinateMatrix.scala:99) 14/07/11 05:15:32 INFO DAGScheduler: Parents of final stage: List() 14/07/11 05:15:32 INFO DAGScheduler: Missing parents: List() 14/07/11 05:15:32 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[5] at map at CoordinateMatrix.scala:99), which has no missing parents 14/07/11 05:15:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MappedRDD[5] at map at CoordinateMatrix.scala:99) 14/07/11 05:15:33 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks 14/07/11 05:15:33 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on executor localhost: localhost (PROCESS_LOCAL) 14/07/11 05:15:33 INFO TaskSetManager: Serialized task 0.0:0 as 1936 bytes in 1 ms 14/07/11 05:15:33 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on executor localhost: localhost (PROCESS_LOCAL) 14/07/11 05:15:33 INFO TaskSetManager: Serialized task 0.0:1 as 1936 bytes in 0 ms 14/07/11 05:15:33 INFO Executor: Running task ID 1 14/07/11 05:15:33 INFO Executor: Running task ID 0 14/07/11 05:15:33 INFO BlockManager: Found block broadcast_0 locally 14/07/11 05:15:33 INFO BlockManager: Found block broadcast_0 locally 14/07/11 05:15:33 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:9+9 14/07/11 05:15:33 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9 14/07/11 05:15:33 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id 14/07/11 05:15:33 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 14/07/11 05:15:33 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap 14/07/11 05:15:33 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition 14/07/11 05:15:33 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id 14/07/11 05:15:33 INFO Executor: Serialized size of result for 1 is 724 14/07/11 05:15:33 INFO Executor: Serialized size of result for 0 is 724 14/07/11 05:15:33 INFO Executor: Sending result for 0 directly to driver 14/07/11 05:15:33 INFO Executor: Sending result for 1 directly to driver 14/07/11 05:15:33 INFO Executor: Finished task ID 1 14/07/11 05:15:33 INFO Executor: Finished task ID 0 14/07/11 05:15:33 INFO TaskSetManager: Finished TID 1 in 82 ms on localhost (progress: 1/2) 14/07/11 05:15:33 INFO DAGScheduler: Completed ResultTask(0, 1) 14/07/11 05:15:33 INFO DAGScheduler: Completed ResultTask(0, 0) 14/07/11 05:15:33 INFO TaskSetManager: Finished TID 0 in 91 ms on localhost (progress: 2/2) 14/07/11 05:15:33 INFO DAGScheduler: Stage 0 (reduce at CoordinateMatrix.scala:99) finished in 0.097 s 14/07/11 05:15:33 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 14/07/11 05:15:33 INFO SparkContext: Job finished: reduce at CoordinateMatrix.scala:99, took 0.170012294 s m: Long = 6