MLlib-Basics (II)

最新推荐文章于 2024-07-19 16:04:56 发布
最新推荐文章于 2024-07-19 16:04:56 发布
阅读量1.2k 收藏
点赞数
分类专栏： scala matrix breeze 文章标签： spark MLlib RowMatrix SVD PCA

MLlib-Basics

3 Local matrix 本地矩阵

             目前只支持DenseMatrix.SparseMatrix有待在以后版本出现。

scala> import org.apache.spark.mllib.linalg.{Matrix, Matrices}
import org.apache.spark.mllib.linalg.{Matrix, Matrices}

// Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))
scala> val dm: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
dm: org.apache.spark.mllib.linalg.Matrix =
1.0  2.0  
3.0  4.0  
5.0  6.0

4 Distributed matrix 分布式的矩阵

       一个分布式的矩阵有long型的行列数和double型的值，以一个或者多个RDD形式分布式存。对于存储打的分布式的矩阵来说，选择正确的格式非常重要。把一个分布式的矩阵转换成一个不同的格式需要全体打乱，这代价非常高。这里只讲了三种类型的分布式矩阵。

注释：

       一个分布式矩阵的RDD必须是确定性的，因为我们要缓存矩阵的大小。

    4.1 RowMatrix

    一个行矩阵就是把每行对应一个RDD，将矩阵的每行分布式存储，矩阵的每行是一个本地向量。这和多变量统计的数据矩阵比较相似。因为每行以一个本地向量表示，那么矩阵列的数量被限制在整数范围内，但是实际应用中列数很小。

      这个要仔细说一下：

import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.mllib.linalg.distributed.RowMatrix

val rows: RDD[Vector] = ... // an RDD of local vectors
// Create a RowMatrix from an RDD[Vector].
val mat: RowMatrix = new RowMatrix(rows)

// Get its size.
val m = mat.numRows()
val n = mat.numCols
    
       思考：考虑RDD[Vector]该如何生成呢？特别是从一个文件作为数据来源，比如说text文件。

       解答：

       假设：    现在在hdfs上，路径为hdfs://node001:9000/spark/input/data.txt 有个data.txt文件是一个三行三列的矩阵，元素之间用制彪符\t 隔开。

                        1    1    2
                        2    3    4
                        5    6    7

*         这个时候RDD，可以看到有text中的每行算一个String                                --------------------------->RDD[String].

scala> val textfile=sc.textFile("hdfs://node001:9000/spark/input/data.txt")
14/07/11 00:45:33 INFO MemoryStore: ensureFreeSpace(82268) called with curMem=249772, maxMem=309225062
14/07/11 00:45:33 INFO MemoryStore: Block broadcast_2 stored as values to memory (estimated size 80.3 KB, free 294.6 MB)
textfile: org.apache.spark.rdd.RDD[String] = MappedRDD[13] at textFile at <console>:21

 *       去掉制表符，每行的每个数都是String，一行是一个Array[String],         ------------------------------->RDD[Array[String]]

           scala> val middle=textfile.map((arg) =>arg.split("\\t"))
           middle: org.apache.spark.rdd.RDD[Array[String]] = MappedRDD[14] at map at <console>:23

 *       每行每个数由String -> Double                                                                                                ------------------------------->RDD[Array[Double]]

           scala> val mid=middle.map((arg)=>arg.map((args)=>args.toDouble))
           mid: org.apache.spark.rdd.RDD[Array[Double]] = MappedRDD[17] at map at <console>:25

 *       定义函数change ，将Array[Double]变成Vector。

          scala> def change(t:Array[Double]):Vector={
                   | val x=Vectors.dense(t)
                   | x
                   | }
           change: (t: Array[Double])org.apache.spark.mllib.linalg.Vector


*        通过change则text中的每行通过Array[String]生成Vector.     --------------------------------------------->RDD[Vector]                 

 scala> val ha=mid.map(change)
ha: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = MappedRDD[18] at map at <console>:29

*        通过RDD[Vector]创建RowMatrix。           

           scala> val mat:RowMatrix=new RowMatrix(ha)
           mat: org.apache.spark.mllib.linalg.distributed.RowMatrix = org.apache.spark.mllib.linalg.distributed.RowMatrix@2ec01ae5
                  

*         下面是计算矩阵的行数和列数。

scala> val m=mat.numRows()
14/07/11 01:05:50 INFO SparkContext: Starting job: count at RowMatrix.scala:194
14/07/11 01:05:50 INFO DAGScheduler: Got job 2 (count at RowMatrix.scala:194) with 2 output partitions (allowLocal=false)
14/07/11 01:05:50 INFO DAGScheduler: Final stage: Stage 2(count at RowMatrix.scala:194)
14/07/11 01:05:50 INFO DAGScheduler: Parents of final stage: List()
14/07/11 01:05:50 INFO DAGScheduler: Missing parents: List()
14/07/11 01:05:50 INFO DAGScheduler: Submitting Stage 2 (MappedRDD[18] at map at <console>:29), which has no missing parents
14/07/11 01:05:50 INFO DAGScheduler: Submitting 2 missing tasks from Stage 2 (MappedRDD[18] at map at <console>:29)
14/07/11 01:05:50 INFO TaskSchedulerImpl: Adding task set 2.0 with 2 tasks
14/07/11 01:05:50 INFO TaskSetManager: Starting task 2.0:0 as TID 4 on executor localhost: localhost (PROCESS_LOCAL)
14/07/11 01:05:50 INFO TaskSetManager: Serialized task 2.0:0 as 2827 bytes in 0 ms
14/07/11 01:05:50 INFO TaskSetManager: Starting task 2.0:1 as TID 5 on executor localhost: localhost (PROCESS_LOCAL)
14/07/11 01:05:50 INFO TaskSetManager: Serialized task 2.0:1 as 2827 bytes in 0 ms
14/07/11 01:05:50 INFO Executor: Running task ID 4
14/07/11 01:05:50 INFO Executor: Running task ID 5
14/07/11 01:05:50 INFO BlockManager: Found block broadcast_2 locally
14/07/11 01:05:50 INFO BlockManager: Found block broadcast_2 locally
14/07/11 01:05:50 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9
14/07/11 01:05:50 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:9+9
14/07/11 01:05:50 INFO Executor: Serialized size of result for 5 is 597
14/07/11 01:05:50 INFO Executor: Serialized size of result for 4 is 597
14/07/11 01:05:50 INFO Executor: Sending result for 5 directly to driver
14/07/11 01:05:50 INFO Executor: Sending result for 4 directly to driver
14/07/11 01:05:50 INFO Executor: Finished task ID 5
14/07/11 01:05:50 INFO Executor: Finished task ID 4
14/07/11 01:05:50 INFO DAGScheduler: Completed ResultTask(2, 1)
14/07/11 01:05:50 INFO TaskSetManager: Finished TID 5 in 26 ms on localhost (progress: 1/2)
14/07/11 01:05:50 INFO DAGScheduler: Completed ResultTask(2, 0)
14/07/11 01:05:50 INFO TaskSetManager: Finished TID 4 in 28 ms on localhost (progress: 2/2)
14/07/11 01:05:50 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
14/07/11 01:05:50 INFO DAGScheduler: Stage 2 (count at RowMatrix.scala:194) finished in 0.028 s
14/07/11 01:05:50 INFO SparkContext: Job finished: count at RowMatrix.scala:194, took 0.050264151 s
m: Long = 3

scala> val n=mat.numCols()
14/07/11 01:06:02 INFO SparkContext: Starting job: first at RowMatrix.scala:186
14/07/11 01:06:02 INFO DAGScheduler: Got job 3 (first at RowMatrix.scala:186) with 1 output partitions (allowLocal=true)
14/07/11 01:06:02 INFO DAGScheduler: Final stage: Stage 3(first at RowMatrix.scala:186)
14/07/11 01:06:02 INFO DAGScheduler: Parents of final stage: List()
14/07/11 01:06:02 INFO DAGScheduler: Missing parents: List()
14/07/11 01:06:02 INFO DAGScheduler: Computing the requested partition locally
14/07/11 01:06:02 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9
14/07/11 01:06:02 INFO SparkContext: Job finished: first at RowMatrix.scala:186, took 0.011817442 s
n: Long = 3

4.1.1 Multivariate summary statistics

      我们为RowMatrix提供了列的总和统计,如果列的数量不是太大，比如小于3000，你就能够计算出 协方差矩阵，作为本地矩阵，这需要 O(n2) 的存储空间，其中n是矩阵的列数。总共的CPU时间是 O(mn2) , m是矩阵的行数，如果是稀疏矩阵的话，会更快。

import org.apache.spark.mllib.linalg.Matrix

import org.apache.spark.mllib.linalg.distributed.RowMatrix

import org.apache.spark.mllib.stat.MultivariateStatisticalSummary

val mat:RowMatrix = ...// a RowMatrix


// Compute column summary statistics.

val summary:MultivariateStatisticalSummary = mat.computeColumnSummaryStatistics()

println(summary.mean)// a dense vector containing the mean value for each column

println(summary.variance)// column-wise variance

println(summary.numNonzeros)// number of nonzeros in each column


// Compute the covariance matrix.

val Cov:Matrix = mat.computeCovariance()

实验：

scala> val summary: MultivariateStatisticalSummary = mat.computeColumnSummaryStatistics()

14/07/11 04:23:50 INFO SparkContext: Starting job: aggregate at RowMatrix.scala:374

14/07/11 04:23:50 INFO DAGScheduler: Got job 7 (aggregate at RowMatrix.scala:374) with 2 output partitions (allowLocal=false)

14/07/11 04:23:50 INFO DAGScheduler: Final stage: Stage 7(aggregate at RowMatrix.scala:374)

14/07/11 04:23:50 INFO DAGScheduler: Parents of final stage: List()

14/07/11 04:23:50 INFO DAGScheduler: Missing parents: List()

14/07/11 04:23:50 INFO DAGScheduler: Submitting Stage 7 (MappedRDD[21] at map at RowMatrix.scala:374), which has no missing parents

14/07/11 04:23:50 INFO DAGScheduler: Submitting 2 missing tasks from Stage 7 (MappedRDD[21] at map at RowMatrix.scala:374)

14/07/11 04:23:50 INFO TaskSchedulerImpl: Adding task set 7.0 with 2 tasks

14/07/11 04:23:50 INFO TaskSetManager: Starting task 7.0:0 as TID 12 on executor localhost: localhost (PROCESS_LOCAL)

14/07/11 04:23:50 INFO TaskSetManager: Serialized task 7.0:0 as 3252 bytes in 1 ms

14/07/11 04:23:50 INFO TaskSetManager: Starting task 7.0:1 as TID 13 on executor localhost: localhost (PROCESS_LOCAL)

14/07/11 04:23:50 INFO TaskSetManager: Serialized task 7.0:1 as 3252 bytes in 1 ms

14/07/11 04:23:50 INFO Executor: Running task ID 12

14/07/11 04:23:50 INFO Executor: Running task ID 13

14/07/11 04:23:50 INFO BlockManager: Found block broadcast_2 locally

14/07/11 04:23:50 INFO BlockManager: Found block broadcast_2 locally

14/07/11 04:23:50 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9

14/07/11 04:23:50 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:9+9

14/07/11 04:23:50 INFO Executor: Serialized size of result for 12 is 1898

14/07/11 04:23:50 INFO Executor: Sending result for 12 directly to driver

14/07/11 04:23:50 INFO Executor: Finished task ID 12

14/07/11 04:23:50 INFO DAGScheduler: Completed ResultTask(7, 0)

14/07/11 04:23:50 INFO Executor: Serialized size of result for 13 is 1898

14/07/11 04:23:50 INFO TaskSetManager: Finished TID 12 in 16 ms on localhost (progress: 1/2)

14/07/11 04:23:50 INFO Executor: Sending result for 13 directly to driver

14/07/11 04:23:50 INFO Executor: Finished task ID 13

14/07/11 04:23:50 INFO DAGScheduler: Completed ResultTask(7, 1)

14/07/11 04:23:50 INFO TaskSetManager: Finished TID 13 in 17 ms on localhost (progress: 2/2)

14/07/11 04:23:50 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool

14/07/11 04:23:50 INFO DAGScheduler: Stage 7 (aggregate at RowMatrix.scala:374) finished in 0.018 s

14/07/11 04:23:50 INFO SparkContext: Job finished: aggregate at RowMatrix.scala:374, took 0.029782399 s

summary: org.apache.spark.mllib.stat.MultivariateStatisticalSummary = org.apache.spark.mllib.linalg.distributed.ColumnStatisticsAggregator@5a30f467


scala> println(summary.mean)                        //是针对每一列

[2.6666666666666665,3.3333333333333335,4.333333333333333]


scala> println(summary.variance)                        //每一列

[4.333333333333333,6.333333333333333,6.333333333333333]


scala> println(summary.numNonzeros)

[3.0,3.0,3.0]


scala> val Cov: Matrix = mat.computeCovariance()

14/07/11 04:25:25 INFO SparkContext: Starting job: aggregate at RowMatrix.scala:312

14/07/11 04:25:25 INFO DAGScheduler: Got job 8 (aggregate at RowMatrix.scala:312) with 2 output partitions (allowLocal=false)

14/07/11 04:25:25 INFO DAGScheduler: Final stage: Stage 8(aggregate at RowMatrix.scala:312)

14/07/11 04:25:25 INFO DAGScheduler: Parents of final stage: List()

14/07/11 04:25:25 INFO DAGScheduler: Missing parents: List()

14/07/11 04:25:25 INFO DAGScheduler: Submitting Stage 8 (MappedRDD[18] at map at <console>:29), which has no missing parents

14/07/11 04:25:25 INFO DAGScheduler: Submitting 2 missing tasks from Stage 8 (MappedRDD[18] at map at <console>:29)

14/07/11 04:25:25 INFO TaskSchedulerImpl: Adding task set 8.0 with 2 tasks

14/07/11 04:25:25 INFO TaskSetManager: Starting task 8.0:0 as TID 14 on executor localhost: localhost (PROCESS_LOCAL)

14/07/11 04:25:25 INFO TaskSetManager: Serialized task 8.0:0 as 3105 bytes in 0 ms

14/07/11 04:25:25 INFO TaskSetManager: Starting task 8.0:1 as TID 15 on executor localhost: localhost (PROCESS_LOCAL)

14/07/11 04:25:25 INFO TaskSetManager: Serialized task 8.0:1 as 3105 bytes in 0 ms

14/07/11 04:25:25 INFO Executor: Running task ID 14

14/07/11 04:25:25 INFO Executor: Running task ID 15

14/07/11 04:25:25 INFO BlockManager: Found block broadcast_2 locally

14/07/11 04:25:25 INFO BlockManager: Found block broadcast_2 locally

14/07/11 04:25:25 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9

14/07/11 04:25:25 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:9+9

14/07/11 04:25:25 INFO Executor: Serialized size of result for 15 is 1140

14/07/11 04:25:25 INFO Executor: Sending result for 15 directly to driver

14/07/11 04:25:25 INFO Executor: Finished task ID 15

14/07/11 04:25:25 INFO Executor: Serialized size of result for 14 is 1140

14/07/11 04:25:25 INFO Executor: Sending result for 14 directly to driver

14/07/11 04:25:25 INFO Executor: Finished task ID 14

14/07/11 04:25:25 INFO DAGScheduler: Completed ResultTask(8, 1)

14/07/11 04:25:25 INFO TaskSetManager: Finished TID 15 in 16 ms on localhost (progress: 1/2)

14/07/11 04:25:25 INFO TaskSetManager: Finished TID 14 in 18 ms on localhost (progress: 2/2)

14/07/11 04:25:25 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool

14/07/11 04:25:25 INFO DAGScheduler: Completed ResultTask(8, 0)

14/07/11 04:25:25 INFO DAGScheduler: Stage 8 (aggregate at RowMatrix.scala:312) finished in 0.019 s

14/07/11 04:25:25 INFO SparkContext: Job finished: aggregate at RowMatrix.scala:312, took 0.026187565 s

14/07/11 04:25:25 INFO SparkContext: Starting job: aggregate at RowMatrix.scala:211

14/07/11 04:25:25 INFO DAGScheduler: Got job 9 (aggregate at RowMatrix.scala:211) with 2 output partitions (allowLocal=false)

14/07/11 04:25:25 INFO DAGScheduler: Final stage: Stage 9(aggregate at RowMatrix.scala:211)

14/07/11 04:25:25 INFO DAGScheduler: Parents of final stage: List()

14/07/11 04:25:25 INFO DAGScheduler: Missing parents: List()

14/07/11 04:25:25 INFO DAGScheduler: Submitting Stage 9 (MappedRDD[18] at map at <console>:29), which has no missing parents

14/07/11 04:25:25 INFO DAGScheduler: Submitting 2 missing tasks from Stage 9 (MappedRDD[18] at map at <console>:29)

14/07/11 04:25:25 INFO TaskSchedulerImpl: Adding task set 9.0 with 2 tasks

14/07/11 04:25:25 INFO TaskSetManager: Starting task 9.0:0 as TID 16 on executor localhost: localhost (PROCESS_LOCAL)

14/07/11 04:25:25 INFO TaskSetManager: Serialized task 9.0:0 as 3057 bytes in 1 ms

14/07/11 04:25:25 INFO TaskSetManager: Starting task 9.0:1 as TID 17 on executor localhost: localhost (PROCESS_LOCAL)

14/07/11 04:25:25 INFO TaskSetManager: Serialized task 9.0:1 as 3057 bytes in 0 ms

14/07/11 04:25:25 INFO Executor: Running task ID 16

14/07/11 04:25:25 INFO Executor: Running task ID 17

14/07/11 04:25:25 INFO BlockManager: Found block broadcast_2 locally

14/07/11 04:25:25 INFO BlockManager: Found block broadcast_2 locally

14/07/11 04:25:25 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:0+9

14/07/11 04:25:25 INFO HadoopRDD: Input split: hdfs://node001:9000/spark/input/data.txt:9+9

14/07/11 04:25:25 INFO Executor: Serialized size of result for 17 is 1037

14/07/11 04:25:25 INFO Executor: Sending result for 17 directly to driver

14/07/11 04:25:25 INFO Executor: Finished task ID 17

14/07/11 04:25:25 INFO DAGScheduler: Completed ResultTask(9, 1)

14/07/11 04:25:25 INFO TaskSetManager: Finished TID 17 in 11 ms on localhost (progress: 1/2)

14/07/11 04:25:25 INFO Executor: Serialized size of result for 16 is 1037

14/07/11 04:25:25 INFO Executor: Sending result for 16 directly to driver

14/07/11 04:25:25 INFO Executor: Finished task ID 16

14/07/11 04:25:25 INFO DAGScheduler: Completed ResultTask(9, 0)

14/07/11 04:25:25 INFO TaskSetManager: Finished TID 16 in 14 ms on localhost (progress: 2/2)

14/07/11 04:25:25 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool

14/07/11 04:25:25 INFO DAGScheduler: Stage 9 (aggregate at RowMatrix.scala:211) finished in 0.015 s

14/07/11 04:25:25 INFO SparkContext: Job finished: aggregate at RowMatrix.scala:211, took 0.019312805 s

Cov: org.apache.spark.mllib.linalg.Matrix =

4.333333333333334  5.166666666666666  5.166666666666668  

5.166666666666668  6.333333333333332  6.333333333333336  

5.166666666666668  6.333333333333332  6.333333333333336


4.2 IndexedRowMatrix      

         这是分布式矩阵的第二种matrix.这种矩阵和RowMatrix非常相似，区别是它带有有一定意义的row indices。It is backed by an RDD of indexed rows, which each row is represented by its index (long-typed) and a local vector.体会一下这句话。

      一个IndexedRowMatrix可以从RDD[IndexedRow] 实例创建，IndexedRow是(Long, Vector)的wrapper.而且这种矩阵可以传换成RowMatrix，通过丢掉它的row indices.


import org.apache.spark.mllib.linalg.distributed.{IndexedRow, IndexedRowMatrix, RowMatrix}

val rows: RDD[IndexedRow] = ...                           // an RDD of indexed rows 


// Create an IndexedRowMatrix from an RDD[IndexedRow].

val mat: IndexedRowMatrix = new IndexedRowMatrix(rows)


// Get its size.

val m = mat.numRows()

val n = mat.numCols()


// Drop its row indices.

val rowMat: RowMatrix = mat.toRowMatrix() 






麦兜NG
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Spark MLlib学习
haoran_yang的博客
11-14 320
《SPARK MLLIB机器学习》_黄美灵 spark主要库 SparkSQL：SQL或Hive查询数据。 Spark Streaming：流数据实时处理，如web日志。 MLlib：机器学习库。 GraphX：图计算库。 目前主要学习使用MLlib和SparkSQL。 MLlib算法库 1.RDD 弹性分布式数据集(Resilient Distributed Dataset)。可...
关于SparkMLlib的基础数据结构Spark-MLlib-Basics
热门推荐
canglingye的专栏
11-20 2万+
此部分主要降价写关于MLlib的集中基础的数据结构
Machine Learning on Spark——第三节 统计基础（一)
weixin_33841503的博客
09-14 151
作者：周志湖 微信号：zhouzhihubeyond 本文主要内容 本文对了org.apache.spark.mllib.stat包及子包中的相关统计类进行介绍，stat包中包括下图中的类或对象: 本文将对其中的部分内容进行详细讲解 获取矩阵列（column-wise）统计信息 Kernel density estima...
Spark 提供的概率与统计算法 - 基本统计算法
BlueSky
04-16 9073
spark-mllib 使用到了Breeze线性代数包，Breeze 采用了 metlib-java包来优化数字处理，但是由于版权问题，Breeze 中缺省不包含 netlib-java 的 native proxy，需要在项目中自行引用。         compile "org.apache.spark:spark-mllib_${scalaMajorVersion}:${s
机器学习（二）基本数据类型
weixin_42438712的博客
09-09 887
原文链接: http://www.wangpengcufe.com/machinelearning/ml2/ MLlib支持存储在单个机器上的局部向量和矩阵，以及由一个或多个RDD支持的分布式矩阵。 局部向量和局部矩阵是用作公共接口的简单数据模型。 底层线性代数操作由Breeze提供。 在监督学习中使用的训练示例在MLlib中称为“标记点”。 一、本地向量 局部向量具有整数类型和基于0的索引和双...
math-basics.rar
08-01
链接： http://www.cis.upenn.edu/~jean/math-basics.pdf 这本书讲了什么？ 这本书的主要目的是介绍线性代数和最优化理论的基础知识以及这些知识在机器学习、机器人学、计算机视觉等领域的应用。 该书包含以下 10...
web-basics
03-18
"web-basics"这个主题显然涵盖了Web开发的基本概念和HTML的相关知识。下面，我们将深入探讨HTML及其在Web开发中的应用。 HTML是一种标记语言，用于定义网页的结构和内容。它的主要任务是将文本、图像、链接等元素...
spring-boot-ui-esb-basics:spring-boot-ui-esb-basics
05-26
基本JAVA-JAKARTA-EE TECH'S。 基本教程ule子红5 Spring启动基础皮草STSTest SampleDROOLS spring-boot-oauth2-master Spring启动休息大师spring-boot-reference.pdf spring-data-jpa-reference.pdf spring-security...
nextjs-basics
03-09
这是一个用引导的项目。入门首先，运行开发服务器： npm run dev# oryarn dev 用浏览器打开以查看结果。 您可以通过修改pages/index.js来开始编辑页面。 页面在您编辑文件时自动更新。 可以在上访问。...
electron-basics:边做边学电子
04-28
**电子基础：超越浏览器的前端开发** ...通过结合Chromium和Node.js，Electron使得前端开发者能够利用熟悉的工具集创建桌面应用，而无需深入学习原生编程语言。 在“边做边学电子”这个项目中，我们将探索如何利用...
《Spark机器学习》笔记——Spark构建聚类模型
陈杰
01-16 974
import breeze.plot.{Figure, hist, plot} import org.apache.spark.mllib.clustering.KMeans import org.apache.spark.mllib.linalg.Vectors import org.apache.spark.mllib.linalg.distributed.RowMatrix import o
Apache Spark 2.x Machine Learning Cookbook(2) ---使用spark学习线性代数
wangjunji34478的专栏
04-19 396
在本章中，我们将介绍以下内容： 包导入和向量和矩阵的初始设置 使用Spark 2.0创建DenseVector并进行设置 使用Spark 2.0创建SparseVector并进行设置 使用Spark 2.0创建DenseMatrix并进行设置 在Spark 2.0中使用稀疏局部矩阵 使用Spark 2.0执行矢量算术 使用Spark 2.0执行矩阵算术 Spark 2.0 ML库中的分布式矩阵 在...
Spark MLlib 核心基础：向量 And 矩阵
Spark MLlib 机器学习
04-23 9261
1、Spark MLlib 核心基础：向量 And矩阵 1.1 Vector 1.1.1 dense vector 源码定义：    * Creates a dense vector from its values.    */   @varargs   def dense(firstValue: Double, otherValues: Double*): Vector =
spark厦大---MLlib基本数据类型(2)
智慧与美貌并存
04-25 616
来源：http://mocom.xmu.edu.cn/article/show/58482057e083c990247075a7/0/1 四、分布式矩阵（Distributed Matrix） 分布式矩阵由长整型的行列索引值和双精度浮点型的元素值组成。它可以分布式地存储在一个或多个RDD上，MLlib提供了三种分布式矩阵的存储方案：行矩阵RowMatrix，索引行矩阵Ind
SparkML之DistributedMatrix(一)
legotime的博客
04-07 2492
===========================目录============================================== 分布矩阵（Distributed matrix） ------行矩阵（RowMatrix） ------标记行矩阵（indexed RowMatrix） ------坐标矩阵（CoordinateMatrix） ------分块矩阵（Bl
spark shell
最新发布
m0_74934794的博客
07-19 273
2.1.1读取linux文件系统的文件创建RDD。2.1.2读取hdfs文件系统上的文件创建RDD。--需要保证每一个worker中都有该文件。--过滤数据,保留每行中含有hello的数据。2.2使用Parallelize创建RDD。2.3从其他的RDD中创建新的RDD。--对RDD中的数据进行过滤操作。--统计RDD中的数据记录表。2.1 读取文件创建RDD。--读取RDD的第一条数据。4.实现wordcount。1.进行shell命令行。3.对RDD进行操作。--读取中的N条数据。
从数据湖到湖仓一体：统一数据架构演进之路
探索大数据世界 - 深入解析数据存储、分布式计算与人工智能
07-17 1363
本文详细探讨了从数据湖到湖仓一体的演进过程及其在现代数据管理中的重要性。湖仓一体由Databricks于2020年提出，旨在融合数据湖和数据仓库的优势，通过无缝集成，实现数据存储和管理的统一架构。 湖仓一体的出现主要是为了降低成本和解决数据孤岛问题。传统的数据湖和数据仓库各自独立建设，导致重复投资和数据管理上的困难，而湖仓一体通过简化基础架构、提升数据存储弹性和质量，有效减少了这些问题。
《基于 CDC、Spark Streaming、Kafka 实现患者指标采集》
山人行
07-18 1385
本篇文章分享一下博主所在公司的患者指标采集的解决方案。主要是基于实现，由于涉及公司隐私，内容主要以方案介绍为主，有需要探讨的可以留言。好，让我们开始。上文介绍了博主所在公司的《基于 CDC、Spark Streaming、Kafka 实现患者指标采集》方案。💗 后续会逐步分享企业实际开发中的实战经验，有需要交流的可以联系博主。
通过 EMR Serverless Spark 提交 PySpark 流任务
科技很有意思
07-19 755
本文将指导您使用EMR Serverless Spark提交PySpark流式任务，展示其在流处理方面的易用性和可运维性。
侧信道分析基础：揭秘加密系统的漏洞
"基础知识：边信道分析" 边信道分析是信息安全领域中的一个重要概念，主要关注的是在经典密码学理论之外，攻击者如何利用超出黑盒模型的额外信息来破解加密系统。传统密码学假设攻击者只能获取加密系统的输入和输出...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值