Spark0.9分布式运行MLlib的二元分类算法

最新推荐文章于 2022-10-23 10:08:22 发布
最新推荐文章于 2022-10-23 10:08:22 发布
阅读量3.3k 收藏
点赞数
分类专栏： 大数据 文章标签： Spark MLlib

MLlib是的Spark实现一些常见的机器学习（ML）的功能并且提供一些相关的测试和数据生成器的功能。 MLlib目前支持4种常见的类型的机器学习问题的设定，即，二元分类，回归，聚类和协同过滤，以及一个原始梯度下降优化。这个指南将概述在MLlib所支持的功能，并且还提供了如何调用MLlib的一些例子。

依赖库

MLlib使用jblas线性代数库,它本身取决于本地Fortran程序。如果不是已经安装在你的节点，你可能需要安装gfortran运行时库。如果不能自动检测到这些库，MLlib将抛出一个链接错误。
使用MLlib在Python中,您将需要安装1.7或更新版本的NumPy和Python 2.7。

二元分类含义

二元分类是一种监督学习算法问题,我们想将实体为两种不同的类别或标签,如,预测邮件是否是垃圾邮件。这个问题涉及到通过执行一组打标签的数据集来进行学习的算法,比如,一组通过(数值)来代表特性以及分类标签的数据实体。算法会返回训练模型,该训练模型可以预测那些未知标签的新实体的标签。
MLlib目前支持两种标准的二元分类模型,即线性支持向量机(向量机)和逻辑回归以及对于每个算法模型的随机变量的L1和L2规则化算法。所有的算法都会利用原始梯度下降训练算法的(在下面描述),并采取作为输入正则化参数(regParam)以及各种数与梯度下降相关的参数(stepSize,numIterations miniBatchFraction)。
可用的二元分类算法:
SVMWithSGD
LogisticRegressionWithSGD

scala模板建立

为了让sbt正确工作,我们需要正确放置SimpleApp位置。scala和simple.scalat根据下面典型的目录结构来放置。一旦建立好了模板,我们可以通过编译来创建一个JAR包，它包含应用程序的代码,然后使用sbt/sbt run来执行我们编写的程序。


find .
./scala/sample/scala/lib
./scala/sample/scala/lib/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar
./scala/sbt
./scala/sbt/sbt-launch-1.jar
./scala/sbt/sbt
./scala/src
./scala/src/main
./scala/src/main/scala
./scala/src/main/scala/SimpleApp.scala

二元分类代码

下面的代码片段说明了如何加载一个样本数据集,对训练数据集执行训练算法，在代码中我们使用静态对象,并计算出实际结果和预测模型计算训练结果之间的误差。 

import org.apache.spark.SparkContext
import org.apache.spark.mllib.classification.SVMWithSGD
import org.apache.spark.mllib.regression.LabeledPoint

object SimpleApp {
  def main(args: Array[String]) {
    val sc = new SparkContext("spark://192.168.159.129:7077", "Simple App", "/root/spark-0.9",
      List("target/scala-2.10/scala_2.10-0.1-SNAPSHOT.jar"))
    val data = sc.textFile("hdfs://master:9000/mllib/sample_svm_data.txt")
    val parsedData = data.map { line =>
    val parts = line.split(' ')
    LabeledPoint(parts(0).toDouble, parts.tail.map(x => x.toDouble).toArray)
    }
  
    // Run training algorithm to build the model
    val numIterations = 20
    val model = SVMWithSGD.train(parsedData, numIterations)
    
    // Evaluate model on training examples and compute training error
    val labelAndPreds = parsedData.map { point =>
      val prediction = model.predict(point.features)
      (point.label, prediction)
    }
    val trainErr = labelAndPreds.filter(r => r._1 != r._2).count.toDouble / parsedData.count
    println("Training Error = " + trainErr)
    }
}
执行结果 

[root@master spark-0.9]# cd /root/sample/scala
[root@master scala]# sbt/sbt package run
[info] Set current project to scala (in build file:/root/sample/scala/)
[info] Compiling 1 Scala source to /root/sample/scala/target/scala-2.10/classes...
[info] Packaging /root/sample/scala/target/scala-2.10/scala_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[success] Total time: 15 s, completed Feb 10, 2014 11:27:51 PM
[info] Running SimpleApp
log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jLogger).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
14/02/10 23:27:54 INFO SparkEnv: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/02/10 23:27:54 INFO SparkEnv: Registering BlockManagerMaster
14/02/10 23:27:54 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140210232754-e3bb
14/02/10 23:27:54 INFO MemoryStore: MemoryStore started with capacity 580.0 MB.
14/02/10 23:27:54 INFO ConnectionManager: Bound socket to port 48916 with id = ConnectionManagerId(master,48916)
14/02/10 23:27:54 INFO BlockManagerMaster: Trying to register BlockManager
14/02/10 23:27:54 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager master:48916 with 580.0 MB RAM
14/02/10 23:27:54 INFO BlockManagerMaster: Registered BlockManager
14/02/10 23:27:54 INFO HttpServer: Starting HTTP Server
14/02/10 23:27:55 INFO HttpBroadcast: Broadcast server started at http://192.168.159.129:49765
14/02/10 23:27:55 INFO SparkEnv: Registering MapOutputTracker
14/02/10 23:27:55 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b309992e-6b24-4823-9ce7-68ff0ee6ec1a
14/02/10 23:27:55 INFO HttpServer: Starting HTTP Server
14/02/10 23:27:56 INFO SparkUI: Started Spark Web UI at http://master:4040
14/02/10 23:27:56 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/02/10 23:27:56 INFO SparkContext: Added JAR target/scala-2.10/scala_2.10-0.1-SNAPSHOT.jar at http://192.168.159.129:35769/jars/scala_2.10-0.1-SNAPSHOT.jar with timestamp 1392046076889
14/02/10 23:27:56 INFO AppClient$ClientActor: Connecting to master spark://192.168.159.129:7077...
14/02/10 23:27:57 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
14/02/10 23:27:57 INFO MemoryStore: ensureFreeSpace(132636) called with curMem=0, maxMem=608187187
14/02/10 23:27:57 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated size 129.5 KB, free 579.9 MB)
14/02/10 23:27:58 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140210232758-0007
14/02/10 23:27:58 INFO AppClient$ClientActor: Executor added: app-20140210232758-0007/0 on worker-20140210205103-slaver01-37106 (slaver01:37106) with 1 cores
14/02/10 23:27:58 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140210232758-0007/0 on hostPort slaver01:37106 with 1 cores, 512.0 MB RAM
14/02/10 23:27:58 INFO AppClient$ClientActor: Executor added: app-20140210232758-0007/1 on worker-20140210205049-slaver02-48689 (slaver02:48689) with 1 cores
9:16 INFO DAGScheduler: Final stage: Stage 15 (reduce at GradientDescent.scala:150)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 15 (MappedRDD[30] at map at GradientDescent.scala:145), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 15 (MappedRDD[30] at map at GradientDescent.scala:145)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 15.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 15.0:0 as TID 28 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 15.0:0 as 2501 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 15.0:1 as TID 29 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 15.0:1 as 2501 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 29 in 59 ms on slaver01 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(15, 1)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 28 in 64 ms on slaver02 (progress: 1/2)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 15.0 from pool
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(15, 0)
14/02/10 23:29:16 INFO DAGScheduler: Stage 15 (reduce at GradientDescent.scala:150) finished in 0.062 s
14/02/10 23:29:16 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.079776485 s
14/02/10 23:29:16 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
14/02/10 23:29:16 INFO DAGScheduler: Got job 16 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 16 (reduce at GradientDescent.scala:150)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 16 (MappedRDD[32] at map at GradientDescent.scala:145), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 16 (MappedRDD[32] at map at GradientDescent.scala:145)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 16.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 16.0:0 as TID 30 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 16.0:0 as 2504 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 16.0:1 as TID 31 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 16.0:1 as 2504 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 31 in 32 ms on slaver01 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(16, 1)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 30 in 65 ms on slaver02 (progress: 1/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(16, 0)
14/02/10 23:29:16 INFO DAGScheduler: Stage 16 (reduce at GradientDescent.scala:150) finished in 0.068 s
14/02/10 23:29:16 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.084612863 s
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 16.0 from pool
14/02/10 23:29:16 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
14/02/10 23:29:16 INFO DAGScheduler: Got job 17 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 17 (reduce at GradientDescent.scala:150)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 17 (MappedRDD[34] at map at GradientDescent.scala:145), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 17 (MappedRDD[34] at map at GradientDescent.scala:145)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 17.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 17.0:0 as TID 32 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 17.0:0 as 2500 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 17.0:1 as TID 33 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 17.0:1 as 2500 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 32 in 47 ms on slaver02 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(17, 0)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 33 in 75 ms on slaver01 (progress: 1/2)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 17.0 from pool
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(17, 1)
14/02/10 23:29:16 INFO DAGScheduler: Stage 17 (reduce at GradientDescent.scala:150) finished in 0.070 s
14/02/10 23:29:16 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.084426168 s
14/02/10 23:29:16 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
14/02/10 23:29:16 INFO DAGScheduler: Got job 18 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 18 (reduce at GradientDescent.scala:150)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 18 (MappedRDD[36] at map at GradientDescent.scala:145), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 18 (MappedRDD[36] at map at GradientDescent.scala:145)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 18.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 18.0:0 as TID 34 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 18.0:0 as 2504 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 18.0:1 as TID 35 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 18.0:1 as 2504 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 34 in 40 ms on slaver02 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(18, 0)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 35 in 81 ms on slaver01 (progress: 1/2)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 18.0 from pool
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(18, 1)
14/02/10 23:29:16 INFO DAGScheduler: Stage 18 (reduce at GradientDescent.scala:150) finished in 0.079 s
14/02/10 23:29:16 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.09669554 s
14/02/10 23:29:16 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
14/02/10 23:29:16 INFO DAGScheduler: Got job 19 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 19 (reduce at GradientDescent.scala:150)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 19 (MappedRDD[38] at map at GradientDescent.scala:145), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 19 (MappedRDD[38] at map at GradientDescent.scala:145)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 19.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 19.0:0 as TID 36 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 19.0:0 as 2502 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 19.0:1 as TID 37 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 19.0:1 as 2502 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 36 in 63 ms on slaver02 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(19, 0)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 37 in 80 ms on slaver01 (progress: 1/2)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 19.0 from pool
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(19, 1)
14/02/10 23:29:16 INFO DAGScheduler: Stage 19 (reduce at GradientDescent.scala:150) finished in 0.076 s
14/02/10 23:29:16 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.090877223 s
14/02/10 23:29:16 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
14/02/10 23:29:16 INFO DAGScheduler: Got job 20 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 20 (reduce at GradientDescent.scala:150)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 20 (MappedRDD[40] at map at GradientDescent.scala:145), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 20 (MappedRDD[40] at map at GradientDescent.scala:145)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 20.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 20.0:0 as TID 38 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 20.0:0 as 2499 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 20.0:1 as TID 39 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 20.0:1 as 2499 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 39 in 57 ms on slaver01 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(20, 1)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 38 in 64 ms on slaver02 (progress: 1/2)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 20.0 from pool
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(20, 0)
14/02/10 23:29:16 INFO DAGScheduler: Stage 20 (reduce at GradientDescent.scala:150) finished in 0.061 s
14/02/10 23:29:16 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.071109426 s
14/02/10 23:29:16 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
14/02/10 23:29:16 INFO DAGScheduler: Got job 21 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 21 (reduce at GradientDescent.scala:150)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 21 (MappedRDD[42] at map at GradientDescent.scala:145), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 21 (MappedRDD[42] at map at GradientDescent.scala:145)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 21.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 21.0:0 as TID 40 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 21.0:0 as 2500 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 21.0:1 as TID 41 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 21.0:1 as 2500 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 41 in 43 ms on slaver01 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(21, 1)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 40 in 55 ms on slaver02 (progress: 1/2)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 21.0 from pool
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(21, 0)
14/02/10 23:29:16 INFO DAGScheduler: Stage 21 (reduce at GradientDescent.scala:150) finished in 0.052 s
14/02/10 23:29:16 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.0626958 s
14/02/10 23:29:16 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
14/02/10 23:29:16 INFO DAGScheduler: Got job 22 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 22 (reduce at GradientDescent.scala:150)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 22 (MappedRDD[44] at map at GradientDescent.scala:145), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 22 (MappedRDD[44] at map at GradientDescent.scala:145)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 22.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 22.0:0 as TID 42 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 22.0:0 as 2503 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 22.0:1 as TID 43 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 22.0:1 as 2503 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 43 in 44 ms on slaver01 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(22, 1)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 42 in 54 ms on slaver02 (progress: 1/2)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 22.0 from pool
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(22, 0)
14/02/10 23:29:16 INFO DAGScheduler: Stage 22 (reduce at GradientDescent.scala:150) finished in 0.051 s
14/02/10 23:29:16 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.060071497 s
14/02/10 23:29:16 INFO GradientDescent: GradientDescent finished. Last 10 stochastic losses 1.973918565153662, 1.8255523040966746, 1.7699816024631967, 1.6469799583886178, 1.625661917005991, 1.5283113889552784, 1.5173506129512995, 1.422277398167446, 1.4154959896484256, 1.3621279370271806
14/02/10 23:29:16 INFO SVMWithSGD: Final model weights 0.14951408585149972,0.03831072711197627,0.037730161810440484,0.18505277569820583,-2.563032483490213E-4,0.07950273502493031,0.0946837869570233,0.007664328764458717,0.12219548598644159,0.12219548598644195,0.034482086651882085,0.035443622005655644,0.02700659703930399,-0.002137650963695721,0.007242361663251616,0.020208016800350677
14/02/10 23:29:16 INFO SVMWithSGD: Final model intercept 0.06977506975495361
14/02/10 23:29:16 INFO SparkContext: Starting job: count at SimpleApp.scala:24
14/02/10 23:29:16 INFO DAGScheduler: Got job 23 (count at SimpleApp.scala:24) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 23 (count at SimpleApp.scala:24)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 23 (FilteredRDD[46] at filter at SimpleApp.scala:24), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 23 (FilteredRDD[46] at filter at SimpleApp.scala:24)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 23.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 23.0:0 as TID 44 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 23.0:0 as 2164 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 23.0:1 as TID 45 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 23.0:1 as 2164 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 45 in 54 ms on slaver01 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(23, 1)
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 44 in 73 ms on slaver02 (progress: 1/2)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Remove TaskSet 23.0 from pool
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(23, 0)
14/02/10 23:29:16 INFO DAGScheduler: Stage 23 (count at SimpleApp.scala:24) finished in 0.069 s
14/02/10 23:29:16 INFO SparkContext: Job finished: count at SimpleApp.scala:24, took 0.089120924 s
14/02/10 23:29:16 INFO SparkContext: Starting job: count at SimpleApp.scala:24
14/02/10 23:29:16 INFO DAGScheduler: Got job 24 (count at SimpleApp.scala:24) with 2 output partitions (allowLocal=false)
14/02/10 23:29:16 INFO DAGScheduler: Final stage: Stage 24 (count at SimpleApp.scala:24)
14/02/10 23:29:16 INFO DAGScheduler: Parents of final stage: List()
14/02/10 23:29:16 INFO DAGScheduler: Missing parents: List()
14/02/10 23:29:16 INFO DAGScheduler: Submitting Stage 24 (MappedRDD[2] at map at SimpleApp.scala:10), which has no missing parents
14/02/10 23:29:16 INFO DAGScheduler: Submitting 2 missing tasks from Stage 24 (MappedRDD[2] at map at SimpleApp.scala:10)
14/02/10 23:29:16 INFO TaskSchedulerImpl: Adding task set 24.0 with 2 tasks
14/02/10 23:29:16 INFO TaskSetManager: Starting task 24.0:0 as TID 46 on executor 1: slaver02 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 24.0:0 as 1726 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Starting task 24.0:1 as TID 47 on executor 0: slaver01 (NODE_LOCAL)
14/02/10 23:29:16 INFO TaskSetManager: Serialized task 24.0:1 as 1726 bytes in 0 ms
14/02/10 23:29:16 INFO TaskSetManager: Finished TID 47 in 35 ms on slaver01 (progress: 0/2)
14/02/10 23:29:16 INFO DAGScheduler: Completed ResultTask(24, 1)
14/02/10 23:29:17 INFO TaskSetManager: Finished TID 46 in 141 ms on slaver02 (progress: 1/2)
14/02/10 23:29:17 INFO TaskSchedulerImpl: Remove TaskSet 24.0 from pool
14/02/10 23:29:17 INFO DAGScheduler: Completed ResultTask(24, 0)
14/02/10 23:29:17 INFO DAGScheduler: Stage 24 (count at SimpleApp.scala:24) finished in 0.144 s
14/02/10 23:29:17 INFO SparkContext: Job finished: count at SimpleApp.scala:24, took 0.165520408 s
Training Error = 0.4968944099378882
14/02/10 23:29:17 INFO ConnectionManager: Selector thread was interrupted!
[success] Total time: 85 s, completed Feb 10, 2014 11:29:17 PM
[root@master scala]#
默认SVMWithSGD.train()方法执行L2正规化算法，正则化参数设置为1.0。如果我们想要配置这个算法,我们可以进一步设定SVMWithSGD的属性,可以直接通过创建一个新的SVMWithSGD对象和调用setter方法。所有其他MLlib算法都可以用这种方式来定制。例如,下面的代码通过正则化参数设置为0.1来产生的L1正规化随机变量向量机，迭代200次来运行这个训练算法。 
import org.apache.spark.mllib.optimization.L1Updater

val svmAlg = new SVMWithSGD()
svmAlg.optimizer.setNumIterations(200)
  .setRegParam(0.1)
  .setUpdater(new L1Updater)
val modelL1 = svmAlg.run(parsedData)



ericliu2017
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
spark写出分布式的训练算法_Spark0.9分布式运行MLlib的线性回归算法
weixin_30877775的博客
12-29 196
1 什么是线性回归线性回归是另一个传统的有监督机器学习算法。在这个问题中,每个实体与一个实数值的标签 (而不是一个像在二元分类的0,1标签),和我们想要预测标签尽可能给出数值代表实体特征。MLlib支持线性回归以及L2(ridge)和L1(lasso)正则化参数调整。Mllib还有一个回归算法,原始梯度下降(在下面描述),和上面描述的有相同的参数二元分类算法。可用线性回归算法:LinearRegr...
基于Spark的分布式深度学习和认知计算
03-13
你是否曾经面对多个优化算法不知所措？或者无法自由选择学习框架？又或许因为Caffe,Tensorflow, Theano, Torch的诸多参数设置而烦恼？或简单的认为只要有大数据就可以训练计算 机了？如果你不懂复杂的数学、统计学理论，还能做训练吗？...... 带着十万个为什么，让我们与深度学习技术讲师一起，了解基于Spark的分布式数据探索、机器学习/深度学习和认知计算。
[机器学习、Spark]Spark MLlib分类
m0_57781407的博客
10-23 3027
线性支持向量机在机器学习领域中是一种常见的判别方法，是一一个有监督学习模型,通常用来进行模式识别，分类以及回归分析。通过找到支持向量从而获得分类平面的方法，称为支持向量机。可以非常成功地处理回归（时间序列分析）和模式识别（分类问题、判别分析）等诸多问题，并可推广到预测和综合评价等领域，因此可应用于理科、工科和管理等多种学科。逻辑回归又称为逻辑回归分析,它是一个概率模型的分类算法，常用于数据挖掘、疾病自动诊断以及经济预测等领域。 逻辑回归和线性回归类似，但它不属于回归分析家族，差异主要是在于变量不同
Spark ML pipeline学习流程 2元分类
a8131357leo的博客
09-16 457
Spark ML使用的数据格式是DataFrame，所以必须使用Dataframe储存处理数据。 准备数据 option('header'):数据是否由标题 option('delimiter')：分隔符 load: 路径 format：读取格式 row_df = sqlContext.read.format('csv')\ .option('header',...
Spark Mllib 下的决策树二元分类 —— 网站分类(1)
尼克不可的博客
04-07 1096
前面一篇文章说了一下基于spark下的协同过滤算法的实现,这篇文章就来讲一下决策树二元分类吧, 这个算法呢主要运用于产品的分类,就好比你要给某人推荐一本书,首先你自己要知道这些书的类型吧,其次你还需要知道你要推荐的这个人他喜欢什么类型,只有书籍的类型和人的喜好匹配上了,这样才能达到推荐的目的;正是在这种场景下就需要我们进行对产品的分类,当然我们人可以很容易的就判断某本书籍是属于什么类型的,但是对于...
spark写出分布式的训练算法_Spark分布式计算执行模型
weixin_39965673的博客
12-29 457
引言相对Hadoop, Spark在处理需要迭代运算的机器学习训练等任务上有着很大性能提升，同时提供了批处理、实时数据处理、机器学习以及图算法等一站式的服务，因此最近大家一起来学习Spark，特别是MLLib。Spark中使用了RDD(Resilient Distributed Datasets, 弹性分布式数据集)抽象分布式计算，即使用RDD以及对应的transform/action等操作来执行...
基于Spark MLlib分布式学习算法的研究.pdf
08-10
#资源达人分享计划#
Spark推荐算法实战MLlib推荐实例
03-07
在大数据分析领域，Spark作为一个高效的分布式计算框架，已经成为许多企业和开发者首选的工具。尤其是它的机器学习库MLlib，为实现各种复杂的数据挖掘任务提供了便利，其中包括推荐系统。本篇文章将深入探讨Spark...
Spark MLlib 机器学习算法与源码
最新发布
09-12
05、Spark MLlib贝叶斯分类算法 06、Spark MLlib决策树算法 07、Spark MLlib KMeans聚类算法 08、Spark MLlib FPGrowth关联规则算法 09、Spark MLlib协同过滤推荐算法 10、Spark MLlib神经网络算法
基于Spark的分布式并行推理算法.pdf
08-09
#资源达人分享计划#
如何使用SparkMLlib构建分类模型?
02-25
1.预测互联网用户对在线广告的点击概率（二分类问题）；2.检测欺诈（二分类问题，欺诈或者不欺诈）；3.预测拖欠贷款（二分类问题）；4.对图片、音频、视频进行分类（多分类问题）；...决策树模型　MLlib构建分类
史上最简单的spark分布式计算教程第一章-快速运行你的第一个spark程序
热门推荐
只有变秃,才能更强
03-08 1万+
快速入门spark: 史上最简单的spark教程 所有代码示例地址:https://github.com/Mydreamandreality/sparkResearch (提前声明:文章由作者:张耀峰结合自己生产中的使用经验整理,最终形成简单易懂的文章,写作不易,转载请注明) (文章参考:Elasticsearch权威指南,Spark快速大数据分析文档,Elasticsearch官方...
spark写出分布式的训练算法_基于Spark的分布式大数据机器学习算法
weixin_39801165的博客
12-19 166
基于Spark的分布式大数据机器学习算法王芮1，韩锐2，贾玉祥1【摘要】摘要：对于大数据而言，机器学习技术是不可或缺的；对于机器学习而言，大规模的数据可以提升模型的精准度。然而复杂的机器学习算法从时间和性能上都急需分布式内存计算这种关键技术。Spark分布式内存计算可以实现算法的并行操作，有利于机器学习算法处理大数据集。因此本文提出在Spark分布式内存环境下实现非线性机器学习算法，其中包括多层可...
lda 可以处理中文_第七篇｜Spark平台下基于LDA的k-means算法实现
weixin_39755003的博客
11-20 346
本文主要在Spark平台下实现一个机器学习应用，该应用主要涉及LDA主题模型以及K-means聚类。通过本文你可以了解到：文本挖掘的基本流程LDA主题模型算法K-means算法Spark平台下LDA主题模型实现Spark平台下基于LDA的K-means算法实现1.文本挖掘模块设计1.1文本挖掘流程文本分析是机器学习中的一个很宽泛的领域，并且在情感分析、聊天机器人、垃圾邮件检测、推荐系统以及自然语言...
Spark大数据分布式图计算处理实战
大数据之眸
02-28 2550
​前言 Spark是一种大规模、快速计算的集群平台，本公众号试图通过学习Spark官网的实战演练笔记提升笔者实操能力以及展现Spark的精彩之处。有关框架介绍和环境配置可以参考以下内容： 大数据处理框架Hadoop、Spark介绍 linux下Hadoop安装与环境配置 linux下Spark安装与环境配置 本文的参考配置为：Deep...
浅谈Spark分布式计算
datacreating的博客
09-24 1789
Spark本身是一个非常庞大的架构，它衍生出了很多框架，每个框架都有不同的设计理念，都值得我们去学习，学无止境，让我们一起学好基础，努力提高自己吧。
spark中local模式与cluster模式使用场景_Spark整合Ray思路漫谈（2）
weixin_39747341的博客
10-21 127
上一篇 关于spark 和ray整合的文章在这：祝威廉：Spark整合Ray思路漫谈​zhuanlan.zhihu.com另外还讲了讲Spark 和Ray 的对比：祝威廉：从MR到Spark再到Ray，谈分布式编程的发展​zhuanlan.zhihu.com现在我们来思考一个比较好的部署模式，架构图大概类似这样：首先，大家可以理解为k8s已经解决一切了，我们spark,ray都跑在K8s上。但是，...
spark写出分布式的训练算法_基于Spark的分布式机器人强化学习训练框架
weixin_35399893的博客
12-29 254
基于Spark的分布式机器人强化学习训练框架方伟1,2，黄增强3，徐建斌4，黄羿1,5，马新强1,5【摘要】摘要：强化学习能够通过自主学习的方式对机器人难以利用控制方法实现的各种任务进行训练完成，有效避免了系统设计人员对系统建模或制定规则。然而，强化学习在机器人开发应用领域中训练成本高昂，需要花费大量时间成本、硬件成本实现学习训练，虽然基于仿真可以一定程度减少硬件成本，但对类似Gazebo这样的复...
分布式系列之分布式分析计算引擎Spark解析
牧羊人的方向
08-04 2209
Spark作为一种通用的大数据分析引擎，集成了批处理、流式查询以及交互式查询于一体，其技术体系相当复杂，本文简要介绍了Spark中的基本架构和基本概念RDD和执行流程，以及Spark on YARN两种模式。...
sparkmllib分类算法之支持向量机
05-27
Spark MLlib中的支持向量机分类算法是一种二元分类器，它将数据集映射到高维空间，然后在该空间中寻找最佳的超平面来区分两个类别。支持向量机最大的优势是在于其对于高维数据的处理能力，可以有效地处理高维数据。 ...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值