Scala实现逻辑回归分类，Titanic

https://yixuan.cos.name/cn/2015/04/spark-beginner-1/

1.读取文件

import scala.io.Source
object myfirst {
  //titanic,LR
  def main(args: Array[String]) {
   val data= Source.fromFile("D:\\IDEA\\_01\\train.csv")
    data.foreach(print)
  }
}

import org.apache.spark.{SparkContext, SparkConf}
object myfirst{
  //titanic,LR
  def main(args: Array[String]) {
    val conf=new SparkConf().setAppName("regression").setMaster("local[1]")
    val sc=new SparkContext(conf)
    val lines = {
      sc.textFile("D:\\IDEA\\_01\\train.csv")
    }
    //lines.foreach(print)
  }
}

2.数据预处理

2.1数据结构

spark MLlib之数据类型http://uohzoaix.github.io/studies/2014/10/14/sparkMLlib(1)/

dense和sparse矩阵

LabeldPoint

import org.apache.spark.mllib.linalg.Vectorsimport org.apache.spark.mllib.regression.LabeledPoint

// Create a labeled point with a positive label and a dense feature vector.

val pos = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))

// Create a labeled point with a negative label and a sparse feature vector.

val neg = LabeledPoint(0.0, Vectors.sparse(3, Array(0, 2), Array(1.0, 3.0)))

2.2特征提取

tfidf

word2vec

countvectorizer

2.3特征转换

https://my.oschina.net/hblt147/blog/1519465

3.建模

• SVMWithSGD
• LogisticRegressionWithLBFGS
• LogisticRegressionWithSGD
• LinearRegressionWithSGD
• RidgeRegressionWithSGD
• LassoWithSGD

模型选择

4.模型训练

5.模型优化

6.模型优化