spark -- PCA

Spark PCA

在机器学习或者数据挖掘中,得到的数据往往维度过高，含有噪音，需要把多指标转化为少数几个综合指标的数据。pca是机器学习框架中常用的一个功能，spark机器模块也实现了这一功能。

---

PCA主要的几个方法

设置输入项的字段

def setInputCol(value: String): this.type = set(inputCol, value)

设置输出项的字段

def setOutputCol(value: String): this.type = set(outputCol, value)

设置转化的维度个数

def setK(value: Int): this.type = set(k, value)

训练模型

def fit(dataset: Dataset[_]): PCAModel

利用模型转化数据

def transform(dataset: Dataset[_]): DataFrame

---

Spark PCA 示例

    val spark = SparkSession.builder().appName("pca").master("local[4]").getOrCreate()
    val file = spark.read.format("csv")
      .option("sep",",")
      .option("header","true")
      .load("boston_house_prices.csv")
    file.show(true)
    import spark.implicits._
    //打乱顺序
    val rand = new Random()
    val data = file.select("MEDV", "CRIM", "ZN", "INDUS", "CHAS", "NOX", "RM", "AGE", "DIS", "RAD", "TAX", "PTRATIO", "B", "LSTAT").map(
      row => (row.getAs[String](0).toDouble, row.getString(1).toDouble, row.getString(2).toDouble, row.getString(3).toDouble, row.getString(4).toDouble, row.getString(5).toDouble, row.getString(6).toDouble, row.getString(7).toDouble, row.getString(8).toDouble, row.getString(9).toDouble, row.getString(10).toDouble, row.getString(11).toDouble, row.getString(12).toDouble, row.getString(13).toDouble, rand.nextDouble()))
      .toDF("price", "crim", "zn", "indus", "chas", "nox", "rm", "age", "dis", "rad", "tax", "ptratio", "b", "lstat", "rand").sort("rand") //强制类型转换过程

    data.show(true)
    val assembler = new VectorAssembler().setInputCols(Array("crim", "zn", "indus", "chas", "nox", "rm", "age", "dis", "rad", "tax", "ptratio", "b", "lstat", "rand")).setOutputCol("features")
    val pca = new PCA().setInputCol("features").setOutputCol("featuresPca").setK(3)
    val assembler_data = assembler.transform(data)
    val pca_model = pca.fit(assembler_data)
    val pca_data = pca_model.transform(assembler_data)
    pca_data.select("features","featuresPca").show(false)

数据从 64 维度降到3 维

---

---

与朱元思书

【作者】吴均 【朝代】南北朝

风烟俱净，天山共色。从流飘荡，任意东西。自富阳至桐庐一百许里，奇山异水，天下独绝。

水皆缥碧，千丈见底。游鱼细石，直视无碍。急湍甚箭，猛浪若奔。

夹岸高山，皆生寒树，负势竞上，互相轩邈，争高直指，千百成峰。

泉水激石，泠泠作响；好鸟相鸣，嘤嘤成韵。蝉则千转不穷，猿则百叫无绝。鸢飞戾天者，望峰息心；

经纶世务者，窥谷忘反。横柯上蔽，在昼犹昏；疏条交映，有时见日。