MLlib学习之Java实现SVD

Singular value decomposition

奇异值分解 具体的数学算法不在本文讨论

实现代码

public class Test {

    public static void main(String[] args) {


        SparkConf conf = new SparkConf().setAppName("svd").setMaster("local[*]");
        JavaSparkContext jsc = new JavaSparkContext(conf);
        double[][] array = {{1.12, 2.05, 3.12}, {5.56, 6.28, 8.94}, {10.2, 8.0, 20.5}};

        LinkedList<Vector> rowsList = new LinkedList<Vector>();

        for (int i = 0; i < array.length; i++) {
          Vector currentRow = (Vector) Vectors.dense(array[i]);
          rowsList.add(currentRow);
        }
        JavaRDD<Vector> rows = jsc .parallelize(rowsList);

        // Create a RowMatrix from JavaRDD<Vector>.
        RowMatrix mat = new RowMatrix(rows.rdd());

        // Compute the top 3 singular values and corresponding singular vectors.
        SingularValueDecomposition<RowMatrix, Matrix> svd = mat.computeSVD(3, true, 1.0E-9d);
        RowMatrix U = svd.U();
        Vector s = svd.s();
        Matrix V = svd.V();

        Vector[] collectPartitions = (Vector[]) U.rows().collect();
        System.out.println("U factor is:");
        for (Vector vector : collectPartitions) {
          System.out.println("\t" + vector);
        }
        System.out.println("s factor is:"+s);
        System.out.println("V factor is:"+V);
        jsc.close();
    }

}
/*输出为:
U factor is:
    [-0.13944608689158527,-0.20702967168806308,0.9683457563759816]
    [-0.4417287662400154,-0.862205021024284,-0.24794797598810003]
    [-0.8862451612994557,0.46232155126194074,-0.028780154830265303]
s factor is:[27.338366804279364,2.4372463015716503,0.6560723589297215]
V factor is:-0.42621025192491035  -0.12721460531662593  -0.8956345601573442  
-0.3712687920670484   -0.8782394900003422   0.30142143626658974  
-0.8249268483878915   0.460990067569871     0.3270838614342604   
*/

具体细节

1.输入若为文件首先要将文件中的数据转化为double类型的二维数组。

double[][] array = {{1.12, 2.05, 3.12}, {5.56, 6.28, 8.94}, {10.2, 8.0, 20.5}};

2.将每一行数据转化为Vector的数据类型，通过Vectors.dense方法完成

LinkedList<Vector> rowsList = new LinkedList<>();
for (int i = 0; i < array.length; i++) {
  Vector currentRow = Vectors.dense(array[i]);
  rowsList.add(currentRow);
}

3.将数据转化为RDD形式

JavaRDD<Vector> rows = jsc.parallelize(rowsList);

4.将RDD变为一个面向行的分布矩阵并计算

RowMatrix mat = new RowMatrix(rows.rdd());
//computeSVD的三个参数分别代表要保留的奇异值数目，是否计算U，一个小于什么值就不计的下限
SingularValueDecomposition<RowMatrix, Matrix> svd = mat.computeSVD(3, true, 1.0E-9d);
RowMatrix U = svd.U();
Vector s = svd.s();
Matrix V = svd.V();