Singular value decomposition
奇异值分解 具体的数学算法不在本文讨论
实现代码
public class Test {
public static void main(String[] args) {
SparkConf conf = new SparkConf().setAppName("svd").setMaster("local[*]");
JavaSparkContext jsc = new JavaSparkContext(conf);
double[][] array = {{1.12, 2.05, 3.12}, {5.56, 6.28, 8.94}, {10.2, 8.0, 20.5}};
LinkedList<Vector> rowsList = new LinkedList<Vector>();
for (int i = 0; i < array.length; i++) {
Vector currentRow = (Vector) Vectors.dense(array[i]);
rowsList.add(currentRow);
}
JavaRDD<Vector> rows = jsc .parallelize(rowsList);
// Create a RowMatrix from JavaRDD<Vector>.
RowMatrix mat = new RowMatrix(rows.rdd());
// Compute the top 3 singular values and corresponding singular vectors.
SingularValueDecomposition<RowMatrix, Matrix> svd = mat.computeSVD(3, true, 1.0E-9d);
RowMatrix U = svd.U();
Vector s = svd.s();
Matrix V = svd.V();
Vector[] collectPartitions = (Vector[]) U.rows().collect();
System.out.println("U factor is:");
for (Vector vector : collectPartitions) {
System.out.println("\t" + vector);
}
System.out.println("s factor is:"+s);
System.out.println("V factor is:"+V);
jsc.close();
}
}
/*输出为:
U factor is:
[-0.13944608689158527,-0.20702967168806308,0.9683457563759816]
[-0.4417287662400154,-0.862205021024284,-0.24794797598810003]
[-0.8862451612994557,0.46232155126194074,-0.028780154830265303]
s factor is:[27.338366804279364,2.4372463015716503,0.6560723589297215]
V factor is:-0.42621025192491035 -0.12721460531662593 -0.8956345601573442
-0.3712687920670484 -0.8782394900003422 0.30142143626658974
-0.8249268483878915 0.460990067569871 0.3270838614342604
*/
具体细节
1.输入若为文件首先要将文件中的数据转化为double类型的二维数组。
double[][] array = {{1.12, 2.05, 3.12}, {5.56, 6.28, 8.94}, {10.2, 8.0, 20.5}};
2.将每一行数据转化为Vector的数据类型,通过Vectors.dense方法完成
LinkedList<Vector> rowsList = new LinkedList<>();
for (int i = 0; i < array.length; i++) {
Vector currentRow = Vectors.dense(array[i]);
rowsList.add(currentRow);
}
3.将数据转化为RDD形式
JavaRDD<Vector> rows = jsc.parallelize(rowsList);
4.将RDD变为一个面向行的分布矩阵并计算
RowMatrix mat = new RowMatrix(rows.rdd());
//computeSVD的三个参数分别代表要保留的奇异值数目,是否计算U,一个小于什么值就不计的下限
SingularValueDecomposition<RowMatrix, Matrix> svd = mat.computeSVD(3, true, 1.0E-9d);
RowMatrix U = svd.U();
Vector s = svd.s();
Matrix V = svd.V();