KMeans将数据分为K类。第一次随机选择k个点做质心。求每个数据与每个质心(迪卡尔积)的欧式距离,并选择最短距离。如此将数据分为k类。每类数据取平均值将结果作为质心重新计算欧式距离。直到质心基本不变(误差平方和最小)为止。 public class KMean { public static void main(String[] args) { if(args.length<3){ System.out.println("error "); System.exit(1); } String inputFile=args[0]; int k=new Integer(args[1]); int iterations=new Integer(args[2]); int runs=1; if(args.length>4){ runs=new Integer(args[3]); } SparkConf conf = new SparkConf(); JavaSparkContext sc = new JavaSparkContext(conf); JavaRDD<String> file = sc.textFile(inputFile); Pattern pattern=Pattern.compile(" ");//预编译 split()每次都会编译,效率低 JavaRDD<Vector> map = file.map(x -> { String[] split = pattern.split(x); double[] point = new double[split.length]; for (int i = 0; i < split.length; i++) { point[i] = new Double(split[i]); } return Vectors.dense(point); }); KMeansModel train = KMeans.train(map.rdd(),k,iterations,runs,KMeans.K_MEANS_PARALLEL()); for(Vector v:train.clusterCenters()){ System.out.println(v);//质心 } double cost = train.computeCost(map.rdd()); System.out.println(cost);//代价 距离平方和 sc.stop();
spark 机器学习包实现KMeans
最新推荐文章于 2021-11-08 11:49:33 发布