mahout使用KMeans算法

mahout提供了内存中和分布式的两种KMeans聚类实现。下面是内存中KMeans的代码示例，示例代码使用了最简单的一维向量作为输入：

    /**
     * Tests KMeans cluster algorithm in memory, note the test uses only 1-D vector
     * i.e., a vector of a single double
     */
    public static void TestMemoryKMeans() {
        int MAX = 10000; //set -Xmx512 can hold 8M+ data
        int k = 10;
        double convergence = 0.01;
        DistanceMeasure distanceMeasure = new EuclideanDistanceMeasure();

        //generate random points
        Random rand = new Random();
        List<Double> points = new ArrayList<Double>(MAX);
        for (int i = 0; i < MAX; i++) {
            points.add((double) rand.nextInt(1000));
        }

        //generate vectors from points
        List<Vector> vectors = new ArrayList<Vector>();
        for (Double pt : points) {
            DenseVector v = new DenseVector(new double[]{pt});
            vectors.add(v);
        }

        //create random initial clusters
        List<org.apache.mahout.clustering.kmeans.Cluster> initClusters = new ArrayList<Cluster>(k);
        double min = points.get(0), max = points.get(points.size() - 1);
        double diff = max - min;
        Random rand = new Random();
        for (int i = 0; i < k; i++) {
            DenseVector v = new DenseVector(new double[]{ rand.nextDouble() * diff + min });
            org.apache.mahout.clustering.kmeans.Cluster c = new
                    org.apache.mahout.clustering.kmeans.Cluster(v, i, measure);
            initClusters.add(c);
        }

        List<List<org.apache.mahout.clustering.kmeans.Cluster>> clusteredClusters =
                KMeansClusterer.clusterPoints(vectors, initClusters,
                        new EuclideanDistanceMeasure(), maxIter, distThreshold);

        List<org.apache.mahout.clustering.kmeans.Cluster> finalCluster = clusteredClusters.get(clusteredClusters.size() - 1);
        Collections.sort(finalCluster, new Comparator<Cluster>() {
            @Override
            public int compare(Cluster o1, Cluster o2) {
                return (int) (o1.getCenter().get(0) - o2.getCenter().get(0));
            }
        });
        for (org.apache.mahout.clustering.kmeans.Cluster c : finalCluster) {
            System.out.println("Cluster:" + c.getId() + ",\tCenter:" + (int) (c.getCenter().get(0)) +
                    ", Radius:" + c.getRadius().get(0));
        }
    }

其中的参数如下：

k是聚类的数量，

convergence即拟合度，

maxIter为最大迭代次数，注意算法有可能在到达最大迭代次数前拟合

mahout的内存版本还是能支撑较大数据的，对于一维向量，调整-Xmx768m，可以支撑1000W+的数据。对于多维的，调整一下内存，应该也能支持百W级别的数据量。

集群版本的KMeans示例代码如下：

    /**
     * Tests KMeans cluster algorithm in hadoop.
     */
    public static void TestHadoopKMeans() {
        String input = "/input_vectors/";
        String output = "/output/";

        try {
            Configuration conf = new Configuration();
            //delete output path
            HadoopUtil.delete(conf, new Path(output));

            int k = 10;
            int maxIter = 10;
            double convergence = 0.1;
            DistanceMeasure distanceMeasure = new EuclideanDistanceMeasure();

            Path convertedInput = new Path(output, DIRECTORY_CONTAINING_CONVERTED_INPUT);
            log.info("Preparing Input");
            InputDriver.runJob(input, convertedInput, "org.apache.mahout.math.RandomAccessSparseVector");

            log.info("Running random seed to get initial clusters");
            Path clusters = new Path(output, Cluster.INITIAL_CLUSTERS_DIR);
            clusters = RandomSeedGenerator.buildRandom(conf, convertedInput, clusters, k, measure);

            log.info("Running KMeans");
            KMeansDriver.run(conf, convertedInput, clusters, output,
                    measure, convergenceDelta, maxIter, true, false);

            //dump clusters to local data file
            ClusterDumper dumper = new ClusterDumper(
                    new Path(output, "clusters-" + maxIter), new Path(output, "clusteredPoints"));
            dumper.setOutputFile("/u01/weiyue.wy/projects/price_aggregate/clusters_dump");

            dumper.printClusters(null);
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

注意输入文件必须为textfile格式，每一行表示一个行量。

KMeans的聚类点/向量输出格式比较特殊，为<IntWritable, WeightedVectorWritable>类型的，这种方式对于其他要使用此数据的可能不太友好，因此需要自己写一个job来转化一下格式。

ClusterDumper是mahout自带的一个辅助类，它能将聚类以它包含的所有向量导出到本地文件中（有时仍然未必是我们想要的，因为数据太大）。

如果只是简单地想要聚类向量本身，可以写一个方法直接读取聚类的sequence file的输出。

由于KMeans算法本身的随机性，多次运行的结果可能会出现较大差异。比较好的方式是在运行kmeans前扫描数据集，得到初始的聚类，然后再运行KMeans算法。