mahout提供了内存中和分布式的两种KMeans聚类实现。下面是内存中KMeans的代码示例,示例代码使用了最简单的一维向量作为输入:
/**
* Tests KMeans cluster algorithm in memory, note the test uses only 1-D vector
* i.e., a vector of a single double
*/
public static void TestMemoryKMeans() {
int MAX = 10000; //set -Xmx512 can hold 8M+ data
int k = 10;
double convergence = 0.01;
DistanceMeasure distanceMeasure = new EuclideanDistanceMeasure();
//generate random points
Random rand = new Random();
List<Double> points = new ArrayList<Double>(MAX);
for (int i = 0; i < MAX; i++) {
points.add((double) rand.nextInt(1000));
}
//generate vectors from points
List<Vector> vectors = new ArrayList<Vector>();
for (Double pt : points) {
DenseVector v = new DenseVector(new double[]{pt});
vectors.add(v);
}
//create random initial clusters
List<org.apache.mahout.clustering.kmeans.Cluster> initClusters = new ArrayList<Cluster>(k);
double min = points.get(0), max = points.get(points.size() - 1);
double diff = max - min;
Random rand = new Random();
for (int i = 0; i < k; i++) {
DenseVector v = new DenseVector(new double[]{ rand.nextDouble() * diff + min });
org.apache.mahout.clustering.kmeans.Cluster c = new
org.apache.mahout.clustering.kmeans.Cluster(v, i, measure);
initClusters.add(c);
}
List<List<org.apache.mahout.clustering.kmeans.Cluster>> clusteredClusters =
KMeansClusterer.clusterPoints(vectors, initClusters,
new EuclideanDistanceMeasure(), maxIter, distThreshold);
List<org.apache.mahout.clustering.kmeans.Cluster> finalCluster = clusteredClusters.get(clusteredClusters.size() - 1);
Collections.sort(finalCluster, new Comparator<Cluster>() {
@Override
public int compare(Cluster o1, Cluster o2) {
return (int) (o1.getCenter().get(0) - o2.getCenter().get(0));
}
});
for (org.apache.mahout.clustering.kmeans.Cluster c : finalCluster) {
System.out.println("Cluster:" + c.getId() + ",\tCenter:" + (int) (c.getCenter().get(0)) +
", Radius:" + c.getRadius().get(0));
}
}
其中的参数如下:
k是聚类的数量,
convergence即拟合度,
maxIter为最大迭代次数,注意算法有可能在到达最大迭代次数前拟合
mahout的内存版本还是能支撑较大数据的,对于一维向量,调整-Xmx768m,可以支撑1000W+的数据。对于多维的,调整一下内存,应该也能支持百W级别的数据量。
集群版本的KMeans示例代码如下:
/**
* Tests KMeans cluster algorithm in hadoop.
*/
public static void TestHadoopKMeans() {
String input = "/input_vectors/";
String output = "/output/";
try {
Configuration conf = new Configuration();
//delete output path
HadoopUtil.delete(conf, new Path(output));
int k = 10;
int maxIter = 10;
double convergence = 0.1;
DistanceMeasure distanceMeasure = new EuclideanDistanceMeasure();
Path convertedInput = new Path(output, DIRECTORY_CONTAINING_CONVERTED_INPUT);
log.info("Preparing Input");
InputDriver.runJob(input, convertedInput, "org.apache.mahout.math.RandomAccessSparseVector");
log.info("Running random seed to get initial clusters");
Path clusters = new Path(output, Cluster.INITIAL_CLUSTERS_DIR);
clusters = RandomSeedGenerator.buildRandom(conf, convertedInput, clusters, k, measure);
log.info("Running KMeans");
KMeansDriver.run(conf, convertedInput, clusters, output,
measure, convergenceDelta, maxIter, true, false);
//dump clusters to local data file
ClusterDumper dumper = new ClusterDumper(
new Path(output, "clusters-" + maxIter), new Path(output, "clusteredPoints"));
dumper.setOutputFile("/u01/weiyue.wy/projects/price_aggregate/clusters_dump");
dumper.printClusters(null);
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
注意输入文件必须为textfile格式,每一行表示一个行量。
KMeans的聚类点/向量输出格式比较特殊,为<IntWritable, WeightedVectorWritable>类型的,这种方式对于其他要使用此数据的可能不太友好,因此需要自己写一个job来转化一下格式。
ClusterDumper是mahout自带的一个辅助类,它能将聚类以它包含的所有向量导出到本地文件中(有时仍然未必是我们想要的,因为数据太大)。
如果只是简单地想要聚类向量本身,可以写一个方法直接读取聚类的sequence file的输出。
由于KMeans算法本身的随机性,多次运行的结果可能会出现较大差异。比较好的方式是在运行kmeans前扫描数据集,得到初始的聚类,然后再运行KMeans算法。