mahout中有很多经典的数据挖掘算法,其中kmeans算法比较简单,学习mahout的道路就从这里开始。
kmeans的原理就不说了,直接来看它的源码。根据运行流程一步一步来
org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
首先:是主函数
public static void main(String[] args) throws Exception {
if (args.length > 0) {
log.info("Running with only user-supplied arguments");
ToolRunner.run(new Configuration(), new Job(), args);
} else {
log.info("Running with default arguments");
Path output = new Path("output");
Configuration conf = new Configuration();
HadoopUtil.delete(conf, output);
run(conf, new Path("testdata"), output, new EuclideanDistanceMeasure(), 6, 0.5, 10);
}
}
没啥好说的,有参数与没参数调用的run方法不同,没参数会自动填上参数。等下会说明kmeans的各个参数什么意思。
下面是有参数的run方法:
public int run(String[] args) throws Exception {
addInputOption();
addOutputOption();
addOption(DefaultOptionCreator.distanceMeasureOption().create());
addOption(DefaultOptionCreator.numClustersOption().create());
addOption(DefaultOptionCreator.t1Option().create());
addOption(DefaultOptionCreator.t2Option().create());
addOption(DefaultOptionCreator.convergenceOption().create());
addOption(DefaultOptionCreator.maxIterationsOption().create());
addOption(DefaultOptionCreator.overwriteOption().create());
Map<String,List<String>> argMap = parseArguments(args);
if (argMap == null) {
return -1;
}
Path input = getInputPath();
Path output = getOutputPath();
String measureClass = getOption(DefaultOptionCreator.DISTANCE_MEASURE_OPTION);
if (measureClass == null) {
measureClass = SquaredEuclideanDistanceMeasure.class.getName();
}
double convergenceDelta = Double.parseDouble(getOption(DefaultOptionCreator.CONVERGENCE_DELTA_OPTION));
int maxIterations = Integer.parseInt(getOption(DefaultOptionCreator.MAX_ITERATIONS_OPTION));
if (hasOption(DefaultOptionCreator.OVERWRITE_OPTION)) {
HadoopUtil.delete(getConf(), output);
}
DistanceMeasure measure = ClassUtils.instantiateAs(measureClass, DistanceMeasure.class);
if (hasOption(DefaultOptionCreator.NUM_CLUSTERS_OPTION)) {
int k = Integer.parseInt(getOption(DefaultOptionCreator.NUM_CLUSTERS_OPTION));
run(getConf(), input, output, measure, k, convergenceDelta, maxIterations);
} else {
double t1 = Double.parseDouble(getOption(DefaultOptionCreator.T1_OPTION));
double t2 = Double.parseDouble(getOption(DefaultOptionCreator.T2_OPTION));
run(getConf(), input, output, measure, t1, t2, convergenceDelta, maxIterations);
}
return 0;
}
有几个重要的参数说明下:
1.Path input 输入路径
2.Path output 输出路径
3.DistanceMeasure measure :数据点间的距离计算方法,默认是SquaredEuclidean方法
可选ChebyshevDistanceMeasure 切比雪夫距离
CosineDistanceMeasure 余弦距离
EuclideanDistanceMeasure 欧氏距离
MahalanobisDistanceMeasure 马氏距离
ManhattanDistanceMeasure 曼哈顿距离
MinkowskiDistanceMeasure 闵可夫斯基距离
SquaredEuclideanDistanceMeasure 欧氏距离 ( 不采取平方根 )
TanimotoDistanceMeasure Tanimoto 系数距离
还有一些基于权重的距离计算方法:
WeightedDistanceMeasure
WeightedEuclideanDistanceMeasure 、 WeightedManhattanDistanceMeasure
4.int numClustersOption(即k):簇个数,如果指定k个数,那么会产生随机的k个中心点。否则会用canopy算法进行预处理
5.int maxIterations:最大迭代次数(重要)
6. Double convergenceDelta: 收敛系数 新的簇中心与上次的簇中心的的距离不能超过 convergenceDelta ,如果超过,则继续迭代,否则停止迭代。参数可缺,默认值是 0.5
可以看到上面有两个run方法,如果指定K个数,会调用第一个run方法;没有指定的话就要调用第二个run方法,它要求输入T1,T2两个参数,它们与canopy算法有关,暂时不考虑
我们指定k的个数,然后来看这个run方法:(kmeans主函数)
public static void run(Configuration conf, Path input, Path output, DistanceMeasure measure, int k,
double convergenceDelta, int maxIterations) throws Exception {
Path directoryContainingConvertedInput = new Path(output, DIRECTORY_CONTAINING_CONVERTED_INPUT);
log.info("Preparing Input");
InputDriver.runJob(input, directoryContainingConvertedInput, "org.apache.mahout.math.RandomAccessSparseVector");
log.info("Running random seed to get initial clusters");
Path clusters = new Path(output, "random-seeds");
clusters = RandomSeedGenerator.buildRandom(conf, directoryContainingConvertedInput, clusters, k, measure);
log.info("Running KMeans with k = {}", k);
KMeansDriver.run(conf, directoryContainingConvertedInput, clusters, output, convergenceDelta,
maxIterations, true, 0.0, false);
// run ClusterDumper
Path outGlob = new Path(output, "clusters-*-final");
Path clusteredPoints = new Path(output,"clusteredPoints");
log.info("Dumping out clusters from clusters: {} and clusteredPoints: {}", outGlob, clusteredPoints);
ClusterDumper clusterDumper = new ClusterDumper(outGlob, clusteredPoints);
clusterDumper.printClusters(null);
}
可以清楚的看到里面调用了4个方法:这里大概说下,详细的内容另外单独说明
一.
InputDriver.runJob(input, directoryContainingConvertedInput, "org.apache.mahout.math.RandomAccessSparseVector");
这个方法将原始文件处理为合适的数据格式
二.
clusters = RandomSeedGenerator.buildRandom(conf, directoryContainingConvertedInput, clusters, k, measure);
这个方法将在原始数据中随机选出k个元素作为中心点
三.
KMeansDriver.run(conf, directoryContainingConvertedInput, clusters, output, convergenceDelta, maxIterations, true, 0.0, false);
kmeans主函数,后面详细说明
四.
ClusterDumper clusterDumper = new ClusterDumper(outGlob, clusteredPoints);
clusterDumper.printClusters(null);
输出聚类结果
以上是对kmeans的一个宏观上的了解。