KMeans算法原理解析与实现

最新推荐文章于 2024-08-07 13:44:45 发布

weixin_33701564

最新推荐文章于 2024-08-07 13:44:45 发布

阅读量230

点赞数

文章标签：数据结构与算法大数据

原文链接：https://my.oschina.net/eager/blog/678155

版权

为什么80%的码农都做不了架构师？>>>

1、首先得先知道KMeans算法原理：

K-means算法，也被称为k-平均或k-均值算法，算法的主要思想是通过迭代过程把数据集划分为不同的类别，使得评价聚类性能的准则函数达到最优（平均误差准则函数E ）。下面用一个直观的图和相应文字来解释。

上图中，其中A, B, C, D, E 是五个在普通点，而灰色的点是种子点，即种子点共2个，所以K=2。

然后，K-Means的算法如下：

1）随机在图中取K（这里K=2）个种子点。

2）对图中的其他所有点求到这K个种子点的距离，若点A离种子点S1最近，那么A属于S1点群。如上图， A,B属于种子点1，C,D,E属于种子点2。

3）移动种子点到属于它的“点群”的中心。如上图中第三幅。

4）重复上述第2）和第3）步骤，直至种子点未移动，得到最终结果。

求点群中心的算法：

Minkowski Distance （闵可夫斯基距离）

Euclidean Distance（欧几里得距离）

Manhattan Distance（曼哈顿距离）

由于欧几里得距离最易理解，此文我们选用此算法来计算点群中心。

2、准备数据kmeans.txt：

3、获取种子点，此处K=3，则可取最大最小值作为其中两个种子点，然后再去均值作为第三个种子点。代码如下：

/**
* 计算种子点
* @author ZD
*
*/
public class ClusterPoint {

   private static class ClusterPointMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

       @Override
       protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context)
               throws IOException, InterruptedException {
           String[] strs = value.toString().split(",");
           for (int i = 1; i <= strs.length; i++) {
               context.write(new Text("c"+i), new IntWritable(Integer.parseInt(strs[i-1])));
           }
       }

   }

   private static class ClusterPointReducer extends Reducer<Text, IntWritable, Text, Text> {

       @Override
       protected void reduce(Text value, Iterable<IntWritable> datas,
               Reducer<Text, IntWritable, Text, Text>.Context context) throws IOException, InterruptedException {
           int min = Integer.MAX_VALUE;
           int max= Integer.MIN_VALUE;
           for (IntWritable data : datas) {
               if(data.get() < min){
                   min = data.get();
               }
               if(data.get() > max){
                   max = data.get();
               }
           }
           context.write(value, new Text(min+", "+max));
       }

   }

   public static void main(String[] args) {
       try {
           Configuration cfg = HadoopCfg.getConfigration();
           Job job = Job.getInstance(cfg);
           job.setJobName("ClusterPoint");
           job.setJarByClass(ClusterPoint.class);
           job.setMapperClass(ClusterPointMapper.class);
           job.setMapOutputKeyClass(Text.class);
           job.setMapOutputValueClass(IntWritable.class);
           job.setReducerClass(ClusterPointReducer.class);
           job.setOutputKeyClass(Text.class);
           job.setOutputValueClass(Text.class);
           FileInputFormat.addInputPath(job, new Path("/input/kmeans.txt"));
           FileOutputFormat.setOutputPath(job, new Path("/ClusterPoint/"));
           System.exit(job.waitForCompletion(true) ? 0 : 1);
       } catch (Exception e) {
           e.printStackTrace();
       }
   }

}

获得种子点后将结果手动写入文件clusterPoint.txt，保存至hdfs的/kmeans/output1/下。如图：

因为我们需要知道kmeans.txt中所有点和种子点c1,c2,c3的距离，并找出其每个点和三个种子点距离最近的种子点，即属于那个点群。然后重新计算三个点群的新种子点，即找到点群的中心点。不断重复上述步骤，得到最终的种子点。

将上述原理转换为MapReduce过程，即在Mapper层找到每个点距离最近的种子点，然后以（种子点名，点）形式传入Reducer层；Reducer层计算该点群的中心点并写入文件，且格式与clusterPoint.txt格式相同。

4、迭代获取最终种子点。原理已经解释清楚，接下来就是具体代码实现：

/**
* 迭代计算生成新种子点
*
* @author ZD
*
*/
public class KMeansExer {

   private static Configuration cfg = HadoopCfg.getConfigration();

   private static class KMeansExerMapper extends Mapper<LongWritable, Text, Text, Text> {

       private Map<String, Vector<Double>> map = new HashMap<String, Vector<Double>>();

       @Override
       protected void setup(Mapper<LongWritable, Text, Text, Text>.Context context)
               throws IOException, InterruptedException {
           FileSystem fs = FileSystem.get(cfg);
           Path path = new Path(cfg.get("seed"));
           RemoteIterator<LocatedFileStatus> rt = fs.listFiles(path, false);
           while (rt.hasNext()) {
               LocatedFileStatus status = rt.next();
               Path filePath = status.getPath();
               // 每次迭代是迭代新生成的种子点文件
               BufferedReader br = new BufferedReader(new InputStreamReader(fs.open(filePath)));
               String line = "";
               while ((line = br.readLine()) != null) {
                   String[] strs = line.split(":");
                   map.put(strs[0], KMeansUtil.str2Vector(strs[1], ",")); //KMeansUtil是自己写的工具类，str2Vector是将字符串按照规定字符分割，并将其保存至Vector中。
               }
               br.close();
           }
       }

       @Override
       protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, Text>.Context context)
               throws IOException, InterruptedException {
           String fileName = ((FileSplit) context.getInputSplit()).getPath().getName();
           if (fileName.equals("kmeans.txt")) {
               Vector<Double> pVector = KMeansUtil.str2Vector(value.toString(), ",");
               String center = "";
               // 第一次距离是最大的值
               double distance = Double.MAX_VALUE;
               Set<String> keys = map.keySet();
               for (String ckey : keys) {
                   double temp = KMeansUtil.getDistance(pVector, map.get(ckey)); //getDistance获得pVector中存的value这个点与所有种子点的距离
                   if (temp < distance) {
                       distance = temp;
                       center = ckey;
                   }
               }
               context.write(new Text(center), value); //将距离最近的种子点和该种子点传入Reducer层
           }
       }

   }

   private static class KMeansExerReducer extends Reducer<Text, Text, Text, NullWritable> {

       @Override
       protected void reduce(Text value, Iterable<Text> datas, Reducer<Text, Text, Text, NullWritable>.Context context)
               throws IOException, InterruptedException {
           List<Vector<Double>> list = new ArrayList<Vector<Double>>();
           int size = 0;
           for (Text data : datas) {
               Vector<Double> v = KMeansUtil.str2Vector(data.toString(), ",");
               size = v.size();
               list.add(v);
           }
           context.write(new Text(value.toString() + ":" + avgDistance(list, size)), NullWritable.get());
       }

   }

   public static String avgDistance(List<Vector<Double>> list, int size) {
       StringBuffer sb = new StringBuffer();
       for (int i = 0; i < size; i++) {
           double sum = 0.0;
           for (Vector<Double> v : list) {
               sum += v.get(i);
           }
           double avg = sum / list.size();
           sb.append(avg).append(",");
       }
       return sb.toString();
   }

   public static void run(int i) throws Exception {
       Job job = Job.getInstance(cfg);
       cfg.set("seed", "/kmeans/output" + i);
       job.setJobName("KMeansExer");
       job.setJarByClass(KMeansExer.class);
       job.setMapperClass(KMeansExerMapper.class);
       job.setMapOutputKeyClass(Text.class);
       job.setMapOutputValueClass(Text.class);
       job.setReducerClass(KMeansExerReducer.class);
       job.setOutputKeyClass(Text.class);
       job.setOutputValueClass(NullWritable.class);
       FileInputFormat.addInputPath(job, new Path("/input/kmeans.txt"));
       FileInputFormat.addInputPath(job, new Path("/kmeans/output" + i));
       FileOutputFormat.setOutputPath(job, new Path("/kmeans/output" + (i + 1)));
       job.waitForCompletion(true);
   }

   public static void main(String[] args) {
       try {
           for (int i = 1; i <= 10; i++) {
               run(i);
           }
       } catch (Exception e) {
           e.printStackTrace();
       }
   }
}

我们可以发现上述程序，迭代到第5次时，结果未发送改变了。但是值得注意的是，结果丢失了一个种子点。如图：

所以，最初选种子点时，是非常重要的。稍作更改，将初始种子点改为：

我们会发现在第8次时，结果稳定，得到最终的种子点。如图：

5、获得最终结果。虽然已获得稳定种子点，但是还未知道每个点属于哪一个点群。实现如下：

/**
* 获得最终三个点群
* @author ZD
*
*/
public class KMeansResult {

   private static class KMeansResultMapper extends Mapper<LongWritable, Text, Text, Text> {

       private Map<String, Vector<Double>> map = new HashMap<String, Vector<Double>>();

       @Override
       protected void setup(Mapper<LongWritable, Text, Text, Text>.Context context)
               throws IOException, InterruptedException {
           Configuration cfg = HadoopCfg.getConfigration();
           FileSystem fs = FileSystem.get(cfg);
           Path path = new Path(cfg.get("seed"));
           RemoteIterator<LocatedFileStatus> rt = fs.listFiles(path, false);
           while (rt.hasNext()) {
               LocatedFileStatus status = rt.next();
               Path filePath = status.getPath();
               BufferedReader br = new BufferedReader(new InputStreamReader(fs.open(filePath)));
               String line = "";
               while ((line = br.readLine()) != null) {
                   String[] strs = line.split(":");
                   map.put(strs[0], KMeansUtil.str2Vector(strs[1], ","));
               }
               br.close();
           }
       }

       @Override
       protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, Text>.Context context)
               throws IOException, InterruptedException {
           String fileName = ((FileSplit) context.getInputSplit()).getPath().getName();
           if (fileName.equals("kmeans.txt")) {
               Vector<Double> pVector = KMeansUtil.str2Vector(value.toString(), ",");
               String center = "";
               double distance = Double.MAX_VALUE;
               Set<String> keys = map.keySet();
               for (String ckey : keys) {
                   double temp = KMeansUtil.getDistance(pVector, map.get(ckey));
                   if (temp < distance) {
                       distance = temp;
                       center = ckey;
                   }
               }
               context.write(new Text(center), value);
           }
       }

   }

   private static class KMeansResultReducer extends Reducer<Text, Text, Text, Text> {

       @Override
       protected void reduce(Text value, Iterable<Text> datas, Reducer<Text, Text, Text, Text>.Context context)
               throws IOException, InterruptedException {
           for (Text data : datas) {
               context.write(value, data);
           }
       }

   }

   public static String avgDistance(List<Vector<Double>> list, int size) {
       StringBuffer sb = new StringBuffer();
       for (int i = 0; i < size; i++) {
           double sum = 0.0;
           for (Vector<Double> v : list) {
               sum += v.get(i);
           }
           double avg = sum / list.size();
           sb.append(avg).append(",");
       }
       return sb.toString();
   }

   public static void main(String[] args) {
       try {
           Configuration cfg = HadoopCfg.getConfigration();
           cfg.set("seed", "/input/kmeans/output8/");
           Job job = Job.getInstance(cfg);
           job.setJobName("KMeansResult");
           job.setJarByClass(KMeansResult.class);
           job.setMapperClass(KMeansResultMapper.class);
           job.setMapOutputKeyClass(Text.class);
           job.setMapOutputValueClass(Text.class);
           job.setReducerClass(KMeansResultReducer.class);
           job.setOutputKeyClass(Text.class);
           job.setOutputValueClass(Text.class);
           FileInputFormat.addInputPath(job, new Path("/input/kmeans.txt"));
           FileOutputFormat.setOutputPath(job, new Path("/kmeans/result/"));
           System.exit(job.waitForCompletion(true) ? 0 : 1);

       } catch (Exception e) {
           e.printStackTrace();
       }
   }
}