Mean Shift算法的核心原理

最新推荐文章于 2024-07-27 12:45:39 发布

TsianLee

最新推荐文章于 2024-07-27 12:45:39 发布

阅读量3.2k

点赞数

1、核函数

在Mean Shift算法中引入核函数的目的是使得随着样本与被偏移点的距离不同，其偏移量对均值偏移向量的贡献也不同。核函数是机器学习中常用的一种方式。核函数的定义如下所示：

X 表示一个 d 维的欧式空间， x 是该空间中的一个点 x={x1,x2,x3⋯,xd} ，其中， x 的模 ∥x∥2=xxT ， R 表示实数域，如果一个函数 K:X→R 存在一个剖面函数 k:[0,∞]→R ，即

$K (x) = k (∥ x ∥ 2)$
并且满足：
(1)、 k 是非负的
(2)、 k 是非增的
(3)、 k 是分段连续的
那么，函数 K(x) 就称为核函数。

常用的核函数有高斯核函数。高斯核函数如下所示：

N (x) = 1 2 π - - \sqrt h e - x 2 2 h 2

其中， h 称为带宽(bandwidth)，不同带宽的核函数如下图所示：

这里写图片描述

2、Mean Shift算法的核心思想

2.1、基本原理

对于Mean Shift算法，是一个迭代的步骤，即先算出当前点的偏移均值，将该点移动到此偏移均值，然后以此为新的起始点，继续移动，直到满足最终的条件。此过程可由下图的过程进行说明(图片来自参考文献3)：

步骤1：在指定的区域内计算偏移均值(如下图的黄色的圈)

这里写图片描述

步骤2：移动该点到偏移均值点处

这里写图片描述

步骤3：重复上述的过程(计算新的偏移均值，移动)

这里写图片描述

步骤4：满足了最终的条件，即退出

这里写图片描述

从上述过程可以看出，在Mean Shift算法中，最关键的就是计算每个点的偏移均值，然后根据新计算的偏移均值更新点的位置。

2.2.2、基本的Mean Shift向量形式

对于给定的 d 维空间 Rd 中的 n 个样本点 xi,i=1,⋯,n ，则对于 x 点，其Mean Shift向量的基本形式为：

M h (x) = 1 k \sum x i \in S h (x i - x)

其中， Sh 指的是一个半径为 h 的高维球区域，如上图中的蓝色的圆形区域。 Sh 的定义为：

S h (x) = (y ∣ (y - x) (y - x) T ⩽ h 2)

这样的一种基本的Mean Shift形式存在一个问题：在 Sh 的区域内，每一个点对 x 的贡献是一样的。而实际上，这种贡献与 x 到每一个点之间的距离是相关的。同时，对于每一个样本，其重要程度也是不一样的。

2.2.3、改进的Mean Shift向量形式

基于以上的考虑，对基本的Mean Shift向量形式中增加核函数和样本权重，得到如下的改进的Mean Shift向量形式：

M h (x) = \sum n i = 1 G H ( x i - x ) w ( x i ) ( x i - x ) \sum n i = 1 G H ( x i - x ) w ( x i )

其中：

G H (x i - x) = | H | - 1 2 G (H - 1 2 (x i - x))

G(x) 是一个单位的核函数。 H 是一个正定的对称 d×d 矩阵，称为带宽矩阵，其是一个对角阵。 w(xi)⩾0 是每一个样本的权重。对角阵 H 的形式为：

H = ⎛ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ h 21 0 ⋮ 0 0 h 22 ⋮ 0 \dots \dots \dots 00 ⋮ h 2 d ⎞ ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ d \times d

上述的Mean Shift向量可以改写成：

M h (x) = \sum n i = 1 G ( x i - x h i ) w ( x i ) ( x i - x ) \sum n i = 1 G ( x i - x h i ) w ( x i )

Mean Shift向量 Mh(x) 是归一化的概率密度梯度。

2.3、Mean Shift算法的解释

在Mean Shift算法中，实际上是利用了概率密度，求得概率密度的局部最优解。

2.3.1、概率密度梯度

对一个概率密度函数 f(x) ，已知 d 维空间中 n 个采样点 xi,i=1,⋯,n ， f(x) 的核函数估计(也称为Parzen窗估计)为：

$f^(x) = \sum n i = 1 K ( x i - x h ) w ( x i ) h d \sum n i = 1 w ( x i )$
其中
w(xi)⩾0 是一个赋给采样点 xi 的权重
K(x) 是一个核函数

概率密度函数 f(x) 的梯度 ▽f(x) 的估计为

▽ f^(x) = 2 \sum n i = 1 ( x - x i ) k ' ( ∥ ∥ x i - x h ∥ ∥ 2 ) w ( x i ) h d + 2 \sum n i = 1 w ( x i )

令 g(x)=−k′(x) ， G(x)=g(∥x∥2) ，则有：

▽ f^(x) = 2 \sum n i = 1 ( x i - x ) G ( ∥ ∥ x i - x h ∥ ∥ 2 ) w ( x i ) h d + 2 \sum n i = 1 w ( x i ) = 2 h 2 ⎡ ⎣ ⎢ \sum n i = 1 G ( x i - x h ) w ( x i ) h d \sum n i = 1 w ( x i ) ⎤ ⎦ ⎥ ⎡ ⎣ ⎢ \sum n i = 1 ( x i - x ) G ( ∥ ∥ x i - x h ∥ ∥ 2 ) w ( x i ) \sum n i = 1 G ( x i - x h ) w ( x i ) ⎤ ⎦ ⎥

其中，第二个方括号中的就是Mean Shift向量，其与概率密度梯度成正比。

2.3.2、Mean Shift向量的修正

M h (x) = \sum n i = 1 G ( ∥ ∥ x i - x h ∥ ∥ 2 ) w ( x i ) x i \sum n i = 1 G ( x i - x h ) w ( x i ) - x

记： mh(x)=∑ni=1G(∥∥xi−xh∥∥2)w(xi)xi∑ni=1G(xi−xh)w(xi) ，则上式变成：

M h (x) = m h (x) + x

这与梯度上升的过程一致。

2.4、Mean Shift算法流程

Mean Shift算法的算法流程如下：

计算 mh(x)
令 x=mh(x)
如果 ∥mh(x)−x∥<ε ，结束循环，否则，重复上述步骤

三、实验

3.1、实验数据

实验数据如下图所示(来自参考文献1)：

这里写图片描述

画图的代码如下：

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'''
Date:20160426
@author: zhaozhiyong
'''</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> matplotlib.pyplot <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">as</span> plt

f = open(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"data"</span>)
x = []
y = []
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> line <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> f.readlines():
    lines = line.strip().split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>)
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> len(lines) == <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>:
        x.append(float(lines[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]))
        y.append(float(lines[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>]))
f.close()  

plt.plot(x, y, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'b.'</span>, label=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"original data"</span>)
plt.title(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'Mean Shift'</span>)
plt.legend(loc=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"upper right"</span>)
plt.show()
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li></ul>

3.2、实验的源码

<code class="hljs python has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#!/bin/python</span>
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#coding:UTF-8</span>
<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'''
Date:20160426
@author: zhaozhiyong
'''</span>

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> math
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> sys
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> numpy <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">as</span> np

MIN_DISTANCE = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.000001</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#mini error</span>

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">load_data</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(path, feature_num=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)</span>:</span>
    f = open(path)
    data = []
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> line <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> f.readlines():
        lines = line.strip().split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"\t"</span>)
        data_tmp = []
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> len(lines) != feature_num:
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">continue</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(feature_num):
            data_tmp.append(float(lines[i]))

        data.append(data_tmp)
    f.close()
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> data

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">gaussian_kernel</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(distance, bandwidth)</span>:</span>
    m = np.shape(distance)[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]
    right = np.mat(np.zeros((m, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)))
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(m):
        right[i, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>] = (-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.5</span> * distance[i] * distance[i].T) / (bandwidth * bandwidth)
        right[i, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>] = np.exp(right[i, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>])
    left = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> / (bandwidth * math.sqrt(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> * math.pi))

    gaussian_val = left * right
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> gaussian_val

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">shift_point</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(point, points, kernel_bandwidth)</span>:</span>
    points = np.mat(points)
    m,n = np.shape(points)
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#计算距离</span>
    point_distances = np.mat(np.zeros((m,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)))
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(m):
        point_distances[i, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>] = np.sqrt((point - points[i]) * (point - points[i]).T)

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#计算高斯核      </span>
    point_weights = gaussian_kernel(point_distances, kernel_bandwidth)

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#计算分母</span>
    all = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.0</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(m):
        all += point_weights[i, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#均值偏移</span>
    point_shifted = point_weights.T * points / all
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> point_shifted

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">euclidean_dist</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(pointA, pointB)</span>:</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#计算pointA和pointB之间的欧式距离</span>
    total = (pointA - pointB) * (pointA - pointB).T
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> math.sqrt(total)

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">distance_to_group</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(point, group)</span>:</span>
    min_distance = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10000.0</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> pt <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> group:
        dist = euclidean_dist(point, pt)
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> dist < min_distance:
            min_distance = dist
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> min_distance

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">group_points</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(mean_shift_points)</span>:</span>
    group_assignment = []
    m,n = np.shape(mean_shift_points)
    index = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>
    index_dict = {}
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(m):
        item = []
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> j <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(n):
            item.append(str((<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"%5.2f"</span> % mean_shift_points[i, j])))

        item_1 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_"</span>.join(item)
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> item_1
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> item_1 <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">not</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> index_dict:
            index_dict[item_1] = index
            index += <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(m):
        item = []
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> j <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(n):
                        item.append(str((<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"%5.2f"</span> % mean_shift_points[i, j])))

                item_1 = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_"</span>.join(item)
        group_assignment.append(index_dict[item_1])

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> group_assignment

<span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> <span class="hljs-title" style="box-sizing: border-box;">train_mean_shift</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(points, kenel_bandwidth=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)</span>:</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#shift_points = np.array(points)</span>
    mean_shift_points = np.mat(points)
    max_min_dist = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
    iter = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>
    m, n = np.shape(mean_shift_points)
    need_shift = [<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">True</span>] * m

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#cal the mean shift vector</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">while</span> max_min_dist > MIN_DISTANCE:
        max_min_dist = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>
        iter += <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"iter : "</span> + str(iter)
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> range(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>, m):
            <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#判断每一个样本点是否需要计算偏置均值</span>
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">not</span> need_shift[i]:
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">continue</span>
            p_new = mean_shift_points[i]
            p_new_start = p_new
            p_new = shift_point(p_new, points, kenel_bandwidth)
            dist = euclidean_dist(p_new, p_new_start)

            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> dist > max_min_dist:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#record the max in all points</span>
                max_min_dist = dist
            <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> dist < MIN_DISTANCE:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#no need to move</span>
                need_shift[i] = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">False</span>

            mean_shift_points[i] = p_new
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#计算最终的group</span>
    group = group_points(mean_shift_points)

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> np.mat(points), mean_shift_points, group

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> __name__ == <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"__main__"</span>:
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#导入数据集</span>
    path = <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"./data"</span>
    data = load_data(path, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#训练，h=2</span>
    points, shift_points, cluster = train_mean_shift(data, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> i <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> xrange(len(cluster)):
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">print</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"%5.2f,%5.2f\t%5.2f,%5.2f\t%i"</span> % (points[i,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>], points[i, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>], shift_points[i, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>], shift_points[i, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>], cluster[i])</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li><li style="box-sizing: border-box; padding: 0px 5px;">120</li><li style="box-sizing: border-box; padding: 0px 5px;">121</li><li style="box-sizing: border-box; padding: 0px 5px;">122</li><li style="box-sizing: border-box; padding: 0px 5px;">123</li><li style="box-sizing: border-box; padding: 0px 5px;">124</li><li style="box-sizing: border-box; padding: 0px 5px;">125</li><li style="box-sizing: border-box; padding: 0px 5px;">126</li><li style="box-sizing: border-box; padding: 0px 5px;">127</li><li style="box-sizing: border-box; padding: 0px 5px;">128</li><li style="box-sizing: border-box; padding: 0px 5px;">129</li><li style="box-sizing: border-box; padding: 0px 5px;">130</li><li style="box-sizing: border-box; padding: 0px 5px;">131</li><li style="box-sizing: border-box; padding: 0px 5px;">132</li><li style="box-sizing: border-box; padding: 0px 5px;">133</li><li style="box-sizing: border-box; padding: 0px 5px;">134</li><li style="box-sizing: border-box; padding: 0px 5px;">135</li><li style="box-sizing: border-box; padding: 0px 5px;">136</li><li style="box-sizing: border-box; padding: 0px 5px;">137</li><li style="box-sizing: border-box; padding: 0px 5px;">138</li><li style="box-sizing: border-box; padding: 0px 5px;">139</li><li style="box-sizing: border-box; padding: 0px 5px;">140</li><li style="box-sizing: border-box; padding: 0px 5px;">141</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li><li style="box-sizing: border-box; padding: 0px 5px;">120</li><li style="box-sizing: border-box; padding: 0px 5px;">121</li><li style="box-sizing: border-box; padding: 0px 5px;">122</li><li style="box-sizing: border-box; padding: 0px 5px;">123</li><li style="box-sizing: border-box; padding: 0px 5px;">124</li><li style="box-sizing: border-box; padding: 0px 5px;">125</li><li style="box-sizing: border-box; padding: 0px 5px;">126</li><li style="box-sizing: border-box; padding: 0px 5px;">127</li><li style="box-sizing: border-box; padding: 0px 5px;">128</li><li style="box-sizing: border-box; padding: 0px 5px;">129</li><li style="box-sizing: border-box; padding: 0px 5px;">130</li><li style="box-sizing: border-box; padding: 0px 5px;">131</li><li style="box-sizing: border-box; padding: 0px 5px;">132</li><li style="box-sizing: border-box; padding: 0px 5px;">133</li><li style="box-sizing: border-box; padding: 0px 5px;">134</li><li style="box-sizing: border-box; padding: 0px 5px;">135</li><li style="box-sizing: border-box; padding: 0px 5px;">136</li><li style="box-sizing: border-box; padding: 0px 5px;">137</li><li style="box-sizing: border-box; padding: 0px 5px;">138</li><li style="box-sizing: border-box; padding: 0px 5px;">139</li><li style="box-sizing: border-box; padding: 0px 5px;">140</li><li style="box-sizing: border-box; padding: 0px 5px;">141</li></ul>