K-均值聚类
1 假设
对于m个样本点,假设有K个类,每个类的中心分别为 μ 1 , ⋯ , μ K ∈ R n \mu_{1},\cdots,\mu_{K}\in \mathbb{R}^{n} μ1,⋯,μK∈Rn,样本点的分类为 C = { c i } , c i ∈ 1 , ⋯ , K C=\{c_{i}\}, c_{i}\in{1,\cdots,K} C={ci},ci∈1,⋯,K.
2 代价函数
J ( c 1 , ⋯ , c m , μ 1 , ⋯ , μ K ) = 1 m ∑ i = 1 m ∣ ∣ x i − μ c i ∣ ∣ 2 J(c_{1},\cdots,c_{m},\mu_{1},\cdots,\mu_{K}) = \frac{1}{m}\sum_{i=1}^{m}||x_{i}-\mu_{c_{i}}||^2 J(c1,⋯,cm,μ1,⋯,μK)=m1i=1∑m∣∣xi−μci∣∣2
3 目标函数
min C , μ J ( c 1 , ⋯ , c m , μ 1 , ⋯ , μ K ) \min_{C,\mu}J(c_{1},\cdots,c_{m}, \mu_{1},\cdots,\mu_{K}) C,μminJ(c1,⋯,cm,μ1,⋯,μK)
4 算法
Step1 初始化。随机选取K个样本点作为初始聚类中心 μ 1 , ⋯ , μ K ∈ R n \mu_{1},\cdots,\mu_{K}\in \mathbb{R}^{n} μ1,⋯,μK∈Rn.
Step2 对样本进行聚类。计算每个样本点到每个类中心的距离 ∣ ∣ x i − μ k ∣ ∣ 2 ||x_{i}-\mu_{k}||^2 ∣∣xi−μk∣∣2,指派到与其最近的中心的类,类指标 c i : = arg min k ∣ ∣ x i − μ k ∣ ∣ 2 c_{i} := \mathop{\arg\min}\limits_{k}||x_{i}-\mu_{k}||^2 ci:=kargmin∣∣xi−μk∣∣2,构成聚类结果.
Step3 计算新的类的中心, μ k : = a v g { x i ∣ c i = k } \mu_{k} := avg\{x_{i} \mid c_{i}=k\} μk:=avg{xi∣ci=k}.
Step4 如果符合迭代收敛或符合停止条件,输出聚类结果 C = { c i } C=\{c_{i}\} C={ci}. 否则重复step2 ~ step4.
伪代码:
R a n d o m l y i n i t i a l i z e K c l u s t e r c e n t r o i d s m 1 , ⋯ , m K ∈ R n R e p e a t { f o r i = 1 t o m c i : = arg min k ∣ ∣ x i − μ k ∣ ∣ 2 → 聚 类 , 即 对 固 定 的 聚 类 中 心 , min C J ( C , μ ) f o r k = 1 t o K μ k : = a v g { x i ∣ c i = k } → 移 动 中 心 , 即 min μ J ( C , μ ) } R e t u r n { c i } \begin{aligned} &Randomly\ initialize\ K\ cluster\ centroids\ m_{1},\cdots,m_{K}\in \mathbb{R}^{n} \\ \\ &Repeat \{\\ & \qquad for\ i = 1\ to\ m \\ &\qquad\qquad c_{i} := \mathop{\arg\min}_{k}||x_{i}-\mu_{k}||^2 \rightarrow聚类,即对固定的聚类中心, \min_{C}J(C,\mu) \\ & \qquad for\ k = 1\ to\ K\\ &\qquad \qquad \mu_{k} := avg\{x_{i} \mid c_{i}=k\} \rightarrow移动中心,即\min_{\mu}J(C,\mu)\\ &\}\\ &Return \ \{c_{i}\} \end{aligned} Randomly initialize K cluster centroids m1,⋯,mK∈RnRepeat{for i=1 to mci:=argmink∣∣xi−μk∣∣2→聚类,即对固定的聚类中心,CminJ(C,μ)for k=1 to Kμk:=avg{xi∣ci=k}→移动中心,即μminJ(C,μ)}Return {ci}
5 补充
5.1 随机初始化
当K比较小时,随机初始化可能造成局部最优,可以使用多次随机初始化获得多个聚类结果,选择代价最小的结果;当K比较大时,随机初始化方法一般能给到一个合理的初始值,从而获得较好的结果。
5.2 K的选择——“肘部法则”
6 代码
# 自编代码
class K_means_clustering:
def __init__(self, data, K=2, n_iter=5):
self.data = data
self.K = K
self.centroid = None
self.cluster = None
self.n_iter = n_iter
def random_init(self):
"""初始化K个聚类中心"""
self.centroid = self.data[np.random.choice(len(self.data), size=self.K)]
return self
def dist(self):
"""计算每个样本点到每个聚类中心的距离"""
dist = np.array([[np.power(x-y,2).sum() for y in self.centroid]
for x in self.data])
return dist
def updateCluster(self):
"""找最近的聚类中心,返回类指标"""
self.cluster = np.argmin(self.dist(), axis=1)
return self
def updateCentroid(self):
"""移动中心"""
self.centroid = np.array([np.mean(self.data[np.where(self.cluster==k)],axis=0) for k in range(self.K)])
return self
def computeCost(self):
cost = np.min(self.dist(), axis=1).mean()
return cost
def fit(self):
self.random_init()
best_centroid = self.centroid # 记录最优的聚类中心
least_cost = self.computeCost()
for iter in range(self.n_iter):
self.updateCluster()
self.updateCentroid()
if (self.centroid == best_centroid).all(): # 聚类中心不再变动
# print('Done clustering!')
break
elif self.computeCost() < least_cost:
least_cost = self.computeCost()
best_centroid = self.centroid
self.centroid = best_centroid
self.updateCluster()
return self
def multi_fit(self, n_fit=10):
self.random_init()
best_centroid = self.centroid # 记录最优的聚类中心
least_cost = self.computeCost()
for n in range(1, n_fit+1):
self.fit()
# print('cost:', self.computeCost())
# plt.subplots()
# plt.scatter(data[:,0], data[:,1], c=model.cluster, cmap=plt.cm.Paired)
# plt.scatter(model.centroid[:,0], model.centroid[:,1],
# marker='o', s=200, c=np.arange(3), edgecolor='b', cmap=plt.cm.Paired)
# plt.show()
if self.computeCost() < least_cost:
least_cost = self.computeCost()
best_centroid = self.centroid
self.centroid = best_centroid
self.updateCluster()
return self
# 调用
model = K_means_clustering(data, K=3, n_iter=5)
model.multi_fit(n_fit=5)
print('the best:', model.computeCost())
plt.figure(figsize=(8,6))
plt.scatter(data[:,0], data[:,1], c=model.cluster, cmap=plt.cm.Paired)
plt.scatter(model.centroid[:,0], model.centroid[:,1],
marker='o', s=200, c=np.arange(3), edgecolor='b', cmap=plt.cm.Paired)
plt.show()
# 上述算法太慢了,直接用sklearn
from sklearn.cluster import KMeans
model = KMeans(n_clusters=16)
model.fit(data) # data.shape: (128*128,3)
centroid = model.cluster_centers_
cluster = model.predict(data)
compressed_data = centroid[cluster].reshape(128, 128, 3)
io.imshow(compressed_data)
plt.show()