通过sse 寻找kmean 的k 值

最新推荐文章于 2024-01-19 20:49:02 发布

samoyan

最新推荐文章于 2024-01-19 20:49:02 发布

阅读量1.9k

点赞数

分类专栏：机器学习 python 文章标签： sklearn python 机器学习

本文链接：https://blog.csdn.net/baoyan2015/article/details/123546744

版权

python 同时被 2 个专栏收录

28 篇文章 0 订阅

订阅专栏

机器学习

22 篇文章 1 订阅

订阅专栏

该博客主要介绍了如何使用Python中的sklearn库进行K-Means聚类分析。首先，通过加载数据并进行文本向量化，利用TfidfVectorizer进行词频矩阵的构建，并通过PCA进行降维处理。然后，通过遍历不同数量的聚类簇，计算误差平方和（SSE），最终选择合适的簇数，并绘制SSE与簇数的关系图。整个过程详细展示了数据预处理和K-Means聚类在文本分类中的应用。

摘要由CSDN通过智能技术生成

# -*- coding: utf-8 -*-
"""
K-means-Single-Test
"""
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from tools.preprocess import *
from tools.visualizer import plot_result
from sklearn.feature_extraction.text import TfidfTransformer, TfidfVectorizer, CountVectorizer
from sklearn.decomposition import PCA, TruncatedSVD
from tools.labelText import LabelText
import settings
import time
import pandas as pd
from sklearn import metrics

"""
loading source
载入资源

"""
print('------Loading Source...')
ori_path = settings.SOURCE_DATA + 'goods_data.csv'
# sentences = loading_source(file_name=ori_path)
sentences = []
# content_lines = loading_source(file_name=ori_path)
# ori_path = settings.SOURCE_DATA + 'cut_data.csv'
sentences = loading_source(file_name=ori_path)
# start = time.time()
# cut_source(content_lines, sentences, write=True)
# end = time.time()
# print('------- cutting cost', end - start)


"""
Vertorizer
向量化
"""
print('------Vertorizer...')
start = time.time()

# 词频矩阵 Frequency Matrix Of Words
vertorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.46)
transformer = TfidfTransformer()
# Fit Raw Documents
freq_words_matrix = vertorizer.fit_transform(sentences)
# Get Words Of Bag
words = vertorizer.get_feature_names()
tfidf = transformer.fit_transform(freq_words_matrix)
weight = freq_words_matrix.toarray()

end = time.time()

print("Shape: Documents(Class) / Words")
print(weight.shape)

print('------ vectorizer cost', end-start)


"""
Dimension Reduction
降维
"""
pca = PCA(n_components=10)
trainingData = pca.fit_transform(weight)

# svd = TruncatedSVD(n_components=10, n_iter=10, random_state=42)
# trainingData = svd.fit_transform(weight)


"""
Compute K-Means
"""

numOfClass: int = 10
SSE = []  # 存放每次结果的误差平方和
for i in range(20,300,20):
    start = time.time()
    clf = KMeans(n_clusters=i, max_iter=10000, init="k-means++", tol=1e-6)

    result = clf.fit(trainingData)
    end = time.time()
    print("n_cluster is ",i," time is",end-start)
    SSE.append(clf.inertia_)
X = range(20,300,20)
plt.xlabel('k')
plt.ylabel('SSE')
plt.plot(X, SSE, 'o-')
plt.savefig('sse.png')

结果如下：