聚类尝试_kprototypes

最新推荐文章于 2024-04-19 16:26:52 发布

nikita_zj

最新推荐文章于 2024-04-19 16:26:52 发布

阅读量2.6k

点赞数 2

分类专栏：模型文章标签：聚类机器学习数据挖掘

本文链接：https://blog.csdn.net/nikita_zj/article/details/122321665

版权

模型专栏收录该内容

15 篇文章 5 订阅

订阅专栏

在链家爬了一些二手房数据，尝试用聚类进行分析。

1. 一些简单的准备工作：导包、导数据、简单的数据提取

# 导入常用的包
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import preprocessing
import pickle
import time
import multiprocessing
plt.rcParams['font.sans-serif']=['SimHei']
plt.rcParams['axes.unicode_minus'] = False

# 导入数据
df_shanghai = pd.read_csv('sh_ershou_clean.csv', index_col = 0)
df_hangzhou = pd.read_csv('hz_ershou_clean.csv', index_col = 0)

print('df_shanghai.shape:{}'.format(df_shanghai.shape))
print('df_hangzhou.shape:{}'.format(df_hangzhou.shape))

# 提取建房时间
df_shanghai['year'] = df_shanghai['info'].str.extract('([\d]+)年').astype('float')
df_shanghai['age'] = 2022-df_shanghai.year

2. 数据预处理

2.1 数值型：标准化

# 对数值型数据进行标准化
t= preprocessing.StandardScaler().fit(df_shanghai[df_shanghai.describe().columns])
df_shanghai_processed = pd.DataFrame(t.transform(df_shanghai[df_shanghai.describe().columns]))
df_shanghai_processed.columns = df_shanghai.describe().columns

2.2 字符型：labelencoding

# 类别型变量进行labelenconding
df_shanghai_processed['location1'] = df_shanghai.location1
le = preprocessing.LabelEncoder()
df_shanghai_processed['location1'] = le.fit_transform(df_shanghai_processed['location1'])

# 将label_encoding的映射关系记录下来
def pkl_save(filename, file):
    #将file保存到filename中
    output = open(filename, 'wb')
    pickle.dump(file, output)
    output.close()

pkl_save('shanghai_location1_labelencoding.pkl', le)

2.3 缺失值填充

# 缺失值填充
df_shanghai_processed.year.fillna(df_shanghai_processed.year.mean(), inplace = True)
df_shanghai_processed.age.fillna(df_shanghai_processed.age.mean(), inplace = True)

3. 调用kmodes包进行聚类

3.1 模型训练

from kmodes.kprototypes import KPrototypes

cate_index = [5] # 为类别型变量在columns中的索引
def TrainCluster(df, model_name=None, start_k=2, end_k=10):
    print('training cluster')
    K = []
    SSE = []
    silhouette_all=[]
    models = [] #保存每次的模型
    for i in range(start_k, end_k):
        start = time.perf_counter ()
        kproto_model = KPrototypes(n_clusters=i, n_jobs=multiprocessing.cpu_count())
        kproto_model.fit(df_shanghai_processed,categorical=cate_index)
        SSE.append(kproto_model.cost_)  # 保存每一个k值的SSE值
        K.append(i)
        print('{}-prototypes SSE loss = {}'.format(i, kproto_model.cost_))
        models.append(kproto_model) #保存每个k值对应的模型
        end = time.perf_counter ()
        print('Running time: %s Seconds'%(end-start))

    return(K,SSE,models)

K, SSE, models = TrainCluster(df = df_shanghai_processed)

3.2 类别数量选择

# 决定选择5
plt.plot(K, SSE)
plt.xlabel('聚类类别数k')
plt.ylabel('SSE')
plt.xticks(K)
plt.title('用肘部法则来确定最佳的k值')
plt.show()

结果如下图：

3.3 确定类别数量k后，进行可视化等后续的分析处理

#确定了最佳的k值后：5
best_model = models[K.index(5)]

# 对样本类别进行标记
df_shanghai['labels_'] = best_model.labels_

# 提取每一簇的中心
center = pd.DataFrame(best_model.cluster_centroids_ )
center.columns = df_shanghai_processed.columns

# 还原每一簇中心，得到标准化之前对应的原始值
for col in center.columns[:-1]:
    col_new = col+'_raw'
    mean = df_shanghai[col].mean()
    std = df_shanghai[col].std()
    center[col_new] = center[col]*std+mean
    print(col_new)

# 绘制散点图
colors1 = '#00CED1' #点的颜色
colors2 = '#DC143C'
col1 = 'total_price'
col2 = 'area'

for label in df_shanghai.labels_.unique():
    if label != 3:
        x = df_shanghai.loc[df_shanghai.labels_ == label, col1]
        y = df_shanghai.loc[df_shanghai.labels_ == label, col2]
        plt.scatter(x, y, label = str(label), alpha = 0.2)
plt.legend()

簇中心各分量如下：