层次聚类pythonscipy,在Python中使用Scipy层次结构聚类的文本聚类

I have a text corpus that contains 1000+ articles each in a separate line. I am trying to use Hierarchy Clustering using Scipy in python to produce clusters of related articles.

This is the code I used to do the clustering

# Agglomerative Clustering

import matplotlib.pyplot as plt

import scipy.cluster.hierarchy as hac

tree = hac.linkage(X.toarray(), method="complete",metric="euclidean")

plt.clf()

hac.dendrogram(tree)

plt.show()

and I got this plot

Then I cut off the tree at the third level with fcluster()

from scipy.cluster.hierarchy import fcluster

clustering = fcluster(tree,3,'maxclust')

print(clustering)

and I got this output:

[2 2 2 ..., 2 2 2]

My question is how can I find the top 10 frequent words in each cluster in order to suggest a topic for each cluster?

解决方案

You can do the following:

Align your results (your clustering variable) with your input (the 1000+ articles).

Using pandas library, you can use a groupby function with the cluster # as its key.

Per group (using the get_group function), fill up a defaultdict of integers for every

word you encounter.

You can now sort the dictionary of word counts in descending order and get your desired number of most frequent words.

Good luck with what you're doing and please do accept my answer if it's what you're looking for.