参考: numpy“手撕”文本主题模型之LSA
环境
scikit-learn == 0.22.2
scipy == 1.4.1
numpy == 1.18.1
CountVectorizer和TfidfVectorizer
CountVectorizer
# 将文本中的词语转换为词频矩阵
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(min_df=1)
corpus = [ 'This is the first document.',
'This is the second second document.',
'And the third one.',
'Is this the first document?',
]
X = vectorizer.fit_transform(corpus)
feature_name = vectorizer.get_feature_names()
print (feature_name)
print (X)
print (X.toarray())
结果:
['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']
(0, 8) 1
(0, 3) 1
(0, 6) 1
(0, 2) 1
(0, 1) 1
(1, 8) 1
(1, 3) 1
(1, 6) 1
(1, 1) 1
(1, 5) 2
(2, 6) 1