gensim快速使用简介

corpus

raw_corpus = ["Human machine interface for lab abc computer applications",
             "A survey of user opinion of computer system response time",
             "The EPS user interface management system",
             "System and human system engineering testing of EPS",              
             "Relation of user perceived response time to error measurement",
             "The generation of random binary unordered trees",
             "The intersection graph of paths in trees",
             "Graph minors IV Widths of trees and well quasi ordering",
             "Graph minors A survey"]

英文大写转小写，去除停用词，去除只出现一次的单词（可选择）

# Create a set of frequent words
stoplist = set('for a of the and to in'.split(' '))
# Lowercase each document, split it by white space and filter out stopwords
texts = [[word for word in document.lower().split() if word not in stoplist]
         for document in raw_corpus]
# Count word frequencies
from collections import defaultdict
frequency = defaultdict(int)
for text in texts:
    for token in text:
        frequency[token] += 1
# Only keep words that appear more than once
processed_corpus = [[token for token in text if frequency[token] > 1] for text in texts]

把语料库的每个单词作为一个唯一id，使用 gensim.corpora.Dictionary类创建字典

from gensim import corpora
dictionary = corpora.Dictionary(processed_corpus)

输出：Dictionary(12 unique tokens: [u’minors’, u’graph’, u’system’, u’trees’, u’eps’]…)
这里输出只有12个单词，而一般做字典库时候都是上千万级别的

vector

要了解语料库的内在结构，需要用数学方法表达文档。一种方法，将文档转向量。bag-of-words模型将文档用一个包含字典内所有词在文档中频率的向量表示。向量长度是等于字典长度，bag-of-words忽略了token（特征）在文档中出现顺序。
使用dictionary将tokenized documents(已标记文档)转为12维的向量

print (dictionary.token2id)

输出：{u’minors’: 11, u’graph’: 10, u’system’: 6, u’trees’: 9, u’eps’: 8, u’computer’: 1, u’survey’: 5, u’user’: 7, u’human’: 2, u’time’: 4, u’interface’: 0, u’response’: 3}
使用dictionary.doc2bow将新文档转为bag-of-words表达方式

new_doc = "Human computer interaction"
new_vec = dictionary.doc2bow(new_doc.lower().split())
new_vec

输出：[(1, 1), (2, 1)]
然后我们把原始的词库，再转为向量列表：

bow_corpus = [dictionary.doc2bow(text) for text in processed_corpus]

bow_corpus输出：
[[(0, 1), (1, 1), (2, 1)],
[(1, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)],
[(0, 1), (6, 1), (7, 1), (8, 1)],
[(2, 1), (6, 2), (8, 1)],
[(3, 1), (4, 1), (7, 1)],
[(9, 1)],
[(9, 1), (10, 1)],
[(9, 1), (10, 1), (11, 1)],
[(5, 1), (10, 1), (11, 1)]]

model

向量化词库后我们可以用模型来转换了。
初始化tf-idf模型，使用我们的语料库转一句’system minors’

from gensim import models
tfidf = models.TfidfModel(bow_corpus)
tfidf[dictionary.doc2bow('system minors'.lower().split())]

输出：[(6, 0.5898341626740045), (11, 0.8075244024440723)]
tfidf模型转成tuples类型，第一个是token id，第二个是tf-idf权重，注意system的权重已经比minors权重要小了。