自然语言处理-bag of words 和TF-IDF

词袋模型（bag of words，BOW）

自然语言处理时的数据经常是一个句子或者一篇文档，那么怎么表示这些句子和文档呢？
词袋模型，顾名思义，就是把句子/文档放在一个袋子里，那么把一堆单词丢进一个袋子里后，句子中单词的顺序就体现不出来了。
来看两个来自维基百科的例子：
1.首先，我们有两个句子：

(1) John likes to watch movies. Mary likes movies too.
(2) Mary also likes to watch football games.

2.两个句子包括这些单词

“John”,“likes”,“to”,“watch”,“movies”,“Mary”,“likes”,“movies”,“too”
“Mary”,“also”,“likes”,“to”,“watch”,“football”,“games”

3.把两个句子的单词转成字典的形式，key是单词，value是单词出现的次数

BoW1 = {“John”:1,“likes”:2,“to”:1,“watch”:1,“movies”:2,“Mary”:1,“too”:1};
BoW2 ={“Mary”:1,“also”:1,“likes”:1,“to”:1,“watch”:1,“football”:1,“games”:1};

4.然后合并两个字典，构造新的字典，也是这两个句子的词表，可看出词表共有10个单词，记录了每个单词出现的次数

BoW3 = {“John”:1,“likes”:3,“to”:2,“watch”:2,“movies”:2,“Mary”:2,“too”:1,“also”:1,“football”:1,“games”:1};

5.根据词表，把两个句子用列表表示出来，列表中的值为出现词表中单词的次数

(1) [1, 2, 1, 1, 2, 1, 1, 0, 0, 0]
(2) [0, 1, 1, 1, 0, 1, 0, 1, 1, 1]

代码如下：

//python
from keras.preprocessing.text import Tokenizer

sentence = ["John likes to watch movies. Mary likes movies too."]

tokenizer = Tokenizer()
tokenizer.fit_on_texts(sentence)
sequences = tokenizer.texts_to_sequences(sentence)
word_index = tokenizer.word_index 
# print(word_index)
# print(sequences)
# print(sequences[0])
bow = {}
for key in word_index:
    bow[key] = sequences[0].count(word_index[key])
print(bow)
print(f"Bag of word sentence 1 :\n{bow}")
print(f'We found {len(word_index)} unique tokens.')

"""
{'likes': 2, 'movies': 2, 'john': 1, 'to': 1, 'watch': 1, 'mary': 1, 'too': 1}
Bag of word sentence 1 :
{'likes': 2, 'movies': 2, 'john': 1, 'to': 1, 'watch': 1, 'mary': 1, 'too': 1}
We found 7 unique tokens.
"""

TF-IDF（Term Frequency / Inverse Document Frequency，词频-逆文本频率）

TF（Term Frequency，词频）的公式为：

而IDF（inverse document frequency，逆文本频率）的公式为：

其中，分母之所以加1是为了防止分母为0。所以，TF-IDF的公式为：

TF-IDF值越大说明这个词越重要，也可以说这个词是关键词
sklearn中封装TF-IDF方法，并且也提供了示例

//python
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())

print(X)
print(X.toarray())

"""
['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']
  (0, 8)	0.38408524091481483
  (0, 3)	0.38408524091481483
  (0, 6)	0.38408524091481483
  (0, 2)	0.5802858236844359
  (0, 1)	0.46979138557992045
  (1, 8)	0.281088674033753
  (1, 3)	0.281088674033753
  (1, 6)	0.281088674033753
  (1, 1)	0.6876235979836938
  (1, 5)	0.5386476208856763
  (2, 8)	0.267103787642168
  (2, 3)	0.267103787642168
  (2, 6)	0.267103787642168
  (2, 0)	0.511848512707169
  (2, 7)	0.511848512707169
  (2, 4)	0.511848512707169
  (3, 8)	0.38408524091481483
  (3, 3)	0.38408524091481483
  (3, 6)	0.38408524091481483
  (3, 2)	0.5802858236844359
  (3, 1)	0.46979138557992045
[[0.         0.46979139 0.58028582 0.38408524 0.         0.
  0.38408524 0.         0.38408524]
 [0.         0.6876236  0.         0.28108867 0.         0.53864762
  0.28108867 0.         0.28108867]
 [0.51184851 0.         0.         0.26710379 0.51184851 0.
  0.26710379 0.51184851 0.26710379]
 [0.         0.46979139 0.58028582 0.38408524 0.         0.
  0.38408524 0.         0.38408524]]
"""