NLP-自然语言处理基础

NLP基础知识

1 NLP常见任务

自动摘要
指代消解   et:小明放学了，妈妈去接他
机器翻译   et:小心地滑 Slide carefully
词性标注   et: heat (v.) water (n.) in(p.) a(det.) pot(n.)
分词          大水沟/很/难/过
主题识别    LDA
文本分类
自然语言处理中，采用词向量表示
语料库

2 独热编码

John likes to watch movies. Mary likes too.
John also likes to watch football games.

词典

{"John":1,  "likes":2, "to":3, "watch":4, "movies":5, "Mary":6, "too":7, "also":8, "football":9, "games":10}

One-hot表示

John:[1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
likes：[0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
 ...
games:[0, 1, 0, 0, 0, 0, 0, 0 ,0 ,1]

词典中的每一个单词都有唯一的索引；词典中词的顺序和句子没有关联

3 Bag of Words(词袋表示)

将各词的词向量相加

John likes to watch movies. Mary likes too. [1, 2, 1, 1, 1, 1, 1, 0, 0, 0]
John also likes to watch football games. [1, 1, 1, 1, 0, 0, 0, 1, 1, 1]

4 词权重TF-IDF (Term Frequency - Inverse Document Frequency)

TF-IDF用来评价该词的权重
词t的TF：$TF(t)=\frac{(词t在当前文档出现的次数)}{(t在全部文档中出现的次数)}$
词t的IDF：$IDF(t)=ln(\frac{总文档数}{含t的文档数+1})$
$TF-IDF权重=TF(t)*IDF(t)$

John likes to watch movies. Mary likes too. [0.693, 1.386, 0.693, 1.099, 0.693, 0.693, 0, 0, 0, 0]

缺点：词在文档中的顺序没有被考虑
无法捕捉到：“小王喜欢小红”和“小红喜欢小王”之间的区别

5 N-gram

用来描述一句话 (词组合) 出现的概率。其假定一个词的出现，只与前面$n$个词有关。

$$P(w_1, ... , w_m) = \prod_{i=1}^{m}P(w_i|w_1, ... , w_{i - 1})$$
为2-gram建立的索引

"John likes": 1
"likes to"：2
...
"football games"：10

John likes to watch movies. Mary likes too. [1, 1, 1, 1, 1, 1, 1, 0, 0, 0]
John also likes to watch football games. [0, 1, 1, 0, 0, 0, 1, 1, 1, 1]

优点：考虑了词的顺序
缺点：词表的膨胀
预测Mary likes to 的概率:
Unigram/1-gram 一元模型

$$P(Mary likes too) = P(too|Mark, likes) * P(likes| Mary) * P(Mary)$$ $$=P(too) * P(likes) * P(Mary)$$
Bi-gram/2-gram二元模型
$$P(Mary likes too) = P(too|Mark, likes) * P(likes| Mary) * P(Mary)$$ $$=P(too| likes) * P(likes | Marry) * P(Mary)$$

6.共现矩阵

局域窗中的共现矩阵可以挖掘语法和语义信息

I like deep learning.
I like NLP.
I enjoy flying.

window length设为1（一般设为5 ~ 10）

counts	I	like	enjoy	deep	learning	NLP	flying	.
I	0	2	1	0	0	0	0	0
like	2	0	0	1	0	1	0	0
enjoy	1	0	0	0	0	0	1	0
deep	0	1	0	0	1	0	0	0
learning	0	0	0	1	0	0	0	1
NLP	0	1	0	0	0	0	0	1
flying	0	0	1	0	0	0	0	1
.	0	0	0	0	1	1	1	0

共现矩阵的行和列作为词向量。
向量维度随着词典大小线性增长（高维并稀疏）。
解决方法：降维和矩阵压缩。如：SVD降维