word2vec损失函数的数学推导和近似训练以及doc2vec的实战掉包应用

更新：
9/4/20
做了关于SG模型最后计算loss的一些补充。对doc2vec损失计算部分出现的错误进行了订正。

11/17/20
补充了部分近似训练的内容。

词义表示

在NLP中，最基础的问题就是如何表示一个词、句子(Represent the Meaning of a Word)。接下来介绍的几种方法各有优劣，不过也是不断进步的过程。

WordNet

WordNet is a large lexical database of English. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations.

pros

1. can find synonyms. 方便寻找同义词

cons

1. missing new words (impossible to keep up to date). 缺少新词。
2. subjective. 主观化。
3. requires human labor to create and adapt. 需要耗费大量人力去整理。

One Hot Encoding

Discrete representation.

cons

1. dimension is extremely high. 维度爆炸。
2. hard to compute accurate word similarity (all vectors are orthogonal). 无法计算词语相似度。

Bag of Words

Co-occurrence of words with variable window size.

cons

1. dimension is extremely high, will grow as dictionary grows. 维度爆炸，而且会随着字典大小的增大而增大，对下游的ML模型产生影响。

Word2vec

A neural probabilistic language model.

Distributional similarity based representations. Represent a word by means of its neighbors.上下文足以辅助理解一个词的意思。

We will build a dense vector for each word type, chosen so that it is good at predicting other words appearing in its context.

Distributional similarity & Distributed representation (dense vector)

There are certain differences between the two. The Distributional Similarity emphasizes that the meaning of a word shall be inferred from its context. Distributed Representation is opposite to One Hot Encoding, and vector representation is non-sparse. 两者有一定的区别。distributional similarity强调能够用上下文去表示某一个单词的意思，而distributed representation与one hot encoding相对，强调向量的表示是非稀疏的。

pros

1. can compute accurate word similarity. 可以计算词语相似度。

cons

1. The calculation is related word vector instead of semantic word vector, so the polysemous case cannot be solved (1 vector for each word instead of each meaning). 计算出来的是关联词向量，而不是语义词向量，所以无法解决一词多意的情况(每个单词而不是每个词意对应1个向量)。

损失函数Loss Function

Softmax function: map from $R^v$ to a probability distribution(从实数空间到概率分布的标准映射方法)。公式分子部分保证将这个数转化成一个正数，分母部分保证所有概率之和为1。

$p_i=\frac{exp(u_i)}{{\sum }_{j}exp(u_j)}$

我们在求出center/context word的概率分布之后，还需要使用交叉熵来得到loss。

$L(\hat{y},y)=-{\sum }_{j=1}^V{y}_{j}log({\hat{y}}_j)$. 根据公式，在完美预测的情况下，loss是0。

此处以skip-gram的训练为例。

$J=1-p(w_{-t}|w_t)$

$w_{-t}$代表$w_t$的上下文（负号表示除了该词之外）。

$p(o|c)=\frac{exp(u_o^T{v}_c)}{{\sum }_{w=1}^{V}exp(u_w^T{v}_c)}$

o is the outside (or output) word index, c is the center word index. $v_c$ and $u_o$ are center and outside vectors of indices c and o. Softmax uses word c to obtain probability of word o.

According to this formula, the words in the text will be represented by two vectors. There’s one when it’s a center word, and there’s another when it’s a context. 根据这个公式，文中的单词会有两个向量表示。当它作为中心词的时候有一个，当它作为上下文的时候又有一个。

loss的推导过程主要运用softmax的概率分布公式和微积分的链式法则。
$$\begin{gathered} \frac{\partial }{\partial v_c}p(o|c) \\ =\frac{\partial }{\partial v_c}log[exp(u_o^T{v}_c)/\sum _{w=1}^{V}exp(u_w^T{v}_c)] \\ =\frac{\partial }{\partial v_c}log[exp(u_o^T{v}_c)]①-\frac{\partial }{\partial v_c}log[\sum _{w=1}^{V}exp(u_w^T{v}_c)]② \\ =u_o-\sum _{x=1}^{V}p(x|c)u_x \end{gathered}$$
① 表示的是observation，也就是context word实际是什么(true label)。② 表示的是expectation，也就是模型认为概率最高的应该是哪个词(prediction label)。所以，实际上我们就是希望最小化实际和预测之间的差值。
$$\begin{gathered} ②=\frac{\partial }{\partial v_c}log[\sum _{w=1}^{V}exp(u_w^T{v}_c)] \\ =\frac{1}{{\sum }_{w=1}^{V}exp(u_w^T{v}_c)}\frac{\partial }{\partial v_c}[\sum _{x=1}^{V}exp(u_x^T{v}_c)] \\ ={\sum }_{w=1}^{V}exp(u_w^T{v}_c) \end{gathered}$$