Word2Vec模型总结

1.Huffman树的构造
解析：给定n个权值作为n个叶子节点，构造一棵二叉树，若它的带权路径长度达到最小，则称这样的二叉树为最优二叉树，也称Huffman树。数的带权路径长度规定为所有叶子节点的带权路径长度之和。Huffman树构造，如下所示：
[1]将 { w 1 , w 2 , . . . , w 3 } \{w_1,w_2,...,w_3\} {w1​,w2​,...,w3​}看成是有n颗树的森林；
[2]在森林中选出两个根节点的权值最小的树合并，作为一棵新树的左、右子树，且新树的根节点权值为其左、右子树根节点权值之和；
[3]从森林中删除选取的两颗树，并将新树加入森林；
[4]重复[2][3]步，直到森林中只剩一棵树为止，该树即为所求的Huffman树。
说明：利用Huffman树设计的二进制前缀编码，称为Huffman编码，它既能满足前缀编码条件，又能保证报文编码总长最短。

2.基于Hierarchical Softmax的模型[CBOW模型]
解析：

其中参数的物理意义，如下所示：
[1]${\mathbf{X}}_w=\sum _{i=1}^{2c}\mathbf{v}\left(Context{\left(w\right)}_i\right)\in {\mathrm{R}}^{\mathrm{m}}$
[2]$d_j^w$表示路径$p^w$中第$j$结点对应的编码[根结点不对应编码]
[3]${\theta }_j^w$表示路径$p^w$中第$j$非叶子结点对应的向量
[4]$p^w$表示从根结点出发到达$w$对应叶子结点的路径。
[5]$l^w$表示路径$p^w$中包含结点的个数。
Hierarchical Softmax基本思想，如下所示：
$$p\left(w|Context\left(w\right)\right)=\prod _{j=2}^{l^w}p\left(d_j^w|{\mathbf{x}}_w,{\theta }_{j-1}^w\right)$$
$$p\left(d_j^w|{\mathbf{x}}_w,{\theta }_{j-1}^w\right)={\left[\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]}^{1-d_j^w}\cdot {\left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]}^{d_j^w}$$
对于word2vec中基于Hierarchical Softmax的CBOW模型，优化的目标函数，如下所示：
$$L=\sum _{w\in C}\log p\left(w|Context\left(w\right)\right)$$
这样得到对数似然函数，如下所示：
$$\begin{gathered} L=\sum _{w\in C}\log \prod _{j=2}^{l^w}\left\{{\left[\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]}^{1-d_j^w}\cdot {\left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]}^{d_j^w}\right\} \\ =\sum _{w\in C}\sum _{j=2}^{l^w}\left\{\left(1-d_j^w\right)\cdot \log \left[\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]+d_j^w\cdot \log \left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]\right\} \end{gathered}$$
将花括号中的内容简记为$L\left(w,j\right)$，如下所示：
$$L\left(w,j\right)=\left(1-d_j^w\right)\cdot \log \left[\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]+d_j^w\cdot \log \left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]$$
使用随机梯度上升法对${\theta }_{j-1}^w$求偏导，如下所示：
$$\begin{array}{l}\frac{\partial L\left(w,j\right)}{\partial {\theta }_{j-1}^w}=\frac{\partial }{\partial {\theta }_{j-1}^w}\left\{\left(1-d_j^w\right)\cdot \log \left[\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]+d_j^w\cdot \log \left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]\right\}\\ =\left(1-d_j^w\right)\cdot \left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]{\mathbf{x}}_w-d_j^w\cdot \sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right){\mathbf{x}}_w\\ =\left\{\left(1-d_j^w\right)\cdot \left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]-d_j^w\cdot \sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right\}{\mathbf{x}}_w\\ =\left[1-d_j^w-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]{\mathbf{x}}_w\end{array}$$
${\theta }_{j-1}^w$的更新方程，如下所示：
$${\theta }_{j-1}^w:={\theta }_{j-1}^w+\eta \left[1-d_j^w-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]{\mathbf{x}}_w$$
使用随机梯度上升法对${\mathbf{x}}_w$求偏导，如下所示：
$$\frac{\partial L\left(w,j\right)}{\partial {\mathbf{x}}_w}=\left[1-d_j^w-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }_{j-1}^w\right)\right]{\theta }_{j-1}^w$$
对于词典中每个词的词向量$\mathbf{v}\left(\tilde{w}\right)$更新方程，如下所示：
$$\mathbf{v}\left(\tilde{w}\right):=\mathbf{v}\left(\tilde{w}\right)+\eta \sum _{j=2}^{l^w}\frac{\partial L\left(w,j\right)}{\partial {\mathbf{x}}_w},\tilde{w}\in Context\left(w\right)$$

3.基于Hierarchical Softmax的模型[Skip-Gram模型]
解析：

其中，$\mathbf{v}\left(w\right)\in {\mathrm{R}}^m$表示当前样本的中心词$w$的词向量。
对于word2vec中基于Hierarchical Softmax的Skip-Gram模型，优化的目标函数，如下所示：
$$L=\sum _{w\in C}\log p\left(Context\left(w\right)|w\right)$$
Skip-Gram模型中条件概率函数$p\left(Context\left(w\right)|w\right)$，如下所示：
$$p\left(Context\left(w\right)|w\right)=\prod _{u\in Context\left(w\right)}p\left(u|w\right)$$
$$p\left(u|w\right)=\prod _{j=2}^{l^u}p\left(d_j^u|\mathbf{v}\left(w\right),{\theta }_{j-1}^u\right)$$
$$p\left(d_j^u|\mathbf{v}\left(w\right),{\theta }_{j-1}^u\right)={\left[\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }_{j-1}^u\right)\right]}^{1-d_j^u}\cdot {\left[1-\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }_{j-1}^u\right)\right]}^{d_j^u}$$
这样得到对数似然函数，如下所示：
$$\begin{array}{l}L=\sum _{w\in C}\log \prod _{u\in Context\left(w\right)}\prod _{j=2}^{l^u}\left\{{\left[\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }_{j-1}^u\right)\right]}^{1-d_j^u}\cdot {\left[1-\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }_{j-1}^u\right)\right]}^{d_j^u}\right\}\\ =\sum _{w\in C}\sum _{u\in Context\left(w\right)}\sum _{i=2}^{l^u}\left\{\left(1-d_j^u\right)\cdot \log \left[\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }_{j-1}^u\right)\right]+d_j^u\cdot \log \left[1-\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }_{j-1}^u\right)\right]\right\}\end{array}$$
将花括号中的内容简记为$L\left(w,u,j\right)$，如下所示：
$$L\left(w,u,j\right)=\left(1-d_j^u\right)\cdot \log \left[\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }_{j-1}^u\right)\right]+d_j^u\cdot \log \left[1-\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }_{j-1}^u\right)\right]$$

4.基于Negative Sampling的模型[CBOW模型]
Negative Sampling不再使用Huffman树，而是使用随机负采样，能大幅度提高性能。假定已经选好$w$的负样本子集$NEG\left(w\right)\ne \varnothing$，定义词$\tilde{w}$的标签[正样本为1，负样本为0]，如下所示：
$$L^w\left(\tilde{w}\right)=\{\begin{array}{l}1,\tilde{w}=w\\ 0,\tilde{w}\ne w\end{array}$$
对于给定的正样本$\left(Context\left(w\right),w\right)$，最大化$g\left(w\right)$，如下所示：
g ( w ) = ∏ u ∈ { w } ∪ N E G ( w ) p ( u ∣ C o n t e x t ( w ) ) g\left( w \right) = \prod\limits_{u \in \left\{ w \right\} \cup NEG\left( w \right)} {p\left( {u|Context\left( w \right)} \right)} g(w)=u∈{w}∪NEG(w)∏​p(u∣Context(w))
$$p\left(u|Context\left(w\right)\right)={\left[\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]}^{L^w\left(u\right)}\cdot {\left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]}^{\left[1-L^w\left(u\right)\right]}$$
其中，${\mathbf{x}}_w$表示$Context\left(w\right)$中各词的词向量之和，${\theta }^u\in {\mathrm{R}}^m$表示词$u$对应的一个辅助向量，为待训练的参数。简化$g\left(w\right)$方程，如下所示：
$$g\left(w\right)=\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^w\right)\prod _{u\in NEG\left(w\right)}\left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]$$
其中，$\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^w\right)$表示当上下文为$Context\left(w\right)$时，预测中心词为$w$的概率，同样$\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right),u\in NEG\left(w\right)$表示当上下文为$Context\left(w\right)$时，预测中心词为$u$的概率。
对于给定的语料库$C$，目标函数如下所示：
L = log ⁡ G = log ⁡ ∏ w ∈ C g ( w ) = ∑ w ∈ C log ⁡ g ( w ) = ∑ w ∈ C log ⁡ ∏ u ∈ { w } ∪ N E G ( w ) { [ σ ( x w T θ u ) ] L w ( u ) ⋅ [ 1 − σ ( x w T θ u ) ] 1 − L w ( u ) } = ∑ w ∈ C ∑ u ∈ { w } ∪ N E G ( w ) { L w ( u ) ⋅ log ⁡ [ σ ( x w T θ u ) ] + [ 1 − L w ( u ) ] ⋅ log ⁡ [ 1 − σ ( x w T θ u ) ] } = ∑ w ∈ C { log ⁡ [ σ ( x w T θ w ) ] + ∑ u ∈ N E G ( w ) log ⁡ [ 1 − σ ( x w T θ u ) ] } = ∑ w ∈ C { log ⁡ [ σ ( x w T θ w ) ] + ∑ u ∈ N E G ( w ) log ⁡ [ σ ( − x w T θ u ) ] } \begin{array}{l} L = \log G = \log \prod\limits_{w \in C} {g\left( w \right)} = \sum\limits_{w \in C} {\log g\left( w \right)} \\ = \sum\limits_{w \in C} {\log } {\prod _{u \in \left\{ w \right\} \cup NEG\left( w \right)}}\left\{ {{{\left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]}^{{L^w}\left( u \right)}}\cdot{{\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]}^{1 - {L^w}\left( u \right)}}} \right\} \\ = \sum\limits_{w \in C} {{\sum _{u \in \left\{ w \right\} \cup NEG\left( w \right)}}\left\{ {{L^w}\left( u \right)\cdot\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right] + \left[ {1 - {L^w}\left( u \right)} \right]\cdot\log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]} \right\}} \\ = \sum\limits_{w \in C} {\left\{ {\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^w}} \right)} \right] + {\sum _{u \in NEG\left( w \right)}}\log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]} \right\}} \\ = \sum\limits_{w \in C} {\left\{ {\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^w}} \right)} \right] + {\sum _{u \in NEG\left( w \right)}}\log \left[ {\sigma \left( { - {\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]} \right\}} \\ \end{array} L=logG=logw∈C∏​g(w)=w∈C∑​logg(w)=w∈C∑​log∏u∈{w}∪NEG(w)​{[σ(xwT​θu)]Lw(u)⋅[1−σ(xwT​θu)]1−Lw(u)}=w∈C∑​∑u∈{w}∪NEG(w)​{Lw(u)⋅log[σ(xwT​θu)]+[1−Lw(u)]⋅log[1−σ(xwT​θu)]}=w∈C∑​{log[σ(xwT​θw)]+∑u∈NEG(w)​log[1−σ(xwT​θu)]}=w∈C∑​{log[σ(xwT​θw)]+∑u∈NEG(w)​log[σ(−xwT​θu)]}​
记$L\left(w,u\right)=L^w\left(u\right)\cdot \log \left[\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]+\left[1-L^w\left(u\right)\right]\cdot \log \left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]$，使用随机梯度上升法对${\theta }^u$求偏导，如下所示：
$$\begin{array}{l}\frac{\partial L\left(w,u\right)}{\partial {\theta }^u}=\frac{\partial }{\partial {\theta }^u}\left\{L^w\left(u\right)\cdot \log \left[\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]+\left[1-L^w\left(u\right)\right]\cdot \log \left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]\right\}\\ =L^w\left(u\right)\left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]{\mathbf{x}}_w-\left[1-L^w\left(u\right)\right]\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right){\mathbf{x}}_w\\ =\left\{L^w\left(u\right)\left[1-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]-\left[1-L^w\left(u\right)\right]\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right\}{\mathbf{x}}_w\\ =\left[L^w\left(u\right)-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]{\mathbf{x}}_w\end{array}$$
参数${\theta }^u$的更新方程，如下所示：
$${\theta }^u:={\theta }^u+\eta \left[L^w\left(u\right)-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]{\mathbf{x}}_w$$
使用随机梯度上升法对${\mathbf{x}}_w$求偏导，如下所示：
$$\frac{\partial L\left(w,u\right)}{\partial {\mathbf{x}}_w}=\left[L^w\left(u\right)-\sigma \left({\mathbf{x}}_w^{\mathrm{T}}{\theta }^u\right)\right]{\theta }^u$$
参数$\mathbf{v}\left(\tilde{w}\right),\tilde{w}\in Context\left(w\right)$的更新方程，如下所示：
v ( w ~ ) : = v ( w ~ ) + η ∑ u ∈ { w } ∪ N E G ( w ) ∂ L ( w , u ) ∂ x w , w ~ ∈ C o n t e x t ( w ) {\bf{v}}\left( {\tilde w} \right): = {\bf{v}}\left( {\tilde w} \right) + \eta \sum\limits_{u \in \left\{ w \right\} \cup NEG\left( w \right)} {\frac{{\partial L\left( {w,u} \right)}}{{\partial {{\bf{x}}_w}}}} ,\tilde w \in Context\left( w \right) v(w~):=v(w~)+ηu∈{w}∪NEG(w)∑​∂xw​∂L(w,u)​,w~∈Context(w)

5.基于Negative Sampling的模型[Skip-Gram模型]
对于给定的语料库$C$，目标函数如下所示：
$$G=\prod _{w\in C}\prod _{u\in Context\left(w\right)}g\left(u\right)$$
g ( u ) = ∏ z ∈ { u } ∪ N E G { u } p ( z ∣ w ) g\left( u \right) = \prod\limits_{z \in \left\{ u \right\} \cup NEG\left\{ u \right\}} {p\left( {z|w} \right)} g(u)=z∈{u}∪NEG{u}∏​p(z∣w)
$$p\left(z|w\right)={\left[\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }^z\right)\right]}^{L^u\left(z\right)}\cdot {\left[1-\sigma \left(\mathbf{v}{\left(w\right)}^{\mathrm{T}}{\theta }^z\right)\right]}^{1-L^u\left(z\right)}$$
L = log ⁡ G = log ⁡ ∏ w ∈ C ∏ u ∈ C o n t e x t ( w ) g ( u ) = ∑ w ∈ C ∑ u ∈ C o n t e x t ( w ) log ⁡ g ( u ) = ∑ w ∈ C ∑ u ∈ C o n t e x t ( w ) log ⁡ ∏ z ∈ { u } ∪ N E G { u } p ( z ∣ w ) = ∑ w ∈ C ∑ u ∈ C o n t e x t ( w ) ∑ z ∈ { u } ∪ N E G { u } log ⁡ { [ σ ( v ( w ) T θ z ) ] L u ( z ) ⋅ [ 1 − σ ( v ( w ) T θ z ) ] 1 − L u ( z ) } = ∑ w ∈ C ∑ u ∈ C o n t e x t ( w ) ∑ z ∈ { u } ∪ N E G { u } { L u ( z ) ⋅ log ⁡ [ σ ( v ( w ) T θ z ) ] + [ 1 − L u ( z ) ] ⋅ log ⁡ [ 1 − σ ( v ( w ) T θ z ) ] } \begin{array}{l} L = \log G = \log \prod\limits_{w \in C} {\prod\limits_{_{u \in Context\left( w \right)}} {g\left( u \right)} } = \sum\limits_{w \in C} {\sum\limits_{_{u \in Context\left( w \right)}} {\log g\left( u \right)} } \\ = \sum\limits_{w \in C} {\sum\limits_{_{u \in Context\left( w \right)}} {\log \prod\limits_{z \in \left\{ u \right\} \cup NEG\left\{ u \right\}} {p\left( {z|w} \right)} } } \\ = \sum\limits_{w \in C} {\sum\limits_{_{u \in Context\left( w \right)}} {\sum\limits_{z \in \left\{ u \right\} \cup NEG\left\{ u \right\}} {\log \left\{ {{{\left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right]}^{{L^u}\left( z \right)}} \cdot {{\left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right]}^{1 - {L^u}\left( {\rm{z}} \right)}}} \right\}} } } \\ = \sum\limits_{w \in C} {\sum\limits_{_{u \in Context\left( w \right)}} {\sum\limits_{z \in \left\{ u \right\} \cup NEG\left\{ u \right\}} {\left\{ {{L^u}\left( z \right) \cdot \log \left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right] + \left[ {1 - {L^u}\left( {\rm{z}} \right)} \right] \cdot \log \left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right]} \right\}} } } \\ \end{array} L=logG=logw∈C∏​u∈Context(w)​∏​g(u)=w∈C∑​u∈Context(w)​∑​logg(u)=w∈C∑​u∈Context(w)​∑​logz∈{u}∪NEG{u}∏​p(z∣w)=w∈C∑​u∈Context(w)​∑​z∈{u}∪NEG{u}∑​log{[σ(v(w)Tθz)]Lu(z)⋅[1−σ(v(w)Tθz)]1−Lu(z)}=w∈C∑​u∈Context(w)​∑​z∈{u}∪NEG{u}∑​{Lu(z)⋅log[σ(v(w)Tθz)]+[1−Lu(z)]⋅log[1−σ(v(w)Tθz)]}​
对每一个样本$\left(w,Context\left(w\right)\right)$，需要针对$Context\left(w\right)$中的每一个词进行负采样，但是word2vec源码中只是针对$w$进行了$\left|Context\left(w\right)\right|$次负采样。它本质上用的还是CBOW模型，只是将原来通过求和累加做整体用的上下文$Context\left(w\right)$拆成一个一个来考虑。对于给定的语料库$C$，目标函数如下所示：
g ( w ) = ∏ w ~ ∈ C o n t e x t ( w ) ∏ u ∈ { w } ∪ N E G w ~ ( w ) p ( u ∣ w ~ ) {g\left( w \right) = {\prod _{\tilde w \in Context\left( w \right)}}{\prod _{u \in \left\{ w \right\} \cup NE{G^{\tilde w}}\left( w \right)}}p\left( {u|\tilde w} \right)} g(w)=w~∈Context(w)∏​u∈{w}∪NEGw~(w)∏​p(u∣w~)
$$p\left(u|\tilde{w}\right)={\left[\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]}^{L^w\left(u\right)}\cdot {\left[1-\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]}^{1-L^w\left(u\right)}$$
L = log ⁡ G = log ⁡ ∏ w ∈ C g ( w ) = ∑ w ∈ C log ⁡ g ( w ) = ∑ w ∈ C log ⁡ ∏ w ~ ∈ C o n t e x t ( w ) ∏ u ∈ { w } ∪ N E G w ~ ( w ) { [ σ ( v ( w ~ ) T θ u ) ] L w ( u ) ⋅ [ 1 − σ ( v ( w ~ ) T θ u ) ] 1 − L w ( u ) } = ∑ w ∈ C log ⁡ ∑ w ~ ∈ C o n t e x t ( w ) ∑ u ∈ { w } ∪ N E G w ~ ( w ) { L w ( u ) ⋅ log ⁡ [ σ ( v ( w ~ ) T θ u ) ] + [ 1 − L w ( u ) ] ⋅ log ⁡ [ 1 − σ ( v ( w ~ ) T θ u ) ] } \begin{array}{l} L = \log G = \log \prod\limits_{w \in C} {g\left( w \right)} = \sum\limits_{w \in C} {\log g\left( w \right)} \\ = \sum\limits_{w \in C} {\log \prod\limits_{\tilde w \in Context\left( w \right)} {\prod\limits_{u \in \left\{ w \right\} \cup NE{G^{\tilde w}}\left( w \right)} {\left\{ {{{\left[ {\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]}^{{L^w}\left( u \right)}} \cdot {{\left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]}^{1 - {L^w}\left( u \right)}}} \right\}} } } \\ = \sum\limits_{w \in C} {\log \sum\limits_{\tilde w \in Context\left( w \right)} {\sum\limits_{u \in \left\{ w \right\} \cup NE{G^{\tilde w}}\left( w \right)} {\left\{ {{L^w}\left( u \right) \cdot \log \left[ {\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right] + \left[ {1 - {L^w}\left( u \right)} \right] \cdot \log \left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]} \right\}} } } \\ \end{array} L=logG=logw∈C∏​g(w)=w∈C∑​logg(w)=w∈C∑​logw~∈Context(w)∏​u∈{w}∪NEGw~(w)∏​{[σ(v(w~)Tθu)]Lw(u)⋅[1−σ(v(w~)Tθu)]1−Lw(u)}=w∈C∑​logw~∈Context(w)∑​u∈{w}∪NEGw~(w)∑​{Lw(u)⋅log[σ(v(w~)Tθu)]+[1−Lw(u)]⋅log[1−σ(v(w~)Tθu)]}​
记$L\left(w,\tilde{w},u\right)=L^w\left(u\right)\cdot \log \left[\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]+\left[1-L^w\left(u\right)\right]\cdot \log \left[1-\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]$。使用随机梯度上升法，对${\theta }^u$求偏导，如下所示：
$$\begin{array}{l}\frac{\partial L\left(w,\tilde{w},u\right)}{\partial {\theta }^u}=\frac{\partial L}{\partial {\theta }^u}\left\{L^w\left(u\right)\cdot \log \left[\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]+\left[1-L^w\left(u\right)\right]\cdot \log \left[1-\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]\right\}\\ =L^w\left(u\right)\left[1-\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]\mathbf{v}\left(\tilde{w}\right)-\left[1-L^w\left(u\right)\right]\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\mathbf{v}\left(\tilde{w}\right)\\ =\left\{L^w\left(u\right)\left[1-\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]-\left[1-L^w\left(u\right)\right]\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right\}\mathbf{v}\left(\tilde{w}\right)\\ =\left[L^w\left(u\right)-\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]\mathbf{v}\left(\tilde{w}\right)\end{array}$$
${\theta }^u$的更新方程，如下所示：
$${\theta }^u:={\theta }^u+\eta \left[L^w\left(u\right)-\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]\mathbf{v}\left(\tilde{w}\right)$$
使用随机梯度上升法，对$\mathbf{v}\left(\tilde{w}\right)$求偏导，如下所示：
$$\frac{\partial L\left(w,\tilde{w},u\right)}{\partial \mathbf{v}\left(\tilde{w}\right)}=\left[L^w\left(u\right)-\sigma \left(\mathbf{v}{\left(\tilde{w}\right)}^{\mathrm{T}}{\theta }^u\right)\right]{\theta }^u$$
参数$\mathbf{v}\left(\tilde{w}\right)$的更新，如下所示：
v ( w ~ ) : = v ( w ~ ) + η ∑ u ∈ { w } ∪ N E G w ~ ( w ) ∂ L ( w , w ~ , u ) ∂ v ( w ~ ) {\bf{v}}\left( {\tilde w} \right): = {\bf{v}}\left( {\tilde w} \right) + \eta \sum\limits_{u \in \left\{ w \right\} \cup NE{G^{\tilde w}}\left( w \right)} {\frac{{\partial L\left( {w,\tilde w,u} \right)}}{{\partial {\bf{v}}\left( {\tilde w} \right)}}} v(w~):=v(w~)+ηu∈{w}∪NEGw~(w)∑​∂v(w~)∂L(w,w~,u)​
其中，$NE{G}^{\tilde{w}}\left(w\right)$表示处理词$\tilde{w}$时生成的负样本子集。

6.Negative Sampling算法
[1]带权采样原理
设词典$D$中的每一个词$w$对应一个线段$l\left(w\right)$，长度如下所示：
$$len\left(w\right)=\frac{\mathrm{c}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\left(w\right)}{\sum _{u\in D}\mathrm{c}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\left(u\right)}$$
这里$\mathrm{c}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\left(\cdot \right)$表示一个词在语料$C$中出现的次数。现在将这些线段首尾相连地拼接在一起，形成一个长度为1的单位线段。如果随机地往这个单位线段上打点，那么其中长度越长的线段（对应高频词）被打中的概率就越大。
[2]word2vec负采样
记$l_0=0$，$l_k=\sum _{j=1}^{k}len\left(w_j\right),k=1,2,\cdots ,N$，这里$w_j$表示词典$D$中第$j$个词，则以 { l j } j = 0 N \left\{ {{l_j}} \right\}_{j = 0}^N {lj​}j=0N​为剖分结点可得到区间$\left[0,1\right]$上的一个非等距剖分，$I_i=(l_{i-1},l_i],i=1,2,\cdots ,N$为其$N$个剖分区间。进一步引入区间$\left[0,1\right]$上的一个等距离剖分，剖分结点为 { m j } j = 0 M \left\{ {{m_j}} \right\}_{j = 0}^M {mj​}j=0M​，其中$M\gg N$，具体示意图如下所示：

将内部剖分结点 { m j } j = 1 M − 1 \left\{ {{m_j}} \right\}_{j = 1}^{M-1} {mj​}j=1M−1​投影到非等距剖分上，则可建立 { m j } j = 1 M − 1 \left\{ {{m_j}} \right\}_{j = 1}^{M-1} {mj​}j=1M−1​与区间 { I j } j = 1 N \left\{ {{I_j}} \right\}_{j = 1}^{N} {Ij​}j=1N​（或 { w j } j = 1 N \left\{ {{w_j}} \right\}_{j = 1}^N {wj​}j=1N​）的映射关系，如下所示：
$$\mathrm{T}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}\left(i\right)=w_k,m_i\in I_k,i=1,2,\cdots ,M-1$$
根据映射每次生成一个$\left[1,M-1\right]$间的随机整数$r$，$\mathrm{T}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}\left(r\right)$就是一个样本。当对$w_i$进行负采样时，如果采样为$w_i$，那么就跳过去。

参考文献：
[1]word2vec中的数学原理详解