Word2Vec学习笔记（三）续

三、（续）Skip-gram模型介绍

    Skip-gram模型并不是和CBOW模型相反的，它们的目的都是计算出词的向量，只不过在作者的论文中给出的图看样子是反的而已。Skip-gram模型是用每个当前词去预测一定范围内除当前词之外前后的词。同样的，此模型也是输出一颗huffman树，如下图所示：此图也借用下图借用
http://blog.csdn.net/itplus/article/details/37969979

Skip-gram模型的目标函数

    由于Skip-gram的模型输入是当前词，目的是预测它周围的词，因此，此任务的目标函数如下所示：

$$\mathcal{L} = \sum_{w \in C} \log P(context(w)|w)$$
由于 $context(w)$ 是一个句子，因此，可以将 $P(context(w)|w)$写成如下形式：
$$P(context(w)|w) = \prod_{u \in context(w)}P(u|w)$$
根据hierarchical softmax的讨论：
$$P(u|w) = \prod_{j=2}^{l^u}P(d_j^u|v(u); \theta_{j-1})$$
那么：最终的目标函数可以写为：
$$\mathcal{L} = \sum_{w \in C} \log \prod_{u \in context(w)} \prod_{j=2}^{l^u}P(d_j^w|v(u); \theta_{j-1})$$

这里，$\theta, v(w)$的更新公式先留着，待推导完再进行补充。
今天推导了一下，其实和上节中的推导差不多，现将推导过程记录一下

$$\mathcal{L} = \sum_{w \in C} \log \prod_{u \in context(w)} \prod_{j=2}^{l^u}P(d_j^w|v(u); \theta_{j-1}) \\ = \sum_{w \in C}\sum_{u \in context(w)} \sum_{j=2}^{l^u} \log P(d_j^w|v(u); \theta_{j-1})\\ = \sum_{w \in C}\sum_{u \in context(w)} \sum_{j=2}^{l^u} \log \{ [1-\sigma(v(w)^T\theta_{j-1}^{u})]^{d_j^u} \sigma(v(w)^T\theta_{j-1}^{u})]^{1-d_j^u} \}\\ =\sum_{w \in C}\sum_{u \in context(w)} \sum_{j=2}^{l^u} \{d_j^u\log [1-\sigma(v(w)^T\theta_{j-1}^{u})] + (1-d_j^u)\log [\sigma(v(w)^T\theta_{j-1}^{u})]\}$$
令 $f = d_j^u\log [1-\sigma(v(w)^T\theta_{j-1}^{u})] + (1-d_j^u)\log [\sigma(v(w)^T\theta_{j-1}^{u})]$,则分别求出 $f$对$\theta_j$ 和 $v(w)$求偏导数：
$$\frac{\partial{f}}{\partial{\theta_{j-1}^{u}}}=[1-d_j^u-\sigma(v(w)^T\theta_{j-1}^{u})] v(w)\\ \frac{\partial{f}}{\partial{v(w)}} = [1-d_j^u-\sigma(v(w)^T\theta_{j-1}^{u})] \theta_{j-1}^{u}$$
那么 $\theta$和 $v(w)$的更新公式如下：
$$\theta_{j-1}^{u} :=\theta_{j-1}^{u}+\eta [1-d_j^u-\sigma(v(w)^T\theta_{j-1}^{u})] v(w)\\ v(w):=v(w)+\sum_{u \in context(w)} \sum_{j=2}^{l^u}[1-d_j^u-\sigma(v(w)^T\theta_{j-1}^{u})] \theta_{j-1}^{u}$$

Word2Vec 的重点参考文献

1. Efficient Estimation of Word Representations in Vector Spaceh. http://arxiv.org/pdf/1301.3781v3.pdf
2. Distributed Representations ofWords and Phrases and their Compositionality. https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf
3. Exploiting Similarities among Languages for Machine Translation. http://static.googleusercontent.com/media/research.google.com/zh-CN//pubs/archive/44931.pdf
4. http://blog.csdn.net/itplus/article/details/37969979
5. http://www.cnblogs.com/neopenx/p/4571996.html
6. http://blog.csdn.net/zhoubl668/article/details/24319529