CS224n(2019):Assignment2 参考答案

CS224n ： Assignment2 参考答案

本文为cs224n课程(winter,2019)的 assignment2 中的公式推导部分参考答案。如有疑问或者错误之处，欢迎交流。
Assignment2 原文
Assignment2 编码部分参考答案

Variables notation

Attention: All the variables’ dimensions here are consistent with the code part in Assignment 2 for easy understanding.

$\mathbf{U}$ , matrix of shape (vocab_size,embedding_dim) ,all the ‘outside’ vectors .

$\mathbf{V}$, matrix of shape (vocab_size,embedding_dim) ,all the ‘center’ vectors .

$\mathbf{y}$, vector of shape (vocab_size,1), the true empirical distribution $\mathbf{y}$ is a one-hot vector with a 1 for the true outside word o, and 0 everywhere else .

$\hat{\mathbf{y}}$, vector of shape (vocab_size,1), the predicted distribution $\hat{\mathbf{y}}$ is the probability distribution $P(O|C=c)$ given by our model .

question a

Given outside word o and context word c.

The distribution of y is as follows:

$$y_w=\{\begin{array}{ll}1& \text{w=o}\\ 0& \text{w!=o}\end{array}$$

$$-\sum _{w=1}^V{y}_{w}log(\widehat{y_w})=-y_{o}log(\widehat{y_o})=-log(\widehat{y_o})$$

Here , V represents the vocab_size.

question b

$$\begin{gathered} \frac{\partial J_{naive-softmax}({\mathbf{v}}_c,o,\mathbf{U})}{\partial {\mathbf{v}}_c} \\ =-\frac{\partial log(P(O=o|C=c))}{\partial {\mathbf{v}}_c} \\ =-\frac{\partial log(exp({\mathbf{u}}_o^T{\mathbf{v}}_c))}{\partial {\mathbf{v}}_c}+\frac{\partial log({\sum }_{w=1}^{V}exp({\mathbf{u}}_w^T{\mathbf{v}}_c))}{\partial {\mathbf{v}}_c} \\ =-{\mathbf{u}}_o+\sum _{w=1}^V\frac{exp({\mathbf{u}}_w^T{\mathbf{v}}_c)}{{\sum }_{w=1}^{V}exp({\mathbf{u}}_w^T{\mathbf{v}}_c)}{\mathbf{u}}_w \\ =-{\mathbf{u}}_o+\sum _{w=1}^{V}P(O=w|C=c){\mathbf{u}}_w \\ ={\mathbf{U}}^T(\hat{\mathbf{y}}-\mathbf{y}) \end{gathered}$$

question c

$$\begin{gathered} \frac{\partial J_{naive-softmax}({\mathbf{v}}_c,o,\mathbf{U})}{\partial {\mathbf{u}}_w} \\ =-\frac{\partial log(exp({\mathbf{u}}_o^T{\mathbf{v}}_c))}{\partial {\mathbf{u}}_w}+\frac{\partial log({\sum }_{w=1}^{V}exp({\mathbf{u}}_w^T{\mathbf{v}}_c))}{\partial {\mathbf{u}}_w} \end{gathered}$$

when w = o,

$$\begin{gathered} \frac{\partial J_{naive-softmax}({\mathbf{v}}_c,o,\mathbf{U})}{\partial {\mathbf{u}}_w} \\ =-{\mathbf{v}}_c+\frac{1}{{\sum }_{w=1}^{V}exp({\mathbf{u}}_w^T{\mathbf{v}}_c)}\frac{\partial {\sum }_{w=1}^{V}exp({\mathbf{u}}_w^T{\mathbf{v}}_c)}{\partial {\mathbf{u}}_o} \\ =-{\mathbf{v}}_c+\frac{1}{{\sum }_{w=1}^{V}exp({\mathbf{u}}_w^T{\mathbf{v}}_c)}\frac{\partial exp({\mathbf{u}}_o^T{\mathbf{v}}_c)}{\partial {\mathbf{u}}_o} \\ =-{\mathbf{v}}_c+\frac{exp({\mathbf{u}}_o^T{\mathbf{v}}_c)}{{\sum }_{w=1}^{V}exp({\mathbf{u}}_w^T{\mathbf{v}}_c)}{\mathbf{v}}_c \\ =(P(O=o|C=c)-1)){\mathbf{v}}_c \end{gathered}$$

when w != o,

$$\begin{gathered} \frac{\partial J_{naive-softmax}({\mathbf{v}}_c,o,\mathbf{U})}{\partial {\mathbf{u}}_w} \\ =\frac{exp({\mathbf{u}}_w^T{\mathbf{v}}_c)}{{\sum }_{w=1}^{V}exp({\mathbf{u}}_w^T{\mathbf{v}}_c)}{\mathbf{v}}_c \\ =P(O=w|C=c){\mathbf{v}}_c \end{gathered}$$

In summary,

$$\begin{gathered} \frac{\partial J_{naive-softmax}({\mathbf{v}}_c,o,\mathbf{U})}{\partial \mathbf{U}} \\ =(\hat{\mathbf{y}}-\mathbf{y}{)}^T{\mathbf{v}}_c \end{gathered}$$

question d

$$\begin{gathered} \frac{\partial \sigma (x)}{\partial x}=\frac{\partial \frac{e^x}{e^x+1}}{\partial x}=\frac{e^x(e^x+1)-e^x{e}^x}{(e^x+1{)}^2} \\ =\frac{e^x}{(e^x+1{)}^2}=\sigma (x)(1-\sigma (x)) \end{gathered}$$

question e

i)
$$\begin{gathered} \frac{\partial J_{neg-sample}({\mathbf{v}}_c,o,\mathbf{U})}{\partial {\mathbf{v}}_c} \\ =\frac{\partial (-log(\sigma ({\mathbf{u}}_o^T{\mathbf{v}}_c))-{\sum }_{k=1}^{K}log(\sigma (-{\mathbf{u}}_k^T{\mathbf{v}}_c)))}{\partial {\mathbf{v}}_c} \\ =-\frac{\sigma ({\mathbf{u}}_o^T{\mathbf{v}}_c)(1-\sigma ({\mathbf{u}}_o^T{\mathbf{v}}_c))}{\sigma ({\mathbf{u}}_o^T{\mathbf{v}}_c)}\frac{\partial {\mathbf{u}}_o^T{\mathbf{v}}_c}{\partial {\mathbf{v}}_c}-\sum _{k=1}^K\frac{\partial log(\sigma (-{\mathbf{u}}_k^T{\mathbf{v}}_c))}{\partial {\mathbf{v}}_c} \\ =-(1-\sigma ({\mathbf{u}}_o^T{\mathbf{v}}_c)){\mathbf{u}}_o+\sum _{k=1}^K(1-\sigma (-{\mathbf{u}}_k^T{\mathbf{v}}_c)){\mathbf{u}}_k \end{gathered}$$

ii)
$$\begin{gathered} \frac{\partial J_{neg-sample}({\mathbf{v}}_c,o,\mathbf{U})}{\partial {\mathbf{u}}_o} \\ =\frac{\partial (-log(\sigma ({\mathbf{u}}_o^T{\mathbf{v}}_c))}{\partial {\mathbf{u}}_o}=-(1-\sigma ({\mathbf{u}}_o^T{\mathbf{v}}_c)){\mathbf{v}}_c \end{gathered}$$

iii)
$$\begin{gathered} \frac{\partial J_{neg-sample}({\mathbf{v}}_c,o,\mathbf{U})}{\partial {\mathbf{u}}_k} \\ =\frac{\partial (-log(\sigma (-{\mathbf{u}}_k^T{\mathbf{v}}_c))}{\partial {\mathbf{u}}_k}=(1-\sigma (-{\mathbf{u}}_k^T{\mathbf{v}}_c)){\mathbf{v}}_c \end{gathered}$$

qustion f

i)

$$\begin{gathered} \frac{\partial J_{skip-gram}({\mathbf{v}}_c,w_{t-m},...,w_{t+m},\mathbf{U})}{\partial \mathbf{U}} \\ =\sum _{-m<=j<=m,j!=0}\frac{\partial J({\mathbf{v}}_c,w_{t+j},\mathbf{U})}{\partial \mathbf{U}} \end{gathered}$$

ii)

when w=c,

$$\begin{gathered} \frac{\partial J_{skip-gram}({\mathbf{v}}_c,w_{t-m},...,w_{t+m},\mathbf{U})}{\partial {\mathbf{v}}_c} \\ =\sum _{-m<=j<=m,j!=0}\frac{\partial J({\mathbf{v}}_c,w_{t+j},\mathbf{U})}{\partial {\mathbf{v}}_c} \end{gathered}$$

iii)

when w!=c,

$$\begin{gathered} \frac{\partial J_{skip-gram}({\mathbf{v}}_c,w_{t-m},...,w_{t+m},\mathbf{U})}{\partial {\mathbf{v}}_w} \\ =0 \end{gathered}$$