CS 224n Assignment #2: word2vec (written部分）

CS 224n Assignment #2: word2vec (written部分）

understanding word2vec

==The key insight behind word2vec is that ‘a word is known by the company it keeps’. == Concretely, suppose we have a ‘center’ word c and a contextual window surrounding c. We shall refer to words that lie in this contextual window as ‘outside words’. For example, in Figure 1 we see that the center word c is ‘banking’. Since the context window size is 2, the outside words are ‘turning’, ‘into’, ‘crises’, and ‘as’.

The goal of the skip-gram word2vec algorithm is to accurately learn the probability distribution $P(O|C)$. Given a specific word o and a specific word c, we want to calculate $P(O=o|C=c)$, which is the probability that word o is an ‘outside’ word for c, i.e., the probability that o falls within the contextual window of c.

In word2vec, the conditional probability distribution is given by taking vector dot-products and applying the softmax function:

Here, $u_o$ is the ‘outside’ vector representing outside word o, and v_c is the ‘center’ vector representing center word c. To contain these parameters, we have two matrices, $U$ and $V$ . The columns of $U$ are all the ‘outside’ vectors $u_w$. The columns of $V$ are all of the ‘center’ vectors $v_w$. Both $U$ and
$V$ contain a vector for every $w\in Vocabulary{.}^1$

Recall from lectures that, for a single pair of words c and o, the loss is given by:

Another way to view this loss is as the $cross-entrop{y}^2$ between the true distribution y and the predicted distribution $yˆ$. Here, both y and $yˆ$ are vectors with length equal to the number of words in the vocabulary. Furthermore, the $k^{th}$ entry in these vectors indicates the conditional probability of the $k^{th}$ word being an ‘outside word’ for the given c. The true empirical distribution y is a one-hot vector with a 1 for the true outside word o, and 0 everywhere else. The predicted distribution $yˆ$ is the probability distribution $P(O|C=c)$ given by our model in equation (1).

---

${}^1$Assume that every word in our vocabulary is matched to an integer number k. $u_k$ is both the $k^{t}h$ column of U and the ‘outside’ word vector for the word indexed by k. $v_k$ is both the $k^{t}h$ column of V and the ‘center’ word vector for the word indexed by k. In order to simplify notation we shall interchangeably use k to refer to the word and the index-of-the-word.
${}^2$The Cross Entropy Loss between the true (discrete) probability distribution p and another distribution q is$-{\sum }_i{p}_{i}log(q_i)$.

Question and Answer

(a) (3 points) Show that the naive-softmax loss given in Equation (2) is the same as the cross-entropy loss between y and $yˆ$ ;i.e.,show that

Answer:
$y_w$是单位矩阵的一列（one-hot）,因此对于只有中心词$w_0$的位置为1，其余位置为0：
$-{\sum }_{w\in Vocab}{y}_{w}log({\hat{y}}_w)=-{\sum }_{w\in Vocabw\ne w_o}-y_{w_o}log({\hat{y}}_{wo})=0-log({\hat{y}}_{wo})=-log({\hat{y}}_{wo})$

(b) (5 points) Compute the partial derivative of $J_{naive-softmax}(v_c,o,U)$ with respect to $v_c$. Please write your answer in terms of $y,\hat{y}$, and $U$.

Answer:
首先先解释一下：

1. $U$代表一个单词作为上下文的坐标；
2. $V$代表一个单词作为中心词的坐标；
3. $y$是输入（训练集）；
4. $\hat{y}$是输出的估计值（对该训练数据的预测）；

因为这里$u_0=U^{T}y表示取的第o单词的坐标，由于y是one-hot的$，$\sum u_{w}P(w|c)$对于给定的中心词c的概率分布，因此有$P(w|c)={\hat{y}}_w$是我们的预测值，因此上式变为：
$\frac{\partial J_{naive-softmax}}{\partial v_c}=U^T(\hat{y}-y)$

© (5 points) Compute the partial derivatives of $J_{naive-softmax}(vc,o,U)$ with respect to each of the ‘outside’ word vectors, $u_w’s$. There will be two cases: when $w=o$, the true ‘outside’ word vector, and $w\ne o$, for all other words. Please write you answer in terms of $y,\hat{y}$, and $v_c$.

Answer:

(d) (3 Points) The sigmoid function is given by Equation 4:
$\sigma (x)=\frac{1}{1+e^{-x}}=\frac{e^x}{e^x+1}$
Please compute the derivative of σ(x) with respect to x, where x is a vector.

Answer:
$$\frac{\mathrm{d}\sigma (x)}{\mathrm{d}x}=\sigma (x)(1-\sigma (x))$$

(e) (4 points) Now we shall consider the Negative Sampling loss, which is an alternative to the Naive Softmax loss. Assume that K negative samples (words) are drawn from the vocabulary. For simplicity of notation we shall refer to them as $w_1,w_2,...,w_K$ and their outside vectors as $u_1,...,u_K$. Note that $o\notin w_1,...,w_K$. For a center word c and an outside word o, the negative sampling loss function is given by:

for a sample $w_1,...w_K$, where σ(·) is the sigmoid $function{.}^3$ Please repeat parts (b) and ©, computing the partial derivatives of $J_{neg-sample}$ with respect to $v_c$, with respect to $u_o$, and with respect to a negative sample $u_k$. Please write your answers in terms of the vectors $u_o,v_c$, and $u_k$, where k ∈ [1,K]. After you’ve done this, describe with one sentence why this loss function is much more efficient to compute than the naive-softmax loss. Note, you should be able to use your solution to part (d) to help compute the necessary gradients here.

(f) (3 points) Suppose the center word is $c=w_t$ and the context window is $[w_{t-m},...,w_{t-1},w_t,w_{t+1},...,w_{t+m}]$, where m is the context window size. Recall that for the skip-gram version of word2vec, the total loss for the context window is:

Here, $J(v_c,w_{t+j},U)$ represents an arbitrary loss term for the center word $c=w_t$ and outside word $w_{t+j}$. $J(v_c,w_{t+j},U)$ could be $J_{naive-softmax}(v_c,w_{t+j},U)$ or $J_{neg-sample}(v_c,w_{t+j},U)$, depending on your implementation.
Write down three partial derivatives:

Answer: