CS244n NLP with Deep Learning | Winter 2019

Lecture 1

word2vec

每一个单词可以变成一个vector, 相似的word vector也相似
word = $\left[\begin{array}{c}0.23\\ 0.52\\ -0.41\\ -0.31\\ 0.27\\ 0.48\end{array}\right]$
$$J(\theta )=\frac{1}{T}\sum _{t=1}^T\sum _{\begin{array}{c}-m\le j\le m\\ j\ne 0\end{array}}\log p(w_{t+j}|w_{t;\theta })$$
这里m是单词j的sliding window的左右长度

$$\theta =\left[\begin{array}{c}{v}_{apple}\\ v_{banana}\\ v_{zebra}\\ ...\\ v_{apple}\\ v_{banana}\\ v_{zebra}\end{array}\right]={\mathbb{R}}^{2dv}$$
重复因为每个词有center和context两种representation。
derivative: $$\frac{\partial \log (p)(\frac{exp(u_o^T{\dot{V}}_c)}{\sum exp(u_o^T{\dot{V}}_c)})}{\partial v_c}=u_0-$$
$$\begin{gathered} \frac{\partial }{\partial v_c}log\sum _{w=1}^V\exp (u_w^T{\dot{v}}_c)=\frac{1}{{\sum }_{w=1}^v\exp (u_w^T{v}_c)}\ast \sum _{x=1}^V\exp (u_x^T{v}_c)\frac{\partial }{\partial v_c}{u}_x^T{v}_c= \\ {u}_0-\frac{{\sum }_{x=1}^V\exp (u_x^T{v}_c)u_x}{{\sum }_{w=1}^v\exp (u_w^T{v}_c)}=u_0-\sum _{x=1}^{V}p(x|c)u_x \end{gathered}$$
https://medium.com/@zafaralibagh6/a-simple-word2vec-tutorial-61e64e38a6a1
疑问：是不是WI指的是$u_w$, WO指的是$v_w$ ?

Lecture 2

Gradient descent
${\theta }^{new}={\theta }^{old}-\alpha {\nabla }_{\theta }J(\theta )$, $\alpha$ is the learnnig rate
Stochastic Gradient Descnet and negative sampling
$$P(O|C)=\frac{exp(u_o^T{\dot{V}}_c}{{\sum }_{w\in V}exp(u_w^T{\dot{V}}_c)}$$