CS224d-Lecture8

Language Model

probability of a sequence of words

• P(w1, w2, …, wT)

Useful for machine learning:

word - ordering

• p(the cat is small) > p(small the is cat)

word - choice

• p(walking home after school) > p(walking house after school)

Traditional Language Model

条件概率，其中 window size = n

assumption

$$\begin{align*} P(w_1, w_2, ..., w_T) &= \prod_{i=1}^m P(w_i|w_1,w_{i-1}) \\ &\approx \prod_{i=1}^m P(w_i|w_1,w_{i-1}) \end{align*}$$

n-gram

• unigram $p(w_2|w_1) = \frac{count(w_1,w_2)}{count(w_1)}$
• bigram $p(w_3|w_1, w_2) = \frac{count(w_1,w_2,w_3)}{count(w_1, w_2)}$
  n-gram 耗费大量内存

RNN

• 每步权重互联
• 条件依赖于之前所有单词
• RAM 耗费只同单词量相关
  
  

$h_t = \sigma(W^{hh} h_{t-1} + W^{hx}x_t)$
$\hat y_t = softmax(W^{s} h_t)$

训练 RNN is hard

vanishing / exploding gradient problem

total error

$$\tfrac{\partial E}{\partial W} = \sum_{t=1}^T\tfrac{\partial E_t}{\partial W}$$
$$\tfrac{\partial E_t}{\partial W} = \sum_{k=1}^T\tfrac{\partial E_t}{\partial y_t} \cdot \tfrac{\partial y_t}{\partial h_t} \cdot \tfrac{\partial h_t}{\partial h_k} \cdot \tfrac{\partial h_k}{\partial W}$$
其中
$$\tfrac{\partial h_t}{\partial h_k} = \prod_{j=k+1}^t \tfrac{\partial h_j}{\partial h_{j-1}}$$
故

由于取
$$h_t = Wf(h_{t-1}) + W^{(hx)}x_{[t]}$$
则
$$\tfrac{\partial h_t}{\partial h_k} = \prod_{j=k+1}^t \tfrac{\partial h_j}{\partial h_{j-1}} = \prod_{j=k+1}^t W^Tdiag(f'(h_{j-1}))$$
$$||\tfrac{\partial h_j}{\partial h_{j-1}}|| <= ||W^T|| \cdot ||diag(f'(h_{j-1})|| <= \beta_W \beta_h$$
$$||\tfrac{\partial h_t}{\partial h_k}|| = || \prod_{j=k+1}^t\tfrac{\partial h_j}{\partial h_{j-1}}|| <= (\beta_W\beta_h)^{t-k}$$
可能非常快的就变得很大或者很小。

vanishing gradient problem 使得许多步之前的对当前训练的影响微乎其微

exploding gradient clip gradient

vanishing gradient -> Initialization + ReLus

softmax is huge and slow

• class based trick

双向 RNN

• 之前和之后的训练词对当前训练都有影响

深度双向 RNN

F1 度量

precision = tp/(tp+fp)
recall = tp/(tp+fn)
F1 = 2(precision recall)/(precsion + recall)