Attention mechanism

1. Attention with RNN

Attention-based encoder-decoder sequence model architecture (A is RNN or LSTM or GRU）

decoder’s inputs一般用上一时间步预测出的结果。
下一时间步的hidden state s用上一时间步的c，s和这一时间步的new input更新：s_t = func(s_t-1, c_t-1, x’_t)


attention参数包含W_Q, W_K, W_V.

softmax(K^Tq): 可以衡量k1, k2, …, km分别与qj的相似度/匹配程度，值域[0, 1].

attention本质是对value进行加权求和。value表示对应word的信息。权重alpha表示信息的重要性，权重越大越attention于对应的value上。

终极总结图：

我们上面采取的是第一种方式：use dot product to measure similarity

2. Self-attention with RNN

下一时间步的hidden state h用上一时间步的c，h和这一时间步的new input更新：h_t = func(h_t-1, c_t-1, x’_t). 也可以只用c和x’: h_t = func(c_t-1, x’_t)

h0是全零向量，所以c1 = h1

repeat the process until 读完所有input x:
读取x_t, h_t-1, c_t-1，更新h_t
算h_t和所有h_1, …, h_t的align程度, 记作alpha
c = alpha和h对应相乘




repeat the process …

3. Attention without RNN

可用于替代RNN models for seq2seq(many-to-many) tasks e.g. machine translation。

x：input
x’：also input，but use predicted results during inference
每个x‘都对应一个c

4. Self-attention without RNN

可用于替代RNN for any tasks，不局限于seq2seq。

两个input sequence都是x
每个x对应一个c

Reference:
[1] https://www.bilibili.com/video/BV1YK4y1x7PC
[2] https://www.bilibili.com/video/BV1Xf4y127fz
[3] https://www.bilibili.com/video/BV12f4y12777
[4] https://www.bilibili.com/video/BV1E54y1B7hk