transformer笔记

来自李宏毅老师的课程笔记，http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML19.html

0.前置基础

RNN: 优点:可以获取全局信息 缺点:不能并行计算
CNN: 优点:GPU并行计算 缺点:bigram仅能观察到本地信息
self-attention层:
x1: input sequence item
a1: item embedding
q1: query(match others), ==Wq * a1
k1: key(to be matched by query), ==Wk * a1
v1: information(to be exacted), ==Wv *a1

1.self-attention

1.1 scaled dot-product attention
那每一个query，去对每一个key做attention
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-sS8AEWGv-1583759023810)(./DL_picture/transformer1.png)]
1.2 aij softmax
1.3 get bi
b1 = sum(a1i * vi),相当于item i包含了全局信息
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-eFAZzWCs-1583759023815)(./DL_picture/transformer2.png)]
1.4 矩阵运算
self_attention转化为Q,K,V之间的矩阵运算,便于并行计算
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ZkYC7I0p-1583759023825)(./DL_picture/transfor