李宏毅 ML2021 学习笔记 - W5

1. Transformer

Sequence-to-Sequenct (Seq2seq): input a sequence，output a sequence，输出长度由机器决定。

e.g 语音识别，机器翻译，语音翻译

1. Encoder

1.1. 网络架构

每个block更为具体的架构：

1.2. to learn more

2. Decoder

2.1. 网络架构: Autoregressive

Masked Self-attention:

2.2. AT v.s. NAT

2.3. Cross-attention

 

2.4. Train

用ground truth做input来train

2.5. Train Tips

•  Copy Mechanism
• Guided Attention
• Beam Search
• Samlping