Transformer小结

Attention is all you need

Transformer

LayerNorm(x + Sublayer(x))

整理的Transformer 伪代码
输入 Inputs 输出 Outputs

X = Positional_Encoding(Input_Embedding(Inputs))
X = LayerNorm(X + Multi-Head_Attention(X))
X = LayerNorm(X + Feed_Forward(X))

Y = Positional_Encoding(Output_Embedding(Outputs))
Y = LayerNorm(Y + Masked_Multi-Head_Attention(Y))
Y = LayerNorm(Y + Multi-Head_Attention($X_Q$,$X_K$,$Y_V$))
Y = LayerNorm(Y + Feed_Forward(Y))

Y = Linear(Y)
Output Probabilities = Softmax(Y)

Scaled Dot-Product Attention

Attention(Q, K, V) = softmax($\frac{Q{K}^T}{\sqrt{d_k}}$)V

Multi-Head Attention

MultiHead(Q,K,V) = Concat($hea{d}_1$,…,$hea{d}_h$)$W^O$

where $hea{d}_i$ = Attention($Q{W}_i^Q$, $K{W}_i^K$, $V{W}_i^V$)

$W_i^Q\in R^{d_{model}\ast d_k}$
$W_i^K\in R^{d_{model}\ast d_k}$
$W_i^V\in R^{d_{model}\ast d_v}$
$W_i^O\in R^{h{d}_v\ast d_{model}}$

In this work we employ h = 8 parallel attention layers, or heads.
For each of these we use $d_k$ = $d_v$ = $d_{model}$/h = 64
Due to the reduced dimention of each head, the total computational cost
is similar to that of single-head attention with full dimensionality.

Position-wise Feead-Forward Networks

FFN(x) = max(0, $x{W}_1$ + $b_1$)$W_2$ + $b_2$

Positional Encoding

$PE_{(pos, 2i)} = sin(pos/10000^{2i/d_{model}}) $

$PE_{(pos, 2i+1)} = cos(pos/10000^{2i/d_{model}}) $

where pos is the position and i is the dimension.

重新写Transformer的伪代码

输入 Inputs 输出 Outputs

X = Positional_Encoding(Input_Embedding(Inputs))

$Q_X$,$K_X$,$V_X$ = X

X = LayerNorm(X + Multi-Head_Attention($Q_X$, $K_X$, $V_X$))

X = LayerNorm(X + Feed_Forward(X))

$Q_X$,$K_X$,$V_X$ = X

Y = Positional_Encoding(Output_Embedding(Outputs))

$Q_Y$,$K_Y$,$V_Y$ = Y

Y = LayerNorm(Y + Masked_Multi-Head_Attention($Q_Y$,$K_Y$, $V_Y$))

$Q_Y$,$K_Y$,$V_Y$ = Y

Y = LayerNorm(Y + Multi-Head_Attention($Q_X$,$K_X$,$V_Y$))

Y = LayerNorm(Y + Feed_Forward(Y))

Y = Linear(Y)
Output Probabilities = Softmax(Y)

Hardware and Schedule

We trained our models on one machine with 8 NVIDIA P100 GPUs.
We trained the base models for a total of 100,000 steps or 12 hours.
The big models were trained for 300,000 steps(3.5days)

Optimizer

We used the Adam optimizer with with ${\beta }_1$ = 0.9, ${\beta }_2$ = 0.98 and $ϵ$= 10^{−9}$

Regularization

Residual Dropout
Label Smoothing