主要参考苏神blog,传送门,自己整理下加深记忆
绝对位置编码
绝对位置编码一般形式的带绝对位置编码的Attention形式如下:
{ q i = ( x i + p i ) W Q k i = ( x i + k i ) W K v i = ( x i + v i ) W V a i , j = s o f t m a x ( q i k j T ) o i = ∑ j a i , j v j \left\{ \begin{aligned} q_i & = &(x_i+p_i)W_Q \\ k_i & = & (x_i+k_i)W_K\\ v_i & = & (x_i+v_i)W_V\\ a_{i,j} & = & softmax(q_ik_{j}^{T})\\ o_i & = &\sum_ja_{i,j}v_j \end{aligned} \right. ⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧qikiviai,joi=====(xi+pi)WQ(xi+ki)WK(xi+vi)WVsoftmax(qikjT)j∑ai,jvj
其中softmax表示对 j j j那一维归一化,这里的向量都是指行向量。将 q i k j T q_ik_j^T qikjT展开可得:
q i k j T = ( x i + p i ) W Q W K T ( x j + p j ) T = ( x i W Q + p i W Q ) ( W K T x j T + W K T p j T ) q_ik_j^T=(x_i+p_i)W_QW_K^T(x_j+pj)^T=(x_iW_Q+p_iW_Q)(W_K^Tx_j^T+W_K^Tp_j^T) qikjT=(xi