Rotary Position Embedding(RoPE)原理推导和详解

RoPE算法背景

在《Attention is all your need》 中使用了absolute position embedding，且在论文中验证了嵌入位置向量是有效的，所以位置向量的概念渐渐的引入人们的视角里。目前主要的向量编码有俩种，分别是绝对位置编码和相对位置编码。其中绝对位置编码的优点就是直观，可以将token的位置信息直接嵌入，但缺点也比较明显的，长度固定，且在计算时不好考虑token相对位置之间的关系。同理，相对位置编码的优势就是能够兼容变长序列且能够考虑到不同token相对位置的关系，很好地解决了绝对位置的问题，但它的缺点就是计算资源需求变大。Rotary Position Embedding同时考虑了相对位置和绝对位置优势，提出了一种即考虑到相对位置又能包含绝对位置的位置编码方式。

RoPE算法原理

寻找相对位置关系

在做attention计算中，$Q^{T}V$中会涉及多个token之间的内积计算，那么对元素$x_m$和元素$x_n$之间做内积时如何得到考虑到$m-n$这个相对位置的关系。作者将这个问题归纳为如下公式
$$<f_q(x_m,m),f_k(x_n,n)>=g(x_m,x_n,m-n)$$
这里作者先讨论了$x$为二维向量的情况，将二维向量的几何性质和复数的计算考虑进去。给定俩个二维向量$x_q$和$x_k$，所以有
$$\begin{array}{rl}{f}_q(x_q,m)& =R_q(x_q,m)e^{i{\Theta }_q(x_q,m)}\\ f_k(x_k,n)& =R_k(x_k,n)e^{i{\Theta }_k(x_k,n)}\\ g(x_q,x_k,m-n)& =Re(l(x_q,x_k,m-n))=Re(R_g(x_q,x_k,m-n)e^{i{\Theta }_g(x_q,x_k,m-n)})\end{array}$$根据二维向量的性质有
$$R_q(x_q,m)=\left(\begin{array}{c}{q}_m^1\\ q_m^2\end{array}\right)=q_m^1+i{q}_m^2$$根据欧拉公式有
$$e^{i{\Theta }_q(x_q,m)}=\cos ({\Theta }_q(x_q,m))+i\sin ({\Theta }_q(x_q,m))$$然后有$$\begin{array}{rl}{f}_q(x_q,m)& =(q_m^1+i{q}_m^2)(\cos ({\Theta }_q(x_q,m))+i\sin ({\Theta }_q(x_q,m)))\\ & =q_m^1\cos ({\Theta }_q(x_q,m))-q_m^2\sin ({\Theta }_q(x_q,m))+i(q_m^1\sin ({\Theta }_q(x_q,m))+q_m^2\cos ({\Theta }_q(x_q,m)))\\ & =\left(\begin{array}{c}{q}_m^1\cos ({\Theta }_q(x_q,m))-q_m^2\sin ({\Theta }_q(x_q,m))\\ q_m^1\sin ({\Theta }_q(x_q,m))+q_m^2\cos ({\Theta }_q(x_q,m))\end{array}\right)\\ & =\left(\begin{array}{cc}\cos ({\Theta }_q(x_q,m))& -\sin ({\Theta }_q(x_q,m))\\ \sin ({\Theta }_q(x_q,m))& \cos ({\Theta }_q(x_q,m))\end{array}\right)\left(\begin{array}{c}{q}_m^1\\ q_m^2\end{array}\right)\end{array}$$同理$$f_k(x_k,n)=\left(\begin{array}{cc}\cos ({\Theta }_k(x_k,n))& -\sin ({\Theta }_k(x_k,n))\\ \sin ({\Theta }_k(x_k,m))& \cos ({\Theta }_k(x_k,m))\end{array}\right)\left(\begin{array}{c}{k}_n^1\\ k_n^2\end{array}\right)$$然后可以得出 < f q ( x q , m ) , f k ( x k , n ) > = ( ( cos ⁡ ( Θ q ( x q , m ) ) − sin ⁡ ( Θ q ( x q , m ) ) sin ⁡ ( Θ q ( x q , m ) ) cos ⁡ ( Θ q ( x q , m ) ) ) ( q m 1 q m 2 ) ) T ( cos ⁡ ( Θ k ( x k , n ) ) − sin ⁡ ( Θ k ( x k , n ) ) sin ⁡ ( Θ k ( x k , m ) ) cos ⁡ ( Θ k ( x k , m ) ) ) ( k n 1 k n 2 ) = ( q m 1 q m 2 ) ( cos ⁡ ( Θ q ( x q , m ) ) sin ⁡ ( Θ q ( x q , m ) ) − sin ⁡ ( Θ q ( x q , m ) ) cos ⁡ ( Θ q ( x q , m ) ) ) ( cos ⁡ ( Θ k ( x k , n ) ) − sin ⁡ ( Θ k ( x k , n ) ) sin ⁡ ( Θ k ( x k , n ) ) cos ⁡ ( Θ k ( x k , n ) ) ) ( k n 1 k n 2 ) = ( q m 1 q m 2 ) ( cos ⁡ ( Θ q ( x q , m ) ) cos ⁡ ( Θ k ( x k , n ) ) + sin ⁡ ( Θ q ( x q , m ) ) sin ⁡ ( Θ k ( x k , n ) ) sin ⁡ ( Θ q ( x q , m ) ) cos ⁡ ( Θ k ( x k , n ) ) − sin ⁡ ( Θ k ( x k , n ) ) cos ⁡ ( Θ q ( x q , m ) ) sin ⁡ ( Θ k ( x k , n ) ) cos ⁡ ( Θ q ( x q , m ) ) − sin ⁡ ( Θ q ( x q , m ) ) cos ⁡ ( Θ k ( x k , n ) ) cos ⁡ ( Θ q ( x q , m ) ) cos ⁡ ( Θ k ( x k , n ) ) + sin ⁡ ( Θ q ( x q , m ) ) sin ⁡ ( Θ k ( x k , n ) ) ) ( k n 1 k n 2 ) = ( q m 1 q m 2 ) ( cos ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) − sin ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) sin ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) cos ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) ) ( k n 1 k n 2 ) = ( q m 1 ( cos ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) ) + q m 2 ( sin ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) ) − q m 1 ( sin ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) ) + q m 2 ( cos ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) ) ) ( k n 1 k n 2 ) = ( q m 1 k n 1 + q m 2 k m 2 ) cos ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) + ( q m 2 k n 1 − q m 1 k n 2 ) sin ⁡ ( Θ k ( x k , n ) − Θ q ( x q , m ) ) \begin{align*}<f_q(x_q,m) , f_k(x_k,n)>& = \left(\left (\begin{matrix}\cos(\Theta_q(x_q,m)) &-\sin(\Theta_q(x_q,m)) \\ \sin(\Theta_q(x_q,m)) & \cos(\Theta_q(x_q,m)) \end{matrix} \right)\left (\begin{matrix}q_m^1\\ q_m^2\end{matrix} \right)\right)^{T}\left (\begin{matrix}\cos(\Theta_k(x_k,n)) &-\sin(\Theta_k(x_k,n)) \\ \sin(\Theta_k(x_k,m)) & \cos(\Theta_k(x_k,m)) \end{matrix} \right)\left (\begin{matrix}k_n^1\\ k_n^2\end{matrix} \right) \\ & =\left(\begin{matrix} q_m^1 & q_m^2 \end{matrix}\right)\left(\begin{matrix}\cos(\Theta_q(x_q,m)) & \sin(\Theta_q(x_q,m)) \\ -\sin(\Theta_q(x_q,m)) & \cos(\Theta_q(x_q,m)) \end{matrix}\right)\left (\begin{matrix}\cos(\Theta_k(x_k,n)) &-\sin(\Theta_k(x_k,n)) \\ \sin(\Theta_k(x_k,n)) & \cos(\Theta_k(x_k,n)) \end{matrix} \right)\left (\begin{matrix}k_n^1\\ k_n^2\end{matrix} \right) \\ &=\left(\begin{matrix} q_m^1 & q_m^2 \end{matrix}\right)\left (\begin{matrix}\cos(\Theta_q(x_q,m))\cos(\Theta_k(x_k,n))+\sin(\Theta_q(x_q,m))\sin(\Theta_k(x_k,n)) & \sin(\Theta_q(x_q,m))\cos(\Theta_k(x_k,n))-\sin(\Theta_k(x_k,n))\cos(\Theta_q(x_q,m))\\ \sin(\Theta_k(x_k,n)) \cos(\Theta_q(x_q,m)) -\sin(\Theta_q(x_q,m))\cos(\Theta_k(x_k,n)) & \cos(\Theta_q(x_q,m))\cos(\Theta_k(x_k,n))+\sin(\Theta_q(x_q,m))\sin(\Theta_k(x_k,n)) \end{matrix} \right) \left (\begin{matrix}k_n^1\\ k_n^2 \end{matrix} \right) \\ & = \left(\begin{matrix} q_m^1 & q_m^2 \end{matrix}\right) \left (\begin{matrix} \cos(\Theta_k(x_k,n)-\Theta_q(x_q,m)) &-\sin(\Theta_k(x_k,n)-\Theta_q(x_q,m)) \\ \sin(\Theta_k(x_k,n)-\Theta_q(x_q,m)) &\cos(\Theta_k(x_k,n)-\Theta_q(x_q,m)) \end{matrix}\right)\left (\begin{matrix}k_n^1\\ k_n^2 \end{matrix} \right)\\ &= \left(\begin{matrix} q_m^1(\cos(\Theta_k(x_k,n)-\Theta_q(x_q,m)))+q_m^2(\sin(\Theta_k(x_k,n)-\Theta_q(x_q,m))) & -q_m^1(\sin(\Theta_k(x_k,n)-\Theta_q(x_q,m)))+q_m^2(\cos(\Theta_k(x_k,n)-\Theta_q(x_q,m)) )\end{matrix} \right)\left (\begin{matrix}k_n^1\\ k_n^2 \end{matrix} \right) \\ & = (q_m^1k_n^1+q_m^2k_m^2)\cos(\Theta_k(x_k,n)-\Theta_q(x_q,m))+(q_m^2k_n^1-q_m^1k_n^2)\sin(\Theta_k(x_k,n)-\Theta_q(x_q,m)) \end{align*} <fq​(xq​,m),fk​(xk​,n)>​=((cos(Θq​(xq​,m))sin(Θq​(xq​,m))​−sin(Θq​(xq​,m))cos(Θq​(xq​,m))​)(qm1​qm2​​))T(cos(Θk​(xk​,n))sin(Θk​(xk​,m))​−sin(Θk​(xk​,n))cos(Θk​(xk​,m))​)(k