A t t n i j b h × V a l u e s j d b h = Z i d b h Attn_{ijbh} \times Values_{jdbh} = Z_{idbh} Attnijbh×Valuesjdbh=Zidbh (爱因斯坦标记法)
这里的 i , d , b , h i,d,b,h i,d,b,h于‘->’左右都出现了,被称作自由标,而 j j j 只在左边出现,被称为哑标,计算结果则是保留自由标,对哑标进行逐元素相乘并求和
展开为求和公式
Σ j A t t n i j b h × V a l u e s j d b h = Z i d b h \Sigma_{j} Attn_{ijbh} \times Values_{jdbh} = Z_{idbh} ΣjAttnijbh×Valuesjdbh=Zidbh
例如,对于单个head和单个batch的Attention Matrix,可以表示如下:
[ A t t n 11 b h A t t n 12 b h ⋯ A t t n 1 j b h A t t n 21 b h A t t n 22 b h ⋯ A t t n 2 j b h ⋮ ⋮ ⋱ ⋮ A t t n i 1 b h A t t n i 2 b h ⋯ A t t n i j b h ] \begin{bmatrix} Attn_{11bh} & Attn_{12bh} & \cdots & Attn_{1jbh} \\ Attn_{21bh} & Attn_{22bh} & \cdots & Attn_{2jbh} \\ \vdots & \vdots & \ddots & \vdots \\ Attn_{i1bh} & Attn_{i2bh} & \cdots & Attn_{ijbh} \\ \end{bmatrix} Attn11bhAttn21bh⋮Attni1bhAttn12bhAttn22bh⋮Attni2bh⋯⋯⋱⋯Attn1jbhAttn2jbh⋮Attnijbh
相应的Value Matrix表示为
[ V a l u e s 1 d b h V a l u e s 2 d b h ⋯ V a l u e s j d b h ] \begin{bmatrix} Values_{1dbh} \\ Values_{2dbh} \\ \cdots \\ Values_{jdbh} \\ \end{bmatrix} Values1dbhValues2dbh⋯Valuesjdbh
我们可以得到Z
[ Z 11 b h Z 12 b h ⋯ Z 1 d b h ⋮ ⋮ ⋱ ⋮ Z i 1 b h Z i 2 b h ⋯ Z i d b h ] = [ A t t n 11 b h A t t n 12 b h ⋯ A t t n 1 j b h A t t n 21 b h A t t n 22 b h ⋯ A t t n 2 j b h ⋮ ⋮ ⋱ ⋮ A t t n i 1 b h A t t n i 2 b h ⋯ A t t n i j b h ] ⋅ [ V a l u e s 1 d b h V a l u e s 2 d b h ⋯ V a l u e s j d b h ] \begin{bmatrix} Z_{11bh} & Z_{12bh} & \cdots & Z_{1dbh} \\ \vdots & \vdots & \ddots & \vdots \\ Z_{i1bh} & Z_{i2bh} & \cdots & Z_{idbh} \\ \end{bmatrix} = \begin{bmatrix} Attn_{11bh} & Attn_{12bh} & \cdots & Attn_{1jbh} \\ Attn_{21bh} & Attn_{22bh} & \cdots & Attn_{2jbh} \\ \vdots & \vdots & \ddots & \vdots \\ Attn_{i1bh} & Attn_{i2bh} & \cdots & Attn_{ijbh} \\ \end{bmatrix} \cdot \begin{bmatrix} Values_{1dbh} \\ Values_{2dbh} \\ \cdots \\ Values_{jdbh} \\ \end{bmatrix} Z11bh⋮Zi1bhZ12bh⋮Zi2bh⋯⋱⋯Z1dbh⋮Zidbh = Attn11bhAttn21bh⋮Attni1bhAttn12bhAttn22bh⋮Attni2bh⋯⋯⋱⋯Attn1jbhAttn2jbh⋮Attnijbh ⋅ Values1dbhValues2dbh⋯Valuesjdbh
其中
Z i d b h ∣ i = 1 , d = 1 = [ A t t n i j b h ∣ i = 1 , j = 1 A t t n i j b h ∣ i = 1 , j = 2 ⋯ A t t n 1 j b h ] ⋅ [ V a l u e s j d b h ∣ j = 1 V a l u e s j d b h ∣ j = 2 ⋯ V a l u e s j d b h ] Z_{idbh} \vert _{i=1,d=1} = \begin{bmatrix} Attn_{ijbh} \vert _{i=1,j=1} & Attn_{ijbh} \vert _{i=1,j=2} & \cdots & Attn_{1jbh} \end{bmatrix} \cdot \begin{bmatrix} Values_{jdbh}\vert _{j=1} \\ Values_{jdbh}\vert _{j=2} \\ \cdots \\ Values_{jdbh} \\ \end{bmatrix} Zidbh∣i=1,d=1=[Attnijbh∣i=1,j=1Attnijbh∣i=1,j=2⋯Attn1jbh]⋅ Valuesjdbh∣j=1Valuesjdbh∣j=2⋯Valuesjdbh
从中我们可以看出所谓的对哑标 j j j 进行逐元素相乘并求和。