输入向量中的 α u t i \alpha u_{t}^{i} αuti 部分代表的是将标量上下文投射到特征维度 dmodel上的向量,乘以一个平衡因子 α \alpha α
详细解释
数学表示
具体例子
-
标量上下文投射:
对于时间步 (t = 2) 的第一个元素 (x6 = 0.5),通过一维卷积(假设卷积核和偏置如下):
W = [ 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 ] , b = [ 0.1 0.2 0.3 0.4 ] W = \begin{bmatrix} 0.1 & 0.2 & 0.3 \\ 0.4 & 0.5 & 0.6 \\ 0.7 & 0.8 & 0.9 \\ 1.0 & 1.1 & 1.2 \\ \end{bmatrix}, \quad b = \begin{bmatrix} 0.1 \\ 0.2 \\ 0.3 \\ 0.4 \\ \end{bmatrix} W= 0.10.40.71.00.20.50.81.10.30.60.91.2 ,b= 0.10.20.30.4
计算投射向量 (u6):
u 6 = W ⋅ [ x 5 x 6 x 7 ] + b u_6 = W \cdot \begin{bmatrix} x_5 \\ x_6 \\ x_7 \end{bmatrix} + b u6=W⋅ x5x6x7 +b
假设 (x5 = -0.5) 和 (x7 = 1.0),那么:
u 6 = [ 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 ] ⋅ [ − 0.5 0.5 1.0 ] + [ 0.1 0.2 0.3 0.4 ] u_6 = \begin{bmatrix} 0.1 & 0.2 & 0.3 \\ 0.4 & 0.5 & 0.6 \\ 0.7 & 0.8 & 0.9 \\ 1.0 & 1.1 & 1.2 \\ \end{bmatrix} \cdot \begin{bmatrix} -0.5 \\ 0.5 \\ 1.0 \end{bmatrix} + \begin{bmatrix} 0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{bmatrix} u6= 0.10.40.71.00.20.50.81.10.30.60.91.2 ⋅ −0.50.51.0 + 0.10.20.30.4 计算结果:
u 6 = [ 0.1 ⋅ − 0.5 + 0.2 ⋅ 0.5 + 0.3 ⋅ 1.0 0.4 ⋅ − 0.5 + 0.5 ⋅ 0.5 + 0.6 ⋅ 1.0 0.7 ⋅ − 0.5 + 0.8 ⋅ 0.5 + 0.9 ⋅ 1.0 1.0 ⋅ − 0.5 + 1.1 ⋅ 0.5 + 1.2 ⋅ 1.0 ] + [ 0.1 0.2 0.3 0.4 ] u_6 = \begin{bmatrix} 0.1 \cdot -0.5 + 0.2 \cdot 0.5 + 0.3 \cdot 1.0 \\ 0.4 \cdot -0.5 + 0.5 \cdot 0.5 + 0.6 \cdot 1.0 \\ 0.7 \cdot -0.5 + 0.8 \cdot 0.5 + 0.9 \cdot 1.0 \\ 1.0 \cdot -0.5 + 1.1 \cdot 0.5 + 1.2 \cdot 1.0 \\ \end{bmatrix} + \begin{bmatrix} 0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{bmatrix} u6= 0.1⋅−0.5+0.2⋅0.5+0.3⋅1.00.4⋅−0.5+0.5⋅0.5+0.6⋅1.00.7⋅−0.5+0.8⋅0.5+0.9⋅1.01.0⋅−0.5+1.1⋅0.5+1.2⋅1.0 + 0.10.20.30.4 进一步计算:
u 6 = [ − 0.05 + 0.1 + 0.3 − 0.2 + 0.25 + 0.6 − 0.35 + 0.4 + 0.9 − 0.5 + 0.55 + 1.2 ] + [ 0.1 0.2 0.3 0.4 ] u_6 = \begin{bmatrix} -0.05 + 0.1 + 0.3 \\ -0.2 + 0.25 + 0.6 \\ -0.35 + 0.4 + 0.9 \\ -0.5 + 0.55 + 1.2 \\ \end{bmatrix} + \begin{bmatrix} 0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{bmatrix} u6= −0.05+0.1+0.3−0.2+0.25+0.6−0.35+0.4+0.9−0.5+0.55+1.2 + 0.10.20.30.4
u 6 = [ 0.35 0.85 1.25 1.65 ] + [ 0.1 0.2 0.3 0.4 ] u_6 = \begin{bmatrix} 0.35 \\ 0.85 \\ 1.25 \\ 1.65 \\ \end{bmatrix} + \begin{bmatrix} 0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{bmatrix} u6= 0.350.851.251.65 + 0.10.20.30.4
u 6 = [ 0.45 1.05 1.55 2.05 ] u_6 = \begin{bmatrix} 0.45 \\ 1.05 \\ 1.55 \\ 2.05 \\ \end{bmatrix} u6= 0.451.051.552.05 -
乘以平衡因子 α \alpha α:
α u 6 = 1 ⋅ [ 0.45 1.05 1.55 2.05 ] = [ 0.45 1.05 1.55 2.05 ] \alpha u_6 = 1 \cdot \begin{bmatrix} 0.45 \\ 1.05 \\ 1.55 \\ 2.05 \\ \end{bmatrix} = \begin{bmatrix} 0.45 \\ 1.05 \\ 1.55 \\ 2.05 \\ \end{bmatrix} αu6=1⋅ 0.451.051.552.05 = 0.451.051.552.05