20240604日志:Attention

# location:beijing

Attention[^1]

single head of attention

Attention blocks showed in Fig. 1 allow vectors to talk to each other and pass information back and forth to update their values.

Fig. 1.1 single head attention (red rectangle)

there is an example to understand the mechanism of Attention. Fig. 1. shows that vector $\overrightarrow{{\mathbf{E}}_n}$ is a combination of token’s meaning and positional information (M&P). then $\overrightarrow{{\mathbf{E}}_n}$ will be sent to Attention blocks.

Fig. 1.2 vector with information of meaning and position

the input of Attention $\overrightarrow{{\mathbf{E}}_n}$ will be figured by three linear layers: V, K, Q for feature shifting.
We can see Q as a method that adds attributes that can receive information of M&P from in-context words(vectors) to $\overrightarrow{{\mathbf{E}}_n}$. And the outcomes are ${\mathbf{Q}}_n$