Self-Attention

machine-learning transformers attention

Self-Attention

Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. ¹

It is a more specific form of cross-attention where $x_1=x_2$.

Visualisation

Taken from ².

Footnotes

1. Attention is All You Need ↩

2. https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html ↩