Scaled Dot-Product Attention

machine-learning transformers attention

Definition

Scaled Dot-Product Attention

$$\mathrm{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left(\frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{d_k}}\right)\cdot \mathbf{V}$$

where:

• $\mathbf{Q}\in {\mathbb{R}}^{n\times d_q}$ is the query vector representing what the current token is searching for.
• $\mathbf{K}\in {\mathbb{R}}^{n\times d_q}$ is the key vector representing what each token contains.
• $\mathbf{V}\in {\mathbb{R}}^{n\times d_v}$ is the value vector representing the actual content to be aggregated or propagated.
• $\mathrm{softmax}$ is the softmax function.

Note that this is usually computed as a batch across all tokens int he sequence fr efficiency.

Shape Visualisation

Taken from ¹.

Footnotes

1. https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html ↩