Multi-Head Attention
Run 4 attention heads in parallel across 16D subspaces to capture distinct linguistic and semantic relationships.
A single attention head can only focus on one type of relationship at a time. Multiple heads let the model attend to different linguistic features simultaneously - one head tracks grammatical neighbors, another tracks subject-verb bindings, and another tracks poetic metaphors.
GPT-3 uses 96 attention heads (each 128D, totaling 12,288D). Head pruning experiments demonstrate that distinct heads specialize in syntax, rare vocabulary tokens, and long-range coreference.
Multi-Head Attention
Inspect Individual Attention Heads
4 Subspaces (16D each)Head 0: Syntactic Alignment (Adjacent tokens)
Attends heavily to the immediate previous word / grammatical neighbor.
- Multi-Head Attention pipeline: project -> parallel attention -> concatenate -> output projection W^O.
- Each head operates on a lower-dimensional slice: d_k = d_model / h = 64 / 4 = 16 dimensions.
- More heads enable diverse syntactic and semantic patterns, while keeping compute cost identical to single-head 64D attention.
- The output matrix W^O blends all 4 head vectors back into a single unified 64D token representation.
Click through Heads 0, 1, 2, and 3 above. Notice how Head 0 focuses on adjacent words while Head 1 binds long-range poetic subject-rhymes.