LLM Interactive Architecture
Architecture Roadmap64D Bi-LM
0
Architecture Map
1
Tokenization (BPE)
2
Token Embeddings
3
Positional Encoding
4
Self-Attention
5
Multi-Head Attention
6
Transformer Block
7
Full GPT Model
8
Training Loop
9
Backpropagation
10
Live Generation
11
Modern Architectures
RoPE
Model SpecBilingual
dim (d_model):64
Layers / Heads:4 / 4
Vocab Size:256

Self-Attention

Scaled Dot-Product & Causal Mask

Compute contextual token representations through scaled dot-product attention between Q and K matrices with causal masking.

Tokens: 5
Head Dim: 16 (sqrt=4)
Plain Language Intuition

Self-attention lets each word 'ask a question' to every other word in the sentence: 'How relevant are you to me?'. The answer determines how much each word influences the others. For example, in 'The cat sat on the mat because it was tired', when computing the meaning of 'sat', attention focuses heavily on 'cat' (the actor) and 'mat' (the location), dynamically connecting context.

Production Real-World Context

This is the exact mechanism that powered the ChatGPT revolution. In autoregressive decoders (like GPT), causal masking prevents the model from looking ahead at future tokens during training, training it to reliably predict the next token one step at a time.

Prerequisites:
Positional Encoding (Step 3)
Embeddings (Step 2)

Scaled Dot-Product Attention Formula

Vaswani et al. 2017, arXiv:1706.03762 §3.2.1
Intuition: Each token projects a Query to search across all Keys, scales by sqrt(d_k) to stabilize gradients, applies softmax to get a probability distribution, and aggregates a weighted sum of Values.
5 Tokens:

Attention Weight Matrix (A_ij)

Rows: Query tokens (attending from) | Columns: Key tokens (attending to)

Loading visualizer...
Interactive Cell Inspection
Hover over any cell in the attention matrix to inspect query-key interactions.
Scaled Dot-Product Attention - PyTorch Reference Implementationpython
26 lines
Key Takeaways & Core Rules
  • Attention pipeline: Q * K^T -> scale by sqrt(d_k) -> causal mask -> softmax -> multiply by V.
  • Causal masking forces decoders (GPT) to look backward only; removing it creates bidirectional encoders (BERT).
  • Scaling by sqrt(d_k) = 4 prevents gradient vanishing caused by softmax saturation on large dot products.
  • Softmax normalizes attention weights so each query distributes exactly 100% of its representation budget.
Try This Experiment:

Toggle the causal mask on and off. Notice how the upper triangle flips from 0% (masked future) to active bidirectional attention weights.