LLM Interactive Architecture
Architecture Roadmap64D Bi-LM
0
Architecture Map
1
Tokenization (BPE)
2
Token Embeddings
3
Positional Encoding
4
Self-Attention
5
Multi-Head Attention
6
Transformer Block
7
Full GPT Model
8
Training Loop
9
Backpropagation
10
Live Generation
11
Modern Architectures
RoPE
Model SpecBilingual
dim (d_model):64
Layers / Heads:4 / 4
Vocab Size:256

Positional Encoding

Inject word order and sequence position into embeddings using harmonic sinusoidal functions and modern rotary embeddings (RoPE).

Sequence Length:32 tokens
Dimensions:64
Plain Language Intuition

Transformers process all words in a sentence at once in parallel (unlike humans or older models that read one word after another). Without position signals, 'dog bites man' and 'man bites dog' would look completely identical to the model! Positional encoding gives each word a unique seat number so word order is preserved.

Production Real-World Context

The original 2017 Transformer paper introduced these sinusoidal waves because they require zero training. Modern frontier LLMs (such as LLaMA 3 and Mistral) use RoPE (Rotary Position Embeddings), which encode relative distances by rotating vector spaces directly in attention.

Prerequisites:
Token Embeddings (Step 2)

Sinusoidal Positional Encoding Formula

Vaswani et al. 2017, arXiv:1706.03762 §3.5
Intuition: This operates like the hands of a clock: the second hand ticks rapidly, the minute hand moves moderately, and the hour hand creeps slowly. Lower dimensions oscillate quickly while higher dimensions change gently, giving every position a unique geometric fingerprint.
Sequence Length (Number of words):32 tokens
Model Dimension (d_model):64
Inspected Word Position:pos = 0
No matrix data available.
No matrix data available.
Wavelength Pattern at Positionpos = 0
64-dimensional values for the selected token position
Fast freq (Dim 0)Slow freq (Dim 63)

Notice the smooth transition from rapid oscillations at low dimensions to slow, continuous harmonic curves at high dimensions.

Input Vector Sum (Embedding + Position)

Vaswani et al. 2017, arXiv:1706.03762 §3.4
Intuition: The position vector is directly added element-wise to the token's semantic embedding vector. This ensures the token retains its meaning while knowing exactly where it sits in the sentence.
Positional Encoding - PyTorch Reference Implementationpython
26 lines
Key Takeaways & Core Rules
  • Parallel processing: Transformers ingest all tokens simultaneously and require explicit position markers.
  • Geometric fingerprint: Sine and cosine waves across diverse frequencies generate a unique signature for each index.
  • Distance awareness: Neighboring positions have high cosine similarity, allowing attention to gauge distance.
  • Element-wise sum: The input to the first layer is simply word_embedding + position_encoding.
Try This Experiment:

Drag the position slider and observe how the bar pattern shifts smoothly across dimensions.