LLM Interactive Architecture
Architecture Roadmap64D Bi-LM
0
Architecture Map
1
Tokenization (BPE)
2
Token Embeddings
3
Positional Encoding
4
Self-Attention
5
Multi-Head Attention
6
Transformer Block
7
Full GPT Model
8
Training Loop
9
Backpropagation
10
Live Generation
11
Modern Architectures
RoPE
Model SpecBilingual
dim (d_model):64
Layers / Heads:4 / 4
Vocab Size:256

Full Model & Output Logits

Tied Projection (Wu = We)

Project final 64D hidden states into vocabulary logits via tied projection matrix W_u = W_e, followed by Softmax.

Output Dim: V = 256
Tied Params: 16,384
Plain Language Intuition

Now we connect the whole pipeline: token embeddings + position embeddings → 4 stacked transformer blocks → final LayerNorm → linear projection head. The output is an unnormalized score (logit) for each word in the vocabulary. Softmax turns these into probabilities.

Production Real-World Context

Our bilingual model has ~185K parameters with 4 layers and d_model=64. GPT-2 Small has 117M (same architecture, 12 layers, 768D). GPT-3 has 175B. The architectural pipeline is identical - only width and depth scale.

Prerequisites:
Transformer Block (Step 6)
Tokenization (Step 1)

GPT Forward Pass

Radford et al. 2019 (GPT-2)
Intuition: Every GPT model uses this exact pipeline: embed tokens and positions, pass through N transformer blocks, normalize, and project to next-token probabilities.

Model Specifications (Shakespeare + Hafez Bilingual LM)

Vocabulary SizeV = 256
Hidden Dimensiond_model = 64
Layers / BlocksL = 4
Attention Headsh = 4 (16D/head)
Temperature Scaling (Ï„): 0.80
Balanced

Next-Token Probability Distribution (Prompt: "To be or not to...")

"be" (ID 45)
logit: 6.8094.2%
"die" (ID 89)
logit: 4.203.7%
"live" (ID 112)
logit: 3.501.5%
"speak" (ID 140)
logit: 2.100.3%
"think" (ID 167)
logit: 1.800.2%
"grieve" (ID 201)
logit: 1.200.1%
"sleep" (ID 215)
logit: 0.900.1%
Full GPT Architecture - PyTorch Reference Implementationpython
40 lines
Key Takeaways & Core Rules
  • The full GPT pipeline: Embed -> Position -> 4x Block -> Final LayerNorm -> Output Linear Head.
  • Weight tying shares the 256x64 matrix between token embedding and the output projection head.
  • Temperature tau scales logits prior to softmax: tau -> 0 chooses top token deterministically, tau -> inf yields uniform random guessing.
  • The output logits array represents unnormalized energy scores across all 256 vocabulary subwords.
Try This Experiment:

Slide temperature down to 0.1 and notice how probability concentrates on 'be' (98%+), then slide up to 1.8 to see entropy flatten.