LLM Interactive Architecture
Architecture Roadmap64D Bi-LM
0
Architecture Map
1
Tokenization (BPE)
2
Token Embeddings
3
Positional Encoding
4
Self-Attention
5
Multi-Head Attention
6
Transformer Block
7
Full GPT Model
8
Training Loop
9
Backpropagation
10
Live Generation
11
Modern Architectures
RoPE
Model SpecBilingual
dim (d_model):64
Layers / Heads:4 / 4
Vocab Size:256

Autoregressive Generation

Autoregressive Loop

Sample next-token predictions step-by-step using temperature and nucleus sampling.

Strategy: nucleus
Plain Language Intuition

Generation is the ultimate goal - making the model write. We give it a starting prompt, it predicts one token, we append that token and predict again. Each prediction is a probability distribution over the vocabulary. We sample from this distribution using temperature and nucleus strategies.

Production Real-World Context

ChatGPT, Claude, and LLaMA all rely on nucleus (top-p) sampling with temperature. Low temperature (0.2) is ideal for code generation and factual queries; higher temperature (0.8) yields poetic, diverse prose.

Prerequisites:
Training Loop (Step 8)
Full Model (Step 7)

Temperature Sampling

Ackley et al. 1985; Fan et al. 2018
Intuition: Dividing logits by tau scales the entropy of the probability distribution before sampling - the single master dial controlling coherence versus creativity.

Top-p (Nucleus) Sampling

Holtzman et al. 2020, arXiv:1904.09751
Intuition: Dynamically truncates the tail: keeps only the smallest set of high-probability tokens that collectively cover p (e.g. 90%) of the mass.

Top-k Sampling

Fan et al. 2018, arXiv:1805.04833
Intuition: Hard truncation keeping strictly the k most probable tokens regardless of distribution sharpness.
Presets:
Temperature:τ = 0.80
Top-p Threshold:p = 0.90
Max New Tokens:24 tokens
Autoregressive Stream Output
Real-time Tokenization
Click "Run Autoregressive Generation" to witness next-token sampling step-by-step.
Generated by bilingual Shakespeare + Hafez 64D model.
Autoregressive Generation - PyTorch Reference Implementationpython
38 lines
Key Takeaways & Core Rules
  • Autoregressive loop: predict one token -> append -> predict next -> repeat.
  • Temperature scales logits prior to softmax: lower = deterministic, higher = creative diversity.
  • Nucleus (Top-p) dynamically adapts candidate set size based on model confidence.
  • Full end-to-end pipeline completed: text -> BPE -> embeddings -> attention -> blocks -> logits -> generation.
Try This Experiment:

Switch to Greedy mode and generate. Notice how greedy always produces the exact same deterministic continuation.