Loss & Training Dynamics

Cross-Entropy & Perplexity

Track Cross-Entropy loss convergence, perplexity, and gradient descent optimization across Shakespeare and Hafez bilingual corpora.

Initial Loss: ln(256) ≈ 5.545

Converged: 1.82

Plain Language Intuition

Training is how the model learns. We feed it millions of text examples and say 'predict the next token.' When it makes an error, backpropagation calculates gradients and nudges all parameters in the direction that lowers error. The loss curve dropping means the model is acquiring grammar and vocabulary.

Production Real-World Context

GPT-3 was trained on 300 billion tokens costing ~$4.6M in compute. Our bilingual model trains on Shakespeare + Hafez in seconds. The training loop is identical: forward pass, cross-entropy loss, backward pass, AdamW step, and cosine annealing schedule.

Prerequisites:

Full Model (Step 7)

Cross-Entropy Loss & Perplexity

Standard cross-entropy for language modeling

Intuition: Cross-entropy penalizes when the actual ground truth token is assigned low probability. Perplexity (e^loss) represents how many tokens the model is undecided between on average.

Inspect Training Epoch: 25 / 50

Loss: 2.118Perplexity: 8.3

Cross-Entropy Loss Convergence (Shakespeare + Hafez Bilingual Training)

AdamW Optimizer (lr=3e-4)

Training Loop - PyTorch Reference Implementationpython

32 lines

Key Takeaways & Core Rules

Training loop: forward pass -> compute cross-entropy -> backward pass -> AdamW step.
Initial uniform random loss starts at exactly ln(V) = ln(256) = 5.545.
Perplexity exp(loss) drops from 256 (blind guessing) down to ~6.17 (high confidence).
Learning rate schedule with warmup and cosine decay guarantees stable convergence without diverging.

Try This Experiment:

Scrub the epoch slider from 1 to 50 and observe the non-linear drop as the model memorizes frequent bi-grams and poetic rhymes.