LLM Interactive Architecture
Architecture Roadmap64D Bi-LM
0
Architecture Map
1
Tokenization (BPE)
2
Token Embeddings
3
Positional Encoding
4
Self-Attention
5
Multi-Head Attention
6
Transformer Block
7
Full GPT Model
8
Training Loop
9
Backpropagation
10
Live Generation
11
Modern Architectures
RoPE
Model SpecBilingual
dim (d_model):64
Layers / Heads:4 / 4
Vocab Size:256

Token Embeddings (64D)

Turn words into lists of numbers: each word ID gets 64 numbers (a vector) where words with related meanings sit close together.

Dimensions:64
Vocabulary:256
Plain Language Intuition

Computers cannot read letters directly, so they use numbers. The embedding table acts like a catalog: each word ID looks up a row of 64 numbers that represents the meaning of that word, like how 'king' relates to 'queen'.

Production Real-World Context

In giant models like GPT-3, the embedding table holds over 50,000 words with 12,288 numbers each. In our bilingual lab, we use 256 words and 64 numbers so you can inspect every calculation clearly.

Prerequisites:
Tokenization (Step 1)

Token Embedding Lookup

Vaswani et al. 2017, arXiv:1706.03762 §3.4
Intuition: Looking up an embedding is like picking a row from a table: if the token ID is 10, the model simply takes row 10 from the matrix.
Quick Presets:
No matrix data available.

Semantic Space (2D PCA Projection)

Tokens with similar meanings naturally cluster close to each other.

PC1: 42.0%PC2: 28.0%
PC 1 →↑ PC 2
Total projected tokens: 0Click any token to inspect its embedding
Inspected Token:
ID: -
"-"Coordinates: (0.0000, 0.0000)

Notice that words from the same semantic category (like royalty or grammar) sit close to each other in this 64-dimensional space.

Embedding Matrix Balance & Health ⚖️

The numbers in this table are smoothly balanced around zero without extreme spikes or dead columns. This smooth spread proves that the model learned stable, healthy representations for both English and Persian words.

Token Embeddings - PyTorch Reference Implementationpython
24 lines
Key Takeaways & Core Rules
  • ID to Vector: Each word number selects its own row of 64 numbers from the table.
  • Learned Meaning: Training pulls words with similar roles or meanings closer together.
  • Updating Weights: The entire table of numbers is updated step-by-step during training.
  • 2D View: PCA flattens the 64 numbers into 2 dimensions so you can easily see word clusters.
Try This Experiment:

Try comparing 'king' and 'queen' with poetry words. Notice how words from the same topic cluster together in the 2D view.