Tokenization (BPE)
Live BPE Tokenizer
Transform raw text into a sequence of integer token IDs using Byte Pair Encoding (BPE) - the same algorithm used by GPT-2, GPT-3, and GPT-4.
Tokens:0
Bytes:87
Ratio:0.00×
Plain Language Intuition
Tokenization chops raw text into small pieces and gives each piece a number. Think of it like a dictionary where 'the' → 42, 'love' → 187. The model only sees these numbers, never the raw characters.
Production Real-World Context
GPT-4 uses a BPE tokenizer with ~100,000 tokens. Our bilingual Shakespeare + Hafez model uses 256 tokens. The algorithm is identical: more training data and merges produce higher compression for common words.
BPE Compression Ratio
Intuition: Measures how efficiently Byte-Pair Encoding packed raw text into subwords - a ratio of 3× means 3 raw bytes were compressed into 1 token on average.
Presets:
Interactive Token Stream
Click any token to inspect byte representationInteger Token ID Sequence (Input to Embedding Table):
[]
Token Inspector
Select any token in the stream to inspect details.
BPE Tokenizer - Python Reference Implementationpython
38 lines
Key Takeaways & Core Rules
- Text is converted to UTF-8 bytes, then BPE iteratively merges the most frequent pairs into subwords.
- Compression ratio tells how efficient the vocabulary is - higher ratio means fewer tokens per character.
- Vocabulary size V=256 is a key architectural hyperparameter bounding the embedding matrix row count.
- Downstream neural networks never see characters; they operate strictly on integer token IDs.
Try This Experiment:
Try typing Persian poetry or emoji. Notice how Persian UTF-8 characters are 2 bytes each, and frequently merged into Persian subword tokens.