LLM Interactive Architecture
Architecture Roadmap64D Bi-LM
0
Architecture Map
1
Tokenization (BPE)
2
Token Embeddings
3
Positional Encoding
4
Self-Attention
5
Multi-Head Attention
6
Transformer Block
7
Full GPT Model
8
Training Loop
9
Backpropagation
10
Live Generation
11
Modern Architectures
RoPE
Model SpecBilingual
dim (d_model):64
Layers / Heads:4 / 4
Vocab Size:256

Modern LLM Architectures

Step 11

Explore the 6 production upgrades powering today's frontier models: RoPE, RMSNorm, GQA, MLA, SwiGLU, and MoE.

Era: 2023โ€“2026 Production LLMs
Plain Language Intuition

The vanilla transformer (2017) was a breakthrough, but today's production LLMs are built on 6 evolved components. Each upgrade targets a specific bottleneck: faster normalization, better position encoding, smaller KV caches, more expressive FFNs, and sparse computation. Together they let a model like DeepSeek-V3 match GPT-4 performance while activating only 37B of its 671B parameters per token.

Production Real-World Context

Every open-source frontier model released since 2023 uses RMSNorm + RoPE + SwiGLU as a baseline. GQA and MLA solve the KV cache bottleneck that limits how long a context window you can serve. MoE (Mixture of Experts) lets you scale total parameter count without proportionally scaling compute.

Prerequisites:
Transformer Block (Step 6)
Multi-Head Attention (Step 5)

Which Models Use Which Techniques?

Every formula below is validated from the original research paper. Here is how they map to production models today.

7 Frontier Architectures
ModelRMSNormRoPEGQAMLASwiGLUMoE
LLaMA 3 (Meta)โœ“โœ“โœ“-โœ“-
DeepSeek-V3โœ“โœ“-โœ“โœ“โœ“
Mistral 7Bโœ“โœ“โœ“-โœ“-
Mixtral 8ร—7Bโœ“โœ“--โœ“โœ“
Qwen 2.5โœ“โœ“โœ“-โœ“-
Gemma 2 (Google)โœ“โœ“โœ“-โœ“-
Claude 3 (Anthropic)โœ“โœ“โœ“-โœ“-

Formulas follow the forward-pass order through a modern transformer layer: normalize โ†’ position-encode queries/keys โ†’ attention โ†’ FFN.

1

RMSNorm - Root Mean Square Normalization

Applied before every attention and FFN sub-layer (Pre-Norm). Replaces LayerNorm in nearly all modern LLMs by dropping the mean-centering step - achieving 7โ€“64% speedup with no quality loss.

7โ€“64% Faster

RMSNorm Formulation

Zhang & Sennrich 2019, arXiv:1910.07467
Intuition: Rescale by the root-mean-square - no mean subtraction needed. Simpler, faster, just as stable.
Used by:LLaMA 2/3DeepSeek-V2/V3MistralMixtralQwen 2.5Gemma 2Falcon
RMSNorm - PyTorch (from LLaMA)python
26 lines
2

RoPE - Rotary Position Embedding

Applied to each query and key vector before computing attention scores. Encodes absolute position as a rotation so that the dot product QยทKแต€ naturally produces relative positional information.

Relative Attention for Free

RoPE Rotation Matrix

Su et al. 2021, arXiv:2104.09864 (RoFormer)
Intuition: Rotate each query/key vector by an angle proportional to its position. Relative angles cancel out perfectly in the dot product.
Used by:LLaMA 2/3DeepSeek-V2/V3MistralMixtralQwen 2.5Gemma 2PaLM 2GPT-NeoX
RoPE - PyTorch (LLaMA implementation)python
29 lines
3

GQA - Grouped-Query Attention

Standard multi-head attention requires one KV pair per query head. GQA shares a single KV pair across a group of query heads - dramatically shrinking the KV cache during inference.

4โ€“8ร— KV Cache Reduction

Grouped-Query Attention Equation

Ainslie et al. 2023, arXiv:2305.13245
Intuition: Why give every query head its own K and V when neighbouring heads learn similar things? Group them - massive KV cache savings, near-zero quality loss.
Used by:LLaMA 2/3Mistral 7BMixtralQwen 2.5Gemma 2Falcon-40B
GQA - PyTorchpython
39 lines
4

MLA - Multi-Head Latent Attention (DeepSeek)

Instead of caching full-dimension K and V vectors, MLA compresses them into a tiny low-rank latent vector. During inference this latent is all that needs to be stored - 93.3% fewer KV elements than MHA.

93.3% KV Cache Reduction

MLA Low-Rank KV Compression

DeepSeek-V2 2024, arXiv:2405.04434, ยง2.1.2
Intuition: Project KV down into a tiny bottleneck vector. Only cache the bottleneck - reconstruct K and V on the fly, or absorb the up-projection into the attention weights.
Used by:DeepSeek-V2DeepSeek-V3DeepSeek-R1
MLA - PyTorch (DeepSeek architectural pattern)python
34 lines
5

SwiGLU - Swish-Gated Linear Unit

The feed-forward sublayer in every modern LLM uses SwiGLU instead of the original GELU FFN. Two parallel linear projections - one acts as a gate that controls how much of the other passes through.

+1โ€“2% Quality Benchmark Uplift

SwiGLU Feed-Forward Formulation

Shazeer 2020, arXiv:2002.05202 (GLU Variants)
Intuition: Two linear projections - one passes through Swish, the other is a gate. Their element-wise product lets the network control information flow continuously, not with an on/off ReLU switch.
Used by:LLaMA 2/3DeepSeek-V2/V3PaLM 2MistralMixtralQwen 2.5Gemma 2
SwiGLU FFN - PyTorch (from LLaMA)python
18 lines
6

MoE - Mixture of Experts (DeepSeekMoE)

Replace each FFN with N expert FFNs. A router selects the top-K experts for each token. DeepSeek-V3 activates only 37B of its 671B parameters per token.

37B / 671B Active Compute

DeepSeekMoE Routing & Aggregation

DeepSeek-V2 2024, arXiv:2405.04434, ยง2.2.1
Intuition: Most tokens don't need every FFN. Route each token to the few experts most relevant to it - compute only what's needed, skip the rest.
Used by:DeepSeek-V2DeepSeek-V3DeepSeek-R1Mixtral 8ร—7B/22BSwitch TransformerGrok-1
MoE with Top-K Routing - PyTorchpython
41 lines

Putting It All Together: 2017 vs. 2024+ Frontier LLMs

A modern LLM forward pass through one layer contrasts dramatically with the original Attention Is All You Need paper.

Architecture Evolution
StageVanilla Transformer (2017)Modern LLM (2024+)
NormalizeLayerNorm (mean + var)RMSNorm (var only, faster)
PositionSinusoidal PE (additive, fixed)RoPE (multiplicative, relative)
Attention KVFull KV per head (MHA)Shared KV groups (GQA) or latent (MLA)
FFNGELU: xWโ‚ โ†’ GELU โ†’ Wโ‚‚SwiGLU: (Swish(xWโ‚) โŠ™ xV) Wโ‚‚
FFN ArchitectureDense: same FFN for every tokenMoE: route token to top-K of N expert FFNs
Key Takeaways & Core Rules
  • RMSNorm drops mean-centering from LayerNorm - 7โ€“64% faster with identical training dynamics.
  • RoPE encodes position as rotation, yielding relative positional distances for free via dot-product geometry.
  • GQA shares KV heads across query head groups - 8ร— KV cache reduction in LLaMA 3 vs. standard MHA.
  • MLA (DeepSeek) compresses KV into a low-rank latent - 93.3% KV cache reduction vs. MHA.
  • SwiGLU replaces GELU FFN with a gated bilinear product - consistently +1โ€“2% quality boost across LLM benchmarks.
  • MoE routes each token to top-K of N expert FFNs - DeepSeek-V3 activates 37B of 671B params per token.
Try This Experiment:

Review the Model Adoption table: note how modern frontier models combine RMSNorm + RoPE + SwiGLU as a standard baseline, while scaling models like DeepSeek innovate on MLA and MoE for extreme efficiency.