Transformer Block

Pre-LN & Residual Streams

Compose Pre-LayerNorm, Multi-Head Attention, residual identity highways, and 2-layer FeedForward MLPs.

d_ffn: 256 (4 × 64)

Norm: Pre-LN (Standard)

Plain Language Intuition

A transformer block is a modular sandwich: attention (so tokens communicate with each other) + feed-forward network (so each token processes information independently) + skip connections (so gradients flow freely without vanishing) + normalization (so numbers remain stable). Stack 4 of these and you have our full model.

Production Real-World Context

Modern LLMs simply stack dozens of these identical blocks. GPT-3 stacks 96 blocks; LLaMA-70B stacks 80 blocks. Pre-LayerNorm (normalizing before the sublayer) is now universal because it guarantees stable training without fragile warmup schedules.

Prerequisites:

Multi-Head Attention (Step 5)

Embeddings (Step 2)

Pre-Norm Transformer Block

Vaswani et al. 2017, §3.1 (pre-norm variant)

Intuition: Normalize -> attend -> add residual -> normalize -> feed-forward -> add residual. Information is added onto the residual highway rather than overwritten.

Layer Normalization

Ba et al. 2016, arXiv:1607.06450

Intuition: Normalizes features across the 64 embedding dimensions per token so activations do not explode across 4 stacked blocks.

Position-wise Feed-Forward Network

Vaswani et al. 2017, Eq. 2

Intuition: Expands token features by 4x (64 -> 256) into a higher-dimensional space where non-linear separation is easier, then compresses back.

Dataflow through a Single Transformer Block (Layer l)

Input Tensor X_l ∈ ℝ^(N × 64)

Sublayer 1: Self-Attention

Pre-LN

LN(X) → MHA(Q, K, V)

Residual Stream: X + MHA(LN(X))

Sublayer 2: FeedForward MLPExpand 4x → 256D

LN(X^(1)) → W_1 (256D) → GELU → W_2 (64D)

Residual Stream: X^(1) + MLP(LN(X^(1)))

Output Tensor X_(l+1) ∈ ℝ^(N × 64)

The Residual Highway & Gradient Flow

Intuition: Because the partial derivative contains the identity matrix I, backpropagated gradients flow unimpeded from layer 4 directly back to layer 1 without suffering exponential decay.

In our bilingual model with 4 blocks ($L=4$), this residual connection is vital. It guarantees that the early token representations retain their character identities while deeper layers compose complex syntactic and philosophical relationships.

Transformer Block - PyTorch Reference Implementationpython

35 lines

Key Takeaways & Core Rules

Transformer block formula: x + Attention(LN(x)), followed by x + FFN(LN(x)).
Residual connections (x + f(x)) create an identity gradient highway preventing vanishing gradients.
LayerNorm stabilizes activations across the 64 embedding dimensions at each sublayer.
FeedForward network expands 4x into 256 dimensions with GELU, providing non-linear memorization capacity.

Try This Experiment:

Trace how tensor activations travel through the residual stream without losing their base token features.