Chapter 17.2 - Normalization (LayerNorm vs. RMSNorm)

Overview

Remember how Layer Normalization is like a shower for our data? Well, scientists realized that taking a full shower takes too much math. They invented RMSNorm, which is like a super-fast 30-second rinse that works just as well!

---

🎯 Why we do it

Rationale

LayerNorm calculates the "average" of the numbers and subtracts it. But subtraction takes time on a GPU! RMSNorm skips the subtraction completely and only divides the numbers by their size. It saves 10% of the training time for massive models like LLaMA!

🛠️ How we do it

Methodology

RMSNorm just squares all the numbers, finds the average of those squares, takes the square root, and divides the original numbers by that result. Simple and lightning fast!

import torch

data = torch.tensor([3.0, 4.0])

# RMSNorm in a nutshell:
# 1. Square them: [9.0, 16.0]
# 2. Average them: (9 + 16) / 2 = 12.5
# 3. Square root: sqrt(12.5) ≈ 3.53
# 4. Divide: data / 3.53

print("Super fast rinse complete!")