🧼 Batch Normalization

Deep networks are notoriously unstable. Batch Normalization keeps them in check.

🎢 The Rollercoaster

Imagine Layer 1 suddenly outputs massive numbers (like 1,000,000). Layer 2 receives this, panics, and its math blows up. Batch Normalization is a checkpoint between layers. It grabs the numbers flying out of Layer 1, scrubs them down, and forcefully rescales them so they have a mean of 0 and standard deviation of 1.

It ensures every layer gets clean, stable numbers.

🐍 Python Implementation

We simply insert a nn.BatchNorm1d layer immediately after our Linear layers!

import torch
import torch.nn as nn

class StableNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(10, 64)
        # Add Batch Norm! (64 matches the output size of layer1)
        self.batch_norm = nn.BatchNorm1d(64)
        
    def forward(self, x):
        # 1. Raw Math
        x = self.layer1(x)
        # 2. Scrub the numbers clean!
        x = self.batch_norm(x)
        # 3. Bouncer (Activation)
        x = torch.relu(x)
        return x

model = StableNetwork()
print(model)