Skip to main content

Chapter 17.1 - Activation Functions (ReLU, GELU, SwiGLU)

Overview

Over the years, scientists have invented better and better "bouncers" (activation functions) for the AI's club. We started with harsh bouncers like ReLU, moved to gentle ones like GELU, and now we have super-smart ones like SwiGLU!

🎯 Why we do it​

Rationale

The better the activation function, the faster and smarter the AI learns.

  • ReLU just chopped off negative numbers instantly (a bit too harsh).
  • GELU (what GPT-2 uses) gently bends negative numbers to zero.
  • SwiGLU (what LLaMA 3 uses) is even more complex and helps the AI learn incredibly tricky patterns!

🛠️ How we do it​

Methodology

You can literally swap these functions out in PyTorch with one line of code!

import torch
import torch.nn.functional as F

numbers = torch.tensor([-2.0, 0.0, 2.0])

# The harsh bouncer (ReLU)
print("ReLU:", F.relu(numbers))

# The gentle bouncer (GELU)
print("GELU:", F.gelu(numbers))

⚡ Interactive ReLU Activation Curve

Input x:+1.00
Output ReLU:1.0000

Notice how ReLU strictly clamps all negative values to zero (dead zone), whereas positive values pass through unchanged ($y=x$).

ReLU(x)
y = x (Identity)
(1.00, 1.00)