Skip to main content

Chapter 8.2 - GELU Activation Function

Overview

Imagine a bouncer at a club. If you are a positive number, you get to go in! If you are a negative number, you get kicked out! But the GELU bouncer is very gentle; instead of just kicking negative numbers out instantly, he slowly bends them down to zero.

🎯 Why we do it​

Rationale

Linear layers only draw straight lines. But the real world is curvy and complex! Activation functions like GELU (Gaussian Error Linear Unit) add the "curves" so our AI can learn complex patterns. It keeps the good (positive) numbers and gently squishes the bad (negative) numbers to zero.

🛠️ How we do it​

Methodology

We just pass our numbers through the GELU math function. Big positive numbers stay the same. Numbers close to zero get a little curvy, and big negative numbers turn into almost exactly zero.

import torch
import torch.nn.functional as F

# Some numbers
numbers = torch.tensor([-3.0, -1.0, 0.0, 1.0, 3.0])

# The gentle bouncer
curvy_numbers = F.gelu(numbers)

print(curvy_numbers)
# Notice how -3.0 becomes almost 0!

📈 GELU Activation Curve

Input x:+1.00
Output GELU:0.8412

Click anywhere on the curve or drag the slider above to see exact inputs & outputs simultaneously.

GELU(x)
y = x (Identity)
(1.00, 0.84)

💻 Code Implementation​

Here is the exact PyTorch implementation for the concepts discussed above:

import torch
import torch.nn as nn

class GELU(nn.Module):def __init__(self):
        super().__init__()

    def forward(self, x):
        return 0.5 * x * (1 + torch.tanh(
            torch.sqrt(torch.tensor(2.0 / torch.pi)) * 
            (x + 0.044715 * torch.pow(x, 3))
        ))