Chapter 5.3 - Masking, Softmax Context Vector

Overview

Imagine trying to guess the end of a movie while someone is holding the script right in front of you. It's too easy! We want our AI to learn how to predict the next word without cheating. So, we have to hide the future words. This is called "Masking".

---

🎯 Why we do it

Rationale

Our AI's job is to predict the next word. If we let it see the whole sentence at once, it would just cheat and read the answers! We use a "Mask" to cover up the words that come later in the sentence so the AI can only look at the past and the present.

🛠️ How we do it

Methodology

We create a special "staircase" mask. For any word we aren't allowed to see yet, we change its attention score to negative infinity (-inf). Then, we use a math trick called Softmax which turns these scores into percentages. Because -inf is so small, Softmax turns it into a flat 0%! That means the AI gives exactly 0 attention to future words.

import torch

# Our scores
scores = torch.tensor([[1.0, 2.0], 
                       [0.5, 1.5]])

# The Staircase Mask (we hide the top right corner!)
mask = torch.tensor([[0.0, float('-inf')], 
                     [0.0, 0.0]])

# Apply the mask (no peeking!)
masked_scores = scores + mask

# Softmax turns them into percentages!
percentages = torch.softmax(masked_scores, dim=-1)
print(percentages)

🎭 Interactive Causal Attention Mask Visualizer

Autoregressive language models (like GPT) prevent future tokens from cheating during training. Click any query token below to see which preceding keys it is allowed to attend to!

Query \ Key	Tomorrow	I	am	flying	to
Tomorrow	Score	-∞	-∞	-∞	-∞
I	Score	Score	-∞	-∞	-∞
am	Score	Score	Score	-∞	-∞
flying	Score	Score	Score	Score	-∞
to	Score	Score	Score	Score	Score

Token "to" can attend to tokens up to Position 4. Future positions are blocked with -∞ (which Softmax turns into 0.0 probability).