Skip to main content

Chapter 17.3 - Positional Encodings (RoPE ALiBi)

Overview

If I say "Dog bites man" or "Man bites dog", the words are the same, but the order changes everything! GPT-2 adds a number to each word to tell it its position. But modern models use a super cool spinning trick called RoPE!

🎯 Why we do it​

Rationale

Old models could only read short paragraphs because they ran out of position numbers. RoPE (Rotary Position Embedding) physically "spins" the vectors in a circle based on their position. This lets modern models read entire books (like 100,000 words) without forgetting the order!

🛠️ How we do it​

Methodology

Imagine your vector is the hour hand on a clock. If a word is at position 1, we rotate the hand 10 degrees. If it's at position 2, we rotate it 20 degrees. The AI can just look at the angle of the hand to know exactly where the word is!

print("Word 1: Rotating 10 degrees...")
print("Word 2: Rotating 20 degrees...")
print("Word 3: Rotating 30 degrees...")
print("The AI now knows the exact order without doing extra math!")