Chapter 0.1 - Course Index

Welcome to the complete, step-by-step mathematical breakdown of Large Language Models (Decoder-only Transformer Architecture)! Below is the structured index of all course modules:

Course Repository

All the code, notebooks, and materials for this course can be found on our official GitHub repository: hashcodes7/GPT2-End-to-End-Architecture

Chapter 0: Index

• ○Course Index
• ○Architecture Ingredients

Chapter 1: Pre Requisite Maths

• ○Dot Products
• ○Matrix Multiplication

Chapter 2: Input Processing Tokenization

• ○Tokenizing Text
• ○Data Sampling
• ○Creating Token Embeddings
• ○Encoding Positional Embeddings

Chapter 3: Input Processing and Embeddings (End to End Walkthrough)

• ○Tokenization and Input Prompt
• ○Token Embedding Lookup (We Matrix)
• ○Positional Embeddings and Combined Input Matrix

Chapter 4: Scaled Dot Attention Core Mechanics

• ○Dot Products
• ○Matrix Multiplication
• ○attention scores
• ○normalization
• ○context vector
• ○full matrix conversion

Chapter 5: Single Head Self Attention

• ○Weight Matrices
• ○Raw Attention Scores Scaling
• ○Masking, Softmax Context Vector

Chapter 6: Multi Head Attention

• ○Multi Head Self Attention
• ○Output Projection

Chapter 7: Pre FFN Cleanup

• ○1st Residual Connection
• ○1st Layer Normalization

Chapter 8: Feed Forward Network (FFN)

• ○First Linear Layer
• ○GELU Activation Function
• ○Second Linear Layer

Chapter 9: Post FFN Cleanup

• ○2nd Residual Connection
• ○2nd Layer Normalization

Chapter 10: Next Word Generation

• ○LM Head Projection Logits
• ○Sampling (Temperature, Top K, Top P)
• ○Autoregressive Generation Loop
• ○Complete Inference Walkthrough

Chapter 11: Training

• ○Cross Entropy Loss
• ○Generating Text Batches
• ○Calculating the Batch Loss
• ○Backpropagation
• ○The Optimization Step
• ○Tensors
• ○Weight Optimization (Optimizers)
• ○Complete Training Pipeline

Chapter 12: Pretrained Weights

• ○Saving Loading Weights
• ○Downloading GPT 2 Weights
• ○Mapping TF to PyTorch
• ○Complete Weight Loading Pipeline

Chapter 13: Finetuning

• ○Introduction to Finetuning
• ○Instruction Dataset
• ○Collate Function (Padding Masking)
• ○The Finetuning Pipeline

Chapter 14: Optional Alternatives Modern Variants

• ○Activation Functions (ReLU, GELU, SwiGLU)
• ○Normalization (LayerNorm vs. RMSNorm)
• ○Positional Encodings (RoPE ALiBi)
• ○Attention Variants (GQA FlashAttention)
• ○Scaling Speedups (MoE KV Caching)