Attention
Scaled Dot-Product Attention calculates relevance scores between sequence elements. It takes Query (Q), Key (K), and Value (V) matrices, computes dot products of Q and K, scales them by the square root of query dimension, applies Softmax to yield attention weights, and multiplies the weights with V.
GPT
Generative Pre-trained Transformer (GPT) is a decoder-only Transformer model. It is pre-trained on text corpora to predict the next token given preceding context. It uses causal attention masks to ensure tokens only attend to previous positions during training, enabling rapid auto-regressive generation.
Transformers
Transformers are sequence models introduced in 'Attention Is All You Need'. They discard recurrence and convolutions entirely, relying on Multi-Head Self-Attention layers and Position-wise Feed-Forward Networks. They process sequences in parallel, enabling rapid training on massive web datasets.