Transformers

Transformers are sequence models introduced in 'Attention Is All You Need'. They discard recurrence and convolutions entirely, relying on Multi-Head Self-Attention layers and Position-wise Feed-Forward Networks. They process sequences in parallel, enabling rapid training on massive web datasets.

Complexity Profile

Case	Complexity
Best Case	O(N^2 * D)
Average Case	O(N^2 * D)
Worst Case	O(N^2 * D)
Space Complexity	O(N^2)

Code Implementation

import torch.nn as nn

class TransformerEncoderBlock(nn.Module):
    def __init__(self, dim, heads, mlp_dim, dropout=0.1):
        super().__init__()
        self.attn = nn.MultiheadAttention(embed_dim=dim, num_heads=heads)
        self.norm1 = nn.LayerNorm(dim)
        self.norm2 = nn.LayerNorm(dim)
        
        self.mlp = nn.Sequential(
            nn.Linear(dim, mlp_dim),
            nn.ReLU(),
            nn.Linear(mlp_dim, dim)
        )
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, x):
        # 1. Self Attention with Skip Connection
        attn_out, _ = self.attn(x, x, x)
        x = self.norm1(x + self.dropout(attn_out))
        # 2. Feed-Forward with Skip Connection
        mlp_out = self.mlp(x)
        x = self.norm2(x + self.dropout(mlp_out))
        return x

Real-World Applications

• Large Language Models (Llama, GPT, Gemini).
• Vision Transformers (ViT) for object classification.
• Protein folding predictions (AlphaFold).