42. Language Modeling

A language model learns to predict the next token.

Example:

The cat is
       ↓
     next token

Training creates input-target pairs:

input:  The cat is
target: cat is sleeping

The model produces logits for possible next tokens.

Cross-entropy loss compares those logits with the correct next token.