Chapter 10.4 - Complete Inference Walkthrough

Tomorrow I am flying to after tokenization - [49488, 314, 716, 7348, 284] lets say each of token is represented by 3 dimensional matrix . Each token ID is looked up in the token embedding matrix Wₑ. Wₑ : (50257 × 3)

after looking we found the embeddings to be : token embeddings : (Stored / looked up): Tomorrow →49488 → [ 0.62, -0.15, 0.48]
I → 314 → [-0.31, 0.84, 0.12]
am → 716 → [ 0.17, 0.39, -0.56]
flying → 7348 → [ 0.91, -0.44, 0.27]
to → 284 → [-0.08, 0.71, 0.65]

word		token id			token embeddings
Tomorrow		49488		0.62	--0.15	0.48
I		314		-0.31	0.84	0.12
am		716		0.17	0.39	-0.56
flying		7348		0.91	-0.44	0.27
to		284		-0.08	0.71	0.65

now for each word position we find the positional embeddings (Stored / looked up) Position 0 → [ 0.10, 0.00, -0.10] Position 1 → [ 0.20, -0.10, 0.05] Position 2 → [ 0.30, 0.10, 0.00] Position 3 → [ 0.40, -0.20, 0.10] Position 4 → [ 0.50, 0.00, -0.05]

now we add these both to create ==Input embeddings== [ 0.62, -0.15, 0.48] + [ 0.10, 0.00, -0.10] = [ 0.72, -0.15, 0.38]
[-0.31, 0.84, 0.12] + [ 0.20, -0.10, 0.05] = [-0.11, 0.74, 0.17]
[ 0.17, 0.39, -0.56] + [ 0.30, 0.10, 0.00] = [ 0.47, 0.49, -0.56]
[ 0.91, -0.44, 0.27] + [ 0.40, -0.20, 0.10] = [ 1.31, -0.64, 0.37]
[-0.08, 0.71, 0.65] + [ 0.50, 0.00, -0.05] = [ 0.42, 0.71, 0.60]

so final input embeddings = [ [ 0.72, -0.15, 0.38], [-0.11, 0.74, 0.17], [ 0.47, 0.49, -0.56], [ 1.31, -0.64, 0.37], [ 0.42, 0.71, 0.60] ]

------------------------------transformer story------------------------------

[[Chapter 7.3 transformer story]] - self attention mechanism for 1

---

we repeated the self attention 3 times and got 3 context vector from 3 different KQV Head 1 =
[
[ 0.083, -0.260],
[ 0.026, 0.120],
[ 0.170, 0.212],
[ 0.134, 0.056],
[ 0.127, 0.013]
] Head 2 =
[
[-0.192, 0.381],
[ 0.114, 0.248],
[ 0.292, -0.071],
[ 0.056, 0.319],
[ 0.173, 0.201]
] Head 3 =
[
[ 0.214, 0.090],
[ 0.097, -0.161],
[ 0.333, 0.282],
[ 0.011, -0.084],
[ 0.205, 0.118]
]

Concatinating different context vectors

we simply place them side by side

Each | separates the output from one attention head: Concatenated vectors = [ Head 1 | Head 2 | Head 3 ] [
[ 0.083, -0.260 | -0.192, 0.381 | 0.214, 0.090 ],
[ 0.026, 0.120 | 0.114, 0.248 | 0.097, -0.161 ],
[ 0.170, 0.212 | 0.292, -0.071 | 0.333, 0.282 ],
[ 0.134, 0.056 | 0.056, 0.319 | 0.011, -0.084 ],
[ 0.127, 0.013 | 0.173, 0.201 | 0.205, 0.118 ]
]

output Projection Matrix = W_O (6×3) = (ingredients)

[
[ 0.3, -0.2, 0.5],
[-0.6, 0.1, 0.4],
[ 0.2, 0.7, -0.3],
[ 0.8, -0.5, 0.2],
[-0.1, 0.4, 0.6],
[ 0.5, 0.3, -0.2]
]

so Multi-Head Attention Output = Concatenated vectors x W_O =

[
[ 0.413, -0.149, 0.123 ],
[-0.066, 0.171, -0.027 ],
[ 0.296, 0.291, 0.042 ],
[ 0.291, -0.029, -0.002 ],
[ 0.243, 0.069, 0.048 ]
]

adding residual connection

original x input matrix + attention output matrix original X input = [
[ 0.72, -0.15, 0.38],
[-0.11, 0.74, 0.17],
[ 0.47, 0.49,-0.56],
[ 1.31, -0.64, 0.37],
[ 0.42, 0.71, 0.60]
] + Attention output: [
[ 0.413,-0.149, 0.123],
[-0.066, 0.171,-0.027],
[ 0.296, 0.291, 0.042],
[ 0.291,-0.029,-0.002],
[ 0.243, 0.069, 0.048]
] =

Residual Output= [
[1.133, -0.299, 0.503],
[-0.176, 0.911, 0.143],
[0.766, 0.781,-0.518],
[1.601, -0.669, 0.368],
[0.663, 0.779, 0.648]
]

Layer Normalization Story

we do [[Chapter 7.2 layer normalization]] and once done our LayerNorm Output
≈
[
[ 1.40,-1.52, 0.12],
[-1.15, 1.29,-0.14],
[ 0.71, 0.70,-1.41],
[ 1.31,-1.14,-0.17],
[-0.16, 0.91,-0.75]
]

---

we put LayerNorm Output into [[Chapter 7.1 FFN]]

and we get FFN Output as : [
[ 1.228, -0.564, 0.583],
[-0.810, 1.420, -0.561],
[-0.210, 0.117, -0.015],
[ 0.836, -0.351, 0.390],
[-0.339, 0.563, -0.140]
]

Residual Output 2

Residual Output₂ = LayerNorm Output + FFN Output

LayerNorm Output
[
[ 1.40,-1.52, 0.12],
[-1.15, 1.29,-0.14],
[ 0.71, 0.70,-1.41],
[ 1.31,-1.14,-0.17],
[-0.16, 0.91,-0.75]
] + FFN Output [ [ 1.228,-0.564, 0.583], [-0.810, 1.420,-0.561], [-0.210, 0.117,-0.015], [ 0.836,-0.351, 0.390], [-0.339, 0.563,-0.140] ] =Residual Output 2 = [
[ 2.628,-2.084, 0.703],
[-1.960, 2.710,-0.701],
[ 0.500, 0.817,-1.425],
[ 2.146,-1.491, 0.220],
[-0.499, 1.473,-0.890]
]

Second LayerNorm

Residual Output₂
[
[ 2.628,-2.084, 0.703],
[-1.960, 2.710,-0.701],
[ 0.500, 0.817,-1.425],
[ 2.146,-1.491, 0.220],
[-0.499, 1.473,-0.890]
] after normalizing using [[Chapter 7.2 layer normalization]] we get [
[ 1.28, -1.16, -0.12],
[-1.02, 1.36, -0.34],
[ 0.48, 0.91, -1.39],
[ 1.30, -1.14, -0.16],
[-0.41, 1.38, -0.97]
]

---

now if this is not the last layer then this matrix becomes the input for next layer 's attention mechanism and the cycle continues

if this is the last block then

we keep just the last token from our transformed output generated by layer normalization Final Hidden State = [-0.41, 1.38, -0.97] we generate a thing called **LM Head

**LM HEAD Matrix shape = LM Head = (Model Dimension) × (Vocabulary Size)

eg in real chatgpt 2 it will be 3 x 50,257 lets say this time our vocabulary has 5 words

vocab =

0 49488 "I"
1 314 "am"
2 716 "learning"
3 7348 "to"
4 284 "Delhi"

LM Head = 3x5

Now we multiply both LM Head and last token

[-0.41, 1.38, -0.97] x
[
[ 0.3, -0.2, 0.7, 0.1, -0.4],
[-0.5, 0.8, -0.1, 0.6, 0.2],
[ 0.4, 0.3, -0.6, 0.5, -0.7]
] = = [
-1.201,
0.891,
0.131,
0.307,
1.078
] now the maximum prob is of next word is of word 5 = 1.078 which is "Delhi". we can softmax these scores to convert to percentage but that is not needed and the entire word becomes Tomorrow i am flying to Delhi