Skip to main content

Chapter 4.2 - Matrix Multiplication

✖️ Matrix Multiplication

Matrix multiplication is essentially just doing many Dot Products all at once.

Let's visualize how a 2×22 \times 2 matrix multiplies with another 2×22 \times 2 matrix mathematically. We will use colors to track exactly where each number goes.

The Rule: Rows ×\times Columns​

To find the top-left number of the answer, we take the Top Row of the first matrix and calculate the dot product with the Left Column of the second matrix.

[ABCD]×[EFGH]=[(A×E)+(B×G)(A×F)+(B×H)(C×E)+(D×G)(C×F)+(D×H)]\begin{bmatrix} \color{red}{A} & \color{red}{B} \\ \color{skyblue}{C} & \color{skyblue}{D} \end{bmatrix} \times \begin{bmatrix} \color{green}{E} & \color{orange}{F} \\ \color{green}{G} & \color{orange}{H} \end{bmatrix} = \begin{bmatrix} (\color{red}{A} \times \color{green}{E}) + (\color{red}{B} \times \color{green}{G}) & (\color{red}{A} \times \color{orange}{F}) + (\color{red}{B} \times \color{orange}{H}) \\ (\color{skyblue}{C} \times \color{green}{E}) + (\color{skyblue}{D} \times \color{green}{G}) & (\color{skyblue}{C} \times \color{orange}{F}) + (\color{skyblue}{D} \times \color{orange}{H}) \end{bmatrix}

Example with Numbers​

Let's use some real numbers to see this in action. Suppose we have two 2×22 \times 2 matrices, XX and YY.

X=[1234]Y=[5678]X = \begin{bmatrix} \color{red}{1} & \color{red}{2} \\ \color{skyblue}{3} & \color{skyblue}{4} \end{bmatrix} \quad \quad Y = \begin{bmatrix} \color{green}{5} & \color{orange}{6} \\ \color{green}{7} & \color{orange}{8} \end{bmatrix}

When we multiply X×YX \times Y:

X×Y=[(1×5)+(2×7)(1×6)+(2×8)(3×5)+(4×7)(3×6)+(4×8)]X \times Y = \begin{bmatrix} (\color{red}{1} \times \color{green}{5}) + (\color{red}{2} \times \color{green}{7}) & (\color{red}{1} \times \color{orange}{6}) + (\color{red}{2} \times \color{orange}{8}) \\ (\color{skyblue}{3} \times \color{green}{5}) + (\color{skyblue}{4} \times \color{green}{7}) & (\color{skyblue}{3} \times \color{orange}{6}) + (\color{skyblue}{4} \times \color{orange}{8}) \end{bmatrix} X×Y=[5+146+1615+2818+32]=[19224350]X \times Y = \begin{bmatrix} \color{red}{5} + \color{red}{14} & \color{red}{6} + \color{red}{16} \\ \color{skyblue}{15} + \color{skyblue}{28} & \color{skyblue}{18} + \color{skyblue}{32} \end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix}

Matrix Multiplication in PyTorch​

In Python using PyTorch, we can perform this exact same mathematical operation instantly using the @ symbol:

import torch

X = torch.tensor([
  [1, 2],
  [3, 4]
])

Y = torch.tensor([
  [5, 6],
  [7, 8]
])

# Perform matrix multiplication using the @ operator
result = X @ Y

print(result)

Output:

tensor([[19, 22],
        [43, 50]])
The Ultimate Cheat Code

Matrix multiplication (@) is the engine of all Neural Networks. Whenever you see @ in PyTorch, just remember it means: "Take each row of the matrix on the left, and compute its dot product with each column of the matrix on the right."

A
2
1
3
4
0
2
×
B
5
2
1
4
3
7
=
AB
20
29
26
22
(AB)11 = (2 · 5) + (1 · 1) + (3 · 3) = 20.000