Convolution
Lesson 2 of 7
2Convolution
The big pictureConvolution is one idea: a small 3 by 3 filter slides across the image and asks at every position "is the pattern I look for here?". The same filter everywhere, which is why a convolutional network needs thousands of parameters instead of millions.
What the picture shows: The filter slides over the image and asks at every position: is the pattern here? It cannot sit over the border, so a 4 × 6 image gives a 2 × 4 output. The same nine weights are used at every position, which is weight sharing.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
Convolution is like a small magnifying glass passing over the image square by square, looking for one shape: an edge, a corner, a patch of colour. Early layers see simple lines, later ones combine lines into eyes and wheels, and the top ones see faces and cars.
A kernel is a small weight matrix (like 3×3) sliding over the image, computing a dot product at each position. stride sets the step between positions, padding keeps the size, and pooling shrinks the map. Weight sharing gives far fewer parameters than a dense layer.
Before You Start
🖼️An image as a tensor
You know an image is grids of numbers in (C, H, W) order, and the filter runs over those numbers.
🧱Layers and normalisation
You know the linear layer and how to count its parameters, and here you compare it with a convolutional layer.
The Core Idea
Convolution is a small filter sliding over an image. Here you build the filter by hand, see why the output shrinks and how to control its size, then pooling, then why the whole idea works: weight sharing.
1The filter: multiply and sum
A filter is a small weight grid, usually 3 by 3. It sits over a patch of the image, multiplies each number by the matching weight, and sums the result into one number.
Look at the picture: one row of the output. The number is large where brightness changes and zero where it does not.
import numpy as np
image = np.array([
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
[0, 0, 0, 9, 9, 9],
], dtype=float)
kernel = np.array([[-1, 0, 1],
[-1, 0, 1],
[-1, 0, 1]], dtype=float)
def conv2d(img, k, stride=1):
kh, kw = k.shape
out_h = (img.shape[0] - kh) // stride + 1
out_w = (img.shape[1] - kw) // stride + 1
out = np.zeros((out_h, out_w))
for i in range(out_h):
for j in range(out_w):
patch = img[i * stride:i * stride + kh, j * stride:j * stride + kw]
out[i, j] = (patch * k).sum()
return out
print(conv2d(image, kernel))
The two middle columns are 27 and the edges are zero. The filter found the edge exactly where it is. This is all a network's first convolutional layer does: dozens of filters like this one, except they learn their numbers themselves instead of us writing them.
2Why the output shrinks: padding
The filter never leaves the image, so it has fewer positions to stand on than there are pixels.
A 4 by 6 image and a 3 by 3 filter gave a 2 by 4 output, because the filter cannot sit over the border. Padding adds a frame of zeros so the size is kept, and stride jumps the filter more than one pixel, shrinking the output on purpose.
With a 3 by 3 filter, padding=1 keeps the size unchanged. It is the most common setting in modern networks.
3Stride and the size formula
A stride of 2 jumps the filter two pixels each time, so the output halves. One formula covers it all: (n + 2p - k) / s + 1.
Look at the picture: the same 32 pixel image with three settings.
def out_size(n, k, stride=1, padding=0):
return (n + 2 * padding - k) // stride + 1
print("32 k=3 ->", out_size(32, 3))
print("32 k=3 padding=1 ->", out_size(32, 3, padding=1))
print("32 k=3 stride=2 pad=1 ->", out_size(32, 3, stride=2, padding=1))
print("224 k=7 stride=2 pad=3 ->", out_size(224, 7, stride=2, padding=3))
The last line is exactly ResNet's first layer: a 7 by 7 filter with stride 2 turning 224 into 112. Memorise this formula; you will need it every time you design a network and want to know the size entering the final linear layer.
4Pooling: the largest value in each square
Pooling shrinks the image with no weights to learn. MaxPool2d(2) takes the largest value in each 2 by 2 square.
Look at the picture: 16 numbers become 4.
import numpy as np
x = np.array([[1, 3, 2, 0],
[4, 6, 1, 1],
[0, 2, 9, 7],
[1, 0, 5, 8]])
pooled = x.reshape(2, 2, 2, 2).max(axis=(1, 3))
print(pooled)
Each 2 by 2 square became one number, its largest. If the 9 moved one pixel within its square, the output would not change, which is what "less sensitive to position" means.
5Weight sharing
Convolution assumes two things about images: important patterns are small and local, and the same pattern matters wherever it appears. So we use the same filter at every position.
A layer has only out × in × k × k weights, however large the image. Now compare it with a linear layer giving the same output size.
This code needs PyTorch on your machine, so the output shown is recorded from a real run. Install it with pip install torch and run it yourself.
from torch import nn
conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, padding=1)
dense = nn.Linear(3 * 32 * 32, 16 * 32 * 32)
print("conv weights: ", tuple(conv.weight.shape), sum(p.numel() for p in conv.parameters()))
print("dense weights:", sum(p.numel() for p in dense.parameters()))
A 16-filter convolutional layer: 448 parameters. A linear layer producing the same output size from the same input: over fifty million. A hundred-thousand-fold difference, which is why convolutional networks are possible at all, and why they need less data to learn.
How It Works, Step by Step: how convolution works on an image
Place the 3 × 3 filter over the first patch of the image.
Multiply each weight by the pixel under it and sum the nine products into one number: large if the pattern is there, zero if not.
Move the filter by the stride and repeat until the image is covered, giving an output of size (n + 2p − k) / s + 1.
Stack many layers with pooling, so deeper layers combine edges into corners and textures, then object parts.
Worked example
An image whose left half is black (0) and right half white (9), so it has one vertical edge in the middle. And a filter that subtracts the left column from the right one: it gives a big number where brightness changes and zero where it does not.
mat1 and mat2 shapes cannot be multiplied. Use the formula, or easier: pass a random tensor of the right size through the convolutional part and print its shape.
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| An engineer writes the filter's weights by hand. | We wrote the edge filter by hand only to explain. In a real network the filters learn their numbers by training. |
| A convolutional layer's parameter count grows with image size. | It never depends on image size. Conv2d(16, 32, 3) has 4640 parameters on any image, which is the point of weight sharing. |
| A 3 × 3 filter only ever sees 3 × 3 pixels, whatever the network. | Each layer sees 3 × 3 of the one before, so the receptive field grows with depth, and pooling and strides speed that up. |
Try it yourself
A 64 by 64 image, a 5 by 5 filter, stride 1, no padding. Output size?
(64 + 0 − 5) / 1 + 1 = 60. The filter loses two pixels on each side because it cannot sit over the border. With padding=2 it would stay 64.
How many parameters does nn.Conv2d(16, 32, kernel_size=3) have?
Each of the 32 filters spans all 16 input channels at 3 by 3, plus one bias. And the count does not depend on image size at all, which is exactly the point of weight sharing.
Quick review cards
(n + 2p − k) / s + 1.Check Your Understanding
How can a network with only 3 by 3 filters "see" something covering half the image?
By stacking. Each layer sees 3 by 3 of the layer before it, which in turn saw 3 by 3 of its own predecessor, so the "receptive field" grows with depth. Pooling and strides speed this up because each pixel after downsampling stands for a larger region of the original. After ten layers with some downsampling, each deep unit sees most of the image.
A 32 by 32 image, a 3 × 3 filter, stride 2 and padding 1. What is the output size?
(32 + 2 − 3) // 2 + 1 = 16, so 16 by 16: the stride of 2 halved the image.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs2
Done with this lesson?
Sign in to save your progress.