Convolution Mechanics

A convolution is the single operation a CNN repeats billions of times. Understanding exactly what it computes, how channels stack into the weight tensor, and why the parameter count is independent of image size is the foundation for everything else about CNNs.

Rapid Recall

A filter (kernel) is a small grid of learnable numbers, a pattern template, slid over the image; at each position the dot product (elementwise multiply then sum) gives one output number, and the collection is a feature map. With \(C_{in}\) input channels, one filter is \(K \times K \times C_{in}\) and sums across channels to produce one map; \(C_{out}\) filters give \(C_{out}\) maps, so the weight tensor is \(C_{out} \times C_{in} \times K \times K\), independent of image size. Stacking small kernels (two 3×3 = one 5×5 receptive field) is cheaper and more expressive. A 1×1 convolution mixes channels at a single pixel, used for cheap dimensionality change.

§1 What a convolution actually is

A filter (kernel) is a small grid of learnable numbers, a pattern template. You slide it over the image. At each position you ask: "how much does the patch underneath me look like my pattern?" The answer is one number. Collect these over the whole image and you get a feature map: a heatmap of where the pattern appeared. The "looks like" operation is just elementwise multiply, then sum, a dot product.

With input patch and kernel both \(K \times K\):

\[ y_{i,j} = b + \sum_{m=0}^{K-1}\sum_{n=0}^{K-1} w_{m,n}\, x_{\,i+m,\; j+n} \]

where \(w\) = kernel weights, \(b\) = one bias for the whole filter, \(x\) = input. One dot product per output cell.

Interview note: convolution vs cross-correlation

What deep learning calls "convolution" is technically cross-correlation, true convolution flips the kernel first (\(x_{i-m,j-n}\)). Since the kernel is learned, the flip is irrelevant, the network just learns the flipped weights. If asked: "it's cross-correlation; the flip doesn't matter because the weights are learned."

§2 Fully worked numeric example

Input 4×4, kernel 3×3, stride 1, no padding. Output is 2×2.

Input X            Kernel W (edge-ish), bias b = 0
 1  2  0  1         1  0 -1
 0  1  2  3         1  0 -1
 1  0  1  2         1  0 -1
 2  1  0  1

Top-left cell, overlay W on the top-left 3×3 block:

\[ (1)(1)+(2)(0)+(0)(-1)+(0)(1)+(1)(0)+(2)(-1)+(1)(1)+(0)(0)+(1)(-1) = -1 \]

Slide one right (cols 1 to 3): \(= -3\). Slide down (rows 1 to 3, cols 0 to 2): \(= 0\). Bottom-right: \(= -4\).

Output feature map:
-1  -3
 0  -4

That is a complete convolution. Every CNN is this, billions of times, with learned W.

The kernel slides across every valid position; each placement produces one output number. 4×4 input, 3×3 kernel, stride 1 → 2×2 output.

§3 Channels and the weight tensor

Real images aren't flat. An RGB image is \(H \times W \times 3\). After the first conv layer you have many feature maps (channels). So convolution is almost never 2D in practice, it is 3D over the channel dimension, and this is where the parameter math lives.

A single filter spans all input channels. If the input has \(C_{in}\) channels, one filter is not \(K \times K\), it is \(K \times K \times C_{in}\), with a separate slice per channel. You convolve each slice with its channel, then sum across all channels into a single number. So one filter produces one output feature map.

\[ y_{i,j} = b + \sum_{c=0}^{C_{in}-1}\sum_{m=0}^{K-1}\sum_{n=0}^{K-1} w_{c,m,n}\, x_{c,\,i+m,\,j+n} \]

Want \(C_{out}\) feature maps? Use \(C_{out}\) independent filters, each \(K \times K \times C_{in}\). Stack their outputs along the channel axis. The weight tensor shape is:

\[ \underbrace{C_{out}}_{\#\,\text{filters}} \times \underbrace{C_{in}}_{\text{depth each}} \times \underbrace{K \times K}_{\text{spatial}} \]

The weight tensor is not decided by image size. It is decided by three hyperparameters: kernel size \(K\), input channels \(C_{in}\) (fixed by the previous layer), and output channels \(C_{out}\) (your design choice). The image's height and width never enter the parameter count, that is the whole point of weight sharing.

Input: 3 channels (RGB)        One filter has 3 slices (one per channel)
  R 5×5   G 5×5   B 5×5          W_R 3×3  W_G 3×3  W_B 3×3
    │       │       │               │        │        │
    └─conv──┴─conv──┴─conv──────────┘        │        │
                  │                            │        │
            sum all three ◄────────────────────┴────────┘
                  │
              ONE output feature map (+ bias)
  Want 64 output maps? → 64 such 3-slice filters, stacked.

§4 Kernel sizes and the 1×1 convolution

A \(K \times K\) kernel over \(C_{in}\) channels producing \(C_{out}\) has \(K^2 C_{in} C_{out}\) weights. Cost grows with \(K^2\). A 5×5 is \(25/9 \approx 2.8\times\) more expensive than a 3×3.

Stacking small kernels beats one big kernel. Two stacked 3×3 convs have the same receptive field as one 5×5; three 3×3s ≈ one 7×7, but with fewer parameters and more non-linearities. One 7×7 filter: \(7^2 = 49\) params. Three 3×3 filters: \(3 \times 3^2 = 27\) params. Same receptive field, 45% fewer parameters and two extra ReLUs, so more expressive. This is the entire design philosophy of VGG.

The receptive field grows as:

\[ r_{\ell} = r_{\ell-1} + (K_\ell - 1)\prod_{i=1}^{\ell-1} S_i \]

With all stride-1, K=3 layers, the field grows \(3 \to 5 \to 7 \to 9\), +2 per layer. Add stride or pooling and it grows multiplicatively, that is how a final-layer neuron ends up seeing the whole image with few layers.

1×1 convolutions, genuinely useful

A 1×1 kernel looks at a single spatial position but all channels. It does nothing spatially, it is a per-pixel fully-connected layer across channels. Its job is channel mixing and dimensionality change: squeeze 256→64 channels cheaply before an expensive 3×3 (ResNet's bottleneck), or expand, or add a non-linearity across channels at zero spatial cost. Params: \(C_{in}C_{out}\). Used everywhere from Inception to MobileNet. The architectures page shows where the bottleneck uses them.