MLP vs CNN Comparison

# MNIST: MLP vs. CNN — head-to-head Two architectures, two isolated notebook branches, one shared data split: | | Architecture | Trained for | |---|---|---| | **Branch A** | MLP baseline (784 → 256 → 128 → 10) | 3 epochs | | **Branch B** | Small CNN (Conv16 → Conv32 → FC128 → 10) | 3 epochs | Everything *except the architecture* is identical: same seed, same 54k/6k/10k train/val/test split, same batch order, same optimizer (Adam, lr=1e-3), same 3-epoch budget, same loss (cross-entropy). **Cells below:** setup + fixed split → shared model/training code (the fork point) → two branches run in parallel → side-by-side comparison.

# ---------------------------------------------------------------------------
# Shared scaffold — executed ONCE on the parent branch; both experiment
# branches inherit this exact live state (same tensors, same functions).
# ---------------------------------------------------------------------------
import os, json, time, random
from pathlib import Path

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

DEVICE   = torch.device("cuda" if torch.cuda.is_available() else "cpu")
SEED     = 42
…

device=cpu | torch=2.7.1+cpu
split -> train 54,000 | val 6,000 | test 10,000 (official MNIST test)
train class support: [5328 6099 5357 5506 5277 4868 5333 5638 5219 5375]

# ---------------------------------------------------------------------------
# Shared model zoo + the ONE parameterized training function.
# Both branches call the *same* code with a different `arch` string, so any
# difference in the results is attributable to the architecture alone.
# ---------------------------------------------------------------------------

class MLP(nn.Module):
    """Baseline: flatten the image, no spatial structure assumed."""
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 256), nn.ReLU(),
            nn.Linear(256, 128),     nn.ReLU(),
            nn.Linear(128, 10),
        )
    def forward(self, x):
        return self.net(x)
…

scaffold ready — run_experiment(arch) is the single entry point

# ===========================================================================
# CONVERGE — both branches finished; compare on the frozen 10k test set
# ===========================================================================
import json
from pathlib import Path
from IPython.display import display

RUNS_DIR = Path("/home/user/runs")
ARCHS = ["mlp", "cnn"]
LABEL = {"mlp": "MLP baseline", "cnn": "Small CNN"}
COLOR = {"mlp": "#4C72B0", "cnn": "#DD8452"}

M  = {a: json.load(open(RUNS_DIR / a / "metrics.json")) for a in ARCHS}
H  = {a: json.load(open(RUNS_DIR / a / "history.json")) for a in ARCHS}
CM = {a: np.load(RUNS_DIR / a / "confusion.npy") for a in ARCHS}
PR = {a: np.load(RUNS_DIR / a / f"preds_{a}.npy") for a in ARCHS}   # row0 = true, row1 = pred


…

saved -> /home/user/runs/comparison_curves.png

# ===========================================================================
# CONVERGE — where exactly do the two models disagree?
# ===========================================================================
true          = PR["mlp"][0]                 # identical test order for both branches
pred_m, pred_c = PR["mlp"][1], PR["cnn"][1]
ok_m, ok_c    = pred_m == true, pred_c == true
N             = len(true)

only_m = ~ok_m & ok_c          # CNN fixes these
only_c = ok_m & ~ok_c          # CNN breaks these
bothbad = ~ok_m & ~ok_c
err_m, err_c = int((~ok_m).sum()), int((~ok_c).sum())

display(pd.DataFrame({
    "images":            [int((ok_m & ok_c).sum()), int(only_m.sum()), int(only_c.sum()), int(bothbad.sum())],
    "% of test set":     [round(100 * v / N, 2) for v in
                          [(ok_m & ok_c).sum(), only_m.sum(), only_c.sum(), bothbad.sum()]],
    "% of MLP errors":   [round(100 * v / err_m, 1) for v in
…

MLP errors = 259   CNN errors = 127   net gain = 132 images (51.0% of the MLP's mistakes removed)
CNN rescues 69.1% of the MLP's errors but loses 47 images the MLP had right.
On the 80 images BOTH get wrong, the two models pick the same wrong digit 81% of the time -> that residue is genuinely ambiguous / hard handwriting, not an architecture-specific failure.

Most common shared mistakes: 5→3 (7), 9→4 (6), 6→0 (5), 7→2 (5), 2→8 (5), 2→7 (4)

# ===========================================================================
# CONVERGE — why: does spatial inductive bias explain the gap?
# Same frozen models, same test images, just translated by a few pixels.
# An MLP sees a different input vector; a CNN sees the same local strokes.
# ===========================================================================
import torch.nn.functional as F
from torch.utils.data import TensorDataset

def load_trained(arch):
    m = build_model(arch).to(DEVICE)
    m.load_state_dict(torch.load(RUNS_DIR / arch / "model.pt", map_location=DEVICE))
    return m.eval()

models = {a: load_trained(a) for a in ARCHS}
crit   = nn.CrossEntropyLoss()

x_raw = test_ds.data.float().unsqueeze(1)          # (10000,1,28,28) raw 0..255
y_all = torch.tensor(test_ds.targets.numpy())
…

saved -> /home/user/runs/comparison_translation.png

## Findings — exactly where the two models differ, and why ### 1. Headline: the CNN wins by 1.32 pp using *fewer* parameters | | params | train time | train loss (ep 3) | val loss (final) | **TEST loss** | **TEST acc** | test errors | |---|---|---|---|---|---|---|---| | **MLP baseline** | 235,146 | 36.0 s | 0.0743 | 0.0904 | 0.0801 | **97.41 %** | 259 / 10,000 | | **Small CNN** | 206,922 | 58.8 s | 0.0430 | 0.0528 | 0.0373 | **98.73 %** | 127 / 10,000 | The CNN gets **+1.32 pp accuracy and 2.15× lower test loss** — while having **12 % fewer parameters**. The gain therefore comes from the *inductive bias*, not from capacity. The cost is compute: 1.63× slower per epoch (19 s vs 11.5 s) because convolutions do far more arithmetic per pixel. ### 2. Loss curves: the CNN is already ahead after one epoch * CNN epoch-1 val accuracy (**97.65 %**) already beats the MLP's *epoch-3* val accuracy (97.13 %). Final val accuracy 98.55 % vs 97.13 %. * Final val loss 0.0528 vs 0.0904 — the CNN ends **42 % lower**, and lower than the MLP ever reached. * Both models are still improving at epoch 3 (train loss still falling, val loss still falling for the CNN), so **3 epochs underfits both**. The MLP's per-batch train loss stays ~2× above the CNN's the whole way, i.e. it cannot fit the training set as tightly given the same budget. * The MLP's validation loss has visible dips/bumps (e.g. around step 800); that is Adam step-size noise around a plateau, not a learning-rate problem. ### 3. Conf…