Neural Scaling Laws with Parallel Kernels

# Neural Scaling Laws, Reproduced Live — Across 3 Parallel Isolated Kernels **Why this notebook cannot be run in Colab, Jupyter, or a terminal agent.** A normal notebook has *one* kernel. One namespace, one process, one thing happening at a time. To sweep 6 model scales you run them back-to-back and wait. This notebook does something structurally different: 1. We build a char-level GPT and a training harness **once**, in a base kernel. 2. We then **fork that live kernel 3 ways.** Each fork inherits the fully-warm namespace — corpus already tokenized, model class already defined, optimizer factory ready — and gets its *own isolated Python runtime*. 3. All 3 runtimes **train at the same time**, on the same machine, in the same notebook, each writing its results to shared disk. 4. We **converge** the branches back into the main line and fit the scaling law across all 6 model sizes. The fan-out is the point. `d_model=32` and `d_model=192` are training *concurrently* in separate live kernels, and the notebook renders them inline as sibling branches at the fork point. --- ### The science We hold the token budget fixed and vary only model capacity, then fit the standard parametric form $$L(N) \;=\; L_\infty \;+\; \frac{a}{N^{\,b}}$$ where $N$ is the non-embedding parameter count, $L_\infty$ is the irreducible entropy floor of the data, and $b$ is the scaling exponent. This is the same functional shape used by Kaplan et al. (2020) and Hoffmann et al. (2022) — we are reprodu…

import os, json, math, time, random, urllib.request
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

# ---------------------------------------------------------------- reproducibility
SEED = 1337
def seed_all(s=SEED):
    random.seed(s); np.random.seed(s)
    torch.manual_seed(s); torch.cuda.manual_seed_all(s)
seed_all()

ROOT = '/home/user'
os.makedirs(f'{ROOT}/scaling_results', exist_ok=True)
os.makedirs(f'{ROOT}/checkpoints', exist_ok=True)
os.makedirs(f'{ROOT}/models', exist_ok=True)

…

corpus     : data/input.txt
characters : 1,115,394   vocab: 65
train/val  : 1,003,854 / 111,540 tokens

harness ready — ctx=128 batch=32 fixed val set=24 batches (98,304 tokens)

from tqdm.auto import tqdm

# ---- fixed token budget: capacity is the ONLY thing we vary --------------------
STEPS = 900
TOKENS_PER_STEP = BATCH * CTX
TOKEN_BUDGET = STEPS * TOKENS_PER_STEP
LR = 3e-3
EVAL_EVERY = 75

def train_scale(d_model, n_layer=4, n_head=4, steps=STEPS, tag=None,
                threads=2, verbose=True):
    """Train ONE model scale under the fixed token budget and persist everything.

    Writes  scaling_results/<tag>.json   -> the scaling-law data point
            checkpoints/<tag>/ckpt.pt    -> resumable mid-run checkpoints
            models/<tag>/                -> final deployable bundle
    """
    torch.set_num_threads(threads)          # play nice with sibling kernels
…

timing probe d192:   0%|          | 0/30 [00:00<?, ?it/s]

import glob, json
import numpy as np
from scipy.optimize import curve_fit
import plotly.graph_objects as go
from plotly.subplots import make_subplots

# ---- CONVERGE: pull every branch's result off the shared filesystem -------------
recs = sorted((json.load(open(p)) for p in glob.glob(f'{ROOT}/scaling_results/*.json')),
              key=lambda r: r['N_non_embedding'])
N  = np.array([r['N_non_embedding'] for r in recs], dtype=float)
L  = np.array([r['final_val_loss']  for r in recs], dtype=float)

import pandas as pd
tbl = pd.DataFrame([{'branch scale': r['tag'], 'd_model': r['d_model'],
                     'N (non-emb)': r['N_non_embedding'],
                     'val loss': round(r['final_val_loss'], 4),
                     'bits/char': round(r['bits_per_char'], 3),
                     'train secs': r['wall_seconds']} for r in recs])
…

converged 6 scales from 3 isolated kernels · 996s of training

## What the fit actually says Six models, one fixed 3.69M-token budget, capacity as the only free variable: | fit | form | result | R² | |---|---|---|---| | saturating | $L = L_\infty + a N^{-b}$ | $L = 0.173 + 6.64\,N^{-0.1073}$ | **0.99597** | | pure power law | $L = (N_c/N)^{\alpha}$ | $N_c = 2.06\times10^{8}$, $\alpha = 0.0975$ | 0.99594 | **The headline:** a clean power law emerges from six toy models — $R^2 = 0.996$, MAPE 0.74%. Our fitted exponent $\alpha = 0.0975$ lands in the same neighbourhood as Kaplan et al.'s $\alpha_N = 0.076$ for real LMs [^2], which is a genuinely striking result for a 53k–1.8M parameter char-level model on 1.1 MB of Shakespeare. **Being honest about what this does *not* establish.** Two things deserve scepticism, and the numbers say so plainly: 1. **$L_\infty$ is unidentified.** The fitted floor is $0.173 \pm 1.127$ — the uncertainty is 6.5× the estimate. Six points spanning 1.5 decades cannot pin an irreducible-entropy term. Anyone quoting "0.249 bits/char is Shakespeare's entropy" from this fit would be overreading it badly. 2. **The two functional forms are statistically indistinguishable here** (ΔR² = 3×10⁻⁵). Our data lives entirely in the regime where the additive floor is negligible, which is exactly why Kaplan's floor-free form worked for them too. Distinguishing the forms requires the data-scarce / overtraining extremes — precisely where Videau et al. (2026) show the standard additive assumption breaks, because it treats capacit…

import json, torch

# ---- COLD RELOAD: nothing from the training namespace is reused -----------------
# The d192 model was trained in kernel A, a *different Python process*. We rebuild
# it here on the main branch using only the files on disk.
BUNDLE = f'{ROOT}/models/d192_l4'
cfg  = json.load(open(f'{BUNDLE}/config.json'))
tokj = json.load(open(f'{BUNDLE}/tokenizer.json'))
args = json.load(open(f'{BUNDLE}/training_args.json'))
mets = json.load(open(f'{BUNDLE}/metrics.json'))

stoi_r = tokj['stoi']
itos_r = {i: c for c, i in stoi_r.items()}

model_r = CharGPT(cfg['d_model'], cfg['n_layer'], cfg['n_head'], cfg['vocab'])
missing, unexpected = model_r.load_state_dict(
    torch.load(f'{BUNDLE}/weights.pt', map_location='cpu'), strict=True), None
model_r.eval()
…

bundle        : /home/user/models/d192_l4
config        : {'arch': 'CharGPT', 'd_model': 192, 'n_layer': 4, 'n_head': 4, 'vocab': 65, 'ctx': 128}
trained in    : a separate kernel (branch cb010208 / kernel A)
reported val  : 1.6032 nats  (2.313 bits/char)
token budget  : 3,686,400   seed: 1337   torch: 2.7.1+cpu