Web Engineering Specialist 8-10B Training Pipeline

# Web Engineering & Design Specialist — 8–10B Training Pipeline **Goal (spec §32):** an ~9B open-weight model that turns ideas into production-quality, full-stack, award-level websites — and can rebuild sophisticated components offline when needed. Built by fine-tuning a permissively-licensed 8–9B base. **Research date:** 2026-08-21 · **Approach:** QLoRA SFT (with optional continued-pretraining & DPO stages) · **Compute:** cloud GPU, sized for best value. --- ## Executive recommendation | Decision | Recommendation | Rationale | |---|---|---| | **Base model (strategic)** | **`Qwen/Qwen3.5-9B`** (Apache-2.0, 9B dense + vision encoder, 262K ctx, function calling; released Mar 2026) [^6] | Strongest permissive 8–10B option: LiveCodeBench v6 ≈ 65.6 (≈ 30B-MoE-class per model card), BFCL v3 66.1 [^6]. The **vision encoder** directly enables screenshot→code (spec §12) when paired with WebSight-style data. Caveats: hybrid Gated-DeltaNet architecture + VLM checkpoint need recent runtimes and VLM-aware training — see runbook. | | **Base model (pragmatic / executable default)** | **`ibm-granite/granite-4.1-8b`** (Apache-2.0, 8B plain dense, 131K ctx, tool calling) [^1] | Plain-dense Transformer = maximal compatibility with TRL + QLoRA + bitsandbytes today; Apache-2.0; long-context; strong code + function calling [^1]. Use this for the first end-to-end run; swap `MODEL_ID` for Qwen3.5-9B in a later stage. | | **Method** | QLoRA (4-bit NF4, r=64/α=128, all-linear) → optional full-FT…

## Research appendix (2026-08-21, three parallel research passes + verification searches) ### A. Base-model landscape (7–10B, permissive license, commercial OK) | Model | Params | License | Context | Highlights | Verdict | |---|---|---|---|---|---| | **Qwen3.5-9B** [^6] | 9B + vision | Apache-2.0 | 262K | LiveCodeBench v6 ≈ 65.6, BFCL v3 66.1, function calling, GGUF/AWQ/MLX; hybrid Gated-DeltaNet | **#1 strategic** (vision = design-to-code) | | **Granite 4.1 8B** [^1] | 8B | Apache-2.0 | 131K | Plain dense, tool calling, SFT+RL post-trained, official GGUF/FP8 | **#1 pragmatic** (easiest to fine-tune) | | GLM-4.5-Air | 106B total (MoE) | MIT | — | Too big for the 8–10B target | ✗ size | | Qwen3-Coder family | — | Apache-2.0 | — | No 7–10B dense member in the 8–10B window | ✗ range | | Devstral-Small | 24B | — | — | Above target | ✗ size | | Phi-5 | 14B | MIT | — | Above target | ✗ size | | Gemma 3 / 4 | 8B class | Gemma license | — | Output-use + use restrictions | ✗ not permissive | | Llama 3.1-8B / 4 Scout | 8B class | Llama community | — | 700M-MAU cap + attribution clauses | ✗ not permissive | | LFM2.5-8B-A1B | 8B MoE | LFM Open License | — | $10M revenue cap | ✗ not permissive | ### B. Training-data catalogue (verified licenses; see runbook for gating) | Dataset | Size | License | Format | Fit | |---|---|---|---|---| | **WebSight v0.2** | 1.92M screenshot→HTML pairs | CC-BY-4.0 | vision+text | SFT: design-to-code (needs VLM path) | | **WebGen-Instruct** | ~6.7k instru…

# %% Cell 3 — Environment setup (run once)
# Installs the 2026 training stack: transformers / datasets / accelerate / peft / trl / bitsandbytes.
# NOTE: bitsandbytes 4-bit quantization requires CUDA. On a CPU sandbox this still installs,
# but the pipeline automatically falls back to an unquantized tiny model (see Cell 7).

import subprocess, sys

def pip(pkgs):
    r = subprocess.run([sys.executable, "-m", "pip", "install", "-q", *pkgs],
                       capture_output=True, text=True, timeout=1800)
    if r.returncode != 0:
        print(r.stderr[-3000:])
    return r.returncode

for pkgs in (["transformers", "datasets", "accelerate", "peft", "trl",
              "bitsandbytes", "sentencepiece", "protobuf"],):
    pip(pkgs)

…

flash-attn available: False

# %% Cell 4 — Configuration
# smoke=True  -> CPU-safe end-to-end proof (135M model, 1 step, tiny data)
# smoke=False -> real training on cloud GPU (9B QLoRA). Flip once on the GPU box.

import os, torch
from dataclasses import dataclass

@dataclass
class TrainConfig:
    # --- model --------------------------------------------------------------
    model_id: str = "ibm-granite/granite-4.1-8b"   # pragmatic default (Apache-2.0, plain dense) [^1]
    # strategic alternative: "Qwen/Qwen3.5-9B"     # vision encoder, 262K ctx [^6] — see runbook for
    #                                               VLM/DeltaNet caveats before switching.
    # --- smoke mode ----------------------------------------------------------
    smoke: bool = True
    smoke_model: str = "HuggingFaceTB/SmolLM2-135M-Instruct"
    smoke_steps: int = 1
    smoke_max_examples: int = 12          # per data source
…

smoke: True | model: HuggingFaceTB/SmolLM2-135M-Instruct
max_seq_len: 2048 | LoRA r/a: 64 128

# %% Cell 5 — Data assembly
# Mixes (a) the curated local Aurelia SFT samples with (b) permissively-licensed HF sources,
# normalizing everything into chat messages [{"role": ..., "content": ...}].
# License-safe by construction: every registry entry is Apache-2.0 / MIT / CC-BY-4.0.
# Spec §26: quality over size — dedupe, drop empty rows, cap per source.

import glob, json, os, re
import itertools as it
from datasets import Dataset, load_dataset

# ---- (a) local curated samples (staged from outputs bucket: sft_samples/) ----
# Accepted formats:  sample_N.json  ({"prompt","response"})  OR
#                    sample_N_prompt.txt + sample_N_response.md  (hand-authored pairs)
def load_local_samples(d):
    rows = []
    for p in sorted(glob.glob(os.path.join(d, "*.json"))):
        try:
            s = json.load(open(p, encoding="utf-8"))
…

local-aurelia: 5 curated samples

# %% Cell 6 — Tokenization with response-only loss
# Template-agnostic: works with ANY chat template (Granite, SmolLM2, Qwen...).
# labels = -100 on the prompt prefix -> the model only learns from assistant turns (spec §25).
# Caveat (known): prefix-length alignment assumes prompt tokenization == prefix of full
# tokenization; rows where that breaks (rare, truncation edge) are dropped, not corrupted.

from transformers import AutoTokenizer, DataCollatorForSeq2Seq

MID = CFG.smoke_model if CFG.smoke else CFG.model_id
tok = AutoTokenizer.from_pretrained(MID)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

def tokenize_one(msgs):
    prompt = tok.apply_chat_template(msgs[:-1], tokenize=False, add_generation_prompt=True)
    full   = tok.apply_chat_template(msgs,      tokenize=False)
    ef = tok(full,   truncation=True, max_length=CFG.max_seq_len)
    ep = tok(prompt, truncation=True, max_length=CFG.max_seq_len)
…

tokenized 41 samples, dropped 0 (truncation/alignment edge)
train: 38 | val: 3 | max_seq_len: 2048

# %% Cell 7 — Model + LoRA load
# GPU  : 4-bit NF4 QLoRA with double-quant + bf16 compute (bitsandbytes; CUDA required).
# CPU  : smoke fallback — tiny model in FP32, NO quantization (bitsandbytes is CUDA-only;
#        bf16 CPU matmul is very slow, so fp32 is the right smoke dtype).
# LoRA : all-linear (auto-detect q/k/v/o/gate/up/down) — no hard-coded module names.

import torch
import transformers
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig

def _dtype_kw(dtype):
    # transformers 5.x renamed torch_dtype -> dtype; keep both majors working
    v5 = int(transformers.__version__.split(".")[0]) >= 5
    return {"dtype": dtype} if v5 else {"torch_dtype": dtype}

def load_model():
    mid = CFG.smoke_model if CFG.smoke else CFG.model_id
…

CPU smoke: loading FP32 WITHOUT quantization (4-bit QLoRA needs CUDA)