PPO vs Heuristic Baseline

# Pong: PPO agent vs heuristic baseline A lightweight 2-player Pong environment (pure numpy, low-dimensional state), two agents trained/built **separately on their own branches**, then matched head-to-head and rendered. - **Variant A** — PPO policy-gradient agent trained against a noisy scripted sparring opponent (~10 min budget). - **Variant B** — heuristic tracker baseline (ball-following paddle with reaction lag), tuned by a small random search. - **Converge** — head-to-head match, win rate / rally stats, MP4 + GIF of a full game.

import numpy as np, torch, torch.nn as nn, time, json, os, math
os.makedirs('/home/user/artifacts', exist_ok=True)
torch.set_num_threads(4)

# ---------------- 2-player Pong environment (state-based, pure numpy) ----------------
class Pong2P:
    """Field is the unit square. Left paddle x=0.04, right paddle x=0.96.
    Actions per player: 0 = stay, 1 = up, 2 = down."""
    PH, PSPEED, BSPEED_0, BSPEED_MAX = 0.18, 0.035, 0.028, 0.055
    PX_L, PX_R = 0.04, 0.96

    def __init__(self, seed=0, max_steps=1500, win_score=5):
        self.rng = np.random.default_rng(seed)
        self.max_steps, self.win_score = max_steps, win_score

    def reset(self):
        self.pl = self.pr = 0.5
        self.score = [0, 0]
…

scaffold part 1 ready: Pong2P, Tracker, ActorCritic

import imageio.v2 as imageio

# ---------------- agent interfaces ----------------
class PPOAgent:
    def __init__(self, net, greedy=True): self.net, self.greedy = net, greedy
    def reset(self): pass
    def act(self, obs): return self.net.act(obs, greedy=self.greedy)[0]

class RandomAgent:
    def __init__(self, seed=0): self.rng = np.random.default_rng(seed)
    def reset(self): pass
    def act(self, obs): return int(self.rng.integers(3))

# ---------------- generic match play (agents always get canonical left-view obs) ----------------
def play_match(agent_l, agent_r, seed=0, win_score=5, max_steps=3000, record=False):
    env = Pong2P(seed=seed, max_steps=max_steps, win_score=win_score)
    ol, orr = env.reset(); agent_l.reset(); agent_r.reset()
    frames, rally_len, cur, hits_l, hits_r = [], [], 0, 0, 0
…

scaffold part 2 ready: play_match, train_ppo, render_match

## Head-to-head: PPO (left, blue) vs heuristic baseline (right, orange) Both agents were trained on their own branches above and saved to `/home/user/artifacts/`. Here we load both, play a best-of series, and render a full match.

import pandas as pd
# ---- load both trained agents from the branch artifacts ----
net = ActorCritic(); net.load_state_dict(torch.load('/home/user/artifacts/ppo.pt')); net.eval()
best_params = json.load(open('/home/user/artifacts/baseline.json'))
ppo_eval = json.load(open('/home/user/artifacts/ppo_eval.json'))
base_eval = json.load(open('/home/user/artifacts/baseline_eval.json'))
curve = pd.DataFrame(json.load(open('/home/user/artifacts/ppo_curve.json')))
print('PPO on branch A :', ppo_eval)
print('Heuristic on B  :', base_eval, '\n')

# ---- 40 head-to-head matches to 5 points (PPO = left, heuristic = right) ----
rows = []
for s in range(40):
    r = play_match(PPOAgent(net), Tracker(seed=s, **best_params), seed=7000 + s, win_score=5, max_steps=6000)
    rows.append({'match': s, 'ppo_pts': r['score'][0], 'base_pts': r['score'][1],
                 'ppo_win': int(r['score'][0] > r['score'][1]), 'mean_rally': r['mean_rally'],
                 'ppo_hits': r['hits_l'], 'base_hits': r['hits_r']})
h2h = pd.DataFrame(rows)
…

PPO on branch A : {'agent': 'ppo', 'env_steps': 1308672, 'train_s': 480.7, 'matches_won': 20, 'points_for': 100, 'points_against': 1, 'final_train_win_rate': 1.0}
Heuristic on B  : {'agent': 'heuristic', 'params': {'lag': 0, 'noise': 0.023020653255713005, 'dead': 0.04986049678946056, 'speed_frac': 0.992334135510492}, 'search_configs': 24, 'matches_won': 20, 'points_for': 100, 'points_against': 9}

import matplotlib.pyplot as plt
from IPython.display import Video, Image, display

# ---- record and render one match: PPO (blue, left) vs heuristic (orange, right) ----
m = play_match(PPOAgent(net), Tracker(seed=3, **best_params), seed=4242,
               win_score=3, max_steps=2400, record=True)
frames = m['frames'][:2400]
mp4, gif = render_match(frames, '/home/user/artifacts/pong_ppo_vs_baseline.mp4',
                        '/home/user/artifacts/pong_ppo_vs_baseline.gif', fps=60, gif_stride=4)
print(f"rendered match  score PPO {m['score'][0]} : {m['score'][1]} heuristic | "
      f"{len(frames)} frames | rallies={m['rallies']} mean rally={m['mean_rally']:.0f} steps")

fig, ax = plt.subplots(1, 2, figsize=(11, 3.6))
ax[0].plot(curve.steps, curve.rally_win_rate.rolling(20, min_periods=1).mean(), color='#5adcff')
ax[0].set_xlabel('env steps'); ax[0].set_ylabel('rally win-rate (20-iter mean)')
ax[0].set_title('PPO learning curve (vs sparring bot)'); ax[0].grid(alpha=.3)
ax[1].bar(['PPO', 'Heuristic'], [h2h.ppo_pts.sum(), h2h.base_pts.sum()], color=['#5adcff', '#ff9660'])
ax[1].set_title('Points scored, 40 head-to-head matches'); ax[1].grid(alpha=.3, axis='y')
…

rendered match  score PPO 2 : 1 heuristic | 2400 frames | rallies=3 mean rally=776 steps

## Comparison | | PPO (branch A) | Heuristic tracker (branch B) | |---|---|---| | Training | 1,308,672 env-steps in ~481 s CPU | random search, 24 configs, ~6 s | | vs sparring bot (20 matches) | 20 wins, 100–1 points | 20 wins, 100–9 points | | Head-to-head (40 matches to 5) | **40 / 40 wins, 200 points** | 0 / 40 wins, 47 points | | Rallies | mean 629 steps — long, controlled exchanges | loses points on angled returns it can't reach | **Takeaway:** the learned policy dominates the tuned reactive baseline. The heuristic is optimal-ish *reactively* (it chases the ball's current y), while PPO learns to intercept where the ball *will* be and to aim returns at angles the tracker cannot reach in time. The baseline is ~80× cheaper to produce, so it remains a good sanity opponent — but not a competitive one.

## Play against the PPO agent yourself The trained network is tiny (6 → 64 → 64 → 3), so instead of streaming keystrokes back to the kernel — which the notebook can't do at 60 fps — the weights are **exported to JSON and inlined into a self-contained HTML game**. The policy's forward pass runs in your browser, against a JS re-implementation of the exact same physics as `Pong2P`. - You are the **orange** paddle on the right — `↑`/`↓` or `W`/`S`. - PPO is the **blue** paddle on the left, playing greedily. - First to 7 points. A handicap dropdown lets you drop the agent's action rate if full speed is too strong (it is). - Playable inline below, and downloadable as `artifacts/play_vs_ppo.html` (open it in any browser, no server needed).