The agent was labeling blind

We spent a week convinced we had a model problem.

We had built a labeling agent: point it at a few thousand frames, and it segments the objects, draws the boxes, and — this was the part we were proud of — renders its own work back as an overlay image and looks at it before committing, so it can catch itself when it gets one wrong.

It kept getting them wrong. Confidently. It would report that it had checked the overlay and that the masks looked clean, and the masks did not look clean. Boxes clipped through objects. Whole instances missed. And the agent's own verification step sailed through every time, in a calm and well-calibrated voice, telling us everything was fine.

First we assumed it was the model

This is the part worth admitting, because it burned most of the week.

We rewrote the verification prompt to be more specific about what to look for. We swapped the model. We swapped it again. We added a rubric, then a second pass to check the first pass. We built a small eval harness so we could measure whether any of it helped.

None of it helped. Which, in hindsight, was the loudest signal we got — the interventions were uncorrelated with the outcome, and an intervention that does nothing at all is telling you that you are not touching the actual variable.

The overlay never arrived

The thing that broke it open was mundane. Someone asked to see the exact overlay image the agent had looked at on a frame it got wrong.

There wasn't one.

Not a bad overlay, not a corrupted one, not one with the boxes in the wrong place. The message that was supposed to carry the image from the kernel to the agent had never arrived. The agent had not been checking a bad overlay. It had been checking nothing, finding nothing wrong with nothing, and reporting success.

iopub_data_rate_limit

Jupyter's IOPub channel — the one that carries a kernel's outputs back to whatever is listening — has a default rate limit of 1 MB/s.

A base64-encoded PNG of a labeled frame is comfortably over that.

When you exceed it, the messages are dropped. Not queued, not retried, and — this is the part that cost us the week — not raised. The kernel believes it printed. The client receives nothing. There is no exception on either side of the wire, and nothing in the agent's context where the image should have been. It has no way to notice an absence.

# jupyter_server_config.py
c.ServerApp.iopub_data_rate_limit = 1_000_000  # bytes/sec — the default
c.ServerApp.rate_limit_window = 3.0

A silent drop is worse than a crash

If the image transport had thrown, we would have fixed this in an hour. A stack trace names its own cause and points at the line.

Instead we had a system where every component behaved correctly under its own local contract. The kernel wrote to the channel. The channel enforced its documented limit. The agent reasoned over the context it was given. The failure existed only in the gap between them, and every individual piece could pass its own tests.

And the agent's behaviour under the failure looked exactly like a model problem. Asked to verify an image it could not see, it did not say "I cannot see an image" — it produced a plausible, fluent, confident verification, because that is the shape of the answer that was being asked for. We read that output and concluded the model was bad at looking at pictures. It was never looking at pictures.

When an agent produces confident, wrong output, the instinct is to reach for the prompt. Check whether it received its inputs first. The prompt is the most visible variable and it is very rarely the one that is broken.

What we changed

We raised the limit in the sandbox's Jupyter config, and the quality of the labels changed immediately and dramatically — same model, same prompt, same dataset. The only thing that had changed was that the agent could now see.

Then we went to verify the fix had actually landed everywhere, and found the more embarrassing half of the story: it had only reached one of our two sandbox images.

We run two sandbox images: a GPU one on Modal, and a CPU one on E2B. The CPU one is what you get on the free tier, which is to say it is what most people get.

The GPU image is built in code, so it picked the config change up on the next deploy. The CPU image is baked from a template by a command someone runs by hand — a command our deploy pipeline has never run. So the commit was merged, the pull request was closed, the fix was "shipped", and our default runtime carried the bug for four more days, because a config file in git is not a config file in production until something rebuilds the image.

We found that only because we went looking. Nothing told us.

That is the same failure as the one we had just spent a week on, wearing a different hat: a change that appears to have happened, in a system with no mechanism to say otherwise. A config file in git is not a config file in production until something rebuilds the image — and if nothing in your pipeline rebuilds it, merged and shipped are different words that look the same on a dashboard.

So the template build is now part of the deploy. It rebuilds on any change to the image, and then it does the thing we should have been doing all along: it spawns a real sandbox off the template it just published, reads the config back off the running box, and fails the build if the setting isn't there. The build log is not evidence. The running machine is.

What we'd tell you to check first

If you are debugging an agent that produces confident, wrong output on a task that involves looking at something:

• Log what the model actually received, byte for byte, not what you believe you sent it.
• Look for limits that drop rather than fail — rate limits, truncation, size caps, buffer ceilings. They are common in every message transport and none of them raise.
• Be suspicious when your interventions have no effect. It usually means you are tuning a variable that is not in the causal path.
• Check the transport before you tune the prompt.

The model was fine the entire time. It was the pipe.