The agent retracted its own result

The most useful thing Clusy did for us last week was not getting the right answer. It was getting a wrong one, being challenged, and withdrawing it.

The setup: a demo with a computer-vision startup doing production CV on a live manufacturing line. The task is to take a camera frame of a 3D-printer bed and the print plan, and segment only the objects that should be visible in that frame. We were in calls all day, so we forwarded Clusy the client's raw email and their ZIP file, and left.

The whole session is public, and you can fork it: Auto-labeling: visible object masks.

It labelled the dataset

It came back with all 293 frames labelled.

The pipeline it built is worth being precise about, because the naive summary is wrong. It did not just threshold on colour, and it did not just prompt a foundation model either. It used HSV saturation thresholding plus geometry filters — aspect ratio, extent, an edge margin, a bed crop — to find candidate objects, and then SAM 3 prompted segmentation to turn those candidates into actual masks.

That distinction is the whole ballgame here, because the red material in these frames is not the client's production filament. A pure colour rule would have keyed on a property that will not exist in deployment. What it built keys on colour only to propose regions, and lets a segmentation model decide what the object actually is.

It also iterated on its own filters. An early version of the labeller had a precision around 0.46 — over half its mask area was wrong, mostly tubes and wires being picked up as parts. It added a brightness floor and tightened the geometry, which killed the tube and wire blobs and took precision to roughly 0.70–0.72.

It found three things wrong with the data

The interesting part is what it noticed while it was in there — none of which we asked about.

The print bed moves. It shifts 100–200px relative to the camera between frames. Anyone who assumed a fixed transform from plan coordinates to image coordinates — the obvious first approach, and the one we would have tried — would have spent a day building something that cannot work.

Auto-exposure drift changes about 85% of pixels between consecutive frames. Which kills frame differencing, the other obvious approach. Two of the three things you would naturally try are dead on arrival, and it established that before we would have finished setting up the first one.

Eight of the "images" were 404 pages saved as .jpg. The dataset had holes in it, and the holes were shaped like JPEGs.

These are the things a good researcher notices in the first hour and a careless one discovers in week three.

Then it reported a result that was too good

It trained a U-Net to predict the masks directly, and ran an ablation over five configurations: RGB alone, RGB plus the bed plan as an extra input channel (filled, and as contours), each with and without hue augmentation. 293 frames, split 205 train / 44 validation / 44 test, with held-out hue-shift tests at ±60°, ±120° and ±180° to probe robustness to camera colour drift.

And it came back with a dramatic finding: under a +60° hue shift, the models collapsed to an IoU of 0.0000. Not degraded — zero. Predicted mask: 0 positive pixels out of 65,536. A clean, alarming, publishable-sounding result about colour robustness being an architectural property.

We pushed back. It was too clean.

It re-derived the experiment and found the flaw itself. The collapsing models had been trained with roughly ±30° of hue augmentation and evaluated at +60° — outside any range they had ever seen. They had not revealed a colour vulnerability. They had been tested on a distribution they were never trained for.

It withdrew the conclusion.

The models trained with the full ±180° augmentation hold 0.715–0.731 IoU across every colour shift tested. The collapse was a training-budget artifact. There was no vulnerability.

The result that survived

With the artifact cleared away, the same ablation produced something that was real — and that nobody wanted:

Run	Conditioning	Hue aug	Test IoU
A	RGB only	none	0.7384
C	RGB + plan (contours)	yes	0.7352
B	RGB + plan (filled)	none	0.7252
D0	RGB only	±180°	0.7152
D	RGB + plan (contours)	±180°	0.6707

Giving the model the print plan makes it worse. In every matched comparison, RGB-only beats RGB-plus-plan. Under aggressive hue augmentation it is not close: D0 beats D by 0.04–0.05 IoU at every colour shift tested.

This is a genuinely useful negative result, and it is the opposite of what everyone expected — the plan is the extra information, and concatenating it as a channel is the obvious way to use it. It says the obvious way is worse than not using it at all. The plan contours add noise, not signal. If the plan is going to help, it has to enter the model some other way.

That finding saves a team weeks, and it only became visible once the exciting wrong result was out of the way.

Why the retraction is the story

An agent that is right most of the time is a tool you have to check. An agent that can be argued out of a seductive, incorrect result is a tool you can work with.

The failure mode that matters is not the model being wrong. It is the model being wrong fluently — producing a confident, well-structured, plausible write-up of a result that does not exist, in a register that makes it expensive to doubt. That is what quietly costs a research team a quarter.

What we want is an agent that treats its own output as evidence rather than as a conclusion: one that re-derives when challenged, checks what actually varied between conditions, and is willing to say the previous message was wrong.

It was wrong. We said so. It checked, agreed, and told us why.

It is also, to its credit, honest about what it has not done. Its own closing note says no model here has had a reload-from-checkpoint and inference smoke test, and should not be treated as deployable until it has.

The shape of the pipeline

The loop in the diagram at the top is where we have landed for labelling work:

1. Auto-label — propose regions cheaply, segment them with a promptable model.
2. Review, in the notebook. The agent hands the labels back as a review cell and blocks. You keep, fix, or drop each mask.
3. Fine-tune on what survived review.
4. Loop. The model you just trained does the next pass of labelling, and the human reviews a better set of guesses.

Step two is the one people skip, and it is the one that decides whether any of this works. An auto-labeller at 0.70 precision is not a labelled dataset. It is a labelled dataset with a 30% lie in it. The human is not there to be a bottleneck — they are there because the model cannot tell you which of its answers were wrong.

Which is the same lesson as the retraction, arriving from the other direction.