My Agent Reported Evidence It Never Captured

There is one rule in this pipeline I care about more than the rest: reproduce the defect before changing any code, verify the fix afterwards, and record both. The recording is what makes a run checkable by someone who wasn't watching it happen.

A setting in my configuration said video recording was on. The skill I had written said it happened automatically. Then a single run drove 144 browser calls and produced no recordings at all.

The symptom I logged twice

The uncomfortable part isn't the failure. It's that my own notes had already caught it. Twice.

The first time, a run finished and there was no video file. What I wrote down was a workaround: check that the file exists before claiming a video exists. A week later, the same thing — an evidence run that yielded stills and nothing else. Again I wrote down advice for next time: don't put "video attached" in a pull request body, because you will be retracting it.

Twice I documented how to live with it. Neither time did I ask why it was happening. For the best part of two weeks, the setting said one thing and the filesystem said another.

An accepted quirk is a bug you have stopped counting.

What it cost, directly: a pull request body pointing at evidence that did not exist, corrected after the fact, and a full run's worth of browser driving that produced nothing anyone could review. Indirectly, and worse: for two weeks, "verified in the browser" was a claim with nothing behind it — on every run, not just that one.

What finally broke the pattern was the size of the run in the opening. 144 browser calls is a great deal of evidence to not have, and an empty directory after a run that long cannot be filed under quirk.

Why nothing errored

The cause was not subtle, once I looked. My configuration passed two command-line flags to the recording server. Neither flag exists in that tool. So the server did exactly what it should: it printed unknown option and exited at startup.

That should have been the end of the run — no browser, an obvious failure, a loud stop. Instead everything proceeded normally, because a second browser provider was installed alongside mine. It starts a bare server with no recording configured. When the agent asked for a browser, it got one.

The two providers exposed the same tools under different name prefixes. Capability present, evidence absent, and no error anywhere in between.

A stand-in that answers is worse than none

Redundancy is usually a virtue. Here it removed the only signal I had.

If exactly one component can serve a request, its absence is an error you can see. When two can, a dead one becomes invisible: the request still returns something, so nothing upstream has any reason to complain. The failure wasn't the invalid flag. It was that something else was willing to answer.

That shape is specifically dangerous with an agent in the loop, because nothing in my workflow required the model to care which provider had answered. The distinction was visible — the tool names carried different prefixes — but no step asked it to look. It requested a browser, it got a browser, and it truthfully reported having driven one. Then it wrote a summary mentioning evidence, because the instructions I had given it said evidence was being recorded.

What I changed

• Recording is only configurable through the server's own config file, and that file is read once, at startup. So the launcher writes it per run, before the session begins, pointing the recording at this run's own directory.

• The competing provider is disabled for this project, so it can never quietly stand in again.

• Preflight asserts that every flag in my configuration actually appears in the tool's own help output, that the config file exists, and that recording is enabled whenever the setting says it should be. Its exit code is the number of failures — capped, obviously. Matching against help text is a heuristic and it will drift between tool versions, but it catches the failure that actually happened.

• A test locks that behaviour, and the skill now tells the agent to check which provider answered and to stop if it's the wrong one — so the diagnosis lives in the run rather than only in my head.

The preflight check is the part I expect to generalise. The others fix this incident; validating my configuration against the tool's actual interface is what catches the next one.

What I couldn't fix

Traces aren't obtainable on this path at any setting — what the tool writes is a session log, which is not a trace. There was no clever workaround available, so the honest change was to delete the claim: the skill now says never write "trace attached" on a ticket or a pull request. Hardening a claim sometimes means removing it.

What I took from it

A capability that reports success without producing its artifact is worse than one that fails outright.

Three things agreed that recording was working: the setting, the skill, and the run's own summary. But they were never three sources. They were one assumption, echoed three times — and correlated confirmations are not independent ones. Not one of them was looking at the filesystem. Configuration that nothing reads is a lie you will believe, precisely because you wrote it yourself.

Which is what the title is really about. The agent reported evidence it never captured because I had, in effect, told it to.

An agent's self-report is not a measurement.