Everything below actually happened, against a live API, on July 10, 2026. The recordings are committed to the Reflight repo — every run in this post replays offline, byte-identical, for $0.00.
The setup: a scheduling agent (gpt-4o-mini, function calling) with three tools — get_today, check_availability, book_meeting — and one honest task: book a 45-minute design sync next Wednesday at 3:30pm, and if that slot conflicts, take the next free slot that afternoon. The calendar fixture anchors “today” at Friday, July 10 — so next Wednesday is the 15th, the requested 15:30 conflicts, and the only correct answer is 17:15.
I ran it five times through Reflight's N-run executor. Then, because the result was hard to believe, ten more. Total spend for fifteen live runs: under a cent.
What happened
Fifteen out of fifteen runs booked July 12 — a Sunday — at 15:30, and confirmed it to the user as “Wednesday, July 12th.” The recorded transcript makes the failure exact. The agent did call get_today first, did receive the correct date — and computed “next Wednesday” as July 12 anyway:
Here is the uncomfortable part: every tool-level check passed. The wrong day has an empty calendar, so availability said yes. The booking succeeded. No tool errored, no loop, no crash. The rule classifiers — correctly — found nothing. Fifteen green verdicts, 100% pass rate, one distinct answer. The agent isn't flaky. It is consistently, confidently wrong, which no amount of retry-and-compare can surface.
What caught it — and what almost didn't
Two layers, both run on the recordings after the fact — no re-execution, no additional agent spend.
The LLM judge (gpt-4o-mini judging itself) read the transcripts and flagged the failure at 0.90 confidence — “the agent incorrectly stated the date of the meeting as Wednesday, July 12, 2026, when it is actually a Sunday.” Impressive. Except across three batches of the same failure, the same judge caught 5 of 5, then 3 of 5, then 1 of 5. A judge is a probabilistic net — cheap, useful, and exactly as nondeterministic as the agents it judges. Measuring that variance took thirty seconds precisely because the runs were recordings: re-judging is free re-reading, not re-running.
A deterministic assertion — fifteen lines of Python that read each recording and check the booked slot against ground truth — caught every run in every batch, and folded the verdicts under one shared failure signature: wrong_slot ×15, same bug, every run. Encode ground truth once, and every future recording — CI runs included — gets checked against it for $0.00.
Why this needed a flight recorder
- The failure would otherwise be a support ticket. “Agent booked the wrong day,” from a user, days later, with nothing to inspect. Instead it's fifteen committed recordings that reproduce the exact moment, offline, with the network hard-blocked.
- Pass rates lied; the recording didn't. Every metric short of reading the transcript said this agent works. The transcript is the only place the bug exists — which is an argument for keeping every transcript.
- Recurrence turned fifteen anecdotes into one bug. A shared fingerprint groups every run under a single finding. One defect, not fifteen incidents.
The layering is the actual thesis of Reflight: the judge and the assertion are both just consumers of the same recording. An open, replayable format is the substrate; detectors, evals, and regression tests are things you run on top of it — as many times as you like, for free.
Footnote: what the microscope caught in itself
Running this study also surfaced two real bugs in Reflight: the pricing table had no OpenAI models, and cost computation didn't read OpenAI's usage keys — so the first batch of live runs ingested at $0.0000. Both fixed in the same commit. A case study that finds bugs in the microscope too is a good day.
Reflight is open source at github.com/pauti04/reflight, with the full case study, recordings, and the 15-line assertion in the repo. The hosted demo replays real recorded runs in your browser.