Phase 3 · complete · the milestone that mattered

Your Chapter 5, rebuilt from raw logs, in one command

Every table and figure in Chapter 5 was produced by a script that read your log files. Phase 3 writes a second, independent program that reads the same logs — and checks whether the two agree, character for character.

Cells recovered51
Previously invisible22
Tests191
Statedone

Two commands

$ bun run replay
replay — 51 cells from archive/legacy_runs

  test-verified (report.json)     42
  apply-verified (no report)       9   of which APPLY_FAIL: 9

  baseline resolved            12/20
  failure pool                     8   evaluable: 7
  recovered                      3/7
    logic-class                  0/4
    format-class                 3/3

  failure pool matches declared   yes
  failure class matches hardcoded yes

$ bun run report
  byte-identical to the committed master_table.md

That last line is the whole point of the phase.

A program written from scratch, months later, in a different language, reading the same evidence, produced your submitted table with not one character different. Your Chapter 5 is now reproducible rather than merely written down.

The 22 cells nobody could see

The first thing the new reader noticed was an arithmetic problem. The archive holds 95 evaluated task folders and 95 patch files — but only 73 result files.

The reason is a detail of how the evaluation harness works. When git apply refuses a patch, the harness throws an error before it writes its report. Your table-building script looks for report files. So those 22 attempts — every one of them a patch that failed to apply — were invisible to every table you produced.

THE OLD READER — COLLECTS RESULT FILES 73 seen 22 invisible THE NEW READER — WALKS TASK FOLDERS 73 tested 22 failed A missing result file is not missing data. It is the strongest possible evidence that the patch never applied — which is exactly the failure class the thesis is about. Of the 22, nine fall inside runs the thesis counts; the rest are in excluded smoke tests.
This is the defect the whole pipeline was designed against — “a failed attempt leaves no record” — discovered in the project's own evidence, not argued in the abstract.

Those cells can never be mistaken for successes. Each is marked as verified by patch-application only, its “resolved” field is forced to empty, and the function that computes a success rate will not accept them — the safeguard built in phase 1, doing its job on real data.

The classification you made by hand, checked

Your script begins with a promise: “Nothing in this script invents a number. Every cell is read from a report.json.” It keeps that promise for 38 of its 42 records. The four format-class rows are written from a list typed by hand, because — as we now know — no result file exists for them to read.

So phase 3 derives the logic/format split from the outcomes themselves and compares it with your list:

DERIVED FROM OUTCOMES — COMPARED WITH THE HAND-WRITTEN LIST Logic-class — patch applied, tests failed astropy-7746match django-11019match django-11283match django-11564match Format-class — patch never applied astropy-14182match django-11039match django-11583match django-11620match
All eight, both classes, no discrepancy — and the derived failure pool matches failure_pool.json, which is a third independent copy of the same list.

Your judgement was right. What changes is its status: a constant that nothing could contradict is now a result with a test attached.

Proving the check can actually fail

A test that always passes proves nothing. So after everything was green, we deliberately broke the analysis in two ways and confirmed the suite noticed.

Sabotage introducedResult
Score the baseline out of 16 (the tasks with results) instead of 20 — the survivorship error3 tests failed
Go back to collecting result files, dropping the 22 — the original blind spot9 tests failed

Both were then reverted, and all 191 tests pass again. The check has teeth.

What was deliberately not done

The .drawio figure sources and their exported images are not regenerated. Those images are already embedded inside Self.docx, which is final and cannot be re-exported — so rewriting them could only create a difference between the repository and the document you submitted.

Instead, the data behind figures 5.1 to 5.4 is now computed and committed as a file, and tested against the numbers typed into the figures. The pictures stay exactly as submitted; the numbers in them are now checkable. One quantity resisted: the 1× / 2× / 3× input-cost comparison in figure 5.4. The archive contains no token records at all — the original never wrote any — so that figure's data is carried as a labelled input from your configuration notes, marked in the file as not measured by this pipeline. Phase 4 is what fixes that going forward.

Where this leaves the project

Four of nine phases were done at this point, and the repository can now do something the original never could: reproduce the entire results chapter, offline, on any machine, in about three seconds — with no API key, no network and no Docker.

Next was phase 4: the first live model call, with a token budget that is enforced rather than merely written down, and usage recorded per attempt. It found four things wrong with the endpoints we thought we understood.

← Phase 2 · Edit layer What changed since submission →