Phase 2 · complete
The layer your finding actually lives in
Everything the dissertation concludes depends on one narrow question: can the model's proposed edit be turned into a patch that applies? This phase rebuilds that machinery properly, and then measures it against 61 patches your model really wrote.
The retrofit, as a picture
This is the change that produced your result, so it is worth being precise about what moved.
Rebuilt from your own script, bugs included
The parser was ported from day11b_diagnose_revise_searchreplace.py
— the script that produced your retrofitted numbers. Its behaviour had to be
preserved exactly, because changing it would change what the archived results
mean. Three genuine defects were found in the process, and each is now a test:
| Defect | Effect |
|---|---|
A replacement block containing ======= broke the pattern match | Valid edits were rejected as unparseable |
| An empty search block was counted as matching everywhere | Reported as “ambiguous” rather than “empty” — the wrong diagnosis |
| A block opened but never closed produced zero blocks | Indistinguishable from the model proposing no edit at all |
A fairness rule, enforced by a test
The next study compares the two edit formats head to head. That comparison is only about the models if both formats are extracted from the model's reply with identical tolerance. If our reader forgave markdown code fences for one format and not the other, we would be measuring our own code and calling it a property of the model — exactly the confound the thesis criticises elsewhere.
Models habitually end with a sentence like “This should resolve the failing test.” Our diff reader was absorbing that sentence into the patch as if it were code, which made the patch's line counts wrong — producing a malformed verdict the model had not earned.
Left alone, this would have biased the comparison against unified diffs using our bug, and the result would have looked like a finding. The fairness test found it; reading the code had not.
Then: measure it on real output
With a tested parser, the 61 archived patches from phase 0 could finally be judged. Structurally, with no test execution, no Docker and no API calls:
The number we nearly published was wrong
The first attempt at this audit reported 3 of 51 clean — and 0 of 10 for the programmatic era.
The second number is impossible. Those patches were generated by a library that cannot make counting errors, and several are recorded as having applied successfully. A 0% score could only mean the analysis was broken.
It was. Splitting the patch text into lines leaves one empty element at the end, and counting it as a line of content inflated every measurement by exactly one — flagging every patch ever written.
The control caught it. Reviewing the code had not.
This is why the audit script now refuses to print a number if any programmatically generated patch shows a defect, and writes a warning instead. A result that cannot be contradicted by anything is not yet a result.
The corrected figures — 30/51 and 10/10 — come from the tested parser with 138 tests around it, not from a throwaway script. The full audit is committed to the repository, so it can be checked on any machine, even one without the original archive.