Phase 1 · complete

A vocabulary strict enough that the old mistakes cannot recur

The original study lost eight of twenty-four cells because a failure could quietly produce nothing. This phase makes that impossible — not by being careful, but by making silence fail to compile.

Every attempt must end in one of nine outcomes

In the original code, an attempt whose patch was rejected printed Not applied and moved on, writing nothing to disk. The cell had been run, had failed, and left no trace — so the results table had a hole where a failure should have been, and the hole was indistinguishable from a task that was never attempted.

Now every attempt terminates in a named, recorded state:

MALFORMED carries a reason rather than standing alone — eight of them, such as “the quoted text was not found” or “the hunk's declared size does not match its contents.” Without that, every failure looks the same in the results, and a model that misplaced one space scores identically to one that produced gibberish.

The rule that stops a resolve rate being fabricated

This is the most important idea in the phase, and it is worth understanding because it is what keeps the old and new work honest side by side.

Your original results were verified by running the tests in Docker. The new pipeline deliberately does not use Docker at all — it verifies that a patch applies, which is a weaker but far cheaper check that needs no containers and no keys.

Conditions live in one config file, not in thirty scripts

The original experiment's conditions were spread across separate Python files — day11b_diagnose_revise_searchreplace.py and twenty-nine siblings. To know what a run did, you had to read the script, and the script was edited between runs.

Now there is one settings file. Every model, strategy, edit format and budget is declared there, and the file is hashed. If any setting changes — even a comment — the hash changes, so two runs claiming the same configuration can be proven to have used it.

A pre-flight check that fails loudly and early

One command, doctor, runs every precondition before anything expensive happens: the config parses, the dataset has its 300 rows, sampling settings are pinned, API key names are present, and the endpoints respond at a measured speed rather than an assumed one.

One gate is unusual and deserves mention. The pipeline generates unified diffs with a library rather than asking the model to write them — that is the whole mechanism of your retrofit. So doctor proves, against a real known-good patch, that git apply accepts what the library produces. If that ever stopped being true, every result would be wrong, so it is checked before any other code is allowed to run.

What this phase measured about git

Building that gate meant testing what git apply actually tolerates. The answer was more forgiving than expected, and it changed how the thesis's central mechanism should be described — that finding is written up on the findings page as item 2.