The page to read before a viva
Four things we learned after the document was finished
None of these change Self.docx, and none of them make anything in
it wrong. Two add evidence you did not have. One sharpens how the central
mechanism should be described. One confirms a judgement you made by hand.
The short version, if you read nothing else.
Your numbers are right — an independent re-implementation reproduced your Chapter 5 tables exactly. What is new is that the format-class finding can now be measured on 61 real patches instead of argued from 4, and that the reason bad diffs fail is more specific than the document says.
Where this work came from
Worth stating plainly before the findings, because it frames all four of them.
The dissertation is yours. The research question, the experimental design, the runs, the analysis and the writing are your work, submitted and final.
The software described on these pages is not part of it.
recovery-bench was built after submission by Aayush,
working with an AI assistant, as follow-up work on the agenda your Chapter 6
sets out. It has never been submitted for assessment and is not offered as
your own coding.
What it contributes is verification: an independent check that the numbers in your Chapter 5 follow from your logs, plus additional evidence drawn from data your own experiments produced.
That distinction is a strength rather than something to manage. Independent reproduction is worth more precisely because the person doing it was not the person who produced the original result — a check you run on yourself is a weaker check. Say it early and it is a methodological point in your favour. Leave it to be discovered and it becomes a different kind of conversation.
1 · The finding now rests on 61 patches, not 4
Your archive turned out to contain something nobody had noticed: the
predictions*.json files preserve the actual patch text the
model wrote for every instance it touched. Sixty-one of them. That is
the only surviving trace of the model's raw output, since the per-attempt
folder was deleted long ago.
Running all 61 through a tested validator separates them by era — the ones the model wrote as unified diffs by hand, and the ones generated programmatically after your retrofit:
If asked why this matters: the thesis argued from four tasks that the failure was mechanical. This shows the same mechanism operating across 51 independent generations, and shows the fix working on all 10 it was applied to.
2 · “The model cannot count lines” is too coarse
The document explains format-class failure as the model being unable to compute
line numbers correctly. We tested that against real git apply, and
it turns out git is far more forgiving than that framing assumes.
| What the patch got wrong | git's verdict |
|---|---|
| Hunk claims to start at line 999 in a 12-line file | accepted — git rescans for the context |
| Prose or junk text before the file header | accepted |
| Declared line counts disagree with the hunk body | rejected — “corrupt patch” |
| The quoted surrounding source does not match the real file | rejected |
| No file header at all | rejected — nothing to patch |
The decisive weakness is not arithmetic in general — it is reproducing surrounding source code exactly, and keeping a hunk's declared size consistent with its contents. Six of the 51 real diffs had no file header whatsoever, which is not an arithmetic error of any kind.
This makes your conclusion stronger, not weaker.
SEARCH/REPLACE removes both failure modes at once:
the model quotes a snippet instead of reconstructing context, and a program
does the counting.
3 · Twenty-two evaluations were invisible to your tables
The evaluation harness writes a report.json for each task. Your
table-building script collects those files. But when git apply
refuses a patch, the harness raises an error before writing the
report — so those runs left behind a patch file and a log, and nothing
the script could find.
The replay now reads directories rather than report files, so those cells exist again. They can never be counted as successes: the type system marks them as “applied-check only, never tested”, and the function that computes a resolve rate refuses to accept them.
4 · Your logic/format split was correct — and now something checks it
Your script says at the top: “Nothing in this script invents a number. Every
cell is read from a report.json.” That holds for 38 of its 42 records. The
other four — the format-class rows — are written from a list typed by hand, and
the count 4/20 is a fixed piece of text.
That was unavoidable. An apply failure produces no report, and the script reads nothing else, so it could not have derived them. The evidence was in the logs the whole time.
So we derived the classification independently, from outcomes, and compared:
failure pool matches declared yes
failure class matches hardcoded yesAll eight tasks, both classes, no discrepancy.
The hand-written judgement was right. What is new is its status: a constant
that nothing could contradict has become a result with a test attached. The
same now applies to the whole table — a second, independently written
implementation regenerates master_table.md
byte-for-byte.
Two smaller corrections, for completeness
-
SWE-bench Lite spans 12 repositories, not 11. The missing one
was
pallets/flask, with 3 instances — which happens to be exactly the floor the sampling plan was written against. - An early note claimed the original code had never been version-controlled. That was wrong; the check had been run in the wrong folder. All 30 scripts and 427 log files are in your repository. The note is struck through rather than deleted, so the error stays visible.
5 · The reasoning setting was never actually sent
Every model in the configuration is marked thinking: false. When
phase 4 made the first real API call, it turned out that setting had
never been transmitted — the request simply did not mention
it, so the service applied its own default, which is thinking on.
This was true of your original runs too.
Measured on a one-word question, the cost difference is large, and one model refuses to comply at all:
| Model | Default | Thinking off | Effect |
|---|---|---|---|
nemotron_ultra | 38 | 2 | 19× cheaper |
nemotron_nano | 49 | 3 | 16× cheaper |
muse_30b | 84 | 41 | still thinking |
gemini_flash | 93 | 0 | obeys, different setting name |
muse_30b is the one that matters for the argument. A
Reflection-only condition is only meaningful if the arm it is compared
against is not reflecting — and on that model, asking for no thinking still
leaves it thinking. Phase 4 makes the check report
this rather than pass over it.
This is the same shape of defect as the token ceiling written
8_000_000, which YAML read as text so that no comparison against
it ever did anything: a setting that existed, was believed to be in force, and
was never actually applied. Both now have tests that fail if they regress.
6 · A “success” response that contained nothing
Probing four NVIDIA models at once returned, for two of them, an HTTP
200 success with an entirely empty body — while a request
beside them returned the honest form of the same condition,
503 ResourceExhausted: Worker local total request limit reached.
Run one at a time, both models answer in under half a second.
Worth knowing because of what it would have caused. An empty success reaching the full experiment would have been recorded as the model produced no output — a claim about model behaviour. It was a full queue at the provider's end. The pipeline now names it as provider capacity and retries.
The pattern across findings 5 and 6.
Neither is visible without spending a token. Both survived three phases of careful offline work, and both would have quietly distorted a result rather than announcing themselves. That is the argument for building the instrument before trusting the measurement — which is, in the end, what your Chapter 6 proposes.
If you are asked a hard question
Three that are worth having an answer ready for.
“Your sample is twenty tasks. Isn't that too small?”
Yes, and the document says so. The honest answer is that the finding is not statistical — it is mechanical. The claim is not “format failures are 20% of failures in general”; it is “these four failures were caused by patch representation, and fixing the representation fixed three of them, while no amount of extra reasoning fixed any of the other four.” The 61-patch audit is what extends the mechanism beyond the small sample.
“How do you know the recovery wasn't just luck or resampling?”
On django__django-11019, six independent generations — three
strategies across two rounds — returned an identical sub-test score:
6 of 16 passing, zero regressions, every time. The retrofit moved that task from
0 of 16 with 33 broken tests. Nothing told to the model afterwards moved it again.
“Did you build this verification software yourself?”
The straight answer, which costs nothing:
“No. The dissertation is my work. The verification tooling was built afterwards by a collaborator using AI assistance, to test whether my results reproduce from my logs. It is not submitted work — it is a check on submitted work, and it found no discrepancy.”
Then talk about what it found, which is the interesting part and which you do know thoroughly: the table reproduced exactly, the classification you made by hand was confirmed, and the format finding now rests on 61 patches instead of four.
“Did any strategy make things worse?”
Yes, and it is reported rather than averaged away. On
django__django-11283, Diagnose+Revise broke 8
previously-passing tests where Blind Retry broke none. The extra diagnostic
context made the patch more damaging, not less.