Phase 0 · complete
Rescue the evidence before touching anything else
Before a single line of new code, two things had to be made safe: the data your experiments produced, and the exact set of problems they were run against. Both were more fragile than they looked.
Why this had to come first
Your results exist in exactly one place: the log files your evaluation runs wrote in August. There is no way to regenerate them — the free-tier quota is gone, the model version has moved on, and the per-attempt working folder was already deleted. If those files were lost or quietly altered, the dissertation would have no evidence behind it.
So phase 0 does two unglamorous things, and nothing else.
1 · Checksum every original file
495 files were catalogued and hashed, so any later change is detectable rather than assumed not to have happened.
| What | Files | Why it matters |
|---|---|---|
| Evaluation logs | 427 | The raw output of every run — the source of every number in Chapter 5 |
| Original scripts | 31 | The programs that produced the results, kept as the record of method |
| Aggregate summaries | 25 | Intermediate tables and the two original figures |
| Prediction files | 12 | The patches the model actually wrote — far more valuable than anyone realised |
The prediction files turned out to be the real find.
They preserve the full patch text for all 61 attempts across the 20 tasks that were touched. That is the only surviving trace of what the model actually wrote — and it is what makes the whole format-class finding measurable offline in phase 2, with no API call.
An early note recorded that the original code had never been
version-controlled. That was wrong. The check had been run from the wrong
directory, against an unrelated repository that tracks nothing. All 30 scripts
and all 427 logs are in Lagadnakul/Dissertation, and have been
since August.
The claim is struck through in the record rather than deleted, so the mistake stays visible. Because the files are safe upstream, the local copy is now treated as convenience rather than as the record — only its checksums are kept, which keeps the repository small without risking the evidence.
2 · Freeze the benchmark
SWE-bench Lite is a public dataset that can be revised upstream at any time. If it changed between your run and a future one, the two would not be comparable — and nothing would announce it. So the exact 300 problems were exported to a single file, pinned to the revision they came from:
princeton-nlp/SWE-bench_Lite default/test
revision 6ec7bb89b9342f664a54a6e0a6ea6501d3437cc2
300 instances, sorted by instance_idThe export script refuses to write anything unless upstream reports exactly 300 rows with all nine required fields present. A silent upstream change becomes a loud failure instead of a quiet inconsistency.
What the export revealed
The planning documents said SWE-bench Lite spans 11
repositories. The export measured 12. The missing one was
pallets/flask, with 3 instances.
That is a small error with a neat consequence: the sampling rule written for a future study requires at least 3 tasks per repository, and flask's 3 is exactly the floor that rule was set against. Had the count stayed at 11, the rule would have been impossible to satisfy without anyone noticing why.
What phase 0 did not do
No analysis, no new results, no changes to anything you wrote. Its entire job was to make sure that everything after it is built on evidence that cannot quietly shift underneath it.