Read this first · assumes nothing

An AI fixed the bug correctly. It still scored zero.

That sentence is the whole dissertation. Everything else — the experiments, the tables, the software being built now — exists to establish that it can happen, how often, and what follows from it.

Clearing one thing up first

If you are looking at the GitHub page and see Shell 76.2% · Python 23.8% — ignore it. That is GitHub counting what kinds of files sit in the repository. It has nothing to do with the research, and nobody here is ranking programming languages.

The setting

You can now ask an AI to fix a bug in real software. Sometimes it works. Often it does not. To measure how often, researchers built a test called SWE-bench, and almost everyone in the field uses it. It works like this:

Take a real bug from a real open-source project — an actual issue an actual developer reported on GitHub.
Give the AI the bug report and the code.
The AI writes a fix.
Run the project's own test suite against it.
Tests pass → solved. Tests fail → not solved.

Pass or fail. One bit. No partial credit. This is the scoreboard companies quote when they say a model “scores 49% on SWE-bench”.

The question Nakul asked

Not “how good is the AI” — plenty of people measure that. Something sharper:

When the AI fails, and you tell it that it failed, can it fix itself?

Humans do this constantly. You write code, it breaks, you read the error, you fix it. The question is whether an AI can close that same loop.

So he took 20 real bugs, gave them to one model, and whenever it failed he gave it another attempt — in three different ways, to see which kind of help actually helps:

Strategy	What the model is told before trying again
Blind retry	Nothing. Just “try again.”
Reflection	Its own previous attempt. “What is wrong with this?”
Diagnose & revise	Its previous attempt and the real error output from the failing tests.

The expected result is obvious. More information should mean better outcomes. Showing it the actual error ought to beat telling it nothing.

What he found instead

Of the 20 bugs, the model solved 12. Eight failed. And when he looked at why each one failed, they split into two completely different kinds — which nobody had been separating.

Half of the failures had nothing to do with the AI's understanding of the bug.

Why a correct fix can score zero

When an AI proposes a code change, it cannot simply say “change line 40”. It has to produce a precise technical document called a patch, which states exactly which lines are removed, which are added, how many lines are involved — and it must quote the surrounding existing code character for character.

Get one detail wrong — miscount by one line, mistype a single space — and the tool that applies patches refuses the entire thing. It does not partially apply. It rejects.

The shape of the problem

A doctor diagnoses your illness perfectly and prescribes exactly the right drug — but writes it in handwriting the pharmacist cannot read, so you go home with nothing.

Did the doctor fail at medicine? No. But if the only thing you measure is “did the patient get the right medicine”, it looks identical to a misdiagnosis.

That is those four tasks. The AI may well have had the right fix. Nobody will ever know, because the paperwork was malformed — and the benchmark scores that exactly the same as being wrong.

The experiment that proves it

This is what turns an observation into a finding. Nakul changed nothing about the model, the bug, or the reasoning he asked for. He changed only the format the answer had to be written in. Instead of making the AI produce that fussy patch document itself, he asked it to say two simple things:

here is the code I want to change — quote it
here is what it should become

Then a small program did the fussy part. The program counts the lines. A program cannot miscount.

One of the four paperwork failures had to be excluded: the model refused to generate for it at all, so there was never a patch to judge. That leaves three, and all three recovered.

So the finding is

Self-correction worked only where the problem was clerical.

Where the AI was genuinely wrong about the bug, telling it more about its own mistake changed nothing — not once, across three strategies and two rounds — while costing about three times as much.

This is uncomfortable for a field that largely assumes better feedback produces better results. And it carries a sting for how other work is measured: some published self-correction results may be measuring format recovery rather than reasoning recovery, because nobody separated the two.

Then why build software?

Because the study has three weaknesses that no amount of extra writing can fix.

Weakness	What a reviewer can fairly say
20 tasks, one model, run by hand	“You got lucky” — or “that is one model's quirk”
A badly failed attempt wrote nothing to disk	“Attempts that failed have vanished from your data”
Every number was typed into the tables by hand	“Nothing checks that these match your logs”

Chapter 6 of the dissertation promises that a follow-up will do this properly — at scale, across several models, with everything recorded. That is what is being built. Not new research: the machine that makes the existing research checkable and repeatable by anyone.

What it has already turned up

Three things that justified the effort, before any new experiment was run.

The numbers are right — and now proven so

A completely separate program, written months later in a different language, read the same log files and reproduced the dissertation's results table character for character. That moves Chapter 5 from “trusted” to “reproducible”.

The archive held far more evidence than anyone knew

Buried in old files were 61 actual patches the model had written. Checked: patches the AI wrote by hand were properly formed 30 times out of 51. Patches written by the program: 10 out of 10. A finding that rested on 4 tasks now has 61 data points behind it.

Twenty-two attempts were invisible

Exactly the “leaves no record” problem, found in the project's own data: 22 runs where the patch was rejected, so no result file was ever written, so the tables never saw them. They are recovered now.

What changed since you submitted

The four findings in detail, and three hard questions with answers ready.

The build

8 of 9 phases

What has been built so far, phase by phase, and what comes next.

Run it yourself

Windows

Verify all of this on your own laptop in about ten minutes. Free, safe, and it ends with you regenerating your own Chapter 5.

Nothing here changes Self.docx. The document is final and is not edited again — where the build found something stated differently, it is recorded separately rather than applied silently to the text. The dissertation is Nakul's work; the verification software described here was built afterwards by a collaborator with AI assistance, and is not submitted work — the full note is here.