Phase 6 · complete · the whole grid, no holes
Running everything — and finding two ways the plan could not have worked
Phase 5 proved one cell works. This phase runs all of them, and its single promise is that when hundreds run, none can go missing. Building it surfaced two problems that would have wasted a great deal of time and quota: the experiment did not fit its own budget, and two of your three recovery conditions could never have run at all.
The one promise
Every cell is listed before a single call is made, and afterwards a separate program checks that each one produced exactly one result. That check shares no code with the thing that runs the cells — deliberately, so a bug in the runner cannot hide inside the check that validates it.
VERIFY
planned 28
written 28
COMPLETE — every planned cell has exactly one row.Why this is the whole point of the project.
Your original study lost eight of twenty-four cells because a failure could quietly produce nothing, and phase 3 found twenty-two more evaluations whose patches and logs survived but which no table could see. A sweep that cannot prove completeness would repeat exactly that. So a refused condition, an exhausted budget, a provider outage and a missing file all write a result — the reason is recorded, and nothing is absent.
Problem one: it did not fit its own budget
Your configuration has had a line in it since the beginning —
estimate_before_run: true — with nothing implementing it.
Phase 6 implemented it, and the first run of the estimate said this:
This is the same defect as your token limit written
8_000_000, which the config format read as text so that no
comparison against it ever did anything. There, a ceiling was never
compared to anything. Here, a ceiling was never compared to the
experiment. Both looked complete and neither was checked.
The sweep now refuses to start and names the arm responsible, rather than running until the money stops. The tempting fix is to raise the ceiling — one line, and the warning disappears. That would be meaningless: a ceiling is only a claim about what you are actually allowed to spend.
Your three NVIDIA keys were each created from a separate account, so each has its own allowance — the real capacity is three times eight million, not eight. That makes the grid fit comfortably at 59%.
But it had to be declared, not assumed. The configuration now
says per_key: true, which states the claim so a reader can
see and contradict it, and capacity is multiplied by the keys actually
present rather than by the number hoped for. Two changes were needed to
make it real: the spend ledger now records which key paid (names only,
never values), and the key pool hands out keys in rotation instead of
sticking to the first — otherwise one account would have been drained
while the other two sat idle.
Problem two: two of three conditions could never have run
This is the more serious finding, and it only appeared by actually running things. The first live sweep of 24 cells refused 16 of them:
8 l2_reflection needs a previous attempt to reflect on, and none was produced
8 l2_5_diagnose_revise needs a previous attempt to diagnose, and none was produced
The reason is obvious once seen. Reflection-only and Diagnose+Revise are
defined by looking back at a failed attempt. In your study that
earlier attempt was the baseline — day8_test
is round zero, and each condition's round one reflected on it. The grid had
no round zero at all, so two thirds of your recovery design was
unreachable.
So phase 5's decision to make the baseline replay-only was reversed. It now runs, using your own first-attempt prompts — the pre-retrofit unified-diff prompt for the diff format, the blind-retry prompt for SEARCH/REPLACE, since that one shows no history and so already is a first-attempt prompt.
What stays replay-only is the claim. A baseline run here cannot say "resolved", because nothing here runs tests. Your 12/20 remains the submitted number, regenerated from the logs by phase 3.
It works, verified on a live model.
With the baseline in place, Reflection-only round one runs
against a real model and produces a real attempt — it was handed the
baseline's own response and reflected on it. Before the fix, that cell
refused.
An honest limit worth stating
Running it revealed something sharper about Diagnose+Revise than
phase 5 knew. When the baseline patch
applies cleanly, there is no failure evidence to
diagnose — and without Docker there never will be, because the only
evidence this pipeline can produce is git rejecting a patch.
So those cells are refused and the reason recorded, rather than quietly run as Reflection-only under a Diagnose+Revise label. It is a permanent consequence of the no-Docker decision, not a gap to be closed later, and the results say so cell by cell.
Deliberately broken, and one escape
| Sabotage introduced | Result |
|---|---|
| Skip a cell whose task file is unavailable | nothing failed — then 1 test failed |
| Drop the budget-stop result instead of writing it | 2 tests failed |
| Ignore the three separate accounts when estimating | 1 test failed |
| Put the round into the lineage, so round 2 loses its history | 3 tests failed |
Making the sweep silently skip a cell whose file could not be read broke nothing — because every test ran against a warm cache, so that path was never exercised. That is the phase 3 defect exactly: a cell attempted, no record left.
There is now a test using a task deliberately absent from the committed cache. It asserts that both its cells still produce results, that no request is sent, and that verification passes. The same sabotage now fails.
Where this leaves the project
Seven of nine phases done. The pipeline can enumerate 1,308 cells, tell you what they will cost before spending anything, refuse if that exceeds what you actually have, run them in the right order across three accounts, survive being stopped and restarted without paying twice, and then prove that every single one produced a result.
Next is phase 7: the dashboard. The data is complete; what is missing is a way to read it that is not a command line.