Phase 4 · complete · the first live call
The moment we stopped reading logs and started making them
Phases 0 to 3 never touched the network. This one does — one model call, wrapped in accounting that refuses to overspend. And the act of calling the endpoints immediately proved that four things we believed about them were false.
What this phase had to build
Two things. A way to talk to the three model services, and a ledger that counts what each call costs and stops the run before it exceeds its allowance.
The ledger came first, deliberately — before anything could spend. Its design turns on one decision: the check happens before the request is sent, not after it returns.
Why a ceiling that checks afterwards is not a ceiling.
Your original config set a limit of 8_000_000 tokens. YAML
has no underscore-as-thousands-separator, so it was read as the
text "8_000_000" and every comparison against it was
meaningless. The limit had never once been enforced. That is the defect
this ledger exists to make impossible — and the reason it was built before
the first call rather than after it.
Then we called the endpoints, and four beliefs collapsed
Phase 1 built a pre-flight check called doctor that measures
every model's speed rather than trusting its name. Running it against the
new provider layer showed that the check itself had been
lying:
ok muse_30b ttft n/a n/a tok/s
ok nemotron_nano ttft n/a n/a tok/s
ok nemotron_ultra ttft 1.69s 94094.4 tok/s| What it said | What was true |
|---|---|
Two models ok |
Both had produced no visible output at all. They are reasoning models, the probe allowed 16 tokens, and all 16 were spent thinking before a single word was said. |
94094.4 tok/s |
Speed was measured over the gap between the first visible word and the end. When everything arrives in one burst at the close, that gap is nearly zero — and dividing by it produces a number that measures nothing. |
| Gemini: not implemented | Sitting in the output as a tolerated warning. It had never been tested. |
All three are fixed: the timer now starts at the first token of any kind, speed is measured across the whole answer, a model that says nothing is reported as a warning rather than a pass, and Gemini has a real implementation.
With thinking properly switched off, "reply with ok" costs two or three tokens — and dividing by a sub-millisecond window turned that into 3000.0 tok/s. The same falsehood in a smaller costume.
So speed is now reported as unavailable below twelve tokens or forty milliseconds, because below that there is no measurement to report. The probe asks the model to count to forty instead — long enough to measure honestly, still free.
The finding that matters most
Every model in your configuration is marked thinking: false.
That setting has been there since phase 1.
It was never sent to the model. The request simply did not mention it, so every call — this project's and your original study's — ran with the service's own default, which is thinking on.
The control turned out to be a field called
enable_thinking. Measured on a one-word question, one API key
per model:
Why muse_30b refusing to obey is a real problem, not a curiosity.
Your Reflection-only condition asks whether a model does better when it
reflects on its own failure. That comparison only means something if the
arm it is compared against is not reflecting. On
muse_30b, asking for no thinking still leaves it thinking —
so an arm labelled "no thinking" against that model is not one, and the
comparison would be measuring something other than what it claims.
doctor now reports this instead of letting it pass silently.
A capacity limit that looked exactly like a silent model
Probing four NVIDIA models simultaneously returned, for two of them, a perfectly normal HTTP 200 success — with a completely empty body. A request alongside them returned the honest version of the same condition:
503 ResourceExhausted: Worker local total request limit reached (16/16)Run one at a time, those same two models answer in under half a second.
This one is worth dwelling on, because of what it would have done further
down the line. A success response with nothing in it, reaching the full
experiment, would have been written down as the model produced no
output — a finding about model behaviour. It was a queue being full at
the provider's end. The pipeline now names it as provider capacity and
retries, and doctor probes one model at a time, because a
pre-flight check that manufactures its own failures is worse than none.
A ninth outcome, and what it is kept out of
Phase 1 gave every attempt one of eight endings. The truncation measured here has no name among them: the model was mid-sentence when our token limit cut it off. It did not stay silent, it produced nothing malformed, and it never offered a patch to be rejected.
So TRUNCATED was added. The important part is what it is excluded from.
astropy-14182 stopped
because Gemini thought it was reciting training data, and your
dissertation excludes it from the recovery count. This generalises the
judgement you already made.
Counting a response we severed as the model producing a bad patch would inflate the exact figure your thesis rests on, using our own configuration. It is the same category error as scoring your baseline over only the tasks that happened to produce a result file — the defect phase 3 found in the original analysis.
Proving the checks bite — and finding one that did not
As in phase 3, the finished work was deliberately broken to confirm the tests notice.
| Sabotage introduced | Result |
|---|---|
| Let the ceiling ignore reserved tokens — the concurrent overrun | 2 tests failed |
| Stop sending the thinking setting — the defect above, restored | 3 tests failed |
Count TRUNCATED as a format-class failure | nothing failed |
Breaking the single most consequential rule in the phase — the one the
whole TRUNCATED decision was made to protect — broke
nothing in a 237-test suite. The rule was written down
and explained, and nothing checked it.
There is now a test file dedicated to it, which pins both groups exhaustively, asserts they cannot overlap, and fails if a tenth outcome is ever added without being classified. The same sabotage now kills three tests. This is exactly why deliberately breaking things is part of the method rather than a flourish at the end.
Where this leaves the project
Five of nine phases done. All three model services answer, all five of your API keys work, the token ceiling refuses to overspend before a request is sent, and every call's real cost is recorded in a file that survives a crash and does not restart at zero when a run resumes.
That last part also closes a gap phase 3 had to leave open. Figure 5.4's cost comparison was the one Chapter 5 number that could not be regenerated, because the original never recorded token usage. From now on it is measured.
Next was phase 5: the recovery strategies themselves — Baseline, Blind Retry, Reflection-only and Diagnose+Revise — as four implementations of one interface — the first phase that can produce a genuinely new result rather than reproduce an old one.