A blueprint gives a data shape by pointing at one it never shows

[PierreMardon]

Status

Closed on 2026-10-05: not seen since #83, over 12 runs. Two occurrences in the 12 runs before #83, none in the 12 runs since: five in campaign 3, seven in campaign 4. The measures and their sources are in the comments below, campaign by campaign.

What was seen

02-reservations, blueprint.decidable 4 of 5. The judge: "Every interview answer, the amendment to 5 days, the refusal order, the output lines and the file shape are stated, so the developer has the rules to decide on. One detail points elsewhere: the fields of the hold notice are given only as the same as an existing kind that the page never shows."

Passage: "The outbox gains the notice kind hold, with the same fields as fine."

The developer reads the blueprint and no other file: a shape given by reference sends them to the code.

Sources

• Report of campaign 1, committed: blueprint.decidable.
• Raw data, out of git: .evals/campaign-1/runs/02-reservations/run-01/judge.json, blueprint-rev-02.md.

Suspected cause

None located. The extractor asks for "the existing elements it attaches to, marked as existing" in a diagram, and has no word for an existing data shape the change reuses.

Next

Watch blueprint.decidable in the next campaigns. It pulls the other way from the repetitions: showing the fields of fine again is one more thing said.

[PierreMardon]

Seen again in campaign 2

Campaign 2: 01-overdue-list and 03-overdue-reminders, three runs each, on main with the four first fixes (#78, #79, #80, #81). Report kept: evals/reports/2026-10-03-e830bc236e9b/report.md. Raw data, out of git: .evals/campaign-2/. In the tables, a value is the mean over the three runs, with its lowest and highest when they differ.

01-overdue-list, blueprint.decidable 4: "The one gap is that the days-late figure is defined only as whatever fines.days_late returns, so the developer must open fines.py to know the number printed for a loan due yesterday." Passage: "The number of days late on each line is the one the fine computation already uses (fines.days_late). Nothing new is computed about lateness."

The same shape as the first occurrence: a rule given by pointing at code the page does not show.

Status

Two occurrences, on two cases. Not fixed. To watch with #83, which tells the extractor to leave the names of functions to the plan: a rule given only by the name of a function is what it should stop writing.

[PierreMardon]

Campaign 3

With the fix of the extractor (#83), blueprint.decidable is 5 on three runs of five and 4 on two, and neither 4 is a rule given by pointing at code: one is a criterion that reads as contradicting a later sentence, the other a case "worded confusingly". The blind reader finds one thing missing on each page of 03-overdue-reminders and none elsewhere.

Status

Not seen in campaign 3. Two occurrences before #83, none after, on five runs. Left open until a campaign with more runs confirms it.

[PierreMardon]

Campaign 4: not seen, in seven more runs

Campaign 4: seven runs on main with every fix so far, #87 and #88 included, and the harness of #89, #90 and #91, on the cases no campaign had played again, or once. 04-suspension, 05-fine-cap and 06-borrow-limit: one run played whole and one stopped at the hand over each. 02-reservations: one run stopped at the hand over. Report kept: evals/reports/2026-10-05-d139a1ca0dbe/report.md, compared with campaign 1, the only earlier runs of three of these cases. Raw data, out of git: .evals/campaign-4/. A value is the mean over the runs of a case, with its lowest and highest when they differ.

blueprint.decidable: 4 on 02-reservations, 4 on both runs of 04-suspension, 4.5 (4 to 5) on 05-fine-cap, 5 on both runs of 06-borrow-limit. None of the four 4s is a rule or a shape given by pointing at code:

• 02-reservations: "criterion 3 does not say whether a refused cancel gives its reason on standard error, as criterion 2 does for reserve";
• 04-suspension, both runs: the wording of the new refusal "is given only "for example"", so the developer cannot tell whether it is a contract;
• 05-fine-cap: an assumption "too vague to decide on, since it never says how a price recorded as text is compared with the amount".

So: two occurrences in the 12 runs before #83, none in the 12 runs since, five in campaign 3 and seven here. The count of the judge (#89) now lists the names of functions a page still holds among its details of implementation: 1 or 2 a page in this campaign, 5 to 11 on the same cases in campaign 1.

Status

Closed: not seen since #83, over 12 runs.