An assumption of the plan becomes an acceptance criterion, and is never put to the developer

[PierreMardon]

Status

Open, not fixed. Observed in the four campaigns, on five cases of six: a rule the plan settles on its own reaches the blueprint as settled. Closing it needs a choice of the developer between two ways. It also carries what #59, closed, left open: the rules a session states as settled without asking. #94 bears on the choice: on one case, the developer handed back the very kind of question the first way would send. The measures and their sources are in the comments below, campaign by campaign.

What was seen

01-overdue-list. The judge: "The final blueprint's rules all come from the need, the code, or the developer's amendments, and the ISO date is labeled as an assumption. One small lapse: the empty-output-with-exit-0 behavior was an assumption in the interview, but the blueprint states it as a plain criterion and never offers it to the developer to confirm."

Passage: "5. When no loan is late, nothing is printed and the exit code is 0. The same holds when the data directory has no loans.jsonl."

In the same blueprint another assumption, the ISO date, is labeled as one: what is shown as an assumption varies inside one page.

Sources

• Report of campaign 1, committed: interview.no-invented-rule, scores.
• Raw data, out of git: .evals/campaign-1/runs/01-overdue-list/run-01/judge.json.

Suspected cause

The plan is where an assumption taken instead of a question goes (templates/plan.md line 5), and nothing in agents/surface-extractor.md says how the blueprint shows one: its acceptance criteria are "taken from the specs and the interview", and an assumption is neither.

Next

• Tied to the interview that asks too little: an interview that asks leaves fewer assumptions.
• Watch interview.no-invented-rule in the next campaign before deciding whether the extractor needs a rule.

[PierreMardon]

Seen again in campaign 2

Campaign 2: 01-overdue-list and 03-overdue-reminders, three runs each, on main with the four first fixes (#78, #79, #80, #81). Report kept: evals/reports/2026-10-03-e830bc236e9b/report.md. Raw data, out of git: .evals/campaign-2/. In the tables, a value is the mean over the three runs, with its lowest and highest when they differ.

interview.no-invented-rule: 4 (3 to 5) on 01-overdue-list, 3 (2 to 4) on 03-overdue-reminders, where it was 4. Within the noise of the judge (#74), but the reasons are concrete. The lowest, on 03-overdue-reminders: "Several visible rules were decided without the developer and stated as settled, not as assumptions to confirm: the exact text of the notice members receive, skipping members whose reminder is dated after today, and reminding loans whose member is unknown. None comes from the need, an answer or the code." Passage: "The wording of the notice is fixed by the plan, since the specs leave it open".

So this is no longer one occurrence. With an interview that now asks the rules it sees (#59), what remains comes from the plan: a rule it settles when it is drafted reaches the blueprint as a plain statement.

Status

Observed in both campaigns, on two cases. Not fixed. Two ways, for the developer to choose: the agent that drafts the plan sends such a rule back as a question, which is a new step in planning; or the plan marks it and the blueprint shows it as an assumption to confirm, which the rubric already accepts.

[PierreMardon]

Carried over from #59

#59 is closed: the rule its interview guessed is asked since #81, in 5 runs of 5 of 01-overdue-list over campaigns 2 and 3. What it left open comes here.

On 03-overdue-reminders, in campaign 2, two runs of three still ask nothing, and the judge names rules a user would see applied that nobody put to the developer: "the exact text of the notice members receive, skipping members whose reminder is dated after today, and reminding loans whose member is unknown". They are stated as settled, by the session ("the code's existing conventions") or by the plan when it is drafted.

So what is left has two sources, and only the second is the one this issue was opened on:

• the session, at the interview: it sees the rule, holds that the code settles it, and asks nothing. An interview that asks for the rules a user of the feature would see #81 tells it not to ask what the code settles, its closest precedent included;
• the plan, when it is drafted: the rule is first met there, and nothing lets the Plan agent send a question back.

Which rule came from which source was not counted in campaign 2. The choice between the two ways above closes the second source. Campaign 3, in the comment below, shows the first one at work.

[PierreMardon]

Campaign 3: seen again, and on 03-overdue-reminders the session asks nothing

Campaign 3: five runs on three cases, played from the branch of #83. Report kept: evals/reports/2026-10-03-d765f898c9ad/report.md. Raw data, out of git: .evals/campaign-3/. Written on 2026-10-05 from the data of that campaign: this issue had not been read again after it. The branch it was played from holds the fix of the interview (#81).

	Campaign 2	Campaign 3
01-overdue-list interview.no-invented-rule	4 (3 to 5)	4
02-reservations interview.no-invented-rule		4
03-overdue-reminders interview.no-invented-rule	3 (2 to 4)	3 (2 to 4)
03-overdue-reminders questions	0.33 (0 to 1)	0
03-overdue-reminders interview.right-number	2	1.5 (1 to 2)

The lowest judgement, on 03-overdue-reminders: "The blueprint states rules as settled that come from no source. A member unknown to the library is still reminded, although the README says an unknown member is refused. A reminder dated after today counts as recent. The member order and the wording of the notice appear as criteria, not as assumptions to confirm."

Both runs of that case ask no question, and the judge quotes the session's reason: "None. The specs and the code settle every rule a member or a program reading the outbox would see".

So the first source named in the comment above is at work: with #81 in place, on this case, the session holds that the specs and the code settle every visible rule, and the rules the judge then finds decided alone are those of campaign 2. One of them, as the judge reads it, goes against the README of the host.

Part of the low scores of this case is not the chain's: how often a member is reminded, which the judge counts as a rule the interview should have asked, is stated by the need (#82).

Status

Observed in the three campaigns. Not fixed. On 03-overdue-reminders it has two sources: the session, which asks nothing because it holds that the code settles the rule, and the plan when it is drafted. The choice between the two ways above closes the second. The first is the stop rule of the interview as #81 left it: "What the specs or the code settle, its closest precedent included, is not asked."

[PierreMardon]

Campaign 4: seen again, on three more cases

Campaign 4: seven runs on main with every fix so far, #87 and #88 included, and the harness of #89, #90 and #91, on the cases no campaign had played again, or once. 04-suspension, 05-fine-cap and 06-borrow-limit: one run played whole and one stopped at the hand over each. 02-reservations: one run stopped at the hand over. Report kept: evals/reports/2026-10-05-d139a1ca0dbe/report.md, compared with campaign 1, the only earlier runs of three of these cases. Raw data, out of git: .evals/campaign-4/. A value is the mean over the runs of a case, with its lowest and highest when they differ.

interview.no-invented-rule	Campaign 1	Campaign 4
02-reservations	4	4
04-suspension	5	4
05-fine-cap	4	5
06-borrow-limit	5	4

The rules the judge finds decided without the developer and shown as settled:

• 04-suspension, both runs: where the new check stands among the existing ones. "One visible rule was not asked and is not offered as an assumption to confirm: the debt check runs after the existing checks, so an over-limit member asking for a book already on loan is only told the book is on loan. The blueprint states this as decided." In the other run the page does call it an assumption, "but it never asks the developer to confirm it".
• 06-borrow-limit: that overdue loans count toward the limit, which "the interview files [...] under implementation assumptions and the blueprint states [...] as fixed criterion 3"; and, in the other run, that "the limits are fixed in the code, with no option or data file to change them".
• 02-reservations: "The wording of the hold notice, which reaches real members, came from the plan alone. The blueprint lists it as something the developer might miss, not as an assumption to confirm."

On 06-borrow-limit both rules are what the brief of the case holds, so the plan settled them as the developer wanted: what is faulted is that they were shown as settled, neither asked nor marked.

The opposite sign is new. On 06-borrow-limit the interview did ask where the new check stands, and its developer handed the question back (#94). On 04-suspension it did not ask, and the judge faults that. The same kind of rule, the two ways.

Status

Observed in the four campaigns, on five cases of six. Not fixed. The choice between the two ways is still the developer's, and #94 bears on it: a question sent back for such a rule is one a developer handed back, on the one case where it was seen.