Blueprints say the same fact in the criteria, in the body and in the sensitive zones

[PierreMardon]

Status

Closed. Confirmed, fixed by #83, measured. Three readings confirmed it, a blind count located its cause, and with the fix the facts said more than once fall from 13 to 15 per page to 2 to 4 on the case measured over several runs. The measures and their sources are in the comments below, campaign by campaign.

What was seen

The judge's reasons, one per case:

• 01-overdue-list: "The text tie rule is stated in criterion 3, again in the body, again in the 'Compared as text' paragraph, and once more under sensitive zones; the missing-file case is likewise repeated in the criteria, the body and two sensitive-zone bullets. Implementation details (argparse, counting 'once', main falling back to the clock) and draft history also appear."
• 02-reservations: "Many facts are said three times: the acceptance criteria restate the body sections, and the untouched fines.py/loans.jsonl and the 'holds do not run out on their own' rule recur in scope, the body and sensitive zones. The page also spends a section on implementation choices that are not the developer's to decide: the wrapper, the import cycle and a helper in __main__.py."
• 03-overdue-reminders: "The 7-day rule and the silent skip are restated in the intro, criteria 3 and 7, scope, the rule section, the diagram note and the sensitive zones. The page also spends space on function names, constants and Decimal handling."
• 04-suspension: "Several facts are stated three times: returned loans counting for nothing, the owed output and refusal, and the check order and nothing written appear in the criteria, the body and the sensitive zones."
• 05-fine-cap: "the uncapped book with no price appears in criteria 4-5, the table note, the cap section, the diagram and the sensitive zones [...] Code details such as the constant name and the required-parameter design are not the developer's to decide."
• 06-borrow-limit: "fines.py untouched and the five fields of loans.jsonl appear in criterion 7, out of scope and sensitive zones [...] The page also carries implementation detail (where the constant sits, when the file is saved) and a boilerplate 'No change' line."

Two things are mixed in that score: a fact said several times, and a detail of implementation that is not the developer's to decide.

Sources

• Report of campaign 1, committed: scores, lowest judgement, the judge, checked.
• Raw data, out of git: .evals/campaign-1/runs/*/run-01/judge.json, and the blueprints themselves, .evals/campaign-1/runs/*/run-01/blueprint-rev-*.md.

Suspected cause (a hypothesis)

• The frame asks for it: the acceptance criteria state the rules, the body shows what will be built, the sensitive zones name what touches the developer's control (agents/surface-extractor.md lines 39 to 47). "Say a fact once" (line 61) speaks of the sections of the body only.
• The cross-check counts as an omission what the plan does and the blueprint does not show: the extractor is pushed to show everything, details of implementation included.

Why not fix it now

A blueprint that shows less may raise the omissions of the cross-check, so the planning passes and the cost, and may lower blueprint.decidable, the criterion the chain scores best on (4.7). With one point of range, the judge cannot say whether a change helped.

Next

1. Settle what the judge is worth on this criterion (see the issue on the judge's range).
2. Then decide: the rubric, the frame of the blueprint, or the extractor.

[PierreMardon]

Measured: the repetitions are real, and the frame does not force them

Three readings now agree, with no session of the chain.

1. The judge of the campaign, three passes: blueprint.no-padding 2 of 5 on all six pages, every time.
2. Another model as judge, one pass: 2 on five pages, 3 on 06-borrow-limit.
3. A blind reader, given the six blueprints and the lean and padded pair of the judge's check under neutral names, with no score to anchor on, asked to list every fact said more than once and where:

Page	Words	Facts said more than once	Details of implementation	Its score	Share of the page a developer could skip (its estimate)
the lean page of the judge's check	221	0	0	5	0%
06-borrow-limit	724	10	4	3	22%
05-fine-cap	1165	12	4	2	29%
01-overdue-list	954	12	7	2	31%
02-reservations	2376	29	8	2	35%
04-suspension	1178	19	6	2	38%
03-overdue-reminders	1535	17	8	2	48%
the padded page of the judge's check	572	7	7	1	61%

Where the 106 repeated facts sit, a fact counting in each pair of places it spans: between the criteria and the body 64, inside the body 50, between the scope and another place 45, between the criteria and the sensitive zones 16, between the body and the sensitive zones 16, elsewhere 31.

Its example on 01-overdue-list: "book ids are compared as plain text (B10 before B2), never as numbers" is said in the criteria, in the body and in the sensitive zones; "the fine computation, the format of loans.jsonl and the domain modules are untouched" in the scope, in the sensitive zones and in the closing line.

What this changes

• Status: confirmed by three readings. The reading by the developer is still missing, and stays worth five minutes.
• The cause is not the frame. My hypothesis above was wrong. The lean page keeps the whole frame and says each fact once: its scope holds only what is left out, its body only what the criteria cannot hold (the year as a whole number, the line endings), and its sensitive zone points to "criteria 2 and 3" instead of listing the columns again. On the six pages, the body tells the criteria again, the scope names again what the criteria fix, and the sensitive zones restate both.
• A fix can be measured, which was the reason not to touch it: the count of repeated facts separates the pages where the score of the judge has one point of range (see Evaluations: the judge's blueprint.no-padding has one point of range, and no judgement was checked against a human #73).
• The risk on the cross-check is lower than feared: the checker never counts file layout and naming as an omission, so the details of implementation can leave the page at no cost in passes.

Fix

A pull request follows: the extractor says a fact once on the whole page, the frame included, and shows behavior, not code. It is measured by the next campaign on blueprint.no-padding, the length of the pages and the count of repeated facts, with blueprint.decidable, the omissions of the cross-check and the planning passes as the regressions to watch.

Raw data, out of git: .evals/blind-count/ holds the answer of the blind reader (blind-result.json), the pages as it was given them and the key of their names; the pages come from .evals/campaign-1/runs/*/run-01/work/lending/docs/plans/*/blueprint.md and evals/judge/spoiled/blueprint-lean.md, blueprint-padded.md.

[PierreMardon]

Measured in campaign 3: the fix works, by a count if not by the judge

The fix is #83. It was played before being merged, from its branch: five runs, compared with campaign 2, where the extractor was unchanged. Report kept: evals/reports/2026-10-03-d765f898c9ad/report.md. Raw data, out of git: .evals/campaign-3/, and .evals/blind-count-2/ for the blind reader.

A blind reader was given the eleven blueprints of both campaigns under neutral names, with the lean and padded pages of the judge's check among them, and counted as before. It scored the two reference pages as it had the first time, 5 and 1.

Page	Words	Facts said more than once in prose	Details of implementation	Missing to decide	Its score	Share it would skip
03-overdue-reminders, campaign 2, three runs	1235 to 1425	13, 14, 15	5, 8, 9	0, 1, 2	2, 2, 2	35 to 40%
03-overdue-reminders, with the fix, two runs	905, 986	2, 4	1, 1	1, 1	3, 3	10, 15%
01-overdue-list, campaign 2, three runs	669 to 1545	7, 8, 18	4, 5, 8	0, 0, 1	2, 2, 2	25 to 40%
01-overdue-list, with the fix, two runs	730, 1463	3, 6	4, 5	0, 0	3, 2	15, 25%
02-reservations, with the fix, one run	1572	5	1	0	3	15%
02-reservations, campaign 1, one run, from the first count	2296	29	8		2	35%

• Its ranking puts every page drawn with the fix before every page of campaign 2, between the lean page and the padded one.
• Nothing more is missing to decide: 2 things over the 5 pages with the fix, 4 over the 6 pages before; blueprint.decidable by the judge is 4.5, 5 and 4, against 4.33, 4.67 and 4.
• The cross-check is not blinded: its probes on the toy pass from the branch, and the only omissions of the campaign come from an amendment of the simulated developer (Evaluations: the need of 03-overdue-reminders states a rule its brief contradicts, and the correction is counted against the interview #82), not from a rule left out for brevity.
• blueprint.no-padding, by the judge of the harness, has not moved: 2 on the five runs. It cannot follow this change (Evaluations: the judge's blueprint.no-padding has one point of range, and no judgement was checked against a human #73).

What the fix does not do, or costs:

• 01-overdue-list did not get shorter. One of its two pages carries an amendment that rewrote how every command parses its arguments.
• blueprint.cut is 3 on both runs of 03-overdue-reminders, against 4 on the three runs before: with less to say, the body is one section that mixes what is left (The body of a blueprint is cut by the layout of the code, or left uncut where several things are decided separately #63).
• A page with the fix still scores 3 at best, where the lean reference scores 5: details of implementation remain on 01-overdue-list, and what the frame itself asks for, the closing line and the boundaries between components, is counted as padding by the judge.

Status

Confirmed, fixed by #83, measured on five runs and two blind counts. Closing. What is left has its own issues: the cut (#63), the judge's scale and the missing count in the harness (#73). The reading by the developer of one page before and one after would still be the best check: .evals/blind-count-2/pages/ holds them, page-D.md before and page-A.md after, for the same need.

[PierreMardon]

Read by the developer

Recorded on 2026-10-07. The developer read the two pages kept for it, both drawn for the need of 03-overdue-reminders: page-D.md, before #83, and page-A.md, after.

• The text of the page after the fix is the easier one to digest. The reading that was missing confirms the fix.
• They prefer the diagram of the page before it (A diagram is drawn for a chain with no branch, or where the feature has no shape to draw #64), and could not tell what they would miss with either page.

A comparison was made for them, by one read-only session given the two pages, the two plans, the need and the brief, not told which page came first. Its cost is in no ledger. What it says, its main claims checked against the plans, the interview and the brief:

• The two pages hold the same rules, and neither goes against the brief. No rule is lost from one to the other.
• The differences of content that strike come from the two plans, not from the pages: a wording of the notice left free or fixed by a criterion, two warnings on the mailer and on an amount that grows, a member unknown to members.jsonl.
• What comes from the page: before the fix, each rule said two to four times, blocks with a title, the procedure drawn; after it, each rule said once, a worked example, the two files that feed the decision drawn.
• The worked example of the page after the fix is not in its plan: the page made it, and it is frozen with the contract. It is right here.

Status

Closed, and the reading that was missing is done. What it shows of the cut is in #63, of the diagrams in #64.