Evaluations: the judge's blueprint.no-padding has one point of range, and no judgement was checked against a human

Status

Closed: both halves are done and checked. The count of repeated facts is in the harness since #89, and the rubric says since #104 that what the frame of a blueprint asks for is not padding, as the developer decided on 2026-10-07. Checked in campaign 5: the lean page scores 4 and 4, its padded version 2 and 2, and each count parts them. On real pages the score is 2 or 3, and its reasons now name facts said twice, not the frame. The measures and their sources are in the comments below, campaign by campaign.

What was seen

The check of the judge: the lean blueprint written by hand for it scores 3 and 3, its padded version 2 and 2. The criterion tells them apart, by one point.
Before that lean original existed, the toy's own blueprint scored 2 and 2, the same as its padded version: the judge found the reference page of the repository as padded as a page padded on purpose.
The six blueprints of campaign 1 all score 2. So they score like a page padded on purpose, and the best score seen on this criterion, on any page, is 3.
The rubric: "Be strict: a 5 is earned", and for this criterion "No fact said twice". The frame of a blueprint states a rule in its acceptance criteria and shows it again in its body: a page that follows the frame may be unable to score above 3.
The check rests on three pairs of documents, one per criterion, for fourteen criteria.

Sources

Report of campaign 1, committed: the judge, checked, scores.
Raw data, out of git: .evals/campaign-1/judge-check.json, .evals/trial-1/judge-check-before-the-lean-original.json.
The rubric: evals/judge/rubric.md.
Evaluations of the chain: whether it meets its goals, and what it is like to work with #58, "Risks": "The judge is a model. [...] Its check rests on three pairs."

Why it matters

A change to the extractor or to the frame of the blueprint cannot be steered with a measure that moves between 2 and 3. And whether the blueprints repeat themselves at all, as a developer would feel it, is not known.

The developer reads one short blueprint of campaign 1 next to the judge's reason, and says whether they agree. Free, and the only reference that settles it. Suggested: .evals/campaign-1/runs/01-overdue-list/run-01/, its last blueprint-rev-*.md, against the reason quoted in the issue on the repetitions.
Judge the six blueprints again, several times, to see whether the score holds from one pass to the next.
A second judge, another model, blind to the first scores, on the six blueprints and on the lean and padded pair, listing each fact said more than once: do two judges agree, and does a count separate the pages better than a score?
Then decide: the rubric, the frame, or the extractor.

Measured: the score is stable, across passes and across models

With no session of the chain, the six runs of campaign 1 were judged again (details and the table of every criterion in #74):

the same judge (opus), two more passes: blueprint.no-padding is 2 in all 18 judgements;
another model (sonnet), one pass: 2 on five cases, 3 on 06-borrow-limit.

So the 2 of 5 is neither the noise of the judge nor the strictness of one model: two models, reading the rubric as written, find these pages padded, every time.

What this does not settle: whether a developer would feel it, and whether the criterion can separate a good page from a better one, since its best score on any page so far is 3. A blind count of the facts said more than once, on the six blueprints and on the lean and padded pair of the judge's check, is under way. The reading by the developer stays the one reference that is missing.

A first figure that needs no judge, the length of the pages:

Page	Words
the lean blueprint of the judge's check (the toy, one behavior)	221
its padded version	572
`06-borrow-limit`	724
`01-overdue-list` (one behavior)	954
`05-fine-cap`	1165
`04-suspension`	1178
`03-overdue-reminders`	1535
`02-reservations`	2376

Measured: a count separates the pages where the score does not

A blind reader counted the facts said more than once on the six blueprints of campaign 1 and on the lean and padded pair of the judge's check (table and method in #62).

Its own score on the same criterion and the same scale: 5 for the lean page, 1 for the padded one, 2 for five blueprints and 3 for 06-borrow-limit. That is four points between the lean page and the padded one, where the judge of the harness has one (3 and 2).
It ranks the blueprints the way the second judge does: 06-borrow-limit is the least padded for both.
Its count gives what a score cannot: 0 repeated fact on the lean page, 10 to 29 on the blueprints, and where they sit.

So, on this criterion:

the judge of the harness is right on the direction and stable, and too compressed to steer with: it gives 3 to a page a blind reader finds nothing to skip in. A likely reason is the instruction "Be strict: a 5 is earned, and a document that is merely plausible gets a 3", applied to a page read among three other criteria;
a count of repeated facts, with the length of the page, is the measure to follow a fix with. It is not in the harness: it was done by hand, by one reader, once.

Campaign 2: the same score, again

Campaign 2: 01-overdue-list and 03-overdue-reminders, three runs each, on main with the four first fixes (#78, #79, #80, #81). Report kept: evals/reports/2026-10-03-e830bc236e9b/report.md. Raw data, out of git: .evals/campaign-2/. In the tables, a value is the mean over the three runs, with its lowest and highest when they differ.

blueprint.no-padding is 2 on the six runs. Over the two campaigns and the passes run again, that is 24 judgements by the same judge, all at 2. The measure of the harness that does move is the length: the words of the blueprint, 669 to 1545 on these six runs.

Campaign 3: the judge did not see a change that a count sees plainly

The extractor was changed (#83) and the pages of 03-overdue-reminders went from 13 to 15 repeated facts to 2 and 4, from 1310 words to 946 (table in #62). blueprint.no-padding, by the judge of the harness: 2 before, 2 after, on every run.

Its reasons on the pages after the fix show why the score cannot rise:

"A paragraph lists which module calls which, an implementation choice that is not the developer's to decide." The extractor is told to show the boundaries between components: it is one of the five aspects a blueprint must not leave in the dark.
"the page ends on a filler line about state machines." That is the closing line, which the frame asks for on every page.
"The facts that fines.py and loans.jsonl are left untouched are said three times (criteria, out of scope, sensitive zones)." The statement of the critical zones is always written, and the need itself makes one a criterion.

So the rubric counts as padding three things the chain asks of a blueprint on purpose. On this criterion the judge and the extractor do not hold the same idea of a good page, and the score stays at 2 whatever the page.

Status

Measured over three campaigns: stable, right on the direction, and unable to follow a fix on this criterion. Not fixed. Two things to do, both in the evaluations and neither done:

Say in the rubric what the frame asks for and is therefore not padding: the closing line, the statement of the critical zones, the boundaries between components. Whether the boundaries are the developer's to decide is the developer's call; the chain says yes today.
Add to the harness the count a blind reader made twice by hand: facts said more than once in prose, details of implementation, share of the page to skip, checked on the lean and padded pair as the judge is. It separated every page where the score separated none.

A judge that counts what a blueprint says twice, beside its scores

A judge that counts what a blueprint says twice, beside its scores (#89)

The count is in the harness: #89, checked on the pair

#89, merged. For a blueprint, and beside its scores, the judge now counts by the method the two blind readers were given (evals/judge/count.md):

repeated_in_prose: the facts said more than once outside a diagram;
implementation_details: the details of implementation that are not the developer's to decide;
skippable_percent: the share of the page the developer could skip, to the nearest five.

The judge gives lists, each fact with its statements and the passage of each, copied from the page. The script makes the totals from them and marks a passage that is not on the page. The fourteen criteria, their prompt and the rubric are untouched, so the scores still compare from one campaign to the next. judge counts a run that was judged before without scoring it again. The report shows the counts as measures, and a count that falls is better.

Checked as the judge is, on the lean and padded pair, two passes each (.evals/campaign-4/judge-check.json, out of git; 16 sessions and 0.77 USD, the scores of the three pairs included):

	Lean page	Padded page	Padded page, by hand
`blueprint.no-padding`, the score	3	2 and 2
`repeated_in_prose`	0 and 0	5 and 7	7 then 6
`implementation_details`	0 and 0	7 and 10	7 and 7
`skippable_percent`	0 and 0	45 and 45	61 then 60

Where the score puts one point between the two pages, each count sets them fully apart. The two passes on the padded page differ by 2 facts and 3 details: that is the play to expect of one count. The four answers of the count were in the form asked for; two of the twelve answers of the scores were not, and were refused, which is why the lean page has one score.

On a real page, the blueprint of a run of 06-borrow-limit played today (857 words, .evals/validation-76/): 7 facts repeated in prose, 2 details, 15%, every passage found on the page, in one session and 0.08 USD.

Not done. The pages of .evals/blind-count-2/ were not counted again by the harness: how close its counts land to the blind reader's on those eleven pages, and whether it sees the fall from campaign 2 to campaign 3 that the reader saw, is not known. That is 11 sessions when wanted.

One thing this count says of the first half: by the reader's method, a sentence that says again that a critical zone is unchanged is a repeat, and the bare line of the sensitive zones is not. On the pages drawn since #83, those sentences are most of what is left to count.

Status

Second half done: the count is in the harness (#89) and separates the pair. First half open, and the developer's: what the rubric calls padding.

Campaign 4: the count follows the fix of the blueprint, on four more cases

Campaign 4: seven runs on main with every fix so far, #87 and #88 included, and the harness of #89, #90 and #91, on the cases no campaign had played again, or once. 04-suspension, 05-fine-cap and 06-borrow-limit: one run played whole and one stopped at the hand over each. 02-reservations: one run stopped at the hand over. Report kept: evals/reports/2026-10-05-d139a1ca0dbe/report.md, compared with campaign 1, the only earlier runs of three of these cases. Raw data, out of git: .evals/campaign-4/. A value is the mean over the runs of a case, with its lowest and highest when they differ.

The six blueprints of campaign 1 were counted by the harness for this, on a copy of that campaign (.evals/count-1/, 6 sessions, 0.82 USD). Each cell: facts repeated in prose, details of implementation, share of the page to skip in percent.

	Campaign 1, by a reader	Campaign 1, by the harness	Campaign 4, by the harness
`02-reservations`	26, 8, 35	19, 11, 35	9, 2, 15
`04-suspension`	17, 6, 38	13, 7, 30	4 (3 to 5), 2, 17.5 (15 to 20)
`05-fine-cap`	10, 4, 29	8, 5, 25	1.5 (1 to 2), 1, 12.5 (10 to 15)
`06-borrow-limit`	8, 4, 22	7, 5, 25	6 (5 to 7), 1.5 (1 to 2), 15
`blueprint.no-padding`, the score		2 on every page	2 on every page

The two cases campaign 4 does not play, in campaign 1: 01-overdue-list 12, 7, 31 by the reader and 12, 7, 15 by the harness; 03-overdue-reminders 16, 8, 48 and 12, 12, 35. The reader's repeats in prose are derived from the places it gave, since it was not asked to set diagrams apart: they are the looser of the two.

The harness ranks the six pages of campaign 1 as the reader did, 02-reservations the most padded and 06-borrow-limit the least, and counts fewer repeats on the long pages.
Between the two campaigns the facts said twice fall by half or more on three cases of four, and the details of implementation on all four, where the score is 2 before and after. That is the fix of A blueprint that says a fact once, and shows behavior, not code #83, seen on the cases campaign 3 did not play. 06-borrow-limit, one behavior and the shortest page, keeps its repeats.
Every passage the counts rest on stands on its page, and two runs of one case differ by 1 or 2 facts.

Status

Second half done and at work: the count is in the harness and follows a fix. First half open, and the developer's: what the rubric calls padding.

Decided by the developer: what the frame asks for is not padding

Recorded on 2026-10-07. None of the three is padding: the closing line, the statement of the critical zones, and the boundaries between components, which stay the developer's to decide. The chain stays as it is.

Status

Its second half is done (#89). Its first half is decided, and not done yet: what is left to write is the line of blueprint.no-padding in evals/judge/rubric.md that says what the frame asks for and is therefore not padding, and the same in the method of the count, evals/judge/count.md, where it bears on it. Then the check of the judge on the lean and padded pair, and a campaign to read the score again.

A rubric of the judge that no longer faults what the chain does on pu…

First half done by #104, not checked on the pair

Merged on 2026-10-07.

The rubric. blueprint.no-padding: three things the frame of a blueprint asks for are not padding, the closing line, the statement of the critical zones, and the boundaries between components, which are the developer's to decide. The first two are bare lines, and say no fact twice even when a criterion says the same; the rule of a zone told again in full still does. A diagram that draws what the prose says is no fact said twice (A diagram is drawn for a chain with no branch, or where the feature has no shape to draw #64).
The method of the count, evals/judge/count.md: the closing line joins the bare line of the critical zones among what is no statement of a fact; the boundaries between components, which component calls which, are no detail of implementation; and neither is part of what the developer could skip.
evals/README.md says that the four campaigns kept were judged before these lines, and the one that holds counts counted before the method set them aside: against one of them, a move of the score or of a count may come from the judge, which the hash of the chain does not show. The figures the README gives for scale were counted by the method as it stood.

Read by an independent reviewer before its merge, which changed five sentences. The same pull request carries the three other lines the developer drew in the rubric: #64, #67, #68.

Left to do, billed

The check of the judge on the lean and padded pair, scripts/gate.sh evals judge-check: 16 sessions and 0.77 USD the last time. Reasoned from the two pages, it holds: both carry the same two lines of the frame, and what the padded page adds is still faulted by sentences left as they were. It was not played: the session that did this work had no token for the container.
A campaign to read blueprint.no-padding again, and the counts by the method as it stands.

Status

Open: both halves are done, and the check of the judge is left to play.

Campaign 5: the check of the judge played, and the score read again

The check of the judge, with the rubric of #104 and the method of the count as they stand (.evals/campaign-5/judge-check.json, out of git; 16 sessions, 0.85 USD), two passes each:

	Lean page	Padded page	Padded page, before #104
`blueprint.no-padding`, the score	4 and 4	2 and 2	2 and 2, against 3 for the lean page
`repeated_in_prose`	0 and 0	8 and 7	5 and 7
`implementation_details`	1 and 0	10 and 9	7 and 10
`skippable_percent`	5 and 0	45 and 45	45 and 45

Two points part the two pages, where one did. The lean page gained a point from the lines the rubric drew, and the padded page did not: the lines excuse what the frame asks for and nothing else. The two other pairs hold: the stripped blueprint 2 and 2 against 3 and 4 on blueprint.decidable, the interview with answered questions 2 and 2 against 5 and 5.

The score on real pages. Campaign 5: twelve runs on main with the nine pull requests of 2026-10-07, the six cases twice each. Report kept: evals/reports/2026-10-07-9d5de55f57a9/report.md. blueprint.no-padding is 2 on ten pages and 3 on the two of 04-suspension.

The reasons changed. None faults the closing line, and none faults a diagram for drawing the prose. What they name is said twice on the page:

the statement of the critical zones that goes on to tell a criterion again, on eight pages of twelve: "the critical-zones line restates criterion 9 instead of standing as a bare line";
the scope or the body that tells a criterion again: "The four-step list in "How a fine is computed" restates criteria 1 to 4".

Three reasons still name which module does what as a detail of implementation, "where the limits live", "which module touches files", an import cycle: the line between a boundary and a detail is not sharp for the judge.

The counts agree with the score. Facts repeated in prose: 5.3 a page, from 1 to 13. Details of implementation: 1.3, from 0 to 3. Share to skip: 9%, from 0 to 15, against 15% in campaign 4, counted before the method set aside what the frame asks for. Every passage stands on its page.

So the score and the count now tell the same thing of a page, and neither faults what the chain does on purpose. The score is still coarse, 2 or 3 on real pages: the count is what follows a change of the extractor.

What is left is of the chain, not of the judge: a page still says about five facts twice, and the first of them is the line of the critical zones, which names the zones the plan leaves alone and says again that a criterion holds them. It is small, and it meets #107, where that same line is in question.

Status

Closed: both halves are done and checked. The count is in the harness (#89), the rubric says what the frame asks for (#104), and the check of the judge parts the lean page from the padded one by two points and by each count. The reading by a human is the developer's, of two pages, in #62.

Navigation Menu

Repository navigation

Evaluations: the judge's `blueprint.no-padding` has one point of range, and no judgement was checked against a human #73

Status

What was seen

Sources

Why it matters

Next

Measured: the score is stable, across passes and across models

Measured: a count separates the pages where the score does not

Next

Campaign 2: the same score, again

Campaign 3: the judge did not see a change that a count sees plainly

Status

The count is in the harness: #89, checked on the pair

Status

Campaign 4: the count follows the fix of the blueprint, on four more cases

Status

Decided by the developer: what the frame asks for is not padding

Status

First half done by #104, not checked on the pair

Left to do, billed

Status

Campaign 5: the check of the judge played, and the score read again

Status

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Evaluations: the judge's blueprint.no-padding has one point of range, and no judgement was checked against a human #73

Description

Status

What was seen

Sources

Why it matters

Next

Activity

PierreMardon commented on Oct 3, 2026

Measured: the score is stable, across passes and across models

PierreMardon commented on Oct 3, 2026

Measured: a count separates the pages where the score does not

Next

PierreMardon commented on Oct 3, 2026

Campaign 2: the same score, again

PierreMardon commented on Oct 3, 2026

Campaign 3: the judge did not see a change that a count sees plainly

Status

PierreMardon commented on Oct 5, 2026

The count is in the harness: #89, checked on the pair

Status

PierreMardon commented on Oct 5, 2026

Campaign 4: the count follows the fix of the blueprint, on four more cases

Status

PierreMardon commented on Oct 7, 2026

Decided by the developer: what the frame asks for is not padding

Status

PierreMardon commented on Oct 7, 2026

First half done by #104, not checked on the pair

Left to do, billed

Status

PierreMardon commented on Oct 7, 2026

Campaign 5: the check of the judge played, and the score read again

Status

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Evaluations: the judge's `blueprint.no-padding` has one point of range, and no judgement was checked against a human #73