Repository navigation
A diagram is drawn for a chain with no branch, or where the feature has no shape to draw #64
Description
Activity
Read again in campaign 2
Campaign 2:
01-overdue-listand03-overdue-reminders, three runs each, onmainwith the four first fixes (#78, #79, #80, #81). Report kept:evals/reports/2026-10-03-e830bc236e9b/report.md. Raw data, out of git:.evals/campaign-2/. In the tables, a value is the mean over the three runs, with its lowest and highest when they differ.01-overdue-list, where the case expects none: one run of three drew a diagram.blueprint.diagrams4.67 (4 to 5).03-overdue-reminders: 1 to 3 diagrams,blueprint.diagrams3.67 (3 to 4). The lowest judgement: "The interval flowchart earns its place because the path branches. The sequence diagram is mostly a linear chain of internal reads and calls with one loop [...] the diagram shows module plumbing rather than a shape the developer needs."
Status
Observed again. Not fixed. The same two things as before: a diagram where the feature has no shape, one run in three; and a sequence diagram of internal calls. The second is partly what #83 addresses, by keeping to behavior and leaving the code to the plan.
Campaign 3
With the fix of the extractor (#83):
blueprint.diagrams4 (3 to 5) on01-overdue-list, 3.5 (3 to 4) on03-overdue-reminders, 3 on02-reservations. Both runs of01-overdue-listdrew a diagram where the case expects none. On03-overdue-remindersthe judge still finds "the module-level sequence diagram" among the "implementation choices the developer does not decide".Status
Unchanged by #83. Not fixed. The prompt still allows a diagram to draw what the prose says, and the rubric still faults it: that disagreement is the thing to settle first.
Campaign 4
Campaign 4: seven runs on
mainwith every fix so far, #87 and #88 included, and the harness of #89, #90 and #91, on the cases no campaign had played again, or once.04-suspension,05-fine-capand06-borrow-limit: one run played whole and one stopped at the hand over each.02-reservations: one run stopped at the hand over. Report kept:evals/reports/2026-10-05-d139a1ca0dbe/report.md, compared with campaign 1, the only earlier runs of three of these cases. Raw data, out of git:.evals/campaign-4/. A value is the mean over the runs of a case, with its lowest and highest when they differ.Campaign 1 Campaign 4 02-reservationsdiagrams, a shape expected3 3 02-reservationsblueprint.diagrams4 3 04-suspensiondiagrams2 1.5 (1 to 2) 04-suspensionblueprint.diagrams4 4 05-fine-capdiagrams, none expected2 1 05-fine-capblueprint.diagrams3 3.5 (3 to 4) 06-borrow-limitdiagrams, none expected1 1 06-borrow-limitblueprint.diagrams5 4.5 (4 to 5) - Where the case expects none, every run still draws one: both runs of
05-fine-capand both of06-borrow-limit. - What the judge faults is of two kinds. A diagram of the dependencies between modules, "implementation wiring the developer does not decide":
04-suspension, both runs. And a chain that is nearly linear, which the prose carries: on02-reservations, "The return sequence diagram, though, is a near-linear chain with optional steps that the surrounding prose already conveys"; on05-fine-cap, "a mostly linear chain of internal calls with one optional branch"; on06-borrow-limit, "a near-linear chain of guards whose order criterion 6 already states in prose, as the text itself admits".
Status
Unchanged over four campaigns. Not fixed. The prompt still allows a diagram to draw what the prose says, the rubric still faults it, and that is still to settle first.
- Where the case expects none, every run still draws one: both runs of
Decided by the developer: a diagram may say the prose again, and a chain with a single fork is no shape
Recorded on 2026-10-07. Between the allowance of the prompt and the strictness of the rubric, each gives way on one point:
- A diagram may say again what the prose says. The rubric stops faulting one for that alone, where the shape is real: several actors, states and their transitions, several branches.
- A chain with a single fork is not a shape. A sentence says it better, and the extractor no longer draws it: it joins the single fact, the list and the chain with no branch.
Status
Open, decided, not done yet. What is left to write: the sentence of "The diagrams" in
agents/surface-extractor.md, the criterionblueprint.diagramsofevals/judge/rubric.md, then several runs of05-fine-capand06-borrow-limit, the two cases that expect no diagram and get one in every run.The diagram the developer prefers
Recorded on 2026-10-07. Of the two pages read (#62), the developer prefers the diagram of the page drawn before #83:
- it draws the procedure: a loop over the members, two decisions, one end at
exit 0, and each step marked new or existing; - the diagram of the page after it draws the data: two files that meet on one decision, with a single fork.
That agrees with the decision above: the first is a shape, the second stands at the edge of what the rule sets aside. It is an example to hold the wording against when the sentence of the extractor is written. The comparison checked each box of both diagrams against the other plan: each is true of both.
Status
Unchanged: open, decided, not done yet.
- it draws the procedure: a loop over the members, two decisions, one end at
Done by #101 and #104, not measured
Merged on 2026-10-07, in the same words on both sides:
- The extractor (A blueprint that draws no diagram for a chain with a single fork #101): a chain with a single fork joins the single fact, the list and the chain with no branch, which a sentence says better. Among the shapes, "a path that branches" becomes "a path that forks more than once", so that the two rules no longer meet on one fork. That a diagram may say the prose again stays.
- The rubric (A rubric of the judge that no longer faults what the chain does on purpose #104):
blueprint.diagramsnames the same shapes and the same chains, and no longer faults a diagram, where the shape is real, for drawing what the prose says.blueprint.no-paddingsays that such a diagram is no fact said twice.
What the rule does not reach
Read in the fourth campaign by an independent reviewer of #101: of the four diagrams drawn on the two cases that expect none, one is a chain with a single fork. Those of
06-borrow-limitline up four or five checks that each refuse or go on, and one of05-fine-capis a sequence between five actors. Sodiagram_as_expectedmay move little on these cases. Nothing was added for a row of checks: the decision does not speak of it, and #77 asks that this pass add nothing to what bears on every blueprint. If the measure shows it, that is the next thing to decide here.What to read on the next campaign
diagramsanddiagram_as_expectedon several runs of05-fine-capand06-borrow-limit,blueprint.diagramsby the rubric as it stands, and that a feature with a shape still gets its diagram, on02-reservationsand03-overdue-reminders.Status
Measured in campaign 5
Campaign 5: twelve runs on
mainwith the nine pull requests of 2026-10-07 (#97 to #105), the six cases twice each:02-reservations,04-suspension,05-fine-capand06-borrow-limitplayed whole,01-overdue-listand03-overdue-remindersstopped at the hand over. The eight runs played whole are conformant and pass every hidden test. Report kept:evals/reports/2026-10-07-9d5de55f57a9/report.md, compared with campaign 4. Raw data, out of git:.evals/campaign-5/. A value is the mean over the runs, with its lowest and highest when they differ.Campaign 4 Campaign 5 blueprint.diagrams, all the pages3.9 (3 to 5) on seven 5 on the twelve 05-fine-cap, diagrams, none expected1 and 1 0 and 0 06-borrow-limit, diagrams, none expected1 and 1 1 and 1 diagram_as_expected, the cases that expect something0.2 0.8 - The chain with a single fork is gone. Neither page of
05-fine-capdraws a diagram, where each drew one. - A feature with a shape keeps its diagram.
02-reservationsdraws 2 and 3, the states of a book and who calls whom;03-overdue-remindersand04-suspensionone each. - The judge faults no diagram, by a rubric that changed with the extractor: 5 on every page, a page without one included.
What the rule does not reach, as foreseen. Both pages of
06-borrow-limitstill draw the row of checks of a borrow, four or five decisions that each refuse or go on. By the rule it is a path that forks more than once, and the judge gives it 5; the case expects no diagram, which is the opinion of the corpus. Whether a row of checks is a shape is left as it is: the developer has not spoken of it, and nothing was added for it.How much of this is the draw of the day. The twelve plans of the campaign were drawn again by the same extractor, alone, for the measure of #63:
05-fine-capthen draws none and one,06-borrow-limitnone and one. On one plan, a diagram is there in one draw and not in the next.Status
Closed: measured. No diagram for a chain with a single fork, a diagram where the shape is real, and a judge that holds the same rule.
- The chain with a single fork is gone. Neither page of
Status
Closed: measured in campaign 5. Observed in the first four campaigns: a diagram drawn where the feature has no shape. Decided by the developer on 2026-10-07: a diagram may say the prose again where the shape is real, and a chain with a single fork is not a shape. #101 wrote it in the prompt of the extractor and #104 in the rubric of the judge. In campaign 5,
05-fine-capdraws no diagram in its two runs, the features with a shape keep theirs, andblueprint.diagramsis 5 on the twelve pages. Left as it is: the row of checks06-borrow-limitstill draws. The measures and their sources are in the comments below, campaign by campaign.What was seen
03-overdue-reminders, 3 of 5: "The sequence diagram mostly draws a linear call chain between internal modules, which is implementation detail. Its only branch, skip or remind, is a simple fork that the prose already states."05-fine-cap, 3 of 5: "The flowchart is a straight chain with a single yes/no branch that the prose just above already states in one sentence."diagram_as_expectedis false on05-fine-cap(2 diagrams) and06-borrow-limit(1 diagram), where the case expects none. On06-borrow-limitthe judge gives 5 all the same.01-overdue-listdrew one diagram in the trial run and none in the campaign: the same case, two paths.Sources
diagramsanddiagram_as_expected, scores, lowest judgement.evals/cases/*/case.json,diagram..evals/campaign-1/runs/*/run-01/judge.json,.evals/trial-1/report.md.Suspected cause
agents/surface-extractor.mdline 71 forbids a diagram for "a chain with no branch", and also says "A diagram may say again what the prose says: it earns its place by the shape it gives, not by new facts." The judge faults a fork "that the prose already states": the prompt allows what the rubric penalizes. A chain with one fork sits between the two.Next
Decide which of the two is right before touching either: the prompt's allowance, or the rubric's strictness. Then read again over several runs.