A stop ends on a remark, not on what the developer does next

[PierreMardon]

Status

Open: done by #103, not measured. The step a stop names is right since #78, but the hand back at conformity ended on a remark or on file pointers, with its step in the middle: in 6 of the 7 runs that reached conformity over campaigns 3 and 4. Decided by the developer on 2026-10-07 with #67: the hand back at conformity is its one line and nothing else, and ends on the developer's step. #103 writes it in the prompt of /surface-execute. To close once runs played whole have been read. The measures and their sources are in the comments below, campaign by campaign.

What was seen

• 03-overdue-reminders, stop 4: "ends on a remark about untouched files, not on an action". Passage: "Read that change yourself. lending/fines.py and lending/store.py are untouched."
• 02-reservations, stop 7: "ends on a restatement of the state rather than on the action to take."
• 04-suspension: "Stop 4 also ends on a status remark instead of the action, and Stop 1 ends on what the agent will ask later."

Stops 4 of 04-suspension and 7 of 02-reservations are the replies to "Looks fine to me, go ahead.", the sentence that must approve nothing: the session says it approved nothing, and the judge wants it to end on the launch of /surface-execute.

Sources

• Report of campaign 1, committed: messages.next-step, scores.
• Raw data, out of git: .evals/campaign-1/runs/*/run-01/judge.json, run.json.

Suspected cause

None located: step 11 of /surface-plan does ask to "say that the launch of /surface-execute approves".

Next

Read messages.next-step again once the hand back at conformity is fixed: what is left of the low scores is this issue.

[PierreMardon]

Read again in campaign 2

Campaign 2: 01-overdue-list and 03-overdue-reminders, three runs each, on main with the four first fixes (#78, #79, #80, #81). Report kept: evals/reports/2026-10-03-e830bc236e9b/report.md. Raw data, out of git: .evals/campaign-2/. In the tables, a value is the mean over the three runs, with its lowest and highest when they differ.

messages.next-step: 3.33 (3 to 4) on 01-overdue-list, 4 on 03-overdue-reminders, where both were 3. With the hand back at conformity fixed (#66), what is left is small. The lowest judgement: "Stop 1 ends on what the agent will do rather than on the developer answering Q1", for a question that closes on "I've written the question into interview.md and will record your answer there before drafting the plan."

Status

Mostly gone with #66. One kind left, minor: a question that ends on what the session will do next. Not fixed.

[PierreMardon]

Campaign 3: the hand back at conformity ends on file pointers, in three runs of five

Campaign 3: five runs on three cases, played from the branch of #83. Report kept: evals/reports/2026-10-03-d765f898c9ad/report.md. Raw data, out of git: .evals/campaign-3/. Written on 2026-10-05 from the judgements of that campaign: this issue had not been read again after it.

messages.next-step: 4.5 (4 to 5) on 01-overdue-list, 3 on 02-reservations, 3.5 (3 to 4) on 03-overdue-reminders.

• In three runs of five, the last message, the hand back at conformity, names the right step in its middle, open the pull request and mark it ready, and ends on something else. 02-reservations: "The final message ends on a remark that the critical zones are untouched, and the real next step, opening the pull request and marking it ready, is buried in the middle." Passage: "Here there are none: lending/fines.py and the loans.jsonl format are untouched." Both runs of 03-overdue-reminders end on where the reports are: "The proof is in docs/plans/2026-10-03-overdue-reminders/conformity.md, and the review is in reviews/pass-01.md."
• One run of 01-overdue-list ends a stop on a side check, "Tell me if you meant something wider.", where the step that unblocks the redraft is the answer to its open question.
• Every stop before the last one ends on the right step, in the five runs: answer the question, amend, or launch /surface-execute. The reply to the sentence that agrees is faulted in none of them.

So the step is the right one since #78, and the order of the message is not. This is the first kind this issue was opened on, a stop that "ends on a remark about untouched files, not on an action", at the one stop where every session writes a report of several bullets and the prompt asks to "hand back in one line". Whether that message is held to its line is open in #67.

Status

Open, not fixed, and not gone: seen in three runs of five in campaign 3, at the hand back at conformity. Tied to #67: a hand back held to one line would end on its step.

[PierreMardon]

Campaign 4: three hand backs of three end on something else than their step

Campaign 4: seven runs on main with every fix so far, #87 and #88 included, and the harness of #89, #90 and #91, on the cases no campaign had played again, or once. 04-suspension, 05-fine-cap and 06-borrow-limit: one run played whole and one stopped at the hand over each. 02-reservations: one run stopped at the hand over. Report kept: evals/reports/2026-10-05-d139a1ca0dbe/report.md, compared with campaign 1, the only earlier runs of three of these cases. Raw data, out of git: .evals/campaign-4/. A value is the mean over the runs of a case, with its lowest and highest when they differ.

messages.next-step: 3 on the three runs played whole, where it was 2, 3 and 3 in campaign 1.

• 04-suspension: "Stop 6 ends on file locations rather than on the developer's step (read lending/loans.py:81-88, then mark the PR ready), which sits buried mid-message."
• 05-fine-cap: "Stop 4 ends on a remark about the executor's cd. The real next steps, reading the two critical files and then marking the pull request ready, are buried earlier in the message."
• 06-borrow-limit: "it ends on that file rather than on the step that should come first: read lending/loans.py, then mark the PR ready".

With a pull request open at last (#91), the step is the one the prompt names, mark it ready, and it is still not where the message ends.

Of planning alone, messages-in-planning.next-step is 5, 4, 4 and 4. The one kind left is the first stop, which "ends on a list of later questions", or "on what the agent will do next rather than asking the developer to answer Q1".

Status

Open, not fixed: at the hand back at conformity, in 6 of the 7 runs that reached it over campaigns 3 and 4. Tied to #67.

[PierreMardon]

Decided by the developer: the hand back at conformity is held to its one line

Recorded on 2026-10-07. The hand back at conformity is the line the prompt asks for, and nothing else: the plan is conformant, the files of the critical zones to read when there are any, then the developer's step, last. No report of what was done: the description of the pull request holds it.

The prompt already asks for it "in one line", and every session writes several paragraphs that end on something else (#67, #69). So the sentence is to be made firmer, and the fix measured on runs played whole.

Status

Open, decided, not done yet.

[PierreMardon]

Done by #103, not measured

Merged on 2026-10-07. In the prompt of /surface-execute, the line said at conformity is all the session says at that stop, in this order: the plan is conformant, the changed files of the critical zones by name when there are any, and last the developer's step, or the step that fits when the refresh reached no pull request. "One line" is said to mean a sentence or two, with no list and no heading, and nothing before or after it. The detail is in #67.

What is left of this issue besides: a first stop that ends on what the session will do next, in the runs stopped at the hand over. It was not touched.

What to read on the next campaign

The last message of the runs played whole: what it ends on, and messages.next-step.

Status

Open: done by #103, not measured.