Repository navigation
Since the interview asks for the rules a user would see, it asks some its developer hands back #94
Description
Activity
- added a commit that references this issue
on Oct 5, 2026 Counted by the harness since #96, on every run kept
questions_handed_back(#96, merged): the answers of the developer of a case that say of a question, or of a part of it, that it is "the assistant's call" or "your call", read by the script fromrun.json. The rules of the developer did not change, and no session was played: the measure is read on the runs kept under.evals/, out of git.Answers that hand a question back Campaign 1, before #81 0 of 8 Campaign 2 1 of 4 Campaign 3 2 of 8 Campaign 4 4 of 15 The runs that measured the fix of #93 1 of 4 By case, since #81:
Case Runs Questions Handed back Runs with one 01-overdue-list5 7 2 2 02-reservations2 7 2 2 03-overdue-reminders5 1 0 0 04-suspension2 5 0 0 05-fine-cap5 6 1 1 06-borrow-limit2 5 3 2 The eight answers are of three kinds, which the measure counts alike:
- A whole question of the interview: three, all on
06-borrow-limit, in its two runs. They are what this issue opened on. - A part of a question: three.
02-reservations, in both of its runs: "What the three commands print is your call, since I only rely on their exit codes", to a question that also asked whatreturnprints (campaign 3) or whencancelis refused (campaign 4), which the developer did answer.01-overdue-list, campaign 2, on the tie beyond the book id: "It doesn't say anything about member id, so the rest is your call." - A question the developer's own correction raised: two.
01-overdue-listin campaign 3 and05-fine-capin the runs of the fix of A draft pull request is opened with a line of attribution under the description the state script prints #93. The developer sent the blueprint back, the session asked what the correction left open, what becomes of an extra argument, and the fine of a book whose price is0, where "Point 1 contradicts itself", and the developer handed that back. They are asked while a revision is drafted, not in the interview, and are counted like the others, asquestionscounts them.
One answer leaves the choice without the words, and is not counted: the only question of
03-overdue-reminderssince #81, campaign 2, run 3, "I don't rely on the order of the printed lines, so take your recommendation."What it says:
- Nothing of the kind before An interview that asks for the rules a user of the feature would see #81, in 8 answers. Since, 8 answers of 31 hand something back.
- A whole question of the interview handed back is still seen on one case only. What shows elsewhere is weaker: a question worth asking that carries a part its developer does not care about, what a command prints, on
02-reservationsin 2 runs of 2. - Two of the eight come from the developer the harness plays, whose correction opened the question. They say nothing of the interview.
Sources:
.evals/campaign-1/tocampaign-4/and.evals/fix-93/,runs/*/run-*/run.json, the stops of kindanswer. The measure:evals/surface_evals/developer.py,hands_back.Status
Open, not confirmed: a whole question of the interview handed back on one case, a part of one on two others. The harness counts it from now on, in every campaign.
- A whole question of the interview: three, all on
Decided by the developer: the chain does not change
Recorded on 2026-10-07. The stop rule of the interview stays as it is. Two wordings of an exception were put to the developer, who set both aside:
- "what the precedent of the code nearly settles is not asked": nearly asks the session for a judgement, which is what makes one run ask and another not;
- "the place of a new check among the existing ones is not asked": precise, and specific to the host of the evaluations. It comes from two cases of the same synthetic project, both on the refusals of
borrow. Made general, "where the new rule and an existing one apply to the same case, the existing one goes first", it is harmless for a refusal and a business rule for two discounts or two access rights.
So the sign does not call for a change of the prompts:
- where the question is not asked, as on
04-suspension, An assumption of the plan becomes an acceptance criterion, and is never put to the developer #60 covers it: the blueprint shows the assumption to confirm; - where it is asked and handed back, as on
06-borrow-limit, it costs a turn, and a developer may answer that it is the assistant's call.
The brief of
06-borrow-limitstays: it plays a developer who delegates.questions_handed_backstays in the report as a sign to follow, and is no longer called better or worse (#96 gave it a better way, fewer).Status
Decided: no fault of the chain. To close once the measure has no better way in
evals/surface_evals/report.py.- added a commit that references this issue
on Oct 7, 2026 - added a commit that references this issue
on Oct 7, 2026 Done by #98
Merged on 2026-10-07.
questions_handed_backleaves the measures that have a better way inevals/surface_evals/report.py: a move of it is told as moved, like a move ofquestions, and never called better or worse.evals/README.mdsays why: a developer may leave a choice to the session, and the brief of a case may play one who does.No session was played: nothing of the chain changes, and the measure is read from the runs as before.
The other half of the decision is done too: where the interview does not ask such a rule, the plan marks it and the blueprint shows it as an assumption of the plan, which approving confirms (#100, for #60).
Status
Closed: no fault of the chain.
Status
Closed: no fault of the chain. Decided by the developer on 2026-10-07: the stop rule of the interview stays as it is, and so does the brief of
06-borrow-limit, which plays a developer who delegates.questions_handed_backstays in the report as a sign to follow, with no better way since #98. Where such a question is not asked, the blueprint shows the assumption of the plan, since #100 (#60). The measures and their sources are in the comments below.What was seen
06-borrow-limit, two runs:studentnorstaff?": "There are only the two categories, student and staff, so that case shouldn't come up. What happens with a bad value is your call, and I'll take your recommendation: A."06-borrow-limitquestionsinterview.questions-matterinterview.right-numberplanning_usdThe judge,
interview.questions-matter3: "Q2 asks about the internal order of the checks, which the developer said was not theirs to decide, so it should have been an implementation decision rather than a question."Counted in the answers kept in the
interview.mdof every run since #81: a question handed back whole in 0 answers of 4 in campaign 2, 1 of 8 in campaign 3 (01-overdue-list: "That's your call, and I'm going with your recommendation, A"), 3 of 15 in campaign 4, all three on this case. An answer also hands back a part of its question in each campaign, such as "What the three commands print is your call, since I only rely on their exit codes" on02-reservations, twice.On the three other cases of campaign 4 nothing of the kind:
interview.questions-matteris 5 on04-suspensionand on05-fine-cap, 4 on02-reservations.Why it is not confirmed
04-suspension, where the order of the new check was not asked,interview.no-invented-ruleis 4 on both runs: "One visible rule was not asked and is not offered as an assumption to confirm: the debt check runs after the existing checks". On06-borrow-limit, where it was asked, the question is faulted.borrow", and says of the categories "There are only these two". So the prompt and the case do not hold the same idea of whose that order is: by the rule of An interview that asks for the rules a user of the feature would see #81 it is the developer's, by the case it is the implementer's. A developer at a real desk may care which refusal is read. A run measures the chain and its developer together (Evaluations: one run per case, so the spread between runs is unknown and the next report cannot say what moved #74).So before anything is fixed there is a choice, and it is the developer's: whether the order in which refusals are reported is theirs to decide. Then either the stop rule of the interview or the brief of the case says so.
Why it matters
A question is a turn of the developer, and here a third more of the cost of planning. An interview that asks what its developer does not care about also teaches them to answer "your call", which is the answer a guessed rule hides behind.
Sources
evals/reports/2026-10-05-d139a1ca0dbe/report.md..evals/campaign-4/runs/06-borrow-limit/run-01/andrun-02/, theinterview.mdof the plan folder of each project, andjudge.json.skills/surface-plan/SKILL.md, as An interview that asks for the rules a user of the feature would see #81 left it.evals/cases/06-borrow-limit/brief.md.Next
Read
questions_handed_backper case on the next campaign: the harness counts the answers that hand a question back since #96, from whatrun.jsonkeeps. If the sign holds on other cases, what the stop rule of the interview lacks is a word on what the precedent of the code nearly settles, such as the place of a new check among the existing ones. Tied to #60, which asks the opposite question: how a rule nobody asked reaches the developer.