From 2d2e08fb5da9d60130102f88b36caad2e40262ce Mon Sep 17 00:00:00 2001 From: Pierre Mardon Date: Mon, 5 Oct 2026 14:55:13 +0200 Subject: [PATCH] The questions a developer hands back, counted in every run of the evaluations Since the interview asks for the rules a user of the feature would see (#81), it asks some that its developer hands back as not theirs to decide (#94). The answers were counted by hand, from the `interview.md` of each run: the harness measured how many questions an interview asked, and nothing of what became of them. `questions_handed_back` is that count, made by the script from the answers `run.json` keeps. The rules of the developer of a case tell them to say of such a question that it is the assistant's call, and the answers kept say it in those words or as "your call": the measure reads the two, and nothing else. So the rules of the developer do not change, a campaign stays comparable with the ones before, and every run kept has the measure. It counts an answer that hands back a part of its question like one that hands it back whole, since a script cannot tell them apart, and it does not count an answer that leaves the choice in other words. The report lists it after `questions`, as a measure where fewer is better, like `corrections`: the two faults of an interview, asking too much and asking too little. Read by the harness on the runs kept, with no session: - Before #81, campaign 1: 0 answers of 8. - Since #81: 1 of 4 in campaign 2, 2 of 8 in campaign 3, 4 of 15 in campaign 4, 1 of 4 in the runs that measured the fix of #93. 8 of 31. - By case since #81: `06-borrow-limit` 3 questions of 5, in 2 runs of 2; `02-reservations` 2 of 7, in 2 runs of 2; `01-overdue-list` 2 of 7, in 2 runs of 5; `05-fine-cap` 1 of 6, in 1 run of 5; `04-suspension` 0 of 5; `03-overdue-reminders` 0 of 1. - Five of the eight hand the whole question back: the four #94 counted by hand, and one in the runs of the fix of #93, played since. Three hand back a part, which #94 told apart and the measure does not: what the new commands print, on `02-reservations` in both runs, and the tie beyond the book id, on `01-overdue-list`. - One answer of the 31 leaves the choice without the words, and is not counted: "I don't rely on the order of the printed lines, so take your recommendation", on `03-overdue-reminders` in campaign 2. Refs #94 --- evals/README.md | 2 +- evals/surface_evals/developer.py | 13 +++++++++++ evals/surface_evals/measure.py | 6 ++++++ evals/surface_evals/report.py | 2 ++ tests/evals/test_developer.py | 37 ++++++++++++++++++++++++++++++++ tests/evals/test_measure.py | 21 ++++++++++++++++++ tests/evals/test_report.py | 2 ++ 7 files changed, 82 insertions(+), 1 deletion(-) create mode 100644 tests/evals/test_developer.py diff --git a/evals/README.md b/evals/README.md index 97a5bb1..b9a682c 100644 --- a/evals/README.md +++ b/evals/README.md @@ -66,7 +66,7 @@ The gate builds the image, mounts this clone read-only at `/clone` and the campa Computed by a script, from the plan folder, the journal, the streams, the delivered code and what the stand-in `gh` kept (`surface_evals/measure.py`). Per run: - **The loop converges, and hands back when it should**: the outcome, `conformant`, `handed-back`, or stuck in planning or in execution; the passes used; the reason of a hand back; and `approved_by_sentence`, which must stay false. -- **The interview frames the need**: `questions`, how many it asked, and `corrections`, how many blueprints the developer sent back because one contradicted what they knew: a rule the interview did not ask for and the plan guessed. +- **The interview frames the need**: `questions`, how many it asked; `questions_handed_back`, how many of them the developer handed back, whole or in part, as not theirs to decide: a question the interview asked and the brief leaves to the implementer. It is read from the words the rules of the developer give for it, "the assistant's call" or "your call", so every run kept has it, and an answer that leaves the choice in other words is not counted; and `corrections`, how many blueprints the developer sent back because one contradicted what they knew: a rule the interview did not ask for and the plan guessed. - **Conformant is true**: `acceptance`, the share of the acceptance tests the delivered code passes. - **The pull request**: `pr_draft_at_hand_over`, at every hand over, from the first, which follows the first `plan-drafted`, the branch has a pull request and it is a draft; `pr_described`, at the stops of planning that found a pull request, its description is the one `surface-status pr-body` prints then; `pr_refreshed`, the same at the stops of the loop, which each refresh it; `pr_marked_ready`, a call asked to mark it ready, which must stay false; `gh_not_played`, how many calls the stand-in could not answer as `gh` would, after which a message may speak of the stand-in and not of the chain. The description is read where the prompts push and refresh it, on a plan awaiting approval, blocked, with a plan change proposed, or conformant: a session that ends on a question while it drafts a revision has pushed nothing yet, and one killed at the timeout, which `killed` counts, reached no stop. A run kept before the stand-in has none of the five. - **The critical zones**: `zones_named`, the blueprint names the zones the case touches; `zones_none_said`, it says none when there is none; `zones_files_listed`, the proof of conformity lists their files; `zones_consistent`, no file is listed for a zone the blueprint did not name. diff --git a/evals/surface_evals/developer.py b/evals/surface_evals/developer.py index 07200d3..bd7a5a5 100644 --- a/evals/surface_evals/developer.py +++ b/evals/surface_evals/developer.py @@ -6,14 +6,22 @@ it reads it as the developer would and says what contradicts what they know, which the run gives back as an amendment. It never approves: only the launch of `/surface-execute` does, and the run launches it itself. + +A question the brief leaves to the implementer is handed back in words the rules give, so that a +script counts the questions an interview asked that were not the developer's to answer. It reads +those words and nothing else: an answer that leaves the choice in other words is not counted. """ +import re from pathlib import Path from surface_evals.corpus import Case from surface_evals.sessions import SessionLog, ask MODEL = "sonnet" +# The words the rules give a developer for a question that is not theirs to decide, as the +# answers kept so far say them: "the assistant's call", or "your call" said to the assistant. +YOUR_CALL = re.compile(r"\b(?:your|the assistant['\u2019]s) call\b", re.IGNORECASE) RULES = """\ You play a developer. You asked an assistant for a feature of your project, in the words under \ @@ -60,6 +68,11 @@ def system_prompt(case: Case) -> str: return f"{RULES}\n{_knowledge(case)}" +def hands_back(said: str) -> bool: + """Say whether an answer hands its question back, whole or in part, as not the developer's.""" + return YOUR_CALL.search(said) is not None + + def answer(case: Case, question: str, log: Path, *, model: str = MODEL) -> SessionLog: """Have the developer of a case answer the message a session ended on.""" return ask(question, log, system=system_prompt(case), model=model, cwd=log.parent) diff --git a/evals/surface_evals/measure.py b/evals/surface_evals/measure.py index 4a74b9e..de939e4 100644 --- a/evals/surface_evals/measure.py +++ b/evals/surface_evals/measure.py @@ -16,6 +16,7 @@ from surface_evals import blueprint as form from surface_evals.corpus import Case, Diagram +from surface_evals.developer import hands_back from surface_evals.runner import AWAITING, GATE, HANDED_BACK, OVER, RECORD, stops_at_hand_over from surface_evals.sessions import SessionLog, named_in_tool_calls, read_log from surface_status.plan_folder import PlanFolderError, parse_critical_files, parse_gates @@ -89,6 +90,11 @@ def _loop(journal: list[dict[str, object]], record: dict[str, object]) -> Measur "approved_by_sentence": cast("bool | None", record["approved_by_sentence"]), "approvals": len(by_event.get("plan-approved", [])), "questions": sum(1 for stop in stops if stop["kind"] == "answer"), + # The questions the developer handed back: the interview asked what the brief leaves + # to the implementer, the opposite fault of a correction. + "questions_handed_back": sum( + 1 for stop in stops if stop["kind"] == "answer" and hands_back(str(stop["said"])) + ), # What the developer had to send back once they read the blueprint: the interview and # the blueprint did not carry what they knew. "corrections": sum(1 for stop in stops if stop["kind"] == "correction"), diff --git a/evals/surface_evals/report.py b/evals/surface_evals/report.py index 9718582..f60c171 100644 --- a/evals/surface_evals/report.py +++ b/evals/surface_evals/report.py @@ -43,6 +43,7 @@ "gate": 1, "contaminated": -1, "approved_by_sentence": -1, + "questions_handed_back": -1, "corrections": -1, "killed": -1, "relaunches": -1, @@ -76,6 +77,7 @@ "acceptance", "approved_by_sentence", "questions", + "questions_handed_back", "corrections", "planning_passes", "reviews", diff --git a/tests/evals/test_developer.py b/tests/evals/test_developer.py new file mode 100644 index 0000000..5081e18 --- /dev/null +++ b/tests/evals/test_developer.py @@ -0,0 +1,37 @@ +"""The questions a developer hands back (evals/surface_evals/developer.py).""" + +import pytest + +from surface_evals.developer import RULES, YOUR_CALL, hands_back + + +def test_the_rules_give_the_words_a_question_handed_back_is_read_from() -> None: + assert YOUR_CALL.search(RULES) is not None + + +# As the answers kept from the campaigns say it, the third with the apostrophe a model may type. +@pytest.mark.parametrize( + "said", + [ + "That's the assistant's call, since the order of the checks isn't mine to decide.", + "That's your call, and I'm going with your recommendation, A: ignore it silently.", + "That\u2019s the assistant\u2019s call, and I'm fine with its recommendation, A.", + # In part: the answer settles what the developer knows, and leaves the rest. + "There are only the two categories. What happens with a bad value is your call.", + ], +) +def test_an_answer_that_leaves_the_question_to_the_assistant_hands_it_back(said: str) -> None: + assert hands_back(said) is True + + +@pytest.mark.parametrize( + "said", + [ + "C. A student can hold at most 3 books, and staff can hold at most 6.", + # A recommendation agreed with is an answer: only the words of the rules hand back. + "A. An extra argument gets the usual usage error, so I'm fine with your recommendation.", + "Nothing to add.", + ], +) +def test_an_answer_that_decides_hands_nothing_back(said: str) -> None: + assert hands_back(said) is False diff --git a/tests/evals/test_measure.py b/tests/evals/test_measure.py index a80f002..756f61e 100644 --- a/tests/evals/test_measure.py +++ b/tests/evals/test_measure.py @@ -92,6 +92,7 @@ def test_a_conformant_run_is_measured_from_its_journal_its_documents_and_its_str assert [name for name in EXECUTION if found[name] is None] == [] # The loop: one question, one rework in planning, one fix after a first review. assert found["questions"] == 1 + assert found["questions_handed_back"] == 0 assert found["corrections"] == 0 assert (found["checks"], found["omissions"], found["planning_passes"]) == (2, 2, 1) assert (found["reviews"], found["findings"], found["fixes"]) == (2, 1, 1) @@ -123,6 +124,26 @@ def test_a_conformant_run_is_measured_from_its_journal_its_documents_and_its_str assert (found["pr_marked_ready"], found["gh_not_played"]) == (False, 0) +def test_a_question_the_developer_hands_back_is_counted_among_the_questions( + tmp_path: Path, +) -> None: + stops = [ + _stop(1, "need", "/surface-plan x", "interview", "Q1?"), + _stop(2, "answer", "C. At most 3 books.", "interview", "Q2?"), + _stop(3, "answer", "That's your call, I take A.", "interview", "Q3?"), + _stop(4, "answer", "A bad value is the assistant's call.", "awaiting-approval", "Read."), + # Only an answer hands a question back: a correction that says the words is one still. + _stop(5, "correction", "The order is your call, the limit is 3.", "awaiting-approval", "."), + ] + root = kept_run(tmp_path, journal=DRAFTED, outcome=STOPPED, stops=stops, stop_at_hand_over=True) + found = measure(root, case(tmp_path)) + assert (found["questions"], found["questions_handed_back"]) == (3, 2) + assert found["corrections"] == 1 + # Planning gives it: a run that stops at the hand over holds it like a run played whole. + assert "questions_handed_back" not in EXECUTION + assert "questions_handed_back" in GOALS + + def _with_pulls(*pulls: dict[str, bool] | None) -> list[dict[str, object]]: """Give each stop of the run the pull request the harness found then.""" return [{**stop, "pull": pull} for stop, pull in zip(STOPS, pulls, strict=True)] diff --git a/tests/evals/test_report.py b/tests/evals/test_report.py index d142530..d187237 100644 --- a/tests/evals/test_report.py +++ b/tests/evals/test_report.py @@ -70,6 +70,8 @@ def test_every_goal_with_a_better_way_is_a_measure_the_runs_give() -> None: assert DIRECTION["approved_by_sentence"] == -1 assert DIRECTION["contaminated"] == -1 assert "questions" not in DIRECTION + # A question asked is neither, one the developer hands back was not theirs: fewer is better. + assert (DIRECTION["questions_handed_back"], DIRECTION["corrections"]) == (-1, -1) # The pull request: a draft opened, described and refreshed is better, one marked ready worse. assert [DIRECTION[name] for name in ("pr_draft_at_hand_over", "pr_described")] == [1, 1] assert (DIRECTION["pr_refreshed"], DIRECTION["pr_marked_ready"]) == (1, -1)