The questions a developer hands back, counted in every run of the evaluations

[PierreMardon]

What changes

A run of the evaluations now measures questions_handed_back: among the questions of the interview, how many the developer of the case handed back as not theirs to decide. The report lists it after questions.

Why

Since the interview asks for the rules a user of the feature would see (#81), it asks some that its developer hands back (#94). That issue counted the answers by hand, from the interview.md of each run, and named the count as what the harness lacked: it measured how many questions an interview asked, and nothing of what became of them.

How it is counted

By the script, from the answers run.json keeps, with no session and no judge.

• The rules of the developer of a case already tell them to say of such a question "that it is the assistant's call". The answers kept say it in those words, or as "your call". The measure reads the two, and nothing else.
• So the rules of the developer do not change: a campaign stays comparable with the ones before, and every run kept has the measure.
• It counts an answer that hands back a part of its question like one that hands it back whole: a script cannot tell them apart.
• It does not count an answer that leaves the choice in other words. One of the 31 answers kept since An interview that asks for the rules a user of the feature would see #81 does.
• Fewer is better in the report, like corrections: the two faults of an interview, asking too much and asking too little. Whether a question handed back on 06-borrow-limit is a fault of the interview or of the case is still the choice Since the interview asks for the rules a user would see, it asks some its developer hands back #94 leaves to the developer, and one line of DIRECTION to change.

Files

• evals/surface_evals/developer.py: the words, and hands_back.
• evals/surface_evals/measure.py, report.py: the measure, its place in the goals table, its better way.
• evals/README.md: the measure, and what it does not count.
• tests/evals/test_developer.py, new, test_measure.py, test_report.py.

Measured

Read by the harness of this branch on the runs kept under .evals/, out of git, with no session:

	Answers that hand a question back
Campaign 1, before #81	0 of 8
Campaign 2	1 of 4
Campaign 3	2 of 8
Campaign 4	4 of 15
The runs that measured the fix of #93	1 of 4

By case, since #81:

Case	Runs	Questions	Handed back	Runs with one
01-overdue-list	5	7	2	2
02-reservations	2	7	2	2
03-overdue-reminders	5	1	0	0
04-suspension	2	5	0	0
05-fine-cap	5	6	1	1
06-borrow-limit	2	5	3	2

Against the count made by hand in #94:

• Five of the eight answers hand the whole question back: the four Since the interview asks for the rules a user would see, it asks some its developer hands back #94 counted, one on 01-overdue-list and three on 06-borrow-limit, and one on 05-fine-cap in the runs of the fix of A draft pull request is opened with a line of attribution under the description the state script prints #93, played since.
• Three hand back a part of it, which Since the interview asks for the rules a user would see, it asks some its developer hands back #94 told apart and the measure does not. On 02-reservations, in both runs: "What the three commands print is your call, since I only rely on their exit codes." On 01-overdue-list: "my rule only says book id. It doesn't say anything about member id, so the rest is your call."
• One answer leaves the choice without the words, and is not counted: "I don't rely on the order of the printed lines, so take your recommendation", on 03-overdue-reminders in campaign 2.

Not measured: a campaign played with this harness. The reports kept in evals/reports/ are not written again, so they do not hold the measure.

Refs #94