A stand-in gh in the test image: the pull request steps played, recorded and measured

ARCHITECTURE.md

@@ -41,7 +41,7 @@ host CI    --runs----> surface-status check --require conformant
   - `report.py` (list, `show`, `check`), `pr_body.py`, `settings.py`, and `cli.py` (arguments, exit codes, JSON).
 - `agents/surface-*.md`: the four roles.
 - `install.py`: install, update, `--check`, and the one-line form.
-- `tests/`: one folder per part (`state`, `journal`, `cli`, `git`, `gates`, `install`, `prompts`, `evals`), `ci/`, `e2e/`, and golden files in `fixtures/`.
+- `tests/`: one folder per part (`state`, `journal`, `cli`, `git`, `gates`, `install`, `prompts`, `evals`), `ci/`, `e2e/`, `gh/` for the stand-in `gh` of the end to end image, and golden files in `fixtures/`.
 - `evals/`: the evaluations of the chain on real sessions. `host/`, a synthetic project the chain is installed in; `cases/`, one need per shape, each with a brief, acceptance tests the agents never see and a reference implementation; `judge/`, the rubric, the method of the count and the spoiled documents that check the judge; `surface_evals/`, the harness; `reports/`, the campaigns kept; `run.py`, its command line.
 - `scripts/gate.sh`: every gate. `scripts/check_commit_msg.py`: the commit message hook.
 
@@ -105,11 +105,11 @@ The loop stops at every command the developer's rules or mode do not allow, and
 
 ### Tests
 
-Unit and hypothesis property tests cover each part of the script and the installer, with sockets blocked; the `ci` profile runs 500 examples. Golden journals replay to their expected states. Prompt tests hold the frontmatter, the contract the prompts carry (`tests/prompts/chain_contract.py`), host neutrality and privacy. The privacy check reads plan folders, where a path of the machine is a leak too, while the decision-link check skips them, since a plan names decisions still to come. `tests/ci/` proves the conformity check in a checkout shaped like a CI's, and the `conformity` job of `ci.yml` runs it again on a real runner. End to end tests run real sessions on a toy project, in a container, on demand only, since they are billed ([ADR 0025](docs/adr/0025-end-to-end-on-a-toy-project.md)).
+Unit and hypothesis property tests cover each part of the script and the installer, with sockets blocked; the `ci` profile runs 500 examples. Golden journals replay to their expected states. Prompt tests hold the frontmatter, the contract the prompts carry (`tests/prompts/chain_contract.py`), host neutrality and privacy. The privacy check reads plan folders, where a path of the machine is a leak too, while the decision-link check skips them, since a plan names decisions still to come. `tests/ci/` proves the conformity check in a checkout shaped like a CI's, and the `conformity` job of `ci.yml` runs it again on a real runner. End to end tests run real sessions on a toy project, in a container, on demand only, since they are billed ([ADR 0025](docs/adr/0025-end-to-end-on-a-toy-project.md)). Their image reaches no GitHub, and a bare remote on disk gives the real `gh` nothing to open a pull request on: a stand-in, `tests/e2e/toy/gh_stand_in.py`, first on the `PATH` of the image, answers the pull request calls of the prompts and refuses the others, so that the steps a developer with `gh` lives are played there. It keeps the pull requests and a record of every call in the git directory of the project, which every repository has, which is one project's alone and which stays out of the work tree; the scenarios and the evaluations read them, and `tests/gh/` holds the stand-in without a session. It belongs to the test image only: nothing of it is installed in a host.
 
 ### Evaluations
 
-The tests say that the chain works, not that it is worth using. The evaluations of `evals/` answer that after a change of the prompts, on real sessions and with no human in them, in the container of the end to end tests, on demand and never in CI ([ADR 0036](docs/adr/0036-evaluations-on-real-sessions.md)). A run plays the developer's side on a synthetic host: it types the need of a case, has a model answer each question from a written brief, since written answers cannot meet questions that change from run to run, then read the blueprint and send back what contradicts the brief, replies with a sentence that agrees, which must approve nothing, and launches `/surface-execute`. A campaign that measures a change of planning alone stops its runs after that sentence, at the hand over: what the execution gives is then left empty, never 0, and what the judge says of their stops is kept under another name, so that a run stopped on purpose is not read as one that failed to conform. A script measures what a script can, from the journal, the plan folder, the streams and the delivered code: the outcome, the passes, acceptance tests kept from the agents, the critical zones, the form of the blueprint; and, on the toy, whether a fault put there on purpose is found by the cross-check and classified by the review. A model judges the rest against a written rubric, the blueprint, the interview, the messages, each judgement with the passage it rests on, and is itself checked: a document spoiled in a known way must score below its original. Beside its scores, and in a call of its own that leaves them as they are asked, it counts what the developer would skip on the blueprint: the facts said more than once in prose, the details of implementation that are not theirs to decide, the share of the page, each fact and each detail with its passages, since a score from 1 to 5 gave the same value to pages that a reader who counts tells apart. The script makes the totals from those lists, and the count is checked like the scores: the padded blueprint must count more than the lean one. A measure is kept as its spread over the runs, and the report says what lies outside the spread of the campaign before: there is no threshold, since what to hold is decided once the measures have been observed. A ledger stops a campaign before a ceiling, in USD at list price or in points of the subscription's weekly gauge.
+The tests say that the chain works, not that it is worth using. The evaluations of `evals/` answer that after a change of the prompts, on real sessions and with no human in them, in the container of the end to end tests, on demand and never in CI ([ADR 0036](docs/adr/0036-evaluations-on-real-sessions.md)). A run plays the developer's side on a synthetic host: it types the need of a case, has a model answer each question from a written brief, since written answers cannot meet questions that change from run to run, then read the blueprint and send back what contradicts the brief, replies with a sentence that agrees, which must approve nothing, and launches `/surface-execute`. A campaign that measures a change of planning alone stops its runs after that sentence, at the hand over: what the execution gives is then left empty, never 0, and what the judge says of their stops is kept under another name, so that a run stopped on purpose is not read as one that failed to conform. A script measures what a script can, from the journal, the plan folder, the streams, the delivered code and what the stand-in `gh` kept: the outcome, the passes, acceptance tests kept from the agents, the critical zones, the form of the blueprint, the pull request opened as a draft, its description current wherever the plan is left to the developer, and never marked ready; and, on the toy, whether a fault put there on purpose is found by the cross-check and classified by the review. A model judges the rest against a written rubric, the blueprint, the interview, the messages, each judgement with the passage it rests on, and is itself checked: a document spoiled in a known way must score below its original. Beside its scores, and in a call of its own that leaves them as they are asked, it counts what the developer would skip on the blueprint: the facts said more than once in prose, the details of implementation that are not theirs to decide, the share of the page, each fact and each detail with its passages, since a score from 1 to 5 gave the same value to pages that a reader who counts tells apart. The script makes the totals from those lists, and the count is checked like the scores: the padded blueprint must count more than the lean one. A measure is kept as its spread over the runs, and the report says what lies outside the spread of the campaign before: there is no threshold, since what to hold is decided once the measures have been observed. A ledger stops a campaign before a ceiling, in USD at list price or in points of the subscription's weekly gauge.
 
 ### Toolchain
 

evals/README.md

@@ -8,7 +8,7 @@ Every command but `report` is billed and takes minutes to hours: run them on dem
 
 ## What you need
 
-What the end to end tests need (`tests/e2e/README.md`): Docker, and a token in your environment, `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY`. The sessions bypass permissions, so they run in the same container, and refuse to start anywhere else.
+What the end to end tests need (`tests/e2e/README.md`): Docker, and a token in your environment, `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY`. The sessions bypass permissions, so they run in the same container, and refuse to start anywhere else. That container reaches no GitHub: a stand-in for `gh` plays the pull request steps of the chain and records every call, so that these steps are run and measured, and no message has a missing `gh` to tell. What the stand-in answers and what it does not imitate is in `tests/e2e/README.md`.
 
 ## The corpus
 
@@ -37,11 +37,11 @@ The sessions run in the container, where this clone is mounted at `/clone`: a se
 
 ## A run
 
-A run plays the developer's side of the README: the need, an answer to each question, then a reading of the blueprint handed over, whose corrections go back as amendments, twice at most. It gives the amendment of the case when it has one, replies with a sentence that agrees, "Looks fine to me, go ahead.", which must approve nothing, then launches `/surface-execute`, which approves. It stops when the plan is conformant or the loop hands back, runs the acceptance tests, and keeps everything under `runs/<case>/run-NN/`: the project, the stream of every session, each blueprint handed over, and `run.json`, what was said at each stop and the state of the plan then.
+A run plays the developer's side of the README: the need, an answer to each question, then a reading of the blueprint handed over, whose corrections go back as amendments, twice at most. It gives the amendment of the case when it has one, replies with a sentence that agrees, "Looks fine to me, go ahead.", which must approve nothing, then launches `/surface-execute`, which approves. It stops when the plan is conformant or the loop hands back, runs the acceptance tests, and keeps everything under `runs/<case>/run-NN/`: the project, the stream of every session, each blueprint handed over, and `run.json`, what was said at each stop, the state of the plan then, and the pull request of the branch then, as the stand-in `gh` keeps it: whether it is a draft, and whether its description is the one `surface-status pr-body` prints at that moment. The calls themselves stay in the project, in `.git/gh-stand-in/calls.jsonl`.
 
 `corpus --stop-at-hand-over` stops each run at the hand over, for a campaign that measures a change of planning alone, the interview or the blueprint, without paying for the execution. The run plays planning as above, the sentence that agrees included, since it is one cheap turn and `approved_by_sentence` is a measure of planning. It then launches nothing and runs no acceptance test. Its outcome is `stopped-at-hand-over`, which no run played whole has, and its `run.json` says `stop_at_hand_over`.
 
-Nothing of the execution is read from such a run, whatever its outcome. Every measure the execution gives or adds to is empty for it, not 0 or false, so that it is never read as a run that failed to conform, nor counted with the runs played whole: `conformant`, `acceptance`, `gate`, the approvals, the reviews and their findings, the fixes, the failed gate runs, the sessions killed or relaunched, the files of the critical zones, and `usd`. `planning_usd`, what planning cost, is the one cost both kinds of runs have. The judge is asked the blueprint and the interview as for any run, since both are whole at the hand over: those scores compare with a campaign played whole. It is asked the messages and following along of planning alone, and told that the run stops there; it keeps those scores under names of their own, `messages-in-planning` and `following-in-planning`, since they answer another question than those of a run judged to its end. The report says how many runs played planning alone, and compares with another campaign only what both measured.
+Nothing of the execution is read from such a run, whatever its outcome. Every measure the execution gives or adds to is empty for it, not 0 or false, so that it is never read as a run that failed to conform, nor counted with the runs played whole: `conformant`, `acceptance`, `gate`, the approvals, the reviews and their findings, the fixes, the failed gate runs, the sessions killed or relaunched, the files of the critical zones, `pr_refreshed`, `pr_marked_ready`, and `usd`. Of the pull request, such a run keeps what planning gives: the draft at the hand over, its description there, and the calls the stand-in did not play, which say how far to trust the run as `contaminated` does. Never marked ready is a promise to the end of the chain, and the hand back at conformity, where a chain would break it, is not reached: a false would claim what was not played. A pull request that planning marked ready still shows, since it is no draft at the hand over. `planning_usd`, what planning cost, is the one cost both kinds of runs have. The judge is asked the blueprint and the interview as for any run, since both are whole at the hand over: those scores compare with a campaign played whole. It is asked the messages and following along of planning alone, and told that the run stops there; it keeps those scores under names of their own, `messages-in-planning` and `following-in-planning`, since they answer another question than those of a run judged to its end. The report says how many runs played planning alone, and compares with another campaign only what both measured.
 
 ## The commands
 
@@ -63,11 +63,12 @@ The gate builds the image, mounts this clone read-only at `/clone` and the campa
 
 ## Does it meet its goals
 
-Computed by a script, from the plan folder, the journal, the streams and the delivered code (`surface_evals/measure.py`). Per run:
+Computed by a script, from the plan folder, the journal, the streams, the delivered code and what the stand-in `gh` kept (`surface_evals/measure.py`). Per run:
 
 - **The loop converges, and hands back when it should**: the outcome, `conformant`, `handed-back`, or stuck in planning or in execution; the passes used; the reason of a hand back; and `approved_by_sentence`, which must stay false.
 - **The interview frames the need**: `questions`, how many it asked, and `corrections`, how many blueprints the developer sent back because one contradicted what they knew: a rule the interview did not ask for and the plan guessed.
 - **Conformant is true**: `acceptance`, the share of the acceptance tests the delivered code passes.
+- **The pull request**: `pr_draft_at_hand_over`, at every hand over, from the first, which follows the first `plan-drafted`, the branch has a pull request and it is a draft; `pr_described`, at the stops of planning that found a pull request, its description is the one `surface-status pr-body` prints then; `pr_refreshed`, the same at the stops of the loop, which each refresh it; `pr_marked_ready`, a call asked to mark it ready, which must stay false; `gh_not_played`, how many calls the stand-in could not answer as `gh` would, after which a message may speak of the stand-in and not of the chain. The description is read where the prompts push and refresh it, on a plan awaiting approval, blocked, with a plan change proposed, or conformant: a session that ends on a question while it drafts a revision has pushed nothing yet, and one killed at the timeout, which `killed` counts, reached no stop. A run kept before the stand-in has none of the five.
 - **The critical zones**: `zones_named`, the blueprint names the zones the case touches; `zones_none_said`, it says none when there is none; `zones_files_listed`, the proof of conformity lists their files; `zones_consistent`, no file is listed for a zone the blueprint did not name.
 - **The form of the blueprint**: `frame`, no numbered heading, the sections of its body against what the case expects, its diagrams against whether the feature has a shape, and `cut_kept` across the revisions of an amended case.
 - **The plan**: drafted by the built-in agent, on which model, and holding the minimum the chain reads.
@@ -107,4 +108,6 @@ A campaign judged before the judge counted holds no count. Compared with one tha
 
 `--keep` copies both files to `evals/reports/<date>-<version of the chain>/`, to commit: that is what the next campaign is compared with. The version is a hash of `agents/` and `skills/`. The streams and the projects stay in the campaign folder, out of git.
 
+The campaigns kept before the stand-in `gh`, `evals/reports/2026-10-03-*`, were played in an image with no `gh`: every hand over said so, and the judge scored it. Against one of them, a move of the scores of the messages may come from the stand-in, which the hash of the chain does not show, and not from the prompts.
+
 Read a run before blaming the chain: a model may take another valid path, and an acceptance test may hold a rule the brief states badly.