A stand-in gh in the test image: the pull request steps played, recorded and measured

[PierreMardon]

What changes

The image of the end to end tests and of the evaluations holds a stand-in for gh, first on its PATH. A session there plays the pull request steps of the chain, the stand-in records every call, and the harness asserts and measures what was asked. This is what #75 asks for the evaluations and #8 for the end to end tests: one stand-in serves both.

• The stand-in, tests/e2e/toy/gh_stand_in.py, standard library only, copied in the image as /opt/gh-stand-in/gh. It answers, as gh 2.78.0 does when its output is not a terminal: pr create, pr edit (--body, --body-file <file>, --body-file -), pr list --json, pr view, pr ready, repo view, auth status. A second opening for a branch is refused with the address of the first, an edit on a branch with no pull request fails with no pull requests found for branch "<branch>", an opening is refused while the remote does not hold the last commit of the branch. Every other call gets an error that says the stand-in does not play it, and is recorded with played false.
• The record: the pull requests and one line per call, with its arguments, its standard input, the description given, its exit code and what it printed, in the git directory of the project, .git/gh-stand-in/. A call that asks to mark a pull request ready is recorded as such (marks_ready).
• The prepared states of the toy with a drafted plan hold the draft pull request /surface-plan opens at the first plan-drafted, with the description of that moment, and no call recorded for it.
• The nominal scenario of the end to end tests holds that the stop refreshed the description to what surface-status pr-body prints, by an edit and no second opening, that the pull request is still a draft, and that no call marked it ready.
• The evaluations: each stop of a run keeps the pull request of the branch then, and five measures follow. pr_draft_at_hand_over: at every hand over, from the first, the branch has a pull request and it is a draft. pr_described: at the stops of planning, its description is the one surface-status pr-body prints then. pr_refreshed: the same at the stops of the loop. pr_marked_ready, which must stay false. gh_not_played: the calls the stand-in could not answer as gh would.

Why

No session of the tests ever opened a draft pull request, refreshed its description or read one. Every hand over said that gh was missing, at up to four stops of a run, and the judge counted it as noise: the scores of the messages held what the chain says badly, what it says because of the container, and what the container hides, all at once (#75). On the path most developers live, with gh, none of those sentences is written, and that path had never run (#8).

Decisions

• Where the record lives: the git directory of the project, and not a folder next to its bare remote. Every repository has one, with or without a remote, so a call refused for want of a remote is recorded too. It is one project's alone, where --jobs plays several at the same time in one container. And it is out of the work tree: nothing shows in git status, nothing can be committed, and a session that explores the code does not meet it.
• A prepared state holds an open draft, as a real /surface-plan would have left, and not none. With none, /surface-execute would find no pull request at any stop and take the path of a branch that has none, which is the only path the runs ever took and the one A hand back at conformity that fits a branch with no pull request #78 fixed. A state past the approval keeps the description of the draft, since the loop it prepares has not stopped yet.
• The pull request is read at each stop, by the harness, and not from timestamps afterwards: only then can the description be compared with what surface-status pr-body prints at that moment. The description counts at the stops that leave the plan to the developer, awaiting approval, blocked, with a plan change proposed or conformant, where the prompts push and refresh. A session that ends on a question while it drafts a revision has pushed nothing, and one killed at the timeout reached no stop.
• A run that stops at the hand over (corpus --stop-at-hand-over) keeps what planning gives: pr_draft_at_hand_over, pr_described, and gh_not_played, which says how far to trust the run as contaminated does. pr_refreshed and pr_marked_ready are among the measures of the execution, empty for such a run. Never marked ready is a promise to the end of the chain, and the hand back at conformity, where a chain would break it, is not reached: a false would claim what was not played, and would sit with the runs played whole. A pull request that planning marked ready still shows, since it is no draft at the hand over.
• What the stand-in does not know, it refuses and records, and never guesses. gh_not_played then says how often a run met its limits, so the list of calls can grow from what sessions really ask.

Files

• tests/e2e/toy/gh_stand_in.py (new), tests/e2e/Dockerfile, tests/e2e/toy/toy.py, tests/e2e/test_toy.py, tests/e2e/README.md
• evals/surface_evals/runner.py, measure.py, report.py, evals/README.md
• tests/gh/test_gh_stand_in.py (new): the stand-in run as a script in temporary repositories, in the gates, with no container and no session
• tests/evals/support.py, test_runner.py, test_measure.py, test_report.py
• ARCHITECTURE.md: the codemap, the tests, the evaluations

Nothing under skills/, agents/ or install.py changes: the stand-in belongs to the test image only. ADR 0025 holds as it is.

What the stand-in does not imitate

GitHub itself: no network, no CI, no check, no review, no comment, nothing merges or closes. The remote: it takes origin, wherever it points, for the GitHub repository. A terminal: it never prompts. Every other call and flag: gh api, gh pr merge, gh pr checks, gh pr status, --fill, --web, --repo, --template, and a --jq that is more than a path. The two refusals that are GitHub's own answers to an opening, a branch with nothing to merge and a head the remote does not hold, are worded as remembered and were not checked against GitHub. The whole list is in tests/e2e/README.md.

Not measured

No session was played with the stand-in: nothing here says yet that a session opens its draft through it, nor that the hand overs stop speaking of gh. What was run: every gate; the image built under another tag; the stand-in played by hand in that image on a toy project, with the two gh pr list calls of the branch step, an opening, the refresh of a stop on its standard input, and a gh pr ready; and the two end to end tests that start no session.

To validate, the opening first, on planning alone, then the refresh by the loop:

EVALS_OUT=.evals/stand-in scripts/gate.sh evals corpus --case 01-overdue-list --stop-at-hand-over
python3 evals/run.py --out .evals/stand-in report
scripts/gate.sh e2e -k nominal

The first must leave, in .evals/stand-in/runs/01-overdue-list/run-01/, a run.json whose stops on awaiting-approval hold "pull": {"draft": true, "described": true}, and a work/lending/.git/gh-stand-in/calls.jsonl with a pr create --draft that exits 0 and no pr ready. Its final messages must not speak of a missing gh. The report must show pr_draft_at_hand_over 1, pr_described 1 and gh_not_played 0, with pr_refreshed and pr_marked_ready empty. When gh_not_played is not 0, the lines of calls.jsonl with "played": false name the calls a session made and the stand-in does not answer yet. The end to end scenario must pass with its new assertions: the stop of the loop refreshed the description. A run played whole, the same corpus without --stop-at-hand-over, must then show pr_refreshed 1 and pr_marked_ready 0.

A campaign kept before this one was played with no gh: against it, a move of the scores of the messages may come from the stand-in, which the hash of the chain does not show.

Refs #75, #8