diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 7b8874d..4d9004c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -41,7 +41,7 @@ host CI --runs----> surface-status check --require conformant - `report.py` (list, `show`, `check`), `pr_body.py`, `settings.py`, and `cli.py` (arguments, exit codes, JSON). - `agents/surface-*.md`: the four roles. - `install.py`: install, update, `--check`, and the one-line form. -- `tests/`: one folder per part (`state`, `journal`, `cli`, `git`, `gates`, `install`, `prompts`, `evals`), `ci/`, `e2e/`, and golden files in `fixtures/`. +- `tests/`: one folder per part (`state`, `journal`, `cli`, `git`, `gates`, `install`, `prompts`, `evals`), `ci/`, `e2e/`, `gh/` for the stand-in `gh` of the end to end image, and golden files in `fixtures/`. - `evals/`: the evaluations of the chain on real sessions. `host/`, a synthetic project the chain is installed in; `cases/`, one need per shape, each with a brief, acceptance tests the agents never see and a reference implementation; `judge/`, the rubric, the method of the count and the spoiled documents that check the judge; `surface_evals/`, the harness; `reports/`, the campaigns kept; `run.py`, its command line. - `scripts/gate.sh`: every gate. `scripts/check_commit_msg.py`: the commit message hook. @@ -105,11 +105,11 @@ The loop stops at every command the developer's rules or mode do not allow, and ### Tests -Unit and hypothesis property tests cover each part of the script and the installer, with sockets blocked; the `ci` profile runs 500 examples. Golden journals replay to their expected states. Prompt tests hold the frontmatter, the contract the prompts carry (`tests/prompts/chain_contract.py`), host neutrality and privacy. The privacy check reads plan folders, where a path of the machine is a leak too, while the decision-link check skips them, since a plan names decisions still to come. `tests/ci/` proves the conformity check in a checkout shaped like a CI's, and the `conformity` job of `ci.yml` runs it again on a real runner. End to end tests run real sessions on a toy project, in a container, on demand only, since they are billed ([ADR 0025](docs/adr/0025-end-to-end-on-a-toy-project.md)). +Unit and hypothesis property tests cover each part of the script and the installer, with sockets blocked; the `ci` profile runs 500 examples. Golden journals replay to their expected states. Prompt tests hold the frontmatter, the contract the prompts carry (`tests/prompts/chain_contract.py`), host neutrality and privacy. The privacy check reads plan folders, where a path of the machine is a leak too, while the decision-link check skips them, since a plan names decisions still to come. `tests/ci/` proves the conformity check in a checkout shaped like a CI's, and the `conformity` job of `ci.yml` runs it again on a real runner. End to end tests run real sessions on a toy project, in a container, on demand only, since they are billed ([ADR 0025](docs/adr/0025-end-to-end-on-a-toy-project.md)). Their image reaches no GitHub, and a bare remote on disk gives the real `gh` nothing to open a pull request on: a stand-in, `tests/e2e/toy/gh_stand_in.py`, first on the `PATH` of the image, answers the pull request calls of the prompts and refuses the others, so that the steps a developer with `gh` lives are played there. It keeps the pull requests and a record of every call in the git directory of the project, which every repository has, which is one project's alone and which stays out of the work tree; the scenarios and the evaluations read them, and `tests/gh/` holds the stand-in without a session. It belongs to the test image only: nothing of it is installed in a host. ### Evaluations -The tests say that the chain works, not that it is worth using. The evaluations of `evals/` answer that after a change of the prompts, on real sessions and with no human in them, in the container of the end to end tests, on demand and never in CI ([ADR 0036](docs/adr/0036-evaluations-on-real-sessions.md)). A run plays the developer's side on a synthetic host: it types the need of a case, has a model answer each question from a written brief, since written answers cannot meet questions that change from run to run, then read the blueprint and send back what contradicts the brief, replies with a sentence that agrees, which must approve nothing, and launches `/surface-execute`. A campaign that measures a change of planning alone stops its runs after that sentence, at the hand over: what the execution gives is then left empty, never 0, and what the judge says of their stops is kept under another name, so that a run stopped on purpose is not read as one that failed to conform. A script measures what a script can, from the journal, the plan folder, the streams and the delivered code: the outcome, the passes, acceptance tests kept from the agents, the critical zones, the form of the blueprint; and, on the toy, whether a fault put there on purpose is found by the cross-check and classified by the review. A model judges the rest against a written rubric, the blueprint, the interview, the messages, each judgement with the passage it rests on, and is itself checked: a document spoiled in a known way must score below its original. Beside its scores, and in a call of its own that leaves them as they are asked, it counts what the developer would skip on the blueprint: the facts said more than once in prose, the details of implementation that are not theirs to decide, the share of the page, each fact and each detail with its passages, since a score from 1 to 5 gave the same value to pages that a reader who counts tells apart. The script makes the totals from those lists, and the count is checked like the scores: the padded blueprint must count more than the lean one. A measure is kept as its spread over the runs, and the report says what lies outside the spread of the campaign before: there is no threshold, since what to hold is decided once the measures have been observed. A ledger stops a campaign before a ceiling, in USD at list price or in points of the subscription's weekly gauge. +The tests say that the chain works, not that it is worth using. The evaluations of `evals/` answer that after a change of the prompts, on real sessions and with no human in them, in the container of the end to end tests, on demand and never in CI ([ADR 0036](docs/adr/0036-evaluations-on-real-sessions.md)). A run plays the developer's side on a synthetic host: it types the need of a case, has a model answer each question from a written brief, since written answers cannot meet questions that change from run to run, then read the blueprint and send back what contradicts the brief, replies with a sentence that agrees, which must approve nothing, and launches `/surface-execute`. A campaign that measures a change of planning alone stops its runs after that sentence, at the hand over: what the execution gives is then left empty, never 0, and what the judge says of their stops is kept under another name, so that a run stopped on purpose is not read as one that failed to conform. A script measures what a script can, from the journal, the plan folder, the streams, the delivered code and what the stand-in `gh` kept: the outcome, the passes, acceptance tests kept from the agents, the critical zones, the form of the blueprint, the pull request opened as a draft, its description current wherever the plan is left to the developer, and never marked ready; and, on the toy, whether a fault put there on purpose is found by the cross-check and classified by the review. A model judges the rest against a written rubric, the blueprint, the interview, the messages, each judgement with the passage it rests on, and is itself checked: a document spoiled in a known way must score below its original. Beside its scores, and in a call of its own that leaves them as they are asked, it counts what the developer would skip on the blueprint: the facts said more than once in prose, the details of implementation that are not theirs to decide, the share of the page, each fact and each detail with its passages, since a score from 1 to 5 gave the same value to pages that a reader who counts tells apart. The script makes the totals from those lists, and the count is checked like the scores: the padded blueprint must count more than the lean one. A measure is kept as its spread over the runs, and the report says what lies outside the spread of the campaign before: there is no threshold, since what to hold is decided once the measures have been observed. A ledger stops a campaign before a ceiling, in USD at list price or in points of the subscription's weekly gauge. ### Toolchain diff --git a/evals/README.md b/evals/README.md index 9a30404..97a5bb1 100644 --- a/evals/README.md +++ b/evals/README.md @@ -8,7 +8,7 @@ Every command but `report` is billed and takes minutes to hours: run them on dem ## What you need -What the end to end tests need (`tests/e2e/README.md`): Docker, and a token in your environment, `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY`. The sessions bypass permissions, so they run in the same container, and refuse to start anywhere else. +What the end to end tests need (`tests/e2e/README.md`): Docker, and a token in your environment, `CLAUDE_CODE_OAUTH_TOKEN` or `ANTHROPIC_API_KEY`. The sessions bypass permissions, so they run in the same container, and refuse to start anywhere else. That container reaches no GitHub: a stand-in for `gh` plays the pull request steps of the chain and records every call, so that these steps are run and measured, and no message has a missing `gh` to tell. What the stand-in answers and what it does not imitate is in `tests/e2e/README.md`. ## The corpus @@ -37,11 +37,11 @@ The sessions run in the container, where this clone is mounted at `/clone`: a se ## A run -A run plays the developer's side of the README: the need, an answer to each question, then a reading of the blueprint handed over, whose corrections go back as amendments, twice at most. It gives the amendment of the case when it has one, replies with a sentence that agrees, "Looks fine to me, go ahead.", which must approve nothing, then launches `/surface-execute`, which approves. It stops when the plan is conformant or the loop hands back, runs the acceptance tests, and keeps everything under `runs//run-NN/`: the project, the stream of every session, each blueprint handed over, and `run.json`, what was said at each stop and the state of the plan then. +A run plays the developer's side of the README: the need, an answer to each question, then a reading of the blueprint handed over, whose corrections go back as amendments, twice at most. It gives the amendment of the case when it has one, replies with a sentence that agrees, "Looks fine to me, go ahead.", which must approve nothing, then launches `/surface-execute`, which approves. It stops when the plan is conformant or the loop hands back, runs the acceptance tests, and keeps everything under `runs//run-NN/`: the project, the stream of every session, each blueprint handed over, and `run.json`, what was said at each stop, the state of the plan then, and the pull request of the branch then, as the stand-in `gh` keeps it: whether it is a draft, and whether its description is the one `surface-status pr-body` prints at that moment. The calls themselves stay in the project, in `.git/gh-stand-in/calls.jsonl`. `corpus --stop-at-hand-over` stops each run at the hand over, for a campaign that measures a change of planning alone, the interview or the blueprint, without paying for the execution. The run plays planning as above, the sentence that agrees included, since it is one cheap turn and `approved_by_sentence` is a measure of planning. It then launches nothing and runs no acceptance test. Its outcome is `stopped-at-hand-over`, which no run played whole has, and its `run.json` says `stop_at_hand_over`. -Nothing of the execution is read from such a run, whatever its outcome. Every measure the execution gives or adds to is empty for it, not 0 or false, so that it is never read as a run that failed to conform, nor counted with the runs played whole: `conformant`, `acceptance`, `gate`, the approvals, the reviews and their findings, the fixes, the failed gate runs, the sessions killed or relaunched, the files of the critical zones, and `usd`. `planning_usd`, what planning cost, is the one cost both kinds of runs have. The judge is asked the blueprint and the interview as for any run, since both are whole at the hand over: those scores compare with a campaign played whole. It is asked the messages and following along of planning alone, and told that the run stops there; it keeps those scores under names of their own, `messages-in-planning` and `following-in-planning`, since they answer another question than those of a run judged to its end. The report says how many runs played planning alone, and compares with another campaign only what both measured. +Nothing of the execution is read from such a run, whatever its outcome. Every measure the execution gives or adds to is empty for it, not 0 or false, so that it is never read as a run that failed to conform, nor counted with the runs played whole: `conformant`, `acceptance`, `gate`, the approvals, the reviews and their findings, the fixes, the failed gate runs, the sessions killed or relaunched, the files of the critical zones, `pr_refreshed`, `pr_marked_ready`, and `usd`. Of the pull request, such a run keeps what planning gives: the draft at the hand over, its description there, and the calls the stand-in did not play, which say how far to trust the run as `contaminated` does. Never marked ready is a promise to the end of the chain, and the hand back at conformity, where a chain would break it, is not reached: a false would claim what was not played. A pull request that planning marked ready still shows, since it is no draft at the hand over. `planning_usd`, what planning cost, is the one cost both kinds of runs have. The judge is asked the blueprint and the interview as for any run, since both are whole at the hand over: those scores compare with a campaign played whole. It is asked the messages and following along of planning alone, and told that the run stops there; it keeps those scores under names of their own, `messages-in-planning` and `following-in-planning`, since they answer another question than those of a run judged to its end. The report says how many runs played planning alone, and compares with another campaign only what both measured. ## The commands @@ -63,11 +63,12 @@ The gate builds the image, mounts this clone read-only at `/clone` and the campa ## Does it meet its goals -Computed by a script, from the plan folder, the journal, the streams and the delivered code (`surface_evals/measure.py`). Per run: +Computed by a script, from the plan folder, the journal, the streams, the delivered code and what the stand-in `gh` kept (`surface_evals/measure.py`). Per run: - **The loop converges, and hands back when it should**: the outcome, `conformant`, `handed-back`, or stuck in planning or in execution; the passes used; the reason of a hand back; and `approved_by_sentence`, which must stay false. - **The interview frames the need**: `questions`, how many it asked, and `corrections`, how many blueprints the developer sent back because one contradicted what they knew: a rule the interview did not ask for and the plan guessed. - **Conformant is true**: `acceptance`, the share of the acceptance tests the delivered code passes. +- **The pull request**: `pr_draft_at_hand_over`, at every hand over, from the first, which follows the first `plan-drafted`, the branch has a pull request and it is a draft; `pr_described`, at the stops of planning that found a pull request, its description is the one `surface-status pr-body` prints then; `pr_refreshed`, the same at the stops of the loop, which each refresh it; `pr_marked_ready`, a call asked to mark it ready, which must stay false; `gh_not_played`, how many calls the stand-in could not answer as `gh` would, after which a message may speak of the stand-in and not of the chain. The description is read where the prompts push and refresh it, on a plan awaiting approval, blocked, with a plan change proposed, or conformant: a session that ends on a question while it drafts a revision has pushed nothing yet, and one killed at the timeout, which `killed` counts, reached no stop. A run kept before the stand-in has none of the five. - **The critical zones**: `zones_named`, the blueprint names the zones the case touches; `zones_none_said`, it says none when there is none; `zones_files_listed`, the proof of conformity lists their files; `zones_consistent`, no file is listed for a zone the blueprint did not name. - **The form of the blueprint**: `frame`, no numbered heading, the sections of its body against what the case expects, its diagrams against whether the feature has a shape, and `cut_kept` across the revisions of an amended case. - **The plan**: drafted by the built-in agent, on which model, and holding the minimum the chain reads. @@ -107,4 +108,6 @@ A campaign judged before the judge counted holds no count. Compared with one tha `--keep` copies both files to `evals/reports/-/`, to commit: that is what the next campaign is compared with. The version is a hash of `agents/` and `skills/`. The streams and the projects stay in the campaign folder, out of git. +The campaigns kept before the stand-in `gh`, `evals/reports/2026-10-03-*`, were played in an image with no `gh`: every hand over said so, and the judge scored it. Against one of them, a move of the scores of the messages may come from the stand-in, which the hash of the chain does not show, and not from the prompts. + Read a run before blaming the chain: a model may take another valid path, and an acceptance test may hold a rule the brief states badly. diff --git a/evals/surface_evals/measure.py b/evals/surface_evals/measure.py index 2a9726c..4a74b9e 100644 --- a/evals/surface_evals/measure.py +++ b/evals/surface_evals/measure.py @@ -1,7 +1,8 @@ """Does the chain meet its goals: measured from what a run left, by a script alone. -Every measure is read from files: the journal, the plan folder, the streams of the sessions and -the delivered code. A measure that does not apply to a run is None, and stays out of the means: +Every measure is read from files: the journal, the plan folder, the streams of the sessions, the +delivered code, and what the stand-in `gh` of the container kept of the pull request. A measure +that does not apply to a run is None, and stays out of the means: the proof of conformity of a plan that never got there, the cut kept by a revision that was never asked for, and everything the execution gives when the run stopped at the hand over. """ @@ -11,9 +12,11 @@ from pathlib import Path from typing import cast +import gh_stand_in + from surface_evals import blueprint as form from surface_evals.corpus import Case, Diagram -from surface_evals.runner import GATE, RECORD, stops_at_hand_over +from surface_evals.runner import AWAITING, GATE, HANDED_BACK, OVER, RECORD, stops_at_hand_over from surface_evals.sessions import SessionLog, named_in_tool_calls, read_log from surface_status.plan_folder import PlanFolderError, parse_critical_files, parse_gates from surface_status.plan_folder import parse_slice_markers as parse_slices @@ -26,6 +29,9 @@ PLAN_AGENT = "Plan" # The stops that launch the loop: what their sessions cost is the cost of the execution. LAUNCHES = frozenset({"approval", "relaunch"}) +# The states a stop of the chain leaves the plan to the developer in: it pushed, and so +# refreshed the description of the pull request. +DEVELOPERS_TURN = frozenset({AWAITING}) | HANDED_BACK | OVER # What the execution gives, or adds to. A run that stops at the hand over holds none of it, # whatever its outcome, and each is None for it: a 0 or a False would read as a run that failed # to conform, and would join the spread of the runs played whole. @@ -45,6 +51,12 @@ "critical_files", "zones_files_listed", "zones_consistent", + # The description at the stops of the loop, which such a run has none of. + "pr_refreshed", + # Never marked ready is a promise to the end of the chain, and the hand back at conformity, + # where a chain would break it, is never reached: a False would claim what was not played. + # What planning alone says of it is in `pr_draft_at_hand_over`. + "pr_marked_ready", ) Measures = dict[str, float | bool | str | list[str] | None] @@ -97,6 +109,51 @@ def _loop(journal: list[dict[str, object]], record: dict[str, object]) -> Measur } +def _pull(stop: dict[str, object]) -> dict[str, bool] | None: + return cast("dict[str, bool] | None", stop["pull"]) + + +def _described(stops: Sequence[dict[str, object]]) -> bool | None: + """Say whether each of these stops that found a pull request left its description current.""" + held = [pull for stop in stops if (pull := _pull(stop)) is not None] + return None if not held else all(pull["described"] for pull in held) + + +def _pull_request(project: Path, record: dict[str, object]) -> Measures: + """Whether the pull request steps were played, from what the stand-in `gh` kept and recorded. + + Planning opens a draft at the first `plan-drafted`, with the description the state script + prints, each stop of the loop refreshes that description, and nothing ever marks the pull + request ready. The description is read where the prompts push and refresh it: at a stop that + leaves the plan to the developer. A session that ends on a question while it drafts a + revision has pushed nothing yet, and one killed at the timeout, which `killed` counts, + reached no stop of the chain. A run played before the stops kept the pull request, in an + image with no `gh`, has none of these measures. + """ + kept = [stop for stop in cast("list[dict[str, object]]", record["stops"]) if "pull" in stop] + if not kept: + return {} + stops = [stop for stop in kept if stop["ended"] and stop["state"] in DEVELOPERS_TURN] + planning = [stop for stop in stops if stop["kind"] not in LAUNCHES] + # The hand overs: where planning leaves a revision to the developer's approval. + handed = [_pull(stop) for stop in planning if stop["state"] == AWAITING] + calls = gh_stand_in.calls(project) + return { + # Opened at the first `plan-drafted`, and a draft still at every hand over after it. + "pr_draft_at_hand_over": ( + None if not handed else all(pull is not None and pull["draft"] for pull in handed) + ), + # What planning gives: the description the state script prints, at its own stops. + "pr_described": _described(planning), + # What the execution gives: that description again, at the stops of the loop. + "pr_refreshed": _described([stop for stop in stops if stop["kind"] in LAUNCHES]), + "pr_marked_ready": any(call.marks_ready for call in calls), + # The calls the stand-in could not answer as `gh` would: what follows them is its doing. + # Like `contaminated`, it says how far to trust the run, whatever part of it was played. + "gh_not_played": sum(1 for call in calls if not call.played), + } + + def _plan(folder: Path, logs: Sequence[SessionLog]) -> Measures: """Whether the plan was drafted by the built-in agent, and holds what the chain reads.""" drafts = [launch for log in logs for launch in log.launches if launch.agent == PLAN_AGENT] @@ -213,6 +270,7 @@ def measure(root: Path, case: Case) -> Measures: "usd": round(cost_of(logs), 4), } measures.update(_loop(journal, record)) + measures.update(_pull_request(project, record)) measures.update(_plan(folder, logs)) measures.update(_zones(case, kept[-1] if kept else None, folder, conformant=conformant)) measures.update(_form(case, kept)) diff --git a/evals/surface_evals/report.py b/evals/surface_evals/report.py index 0d681e8..9718582 100644 --- a/evals/surface_evals/report.py +++ b/evals/surface_evals/report.py @@ -48,6 +48,11 @@ "relaunches": -1, "planning_passes": -1, "gate_runs_failed": -1, + "pr_draft_at_hand_over": 1, + "pr_described": 1, + "pr_refreshed": 1, + "pr_marked_ready": -1, + "gh_not_played": -1, "plan_minimum": 1, "zones_named": 1, "zones_none_said": 1, @@ -75,6 +80,11 @@ "planning_passes", "reviews", "fixes", + "pr_draft_at_hand_over", + "pr_described", + "pr_refreshed", + "pr_marked_ready", + "gh_not_played", "zones_named", "zones_none_said", "zones_files_listed", diff --git a/evals/surface_evals/runner.py b/evals/surface_evals/runner.py index ac67bd6..ab3df5c 100644 --- a/evals/surface_evals/runner.py +++ b/evals/surface_evals/runner.py @@ -24,6 +24,7 @@ from pathlib import Path from typing import cast +import gh_stand_in import toy from surface_evals import CLONE @@ -57,6 +58,7 @@ class Stop: state: str | None # the state of the plan once the session ended final: str # the final message of the session ended: bool # False when the session was killed at the timeout + pull: dict[str, bool] | None # the pull request of the branch then, see `pull_of` def build_project(dest: Path) -> Path: @@ -155,6 +157,23 @@ def stops_at_hand_over(root: Path) -> bool: return record.get("stop_at_hand_over") is True +def pull_of(project: Path) -> dict[str, bool] | None: + """Tell the pull request of the branch of a project, as the stand-in `gh` keeps it. + + Whether it is still a draft, and whether its description is what the state script prints now, + which is what a stop of the chain leaves. None when the branch has no open pull request, or + when the project can no longer tell: a stop is kept whatever a session left behind it. + """ + try: + branch = toy.git(project, "branch", "--show-current").strip() + pull = gh_stand_in.pull_of(project, branch) + except (subprocess.CalledProcessError, OSError, ValueError, TypeError, KeyError): + return None + if pull is None: + return None + return {"draft": pull.draft, "described": pull.body.strip() == pr_body(project).strip()} + + @dataclass(slots=True) class Run: case: Case @@ -188,6 +207,7 @@ def _say(self, kind: str, said: str, resume: str | None) -> SessionLog: state=state_of(self.project), final=session.final, ended=session.ended, + pull=pull_of(self.project), ) ) self.save() diff --git a/tests/e2e/Dockerfile b/tests/e2e/Dockerfile index c5554ca..66f1448 100644 --- a/tests/e2e/Dockerfile +++ b/tests/e2e/Dockerfile @@ -23,8 +23,16 @@ RUN python3 -m venv /opt/e2e \ RUN userdel --remove node \ && useradd --create-home --uid "$UID" tester -# SURFACE_E2E_CONTAINER lets toy.py start a session; the clone is mounted read-only at /clone. -ENV PATH=/opt/e2e/bin:$PATH \ +# No GitHub is reached from here, and a project pushes to a bare repository on disk: a stand-in +# for `gh` plays the pull request steps of the chain and records every call in the git directory +# of the project (tests/e2e/README.md). It is copied, and not run from the clone, so that a +# session never meets a path of the clone it must not read. +COPY toy/gh_stand_in.py /opt/gh-stand-in/gh +RUN chmod 0755 /opt/gh-stand-in/gh + +# The stand-in comes first on the PATH. SURFACE_E2E_CONTAINER lets toy.py start a session; the +# clone is mounted read-only at /clone. +ENV PATH=/opt/gh-stand-in:/opt/e2e/bin:$PATH \ PYTHONDONTWRITEBYTECODE=1 \ SURFACE_E2E_CONTAINER=1 USER tester diff --git a/tests/e2e/README.md b/tests/e2e/README.md index a8d110b..03b303e 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -6,7 +6,7 @@ Every scenario is billed and takes minutes: run them on demand, before a release ## What you need -- Docker. The sessions bypass permissions, which is not safe on your machine: a command could reach outside the toy folder. They run in the container of `tests/e2e/Dockerfile`, which holds Python 3.11, git, the `claude` CLI and pytest, under a user that is not root. `toy.py` refuses to start a session anywhere else. +- Docker. The sessions bypass permissions, which is not safe on your machine: a command could reach outside the toy folder. They run in the container of `tests/e2e/Dockerfile`, which holds Python 3.11, git, the `claude` CLI, pytest and a stand-in for `gh`, under a user that is not root. `toy.py` refuses to start a session anywhere else. - A token in your environment: `CLAUDE_CODE_OAUTH_TOKEN`, printed by `claude setup-token`, or `ANTHROPIC_API_KEY`. The container cannot use the login of your `claude` on the host, which macOS keeps in its keychain. The token is passed to `docker run` from your environment and never written in the image. Sessions use its account. ## The toy project @@ -16,7 +16,7 @@ Every scenario is billed and takes minutes: run them on demand, before a release | State | What the project holds | |---|---| | `specs` | the main branch, the chain installed, no plan | -| `awaiting-approval` | the branch `feat/csv-export`, a plan drafted at revision 1 (two slices), pushed | +| `awaiting-approval` | the branch `feat/csv-export`, a plan drafted at revision 1 (two slices), pushed, and its draft pull request, open in the stand-in `gh` | | `blueprint-modified` | revision 1 approved, slice 1 done, then a commit of the developer that edits `blueprint.md` | | `developer-break` | both slices done, then a commit of the developer that adds an `isbn` column the blueprint leaves out | | `ceiling` | both slices done with a defect the tests do not see (lines sorted by title only), a first review that found it, then a fix that missed it, `max_autonomous_passes: 1` | @@ -24,6 +24,36 @@ Every scenario is billed and takes minutes: run them on demand, before a release `toy/toy.py run [--resume ]` runs one headless session in the project until it ends, killed after 30 minutes, keeps its stream of JSON events in ``, then prints its exit code, its session id, its cost and its final message. The session leaves out your user settings and MCP servers, and bypasses permissions (`--permission-mode bypassPermissions`): nobody is there to answer a prompt, and what an agent runs to explore the code is left to the permission mode, not listed by the chain. `run` works in the container only; `build` costs nothing and works anywhere. +## The stand-in `gh` + +The container reaches no GitHub, and a project pushes to a bare repository on disk, on which the real `gh` opens no pull request. So the image holds a stand-in, `toy/gh_stand_in.py`, copied to `/opt/gh-stand-in/gh`, first on the `PATH`: a session plays the pull request steps of the chain, and the harness reads what was asked. The evaluations run in the same image and measure the same record (`evals/README.md`). + +It answers the calls the prompts make, as `gh` 2.78.0 does when its output is not a terminal, on the repository of the current directory: + +| Call | What the stand-in does | +|---|---| +| `gh pr create --title --body <text>`, or `--body-file <file>`, or `--body-file -`, with `--draft`, `--base`, `--head` | opens a pull request for the current branch and prints its address. Refused without a title and a description, when the remote does not hold the last commit of the branch, when the branch has nothing to merge, and when a pull request is already open for it into the same base | +| `gh pr edit`, with a number, an address or a branch, and `--title`, `--body`, `--body-file`, `--base` | replaces the title or the description and prints the address. Fails with `no pull requests found for branch "<branch>"` when the branch has none | +| `gh pr list`, with `--state`, `--author`, `--head`, `--base`, `--draft`, `--limit`, `--json`, `--jq` | lists the pull requests, the last opened first, in the fields asked | +| `gh pr view`, with a number, an address or a branch, and `--json`, `--jq` | shows one | +| `gh pr ready`, with `--undo` | marks one ready for review, or back to a draft | +| `gh repo view`, `gh auth status`, `gh --version`, `--help` | the repository `stand-in/<folder of the project>`, the account `developer`, and the calls it plays | + +What it keeps of a project lives in its git directory, `.git/gh-stand-in/`: `pulls.json`, the pull requests, and `calls.jsonl`, one line per call with its arguments, its standard input when the call reads it, the description it gave, its exit code and what it printed. The git directory, and not a folder next to the bare remote, for three reasons. Every repository has one, with or without a remote, so a call refused for want of a remote is recorded too. It belongs to one project, where several run at the same time in one container. And it is out of the work tree: nothing shows in `git status`, nothing can be committed, and a session that explores the code does not meet it. A `gh pr ready` that is not an `--undo` has `marks_ready` true in its line, whatever became of it: the chain must never make one. A call reads its standard input before it holds that folder, so one `gh` may feed another. `gh_stand_in.calls(project)` and `gh_stand_in.pulls(project)` read both. + +A prepared state with a drafted plan holds its draft pull request, as `/surface-plan` leaves it at the first `plan-drafted`: `toy.py` opens it in the stand-in, with the description `surface-status pr-body` prints then, and no call is recorded for it. Without it, `/surface-execute` would find no pull request at any stop and take the path of a branch that has none, the only one the runs took before the stand-in. A state past the approval keeps that first description, since the loop it prepares has not stopped yet, and a stop is what refreshes it. + +What the stand-in does not imitate: + +- GitHub. There is no network, no account, no CI, no check, no review, no comment and no label, and nothing merges or closes a pull request. The address it prints, `https://github.com/stand-in/<project>/pull/<number>`, leads nowhere. +- The remote. It takes `origin`, wherever it points, for the GitHub repository, and reads the remote-tracking branches of the project, never the remote itself. +- Every other call: `gh api`, `gh pr merge`, `gh pr checks`, `gh issue`, `gh run`, and every flag the table does not name, `--fill`, `--web`, `--repo`, `--template` and `--search` among them. Each gets an error that says the stand-in does not play it, exit code 1, and a line of the record with `played` false, unless it holds `--help`, which prints the calls the stand-in plays whatever the call. `gh pr view` shows no `additions` and no `deletions`, since the stand-in reads no diff. `--jq` is played for paths only, such as `.url` or `.[].headRefName`. `--json` knows the fields the stand-in holds: `number`, `title`, `body`, `state`, `isDraft`, `url`, `headRefName`, `baseRefName`, `author`, `createdAt`, `updatedAt`, and names another as `gh` names an unknown one. +- A terminal. It never prompts, and a usage error is one line, without the usage `gh` prints after it. +- Every wording. The messages of the refusals above are those of `gh`, read in its source or seen on a real repository. The two that are GitHub's own answers to an opening, a branch with nothing to merge and a head the remote does not hold, are written as remembered and were not checked against GitHub. +- Outside a repository, or in one whose git directory cannot be written, it answers `--version` and `--help` only, and records nothing. + +Its unit tests, `tests/gh/`, run it as a script in temporary repositories: they are part of the gates, and need neither the container nor a session. + ## The automated scenarios ```sh @@ -37,7 +67,7 @@ The gate builds the image, mounts this clone read-only at `/clone` and the outpu | Test | Start | Session | What must hold | |---|---|---|---| -| `test_the_nominal_path_reaches_conformant` | `awaiting-approval` | `/surface-execute` | `conformant`, one approval, each slice done once, `conformity.md`, and `shelf/export.py` among the files of the critical zone the description lists | +| `test_the_nominal_path_reaches_conformant` | `awaiting-approval` | `/surface-execute` | `conformant`, one approval, each slice done once, `conformity.md`, and `shelf/export.py` among the files of the critical zone the description lists. The pull request: its description is the one `surface-status pr-body` prints, set by an edit and no second opening, it is still a draft, and no call marked it ready | | `test_a_session_killed_in_a_slice_resumes_it` | `awaiting-approval` | `/surface-execute`, killed once an executor has written code, then relaunched | the journal of the killed session kept as is, one approval, each slice done once, `conformant` | | `test_a_modified_blueprint_stops_the_loop` | `blueprint-modified` | `/surface-execute` | nothing recorded, still `executing`, the final message names the blueprint | | `test_the_ceiling_hands_back_to_the_developer` | `ceiling` | `/surface-execute` | `blocked` after a second review with findings, and no second fix | @@ -75,7 +105,7 @@ python3 $toy run $p $work/plan-1.jsonl "/surface-plan Export the shelf as CSV fo The toy shows no branch practice, no pull request and no merge commit, so the session first asks how to name the branch, recommending `feature/<slug>`, and stops. Answer it with `--resume`: it creates the branch, explores and writes `exploration.md`, then opens `interview.md` with the branch question and its answer under "Git", asks its first interview question and stops. Answer it with `--resume` too. Kill the session that takes the answer as soon as the answer is in `interview.md`. Then relaunch without `--resume`, with `/surface-plan` alone: - it must not explore again, nor ask again a question that has an answer: it asks the next open one; -- answered to the end, it has the plan drafted by the `Plan` agent and writes it as returned, with the three headings the chain reads and the gate of the toy in its `gates` block, has the blueprint drawn and cross-checked, records `plan-drafted`, commits and pushes to the bare remote. `gh` cannot open a pull request on a local remote: the session says so and gives the description. +- answered to the end, it has the plan drafted by the `Plan` agent and writes it as returned, with the three headings the chain reads and the gate of the toy in its `gates` block, has the blueprint drawn and cross-checked, records `plan-drafted`, commits and pushes to the bare remote, then opens a draft pull request in the stand-in `gh`, with the description `surface-status pr-body` prints: from the project, `gh pr view` shows it, and `.git/gh-stand-in/calls.jsonl` holds the calls the sessions made, with no `gh pr ready` among them. The hand over must not say that `gh` is missing. ### An amendment, killed right after it was recorded diff --git a/tests/e2e/test_toy.py b/tests/e2e/test_toy.py index e966ac4..8f7beea 100644 --- a/tests/e2e/test_toy.py +++ b/tests/e2e/test_toy.py @@ -12,6 +12,7 @@ from pathlib import Path from typing import Any +import gh_stand_in import pytest import toy @@ -87,6 +88,16 @@ def test_the_nominal_path_reaches_conformant(tmp_path: Path) -> None: # The toy declares the CSV export a critical zone: its code is listed for the developer. assert "shelf/export.py" in toy.critical_files(project) assert not dirty(project, "shelf", "tests", "docs") + # The draft /surface-plan would have opened, kept by the stand-in `gh`: the stop refreshed + # its description, by an edit and no second opening, and nothing marked it ready. + pull = gh_stand_in.pull_of(project, toy.BRANCH) + assert pull is not None + assert pull.body.strip() == toy.pr_body(project).strip() + assert pull.draft + calls = gh_stand_in.calls(project) + assert {call.command for call in calls if call.pull is not None} == {"pr edit"} + assert not [call.argv for call in calls if call.command == "pr create"] + assert not [call.argv for call in calls if call.marks_ready] def test_a_session_killed_in_a_slice_resumes_it(tmp_path: Path) -> None: diff --git a/tests/e2e/toy/gh_stand_in.py b/tests/e2e/toy/gh_stand_in.py new file mode 100755 index 0000000..2e10380 --- /dev/null +++ b/tests/e2e/toy/gh_stand_in.py @@ -0,0 +1,840 @@ +#!/usr/bin/env python3 +"""A stand-in for `gh`, the GitHub CLI, where the sessions of the tests have no GitHub to reach. + +The toy project and the host of the evaluations push to a bare repository on disk, on which the +real `gh` opens no pull request. The image of `tests/e2e/Dockerfile` puts this script first on the +`PATH` under the name `gh`, so a session plays the pull request steps of the chain. It answers the +calls the prompts make as `gh` 2.78.0 does without a terminal, on the repository of the current +directory, keeps the pull requests it is asked for, and records every call: + + gh pr create --title <title> (--body <text> | --body-file <file or ->) [--draft] + gh pr edit [<pull request>] [--title <title>] [--body <text> | --body-file <file or ->] + gh pr list [--state <state>] [--author <login>] [--limit <n>] [--json <fields>] + gh pr view [<pull request>] [--json <fields>] + gh pr ready [<pull request>] [--undo] + gh repo view [--json <fields>] + gh auth status + +Anything else gets an error that says so, and is recorded as not played. No call reaches a +network. `tests/e2e/README.md` says what the stand-in does not imitate. + +What it keeps of a repository lives in its git directory, under `gh-stand-in/`: `pulls.json`, the +pull requests, and `calls.jsonl`, one line per call. The git directory is there for every +repository, with or without a remote, so every call has a place to be recorded. It belongs to one +repository, where several projects run at the same time in one container. And it is out of the +work tree, where the record would show in `git status`, could be committed, and would be met by a +session that explores the code. + +The harness reads them with `pulls`, `pull_of` and `calls`. `open_pull` gives a prepared state the +pull request a session would have left, with no call recorded. +""" + +import fcntl +import io +import json +import re +import subprocess +import sys +from collections.abc import Callable, Generator, Mapping, Sequence +from contextlib import contextmanager +from dataclasses import asdict, dataclass, field, replace +from datetime import UTC, datetime +from pathlib import Path +from typing import Any, NoReturn, cast + +IMITATES = "2.78.0" # the version of `gh` whose answers were read +FOLDER = "gh-stand-in" # in the git directory of the repository +PULLS = "pulls.json" +RECORD = "calls.jsonl" +LOCK = "lock" # held by one call at a time +OWNER = "stand-in" # owns every repository on the GitHub of the stand-in +LOGIN = "developer" # opens every pull request there +AUTHOR: Mapping[str, object] = {"id": OWNER, "is_bot": False, "login": LOGIN, "name": "Developer"} +OPEN = "OPEN" +STATES = ("open", "closed", "merged", "all") +TIMESTAMP = "%Y-%m-%dT%H:%M:%SZ" # the form GitHub gives, and the journal of a plan +PULL_FIELDS = ( + "author", + "baseRefName", + "body", + "createdAt", + "headRefName", + "isDraft", + "number", + "state", + "title", + "updatedAt", + "url", +) +REPOSITORY_FIELDS = ("defaultBranchRef", "description", "name", "nameWithOwner", "owner", "url") + + +@dataclass(frozen=True, slots=True) +class Pull: + """A pull request, as the stand-in keeps it.""" + + number: int + head: str + base: str + title: str + body: str + draft: bool + state: str # OPEN, CLOSED or MERGED: no call closes or merges one + created: str + updated: str + + +@dataclass(frozen=True, slots=True) +class Call: + """One call of `gh`, as the record keeps it.""" + + at: str + argv: tuple[str, ...] + stdin: str | None # what the call read on its standard input, None when it read nothing + branch: str | None # the branch checked out then + command: str | None # `pr edit`, None when the stand-in does not know the command + played: bool # False when the stand-in could not answer as `gh` would + marks_ready: bool # the call asks to mark a pull request ready for review + pull: int | None # the pull request it opened or changed + body: str | None # the description it gave, to a pull request it reached or not + exit_code: int + stdout: str + stderr: str + + +class RefusedError(Exception): + """A call `gh` refuses too, with the message it prints.""" + + +class NotPlayedError(Exception): + """A call the real `gh` may answer, and the stand-in does not.""" + + +def _refuse(message: str) -> NoReturn: + raise RefusedError(message) + + +def _not_played(what: str) -> NoReturn: + message = ( + f"gh stand-in: {what} is not played here. This stand-in for the GitHub CLI answers:" + f" {', '.join(_FLAGS)}. `gh --help` lists their flags." + ) + raise NotPlayedError(message) + + +# What is kept of a repository, and how the harness reads it. + + +def store(repo: Path) -> Path: + """Name the folder the stand-in keeps of a repository whose git directory is its `.git`.""" + return repo / ".git" / FOLDER + + +def pulls(repo: Path) -> list[Pull]: + """Read the pull requests of a repository, in the order of their numbers.""" + return _read_pulls(store(repo)) + + +def pull_of(repo: Path, branch: str) -> Pull | None: + """Give the open pull request of a head branch, None when it has none.""" + return next((pull for pull in pulls(repo) if pull.state == OPEN and pull.head == branch), None) + + +def calls(repo: Path) -> list[Call]: + """Read the record of a repository: every call made in it, in order.""" + record = store(repo) / RECORD + if not record.is_file(): + return [] + found: list[Call] = [] + for line in record.read_text(encoding="utf-8").splitlines(): + try: + held = cast("dict[str, Any]", json.loads(line)) + del held["v"] + held["argv"] = tuple(held["argv"]) + found.append(Call(**held)) + except (ValueError, TypeError, KeyError): + continue # a line cut by the kill of a session + return found + + +def open_pull( # noqa: PLR0913 (what a pull request is made of, each by its name) + repo: Path, *, head: str, base: str, title: str, body: str, draft: bool +) -> Pull: + """Open a pull request with no call recorded, for a state a harness prepares.""" + kept = store(repo) + with _locked(kept): + held = _read_pulls(kept) + opened = _new(held, head=head, base=base, title=title, body=body, draft=draft) + _write_pulls(kept, [*held, opened]) + return opened + + +def _now() -> str: + return datetime.now(UTC).strftime(TIMESTAMP) + + +def _new( # noqa: PLR0913 (what a pull request is made of, each by its name) + held: Sequence[Pull], *, head: str, base: str, title: str, body: str, draft: bool +) -> Pull: + number = max((pull.number for pull in held), default=0) + 1 + now = _now() + return Pull(number, head, base, title, body, draft, OPEN, created=now, updated=now) + + +def _read_pulls(kept: Path) -> list[Pull]: + path = kept / PULLS + if not path.is_file(): + return [] + held = cast("dict[str, list[dict[str, Any]]]", json.loads(path.read_text(encoding="utf-8"))) + return sorted((Pull(**pull) for pull in held["pulls"]), key=lambda pull: pull.number) + + +def _write_pulls(kept: Path, held: Sequence[Pull]) -> None: + """Replace the file in one move, so that a reader never meets half of it.""" + text = json.dumps({"v": 1, "pulls": [asdict(pull) for pull in held]}, indent=2) + "\n" + scratch = kept / f"{PULLS}.new" + scratch.write_text(text, encoding="utf-8") + scratch.replace(kept / PULLS) + + +def _append(kept: Path, call: Call) -> None: + """Add a call to the record, on a line of its own even after one a kill cut short.""" + record = kept / RECORD + written = record.read_bytes() if record.is_file() else b"" + line = json.dumps({"v": 1, **asdict(call)}, ensure_ascii=False) + with record.open("a", encoding="utf-8") as out: + out.write(f"{line}\n" if not written or written.endswith(b"\n") else f"\n{line}\n") + + +@contextmanager +def _locked(kept: Path) -> Generator[None]: + """Hold the folder for one call: a session may run two `gh` at the same time.""" + kept.mkdir(parents=True, exist_ok=True) + with (kept / LOCK).open("w", encoding="utf-8") as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + yield + + +# One call: the repository it is made in, and what it leaves. + + +def _git(*args: str) -> str | None: + """Ask git in the current directory, None when it refuses.""" + done = subprocess.run( + ["git", *args], # noqa: S607 (git from the PATH, as `gh` runs it) + capture_output=True, + text=True, + check=False, + ) + return done.stdout.strip() if done.returncode == 0 else None + + +def _has(ref: str) -> bool: + return _git("rev-parse", "--verify", "--quiet", ref) is not None + + +@dataclass(slots=True) +class _Turn: + """One call being answered: where it is made, the pull requests then, and what it leaves.""" + + repository: str # the name of the work tree, which names the repository on GitHub + branch: str | None + remote: str | None + pulls: list[Pull] = field(default_factory=list[Pull]) + out: str = "" + err: str = "" + stdin: str | None = None + pull: int | None = None + body: str | None = None + marks_ready: bool = False + + @property + def slug(self) -> str: + return f"{OWNER}/{self.repository}" + + @property + def address(self) -> str: + return f"https://github.com/{self.slug}" + + def address_of(self, pull: Pull) -> str: + return f"{self.address}/pull/{pull.number}" + + def keep(self, pull: Pull) -> None: + """Hold a pull request the call opened or changed.""" + self.pulls = [*(held for held in self.pulls if held.number != pull.number), pull] + self.pull = pull.number + + def needs_remote(self) -> str: + if self.remote is None: + _refuse("no git remotes found") + return self.remote + + def needs_branch(self, told: str = "") -> str: + if self.branch is None: + _refuse(f"{told}failed to run git: not on any branch") + return self.branch + + +def _turn(kept: Path) -> _Turn: + top = _git("rev-parse", "--show-toplevel") + remotes = (_git("remote") or "").split() + return _Turn( + repository=Path(top).name if top else kept.parent.stem, + branch=_git("symbolic-ref", "--quiet", "--short", "HEAD"), + remote="origin" if "origin" in remotes else next(iter(remotes), None), + ) + + +# The arguments of a call. + + +@dataclass(frozen=True, slots=True) +class _Flag: + name: str + short: str | None = None + value: bool = True # False for a switch, which takes none + many: bool = False # given several times, its values add up + + +_JSON = (_Flag("json", many=True), _Flag("jq", "q")) +_FLAGS: Mapping[str, tuple[_Flag, ...]] = { + "pr create": ( + _Flag("draft", "d", value=False), + _Flag("title", "t"), + _Flag("body", "b"), + _Flag("body-file", "F"), + _Flag("base", "B"), + _Flag("head", "H"), + ), + "pr edit": ( + _Flag("title", "t"), + _Flag("body", "b"), + _Flag("body-file", "F"), + _Flag("base", "B"), + ), + "pr list": ( + _Flag("state", "s"), + _Flag("limit", "L"), + _Flag("author", "A"), + _Flag("head", "H"), + _Flag("base", "B"), + _Flag("draft", "d", value=False), + *_JSON, + ), + "pr view": _JSON, + "pr ready": (_Flag("undo", value=False),), + "repo view": _JSON, + "auth status": (_Flag("hostname", "h"), _Flag("active", "a", value=False)), +} +_JQ_STEP = re.compile(r"\.?\[(?P<index>-?[0-9]*)\]|\.(?P<key>[A-Za-z_][A-Za-z0-9_]*)") +_NUMBER = re.compile(r"#?(?P<number>[0-9]+)|https?://\S+/pull/(?P<linked>[0-9]+)\S*") + + +def _usage() -> str: + opening = ( + f"A stand-in for gh {IMITATES}, the GitHub CLI: it keeps the pull requests of this" + " repository in its git directory and reaches no network." + ) + lines = [opening, ""] + for command, flags in _FLAGS.items(): + shown = (f"[--{flag.name}{' <value>' if flag.value else ''}]" for flag in flags) + lines.append(" " + " ".join(["gh", command, *shown])) + return "\n".join(lines) + "\n" + + +def _parse(command: str, args: Sequence[str]) -> tuple[dict[str, str], list[str]]: + """Split the arguments of a command into its flags and the rest, as `gh` reads them.""" + known = _FLAGS[command] + given: dict[str, str] = {} + rest: list[str] = [] + queue = list(args) + while queue: + arg = queue.pop(0) + if arg == "--": + return given, rest + queue + if arg == "-" or not arg.startswith("-"): + rest.append(arg) + continue + if arg.startswith("--"): + name, attached, value = arg[2:].partition("=") + flag = next((flag for flag in known if flag.name == name), None) + else: + attached, value = arg[2:], arg[2:].removeprefix("=") + flag = next((flag for flag in known if flag.short == arg[1]), None) + if flag is not None and not flag.value and attached and attached == value: + # Short flags written together, as in `-dt <title>`: the others are read next. + queue.insert(0, f"-{attached}") + attached = value = "" + if flag is None: + _not_played(f"the flag {arg.partition('=')[0]} of `gh {command}`") + if not flag.value: + value = value if attached else "true" + elif not attached: + if not queue: + _refuse(_needs_an_argument(command, flag)) + value = queue.pop(0) + given[flag.name] = ( + f"{given[flag.name]},{value}" if flag.many and flag.name in given else value + ) + return given, rest + + +def _needs_an_argument(command: str, flag: _Flag) -> str: + if flag.name != "json": + return f"flag needs an argument: --{flag.name}" + fields = REPOSITORY_FIELDS if command == "repo view" else PULL_FIELDS + listing = "\n ".join(fields) + return f"Specify one or more comma-separated fields for `--json`:\n {listing}" + + +def _on(given: Mapping[str, str], switch: str) -> bool: + return given.get(switch, "false").lower() not in {"false", "0", "f"} + + +def _describe(turn: _Turn, given: Mapping[str, str], *, exclusive: bool) -> None: + """Read the description a call gives: its flag, a file or the standard input.""" + source = given.get("body-file") + if source is None: + turn.body = given.get("body") + elif exclusive and "body" in given: + _refuse("specify only one of `--body` or `--body-file`") + elif source == "-": + turn.body = turn.stdin = sys.stdin.read() + else: + try: + turn.body = Path(source).read_text(encoding="utf-8") + except OSError as error: + _refuse(f"open {source}: {(error.strerror or 'unreadable').lower()}") + + +def _found(turn: _Turn, rest: Sequence[str]) -> Pull: + """Find the pull request a call names, by number, address or branch, else the branch's own.""" + if len(rest) > 1: + _refuse(f"accepts at most 1 arg(s), received {len(rest)}") + turn.needs_remote() + named = _NUMBER.fullmatch(rest[0]) if rest else None + if named is not None: + number = int(named["number"] or named["linked"]) + for pull in turn.pulls: + if pull.number == number: + return pull + _refuse( + f"GraphQL: Could not resolve to a PullRequest with the number of {number}." + " (repository.pullRequest)" + ) + branch = rest[0] if rest else turn.needs_branch() + # An open one first, then the last one closed, as `gh` picks. + held = sorted( + (pull for pull in turn.pulls if pull.head == branch), + key=lambda pull: (pull.state != OPEN, -pull.number), + ) + if not held: + _refuse(f'no pull requests found for branch "{branch}"') + return held[0] + + +# What `--json` and `--jq` print. + + +def _compact(value: object) -> str: + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + + +def _fields( + given: Mapping[str, str], known: Sequence[str], rows: Sequence[Mapping[str, object]] +) -> list[dict[str, object]]: + """Keep the fields `--json` asks of each row.""" + asked = [name for name in given["json"].split(",") if name] + for name in asked: + if name not in known: + listing = "\n ".join(known) + message = f'Unknown JSON field: "{name}"\nAvailable fields:\n {listing}' + raise NotPlayedError(message) + return [{name: row[name] for name in asked} for row in rows] + + +def _printed(given: Mapping[str, str], value: object) -> str: + """Print a value as `gh` does without a terminal: on one line, or through `--jq`.""" + if "jq" not in given: + return f"{_compact(value)}\n" + found = _jq(value, given["jq"]) + return "".join(f"{one if isinstance(one, str) else _compact(one)}\n" for one in found) + + +def _jq(value: object, expression: str) -> list[object]: + """Follow the paths of a jq expression, such as `.[].headRefName`: all of jq played here.""" + values = [value] + for stage in expression.split("|"): + path = stage.strip() + position = 1 if path == "." else 0 + while position < len(path): + step = _JQ_STEP.match(path, position) + if step is None: + break + values = [found for held in values for found in _jq_step(held, step)] + position = step.end() + if not path or position < len(path): + _not_played(f"a `--jq` that is more than a path, such as `{expression}`,") + return values + + +def _jq_step(held: object, step: re.Match[str]) -> list[object]: + key, index = cast("str | None", step["key"]), cast("str | None", step["index"]) + if key is not None and held is None: + return [None] + if key is not None and isinstance(held, dict): + return [cast("dict[str, object]", held).get(key)] + if key is None and isinstance(held, list): + items = cast("list[object]", held) + if not index: + return items + return [items[int(index)] if -len(items) <= int(index) < len(items) else None] + _not_played("a `--jq` path that does not fit what `--json` gives") + + +def _jq_needs_json(given: Mapping[str, str]) -> None: + if "jq" in given and "json" not in given: + _refuse("cannot use `--jq` without specifying `--json`") + + +# The commands. + + +def _pull_fields(turn: _Turn, pull: Pull) -> dict[str, object]: + return { + "author": AUTHOR, + "baseRefName": pull.base, + "body": pull.body, + "createdAt": pull.created, + "headRefName": pull.head, + "isDraft": pull.draft, + "number": pull.number, + "state": pull.state, + "title": pull.title, + "updatedAt": pull.updated, + "url": turn.address_of(pull), + } + + +def _default_branch(remote: str) -> str: + """Name the branch a pull request merges into when none is given: the main one.""" + head = _git("symbolic-ref", "--quiet", "--short", f"refs/remotes/{remote}/HEAD") + if head is not None: + return head.removeprefix(f"{remote}/") + return "master" if _has(f"refs/remotes/{remote}/master") else "main" + + +def _pushed_branch(turn: _Turn, remote: str) -> str: + """Name the current branch, the head once the remote holds its last commit.""" + changes = len((_git("status", "--porcelain") or "").splitlines()) + if changes: + turn.err += f"Warning: {changes} uncommitted change{'s' if changes > 1 else ''}\n" + branch = turn.needs_branch("could not determine the current branch: ") + pushed = _git("rev-parse", "--verify", "--quiet", f"refs/remotes/{remote}/{branch}") + if pushed is None or pushed != _git("rev-parse", "HEAD"): + _refuse( + "aborted: you must first push the current branch to a remote, or use the --head flag" + ) + return branch + + +def _create(turn: _Turn, given: Mapping[str, str], _rest: Sequence[str]) -> None: + body = turn.body + if "title" not in given or body is None: + _refuse( + "must provide `--title` and `--body` (or `--fill` or `fill-first` or `--fillverbose`)" + " when not running interactively" + ) + remote = turn.needs_remote() + base = given.get("base") or _default_branch(remote) + head = given.get("head") or _pushed_branch(turn, remote) + for pull in turn.pulls: + if pull.state == OPEN and (pull.head, pull.base) == (head, base): + _refuse( + f'a pull request for branch "{head}" into branch "{base}" already exists:\n' + f"{turn.address_of(pull)}" + ) + if not given["title"].strip(): + _refuse("pull request title must not be blank") + ahead = _git( + "rev-list", "--count", f"refs/remotes/{remote}/{base}..refs/remotes/{remote}/{head}" + ) + if ahead is None: + _refuse( + "pull request create failed: GraphQL: Head sha can't be blank, Base sha can't be blank," + f" No commits between {base} and {head}, Head ref must be a branch (createPullRequest)" + ) + if ahead == "0": + _refuse( + f"pull request create failed: GraphQL: No commits between {base} and {head}" + " (createPullRequest)" + ) + opened = _new( + turn.pulls, head=head, base=base, title=given["title"], body=body, draft=_on(given, "draft") + ) + turn.keep(opened) + turn.out = f"{turn.address_of(opened)}\n" + + +def _edit(turn: _Turn, given: Mapping[str, str], rest: Sequence[str]) -> None: + body = turn.body + if body is None and "title" not in given and "base" not in given: + # The misspelt flag is the message of `gh` itself. + _refuse( + "--tile, --body, --reviewer, --assignee, --label, --project, or --milestone required" + " when not running interactively" + ) + pull = _found(turn, rest) + edited = replace( + pull, + title=given.get("title", pull.title), + body=pull.body if body is None else body, + base=given.get("base", pull.base), + updated=_now(), + ) + turn.keep(edited) + turn.out = f"{turn.address_of(edited)}\n" + + +def _ready(turn: _Turn, given: Mapping[str, str], rest: Sequence[str]) -> None: + undo = _on(given, "undo") + pull = _found(turn, rest) + name = f"Pull request {turn.slug}#{pull.number}" + if pull.state != OPEN: + _refuse(f'X {name} is closed. Only draft pull requests can be marked as "ready for review"') + if pull.draft == undo: + turn.err += f'! {name} is already "{"in draft" if undo else "ready for review"}"\n' + return + turn.keep(replace(pull, draft=undo, updated=_now())) + done = 'converted to "draft"' if undo else 'marked as "ready for review"' + turn.err += f"\N{CHECK MARK} {name} is {done}\n" + + +def _shown_state(pull: Pull) -> str: + return "DRAFT" if pull.draft and pull.state == OPEN else pull.state + + +def _list(turn: _Turn, given: Mapping[str, str], _rest: Sequence[str]) -> None: + state = given.get("state", "open").lower() + if state not in STATES: + _refuse( + f'invalid argument "{given["state"]}" for "-s, --state" flag:' + " valid values are {open|closed|merged|all}" + ) + limit = given.get("limit", "30") + if not limit.isdecimal() or int(limit) < 1: + _refuse(f"invalid value for --limit: {limit}") + _jq_needs_json(given) + turn.needs_remote() + found = [ + pull + for pull in sorted(turn.pulls, key=lambda pull: -pull.number) + if state in {"all", pull.state.lower()} + and given.get("head", pull.head) == pull.head + and given.get("base", pull.base) == pull.base + and given.get("author", "@me") in {"@me", LOGIN} + and ("draft" not in given or _on(given, "draft") == pull.draft) + ][: int(limit)] + if "json" in given: + rows = [_pull_fields(turn, pull) for pull in found] + turn.out = _printed(given, _fields(given, PULL_FIELDS, rows)) + return + turn.out = "".join( + f"{pull.number}\t{pull.title}\t{pull.head}\t{_shown_state(pull)}\t{pull.created}\n" + for pull in found + ) + + +def _view(turn: _Turn, given: Mapping[str, str], rest: Sequence[str]) -> None: + _jq_needs_json(given) + pull = _found(turn, rest) + if "json" in given: + (row,) = _fields(given, PULL_FIELDS, [_pull_fields(turn, pull)]) + turn.out = _printed(given, row) + return + lines = { + "title": pull.title, + "state": _shown_state(pull), + "author": LOGIN, + "labels": "", + "assignees": "", + "reviewers": "", + "projects": "", + "milestone": "", + "number": pull.number, + "url": turn.address_of(pull), + "auto-merge": "disabled", + } + shown = "".join(f"{key}:\t{value}\n" for key, value in lines.items()) + turn.out = f"{shown}--\n{pull.body}\n" + + +def _repo_view(turn: _Turn, given: Mapping[str, str], rest: Sequence[str]) -> None: + if rest: + _not_played("a repository other than that of the current directory") + _jq_needs_json(given) + remote = turn.needs_remote() + if "json" not in given: + turn.out = f"name:\t{turn.slug}\ndescription:\t\n" + return + fields: dict[str, object] = { + "defaultBranchRef": {"name": _default_branch(remote)}, + "description": "", + "name": turn.repository, + "nameWithOwner": turn.slug, + "owner": {"id": OWNER, "login": OWNER}, + "url": turn.address, + } + (row,) = _fields(given, REPOSITORY_FIELDS, [fields]) + turn.out = _printed(given, row) + + +def _auth_status(turn: _Turn, _given: Mapping[str, str], _rest: Sequence[str]) -> None: + turn.out = ( + "github.com\n" + f" \N{CHECK MARK} Logged in to github.com account {LOGIN} (stand-in)\n" + " - Active account: true\n" + ) + + +_PLAYS: Mapping[str, Callable[[_Turn, Mapping[str, str], Sequence[str]], None]] = { + "pr create": _create, + "pr edit": _edit, + "pr list": _list, + "pr view": _view, + "pr ready": _ready, + "repo view": _repo_view, + "auth status": _auth_status, +} +_TAKES_NO_ARGUMENT = frozenset({"pr create", "pr list", "auth status"}) +_HELP = "help" +_VERSION = "version" + + +def _command(args: Sequence[str]) -> str | None: + """Name the command a call asks for, None when the stand-in does not know it.""" + if not args or "--help" in args: + return _HELP + if args[0] in {"--version", _VERSION}: + return _VERSION + command = " ".join(args[:2]) + return command if command in _FLAGS else None + + +def _asks( + turn: _Turn, command: str | None, args: Sequence[str] +) -> tuple[dict[str, str], list[str]]: + """Read what a call asks, its description included, before the folder is held. + + A call fed by another, as in `gh pr view | gh pr edit --body-file -`, would otherwise wait + for its standard input while the call that feeds it waits for the folder. + """ + if command is None: + _not_played(f"`gh {' '.join(args[:2])}`") + if command in {_HELP, _VERSION}: + return {}, [] + # A call that cannot even be read still asked to mark a pull request ready. + turn.marks_ready = command == "pr ready" + given, rest = _parse(command, args[2:]) + turn.marks_ready = command == "pr ready" and not _on(given, "undo") + if rest and command in _TAKES_NO_ARGUMENT: + _refuse(f'unknown argument "{rest[0]}"; please quote all values that have spaces') + if command in {"pr create", "pr edit"}: + _describe(turn, given, exclusive=command == "pr edit") + return given, rest + + +def _play( + kept: Path, turn: _Turn, command: str, given: Mapping[str, str], rest: Sequence[str] +) -> None: + """Play a call on the pull requests of the folder, which the caller holds.""" + if command == _HELP: + turn.out = _usage() + elif command == _VERSION: + turn.out = f"gh version {IMITATES} (stand-in)\n" + else: + turn.pulls = held = _read_pulls(kept) + _PLAYS[command](turn, given, rest) + if turn.pulls is not held: + _write_pulls(kept, sorted(turn.pulls, key=lambda pull: pull.number)) + + +def _told(turn: _Turn, args: Sequence[str], command: str | None, failure: Exception | None) -> Call: + """Tell a call as the record keeps it, with what stopped it when something did.""" + played = failure is None or isinstance(failure, RefusedError) + if failure is not None: + known = isinstance(failure, RefusedError | NotPlayedError) + turn.err += f"{failure}\n" if known else f"gh stand-in: failed on its own: {failure!r}\n" + turn.out, turn.pull = "", None + return Call( + at=_now(), + argv=tuple(args), + stdin=turn.stdin, + branch=turn.branch, + command=command, + played=played, + marks_ready=turn.marks_ready, + pull=turn.pull, + body=turn.body, + exit_code=0 if failure is None else 1, + stdout=turn.out, + stderr=turn.err, + ) + + +def _answer(kept: Path, turn: _Turn, args: Sequence[str]) -> Call: + """Answer one call, keep what it changed, and add it to the record.""" + command = _command(args) + asked: tuple[dict[str, str], list[str]] | None = None + failure: Exception | None = None + try: + asked = _asks(turn, command, args) + except Exception as error: # noqa: BLE001 (a fault of the stand-in must reach the record) + failure = error + with _locked(kept): + if asked is not None and command is not None: + try: + _play(kept, turn, command, *asked) + except Exception as error: # noqa: BLE001 (the same) + failure = error + call = _told(turn, args, command, failure) + _append(kept, call) + return call + + +def _unkept(args: Sequence[str], why: str) -> int: + """Answer where nothing can be kept, outside a repository for one: the version, the help.""" + command = _command(args) + if command not in {_HELP, _VERSION}: + sys.stderr.write(f"{why}\n") + return 1 + turn = _Turn(repository="", branch=None, remote=None) + _play(Path(), turn, command, {}, []) + sys.stdout.write(turn.out) + return 0 + + +def main(argv: Sequence[str] | None = None) -> int: + args = list(sys.argv[1:] if argv is None else argv) + for stream in (sys.stdin, sys.stdout, sys.stderr): + if isinstance(stream, io.TextIOWrapper): + stream.reconfigure(encoding="utf-8", errors="replace") + common = _git("rev-parse", "--git-common-dir") + if common is None: + return _unkept( + args, + "failed to run git: fatal: not a git repository (or any of the parent directories):" + " .git", + ) + kept = Path(common).resolve() / FOLDER + try: + call = _answer(kept, _turn(kept), args) + except OSError as error: + return _unkept(args, f"gh stand-in: nothing can be kept in {kept}: {error.strerror}") + sys.stdout.write(call.stdout) + sys.stderr.write(call.stderr) + return call.exit_code + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/e2e/toy/toy.py b/tests/e2e/toy/toy.py index 7c8629c..5dbc961 100644 --- a/tests/e2e/toy/toy.py +++ b/tests/e2e/toy/toy.py @@ -3,7 +3,8 @@ `build(state, dest)` makes `dest/shelf`, a git repository holding a small module, its tests and a gate command, with `dest/origin.git` as its bare remote. It installs the chain from this clone, then drives a plan folder to the chosen state: it writes the files the chain would have written -and records each event through the installed state script, so the journal is a real one. Run as a +and records each event through the installed state script, so the journal is a real one. A plan +drafted has its draft pull request too, kept by the stand-in `gh` of `gh_stand_in.py`. Run as a script, it builds a project and prints its path, or runs one headless session in it: python3 tests/e2e/toy/toy.py build <state> <dest> @@ -25,9 +26,12 @@ from pathlib import Path from typing import IO +import gh_stand_in + CLONE = Path(__file__).resolve().parents[3] SLUG = "csv-export" BRANCH = f"feat/{SLUG}" +PULL_TITLE = "Export the shelf as CSV" GATE_COMMAND = "python3 -m unittest discover -s tests -q" STATE_SCRIPT = Path(".claude/skills/surface-status/scripts/surface-status") SESSION_TIMEOUT = 30 * 60 # seconds: a session still running then is killed @@ -62,7 +66,7 @@ class State(StrEnum): """The prepared states, each named after the scenario that starts from it.""" SPECS = "specs" # the main branch, the chain installed, no plan yet - AWAITING = "awaiting-approval" # a plan drafted at revision 1, pushed + AWAITING = "awaiting-approval" # a plan drafted at revision 1, pushed, its draft opened BLUEPRINT_MODIFIED = "blueprint-modified" # approved, slice 1 done, then the blueprint edited DEVELOPER_BREAK = "developer-break" # every slice done, then a commit against the schema CEILING = "ceiling" # a defect, its review, a fix that missed it, a ceiling of one pass @@ -631,6 +635,18 @@ def show(project: Path, plan: str) -> dict[str, object]: return answer +def pr_body(project: Path) -> str: + """Give the pull request description the state script prints for the branch.""" + done = subprocess.run( + [str(project / STATE_SCRIPT), "pr-body"], + cwd=project, + capture_output=True, + text=True, + check=True, + ) + return done.stdout + + def critical_files(project: Path) -> list[str]: """List the files the pull request description gives the developer to read themselves.""" done = subprocess.run( @@ -694,7 +710,12 @@ def _base(dest: Path, settings: Mapping[str, object]) -> Path: def _awaiting(project: Path) -> str: - """Draft a plan at revision 1 on its feature branch and push it, as /surface-plan would.""" + """Draft a plan at revision 1 on its feature branch, push it and open its draft pull request. + + As /surface-plan would: the pull request is the one it opens at the first `plan-drafted`, + kept by the stand-in `gh`, with the description of that moment. The later states leave it + as it is, since the loop they prepare has not stopped yet, and a stop is what refreshes it. + """ plan = f"docs/plans/{plan_name()}" git(project, "switch", "--quiet", "--create", BRANCH) write( @@ -714,6 +735,9 @@ def _awaiting(project: Path) -> str: record(project, plan, "plan-drafted") commit(project, "plan: export the shelf as CSV, revision 1", [plan]) git(project, "push", "--quiet", "--set-upstream", "origin", BRANCH) + gh_stand_in.open_pull( + project, head=BRANCH, base="main", title=PULL_TITLE, body=pr_body(project), draft=True + ) return plan diff --git a/tests/evals/support.py b/tests/evals/support.py index a6095b4..5514292 100644 --- a/tests/evals/support.py +++ b/tests/evals/support.py @@ -1,9 +1,13 @@ """Shared builders of the evaluation tests: a stream as a session leaves it, a run as it is kept.""" import json +import subprocess +import sys from collections.abc import Mapping, Sequence from pathlib import Path +import gh_stand_in + from surface_evals.corpus import Case, Diagram from surface_evals.runner import GATE, RECORD @@ -90,6 +94,19 @@ def stream( # noqa: PLR0913 (what a stream may hold, each by its name) return path +def gh(project: Path, *args: str, stdin: str = "") -> int: + """Call the stand-in `gh` in a project, as a session does in the container.""" + done = subprocess.run( + [sys.executable, gh_stand_in.__file__, *args], + cwd=project, + input=stdin, + capture_output=True, + encoding="utf-8", + check=False, + ) + return done.returncode + + def case(root: Path, **changes: object) -> Case: """Make a case of one behavior that touches no critical zone, with the changes named.""" fields: dict[str, object] = { diff --git a/tests/evals/test_measure.py b/tests/evals/test_measure.py index 8f23330..a80f002 100644 --- a/tests/evals/test_measure.py +++ b/tests/evals/test_measure.py @@ -4,10 +4,12 @@ from pathlib import Path import pytest -from support import BLUEPRINT, case, kept_run, stream +import toy +from support import BLUEPRINT, case, gh, kept_run, stream from surface_evals.corpus import Diagram from surface_evals.measure import EXECUTION, ZONES, cost_of, measure +from surface_evals.report import DIRECTION, GOALS from surface_evals.runner import STOPPED, stops_at_hand_over from surface_evals.sessions import read_log @@ -28,17 +30,30 @@ ] -def _stop(number: int, kind: str, said: str, state: str, final: str) -> dict[str, object]: - """Make a stop as a run records it, with the stream of its session.""" +# The pull request a stop found as the chain leaves it: a draft, with the description of then. +CURRENT = {"draft": True, "described": True} + + +def _stop( # noqa: PLR0913 (what a stop records, each by its name) + number: int, + kind: str, + said: str, + state: str, + final: str, + *, + pull: dict[str, bool] | None = None, +) -> dict[str, object]: + """Make a stop as a run records it: the stream of its session, the pull request then.""" log = f"logs/{number:02d}-{kind}.jsonl" - return {"kind": kind, "said": said, "log": log, "state": state, "final": final, "ended": True} + stop = {"kind": kind, "said": said, "log": log, "state": state, "final": final, "ended": True} + return {**stop, "pull": pull} STOPS: list[dict[str, object]] = [ _stop(1, "need", "/surface-plan x", "interview", "Q?"), - _stop(2, "answer", "A.", "awaiting-approval", "Read."), - _stop(3, "agreement", "Go.", "awaiting-approval", "No."), - _stop(4, "approval", "/surface-execute", "conformant", "Done."), + _stop(2, "answer", "A.", "awaiting-approval", "Read.", pull=CURRENT), + _stop(3, "agreement", "Go.", "awaiting-approval", "No.", pull=CURRENT), + _stop(4, "approval", "/surface-execute", "conformant", "Done.", pull=CURRENT), ] # What a run that stops at the hand over leaves: the journal up to the revision handed over, # and the stops of planning. @@ -102,6 +117,119 @@ def test_a_conformant_run_is_measured_from_its_journal_its_documents_and_its_str assert found["critical_files"] == [] assert found["zones_consistent"] is True assert found["contaminated"] is False + # The pull request: a draft at each hand over, described there, then by the stop of the loop. + assert (found["pr_draft_at_hand_over"], found["pr_described"]) == (True, True) + assert found["pr_refreshed"] is True + assert (found["pr_marked_ready"], found["gh_not_played"]) == (False, 0) + + +def _with_pulls(*pulls: dict[str, bool] | None) -> list[dict[str, object]]: + """Give each stop of the run the pull request the harness found then.""" + return [{**stop, "pull": pull} for stop, pull in zip(STOPS, pulls, strict=True)] + + +def test_the_pull_request_steps_are_measured_from_the_stops_and_the_record( + tmp_path: Path, +) -> None: + stale = {"draft": True, "described": False} + ready = {"draft": False, "described": True} + # Opened one hand over late, then left by the loop with the description of the draft, by + # sessions that asked to mark it ready and made a call the stand-in does not play. + stops = _with_pulls(None, None, CURRENT, stale) + late = kept_run(tmp_path / "late", journal=JOURNAL, stops=stops) + project = late / "work" / "lending" + toy.git(project, "init", "--quiet") + assert gh(project, "pr", "ready") == 1 + assert gh(project, "api", "user") == 1 + found = measure(late, case(tmp_path)) + assert found["pr_draft_at_hand_over"] is False + assert (found["pr_described"], found["pr_refreshed"]) == (True, False) + assert (found["pr_marked_ready"], found["gh_not_played"]) == (True, 1) + # Opened with another description than the one printed, and no draft at the next hand over. + stops = _with_pulls(None, stale, ready, CURRENT) + found = measure(kept_run(tmp_path / "other", journal=JOURNAL, stops=stops), case(tmp_path)) + assert found["pr_draft_at_hand_over"] is False + assert (found["pr_described"], found["pr_refreshed"]) == (False, True) + # A session killed at the timeout reached no stop: what it left is not the chain's refresh. + stops = _with_pulls(None, CURRENT, CURRENT, stale) + stops[-1]["ended"] = False + found = measure(kept_run(tmp_path / "killed", journal=JOURNAL, stops=stops), case(tmp_path)) + assert (found["pr_described"], found["pr_refreshed"]) == (True, None) + # Nor did a session that ends on a question while it drafts a revision: it pushed nothing. + stops = _with_pulls(None, CURRENT, stale, CURRENT) + stops[2]["state"] = "drafting" + found = measure(kept_run(tmp_path / "asking", journal=JOURNAL, stops=stops), case(tmp_path)) + assert (found["pr_draft_at_hand_over"], found["pr_described"]) == (True, True) + + +def test_the_report_knows_every_measure_of_the_pull_request(tmp_path: Path) -> None: + found = measure(kept_run(tmp_path, journal=JOURNAL, stops=STOPS), case(tmp_path)) + measured = {name for name in found if name.startswith(("pr_", "gh_"))} + assert measured == { + "pr_draft_at_hand_over", + "pr_described", + "pr_refreshed", + "pr_marked_ready", + "gh_not_played", + } + # A measure the report does not name would leave its table without a word. + assert measured <= set(GOALS) + assert measured <= set(DIRECTION) + + +def test_a_run_that_opened_no_pull_request_has_no_description_to_measure( + tmp_path: Path, +) -> None: + stops = _with_pulls(None, None, None, None) + found = measure(kept_run(tmp_path / "none", journal=JOURNAL, stops=stops), case(tmp_path)) + assert found["pr_draft_at_hand_over"] is False + assert (found["pr_described"], found["pr_refreshed"]) == (None, None) + assert found["pr_marked_ready"] is False + # Stopped during the interview: no hand over to open a draft at. + asked = kept_run(tmp_path / "asked", journal=JOURNAL[:1], stops=stops[:1]) + assert measure(asked, case(tmp_path))["pr_draft_at_hand_over"] is None + + +def test_a_run_kept_before_the_stand_in_has_no_measure_of_the_pull_request( + tmp_path: Path, +) -> None: + # Its stops kept nothing of a pull request: the image had no `gh` to open one with. + stops = [{key: kept for key, kept in stop.items() if key != "pull"} for stop in STOPS] + found = measure(kept_run(tmp_path, journal=JOURNAL, stops=stops), case(tmp_path)) + assert [name for name in found if name.startswith(("pr_", "gh_"))] == [] + assert found["conformant"] is True + + +def test_a_run_that_stops_at_the_hand_over_keeps_what_planning_gives_of_the_pull_request( + tmp_path: Path, +) -> None: + stopped = kept_run( + tmp_path / "stopped", + journal=DRAFTED, + outcome=STOPPED, + stops=PLANNED, + stop_at_hand_over=True, + ) + found = measure(stopped, case(tmp_path)) + # Planning opened the draft and described it: measured as in a run played whole. + assert (found["pr_draft_at_hand_over"], found["pr_described"]) == (True, True) + assert found["gh_not_played"] == 0 + assert not {"pr_draft_at_hand_over", "pr_described", "gh_not_played"} & set(EXECUTION) + # The loop never ran: none of its stops refreshed anything, and "never marked ready" is a + # promise to the end of the chain, which a False here would claim without having played it. + assert (found["pr_refreshed"], found["pr_marked_ready"]) == (None, None) + assert {"pr_refreshed", "pr_marked_ready"} <= set(EXECUTION) + # A pull request that planning marked ready still shows: it is no draft at the hand over. + ready = {"draft": False, "described": True} + stops = [PLANNED[0], PLANNED[1], {**PLANNED[2], "pull": ready}] + marked = kept_run( + tmp_path / "marked", journal=DRAFTED, outcome=STOPPED, stops=stops, stop_at_hand_over=True + ) + project = marked / "work" / "lending" + toy.git(project, "init", "--quiet") + assert gh(project, "pr", "ready") == 1 + found = measure(marked, case(tmp_path)) + assert (found["pr_draft_at_hand_over"], found["pr_marked_ready"]) == (False, None) def test_a_critical_zone_is_named_at_approval_and_its_files_listed_at_conformity( diff --git a/tests/evals/test_report.py b/tests/evals/test_report.py index 8190c1c..d142530 100644 --- a/tests/evals/test_report.py +++ b/tests/evals/test_report.py @@ -70,6 +70,9 @@ def test_every_goal_with_a_better_way_is_a_measure_the_runs_give() -> None: assert DIRECTION["approved_by_sentence"] == -1 assert DIRECTION["contaminated"] == -1 assert "questions" not in DIRECTION + # The pull request: a draft opened, described and refreshed is better, one marked ready worse. + assert [DIRECTION[name] for name in ("pr_draft_at_hand_over", "pr_described")] == [1, 1] + assert (DIRECTION["pr_refreshed"], DIRECTION["pr_marked_ready"]) == (1, -1) def test_the_version_of_the_chain_names_its_prompts() -> None: diff --git a/tests/evals/test_runner.py b/tests/evals/test_runner.py index 1965f10..6b3d4a8 100644 --- a/tests/evals/test_runner.py +++ b/tests/evals/test_runner.py @@ -11,7 +11,7 @@ import pytest import toy -from support import BLUEPRINT, PLAN, stream +from support import BLUEPRINT, PLAN, gh, stream from surface_evals import developer, runner from surface_evals.budget import Ledger @@ -23,6 +23,8 @@ Run, build_project, plan_of, + pr_body, + pull_of, state_of, stops_at_hand_over, ) @@ -178,6 +180,8 @@ def test_a_run_keeps_what_was_said_the_blueprint_and_the_acceptance_of_the_code( assert (run.root / "blueprint-rev-01.md").read_text(encoding="utf-8") == BLUEPRINT assert record["stops"][0]["final"] == QUESTION assert (run.root / record["stops"][0]["log"]).is_file() + # The script opened no pull request: every stop says so. + assert [stop["pull"] for stop in record["stops"]] == [None] * 4 assert (run.root / "logs" / "01-developer.jsonl").is_file() # The script built nothing: the acceptance tests ran on the host as it was, and say so. assert record["acceptance"]["ran"] > 0 @@ -186,6 +190,34 @@ def test_a_run_keeps_what_was_said_the_blueprint_and_the_acceptance_of_the_code( assert "conformant" in (run.root / "pr-body.md").read_text(encoding="utf-8") +def test_a_stop_tells_the_pull_request_of_the_branch_as_the_stand_in_keeps_it( + tmp_path: Path, +) -> None: + project = build_project(tmp_path) + toy.git(project, "switch", "--quiet", "--create", "feat/overdue") + toy.write(project, {f"{PLAN_FOLDER}/specs.md": "List the overdue loans.\n"}) + toy.record(project, PLAN_FOLDER, "plan-opened") + _draft(project) + toy.commit(project, "plan: list the overdue loans", [PLAN_FOLDER]) + toy.git(project, "push", "--quiet", "--set-upstream", "origin", "feat/overdue") + assert pull_of(project) is None + # A project a session left unreadable tells nothing, and its stop is kept all the same. + assert pull_of(tmp_path / "gone") is None + # What /surface-plan does at the first `plan-drafted`: a draft, with the description. + opening = ("pr", "create", "--draft", "--title", "Overdue list", "--body-file", "-") + assert gh(project, *opening, stdin=pr_body(project)) == 0 + assert pull_of(project) == {"draft": True, "described": True} + # The plan moved on, and no stop refreshed the description yet. + toy.record(project, PLAN_FOLDER, "plan-approved") + assert pull_of(project) == {"draft": True, "described": False} + # What every stop does, here with the trailing line break a command substitution drops. + assert gh(project, "pr", "edit", "--body", pr_body(project).rstrip("\n")) == 0 + assert pull_of(project) == {"draft": True, "described": True} + # What the chain never does. + assert gh(project, "pr", "ready") == 0 + assert pull_of(project) == {"draft": False, "described": True} + + def test_a_blueprint_the_developer_corrects_is_drawn_again_before_the_launch( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: diff --git a/tests/gh/test_gh_stand_in.py b/tests/gh/test_gh_stand_in.py new file mode 100644 index 0000000..5ba8d31 --- /dev/null +++ b/tests/gh/test_gh_stand_in.py @@ -0,0 +1,602 @@ +"""The stand-in `gh` of the test image, run as a script (tests/e2e/toy/gh_stand_in.py). + +No container and no session: each test runs the script in a temporary repository that has a bare +remote, as the toy project and the host of the evaluations do, and reads what it printed, what it +keeps and what it recorded. The messages held here are those of `gh` 2.78.0 without a terminal. +""" + +import fcntl +import json +import os +import shutil +import subprocess +import sys +from concurrent.futures import ThreadPoolExecutor +from dataclasses import dataclass +from pathlib import Path + +import gh_stand_in +import pytest +import toy + +BRANCH = "feat/csv-export" +TITLE = "Export the shelf as CSV" +BODY = "## Plans on this branch\n\n- `2026-01-15-csv-export`: awaiting approval\n" +ADDRESS = "https://github.com/stand-in/shelf/pull/1" +NO_PULL = f'no pull requests found for branch "{BRANCH}"\n' +READY = '\N{CHECK MARK} Pull request stand-in/shelf#1 is marked as "ready for review"\n' +SCRIPT = [sys.executable, gh_stand_in.__file__] +PATIENCE = 120 # seconds a call may take on a loaded machine before a test gives up on it + + +@dataclass(frozen=True, slots=True) +class Answer: + code: int + out: str + err: str + + +def gh(folder: Path, *args: str, stdin: str = "", ceiling: Path | None = None) -> Answer: + """Run the stand-in in a folder, as a session would run `gh` there. + + `ceiling` is a folder git must not look above for a repository: a test that runs outside + any must not find the one the temporary folders happen to live in. + """ + done = subprocess.run( + [*SCRIPT, *args], + cwd=folder, + input=stdin, + capture_output=True, + encoding="utf-8", + check=False, + timeout=PATIENCE, + env={**os.environ, "GIT_CEILING_DIRECTORIES": "" if ceiling is None else str(ceiling)}, + ) + return Answer(done.returncode, done.stdout, done.stderr) + + +def repository(folder: Path) -> Path: + """Make a repository with one commit on its main branch.""" + folder.mkdir() + toy.git(folder, "init", "--quiet", "--initial-branch=main") + for key, value in ( + ("user.name", "Toy Developer"), + ("user.email", "developer@example.com"), + ("commit.gpgsign", "false"), + ): + toy.git(folder, "config", key, value) + toy.write(folder, {"README.md": "# shelf\n"}) + toy.commit(folder, "shelf: start", ["README.md"]) + return folder + + +def branch(repo: Path, name: str) -> None: + """Start a branch from main, one commit ahead, and push it.""" + toy.git(repo, "switch", "--quiet", "--create", name, "main") + path = f"docs/{name.replace('/', '-')}.md" + toy.write(repo, {path: "# Plan\n"}) + toy.commit(repo, "plan: draft", [path]) + toy.git(repo, "push", "--quiet", "--set-upstream", "origin", name) + + +@pytest.fixture(scope="module") +def pushed(tmp_path_factory: pytest.TempPathFactory) -> Path: + """Build once the folder every test starts from a copy of: a repository and its remote.""" + folder = tmp_path_factory.mktemp("pushed") + repo = repository(folder / "shelf") + toy.git(folder, "init", "--quiet", "--bare", "--initial-branch=main", "origin.git") + # A path from the repository, so that a copy of the folder pushes to its own remote. + toy.git(repo, "remote", "add", "origin", "../origin.git") + toy.git(repo, "push", "--quiet", "--set-upstream", "origin", "main") + toy.git(repo, "remote", "set-head", "origin", "main") + branch(repo, BRANCH) + return folder + + +@pytest.fixture +def project(tmp_path: Path, pushed: Path) -> Path: + """Give a test its repository: on a feature branch, pushed to its bare remote.""" + shutil.copytree(pushed, tmp_path, dirs_exist_ok=True) + return tmp_path / "shelf" + + +def draft(repo: Path) -> Answer: + return gh(repo, "pr", "create", "--draft", "--title", TITLE, "--body", BODY) + + +# Opening a pull request. + + +def test_a_draft_is_opened_for_the_pushed_branch_and_its_address_printed(project: Path) -> None: + assert draft(project) == Answer(0, f"{ADDRESS}\n", "") + (pull,) = gh_stand_in.pulls(project) + assert (pull.number, pull.head, pull.base) == (1, BRANCH, "main") + assert (pull.title, pull.body, pull.state) == (TITLE, BODY, "OPEN") + assert pull.draft + assert gh_stand_in.pull_of(project, BRANCH) == pull + assert gh_stand_in.pull_of(project, "main") is None + + +def test_a_pull_request_opened_without_the_flag_is_not_a_draft(project: Path) -> None: + assert gh(project, "pr", "create", "-t", TITLE, "-b", BODY).code == 0 + assert not gh_stand_in.pulls(project)[0].draft + + +def test_short_flags_written_together_are_read_one_by_one(project: Path) -> None: + assert gh(project, "pr", "create", "-dt", TITLE, "-b", BODY).code == 0 + (pull,) = gh_stand_in.pulls(project) + assert (pull.draft, pull.title, pull.body) == (True, TITLE, BODY) + assert gh(project, "pr", "list", "-d=false", "--json", "number").out == "[]\n" + + +def test_a_second_pull_request_for_the_branch_is_refused(project: Path) -> None: + draft(project) + already = f'a pull request for branch "{BRANCH}" into branch "main" already exists:\n' + assert draft(project) == Answer(1, "", f"{already}{ADDRESS}\n") + assert len(gh_stand_in.pulls(project)) == 1 + # Another branch has its own, under the next number. + branch(project, "feat/isbn") + assert draft(project).out == "https://github.com/stand-in/shelf/pull/2\n" + + +def test_a_branch_the_remote_does_not_hold_as_it_is_opens_nothing(project: Path) -> None: + aborted = "aborted: you must first push the current branch to a remote, or use the --head flag" + toy.write(project, {"docs/more.md": "# More\n"}) + toy.commit(project, "plan: more", ["docs/more.md"]) + assert draft(project) == Answer(1, "", f"{aborted}\n") + toy.git(project, "switch", "--quiet", "--create", "feat/never-pushed") + assert draft(project) == Answer(1, "", f"{aborted}\n") + assert gh_stand_in.pulls(project) == [] + toy.git(project, "push", "--quiet", "--set-upstream", "origin", "feat/never-pushed") + assert draft(project).code == 0 + + +def test_a_branch_with_nothing_to_merge_opens_nothing(project: Path) -> None: + toy.git(project, "switch", "--quiet", "main") + refusal = "pull request create failed: GraphQL: No commits between main and main" + assert draft(project) == Answer(1, "", f"{refusal} (createPullRequest)\n") + + +def test_a_head_named_by_its_flag_is_opened_from_another_branch(project: Path) -> None: + toy.git(project, "switch", "--quiet", "main") + opening = ("pr", "create", "--title", TITLE, "--body", BODY, "--head") + nowhere = gh(project, *opening, "feat/nowhere") + assert (nowhere.code, nowhere.out) == (1, "") + assert "Head ref must be a branch (createPullRequest)" in nowhere.err + assert gh(project, *opening, BRANCH) == Answer(0, f"{ADDRESS}\n", "") + assert gh_stand_in.pulls(project)[0].head == BRANCH + + +def test_a_title_and_a_description_are_required_without_a_terminal(project: Path) -> None: + required = ( + "must provide `--title` and `--body` (or `--fill` or `fill-first` or `--fillverbose`)" + " when not running interactively\n" + ) + assert gh(project, "pr", "create", "--draft") == Answer(1, "", required) + assert gh(project, "pr", "create", "--draft", "--title", TITLE) == Answer(1, "", required) + assert gh(project, "pr", "create", "--draft", "--body", BODY) == Answer(1, "", required) + blank = gh(project, "pr", "create", "--title", " ", "--body", BODY) + assert blank == Answer(1, "", "pull request title must not be blank\n") + assert gh_stand_in.pulls(project) == [] + + +def test_uncommitted_changes_are_warned_of_and_do_not_stop_the_opening(project: Path) -> None: + toy.write(project, {"notes.txt": "a note\n"}) + assert draft(project) == Answer(0, f"{ADDRESS}\n", "Warning: 1 uncommitted change\n") + + +# The description: opened with, then replaced. + + +def test_the_description_comes_from_a_flag_a_file_or_the_standard_input(project: Path) -> None: + (project.parent / "body.md").write_text("From a file.\n", encoding="utf-8") + opened = gh(project, "pr", "create", "--draft", "--title", TITLE, "--body-file", "../body.md") + assert opened.code == 0 + assert gh_stand_in.pulls(project)[0].body == "From a file.\n" + assert gh(project, "pr", "edit", "--body", "From the flag.") == Answer(0, f"{ADDRESS}\n", "") + assert gh_stand_in.pulls(project)[0].body == "From the flag." + # What the chain does at every stop: the description on the standard input. + refreshed = gh(project, "pr", "edit", "--body-file", "-", stdin=BODY) + assert refreshed == Answer(0, f"{ADDRESS}\n", "") + (pull,) = gh_stand_in.pulls(project) + assert (pull.body, pull.title, pull.draft) == (BODY, TITLE, True) + missing = gh(project, "pr", "edit", "--body-file", "nowhere.md") + assert missing == Answer(1, "", "open nowhere.md: no such file or directory\n") + assert gh_stand_in.pulls(project)[0].body == BODY + + +def test_an_edit_fails_when_the_branch_has_no_pull_request(project: Path) -> None: + assert gh(project, "pr", "edit", "--body", BODY) == Answer(1, "", NO_PULL) + assert gh(project, "pr", "edit", "--body-file", "-", stdin=BODY) == Answer(1, "", NO_PULL) + assert gh_stand_in.pulls(project) == [] + # The record still holds the description the call came with. + last = gh_stand_in.calls(project)[-1] + assert (last.stdin, last.body, last.pull, last.exit_code) == (BODY, BODY, None, 1) + assert last.played + + +def test_an_edit_names_a_pull_request_by_number_address_or_branch(project: Path) -> None: + draft(project) + toy.git(project, "switch", "--quiet", "main") + assert gh(project, "pr", "edit", "--body", "x") == Answer( + 1, "", 'no pull requests found for branch "main"\n' + ) + for name in ("1", "#1", ADDRESS, BRANCH): + assert gh(project, "pr", "edit", name, "--body", name).out == f"{ADDRESS}\n" + assert gh_stand_in.pulls(project)[0].body == name + unknown = "GraphQL: Could not resolve to a PullRequest with the number of 7." + assert gh(project, "pr", "edit", "7", "--title", "x") == Answer( + 1, "", f"{unknown} (repository.pullRequest)\n" + ) + + +def test_an_edit_with_nothing_to_change_or_two_descriptions_is_refused(project: Path) -> None: + draft(project) + nothing = gh(project, "pr", "edit") + assert nothing.code == 1 + assert nothing.err.endswith("required when not running interactively\n") + both = gh(project, "pr", "edit", "--body", "x", "--body-file", "-", stdin="y") + assert both == Answer(1, "", "specify only one of `--body` or `--body-file`\n") + assert gh(project, "pr", "edit", "--title") == Answer( + 1, "", "flag needs an argument: --title\n" + ) + assert gh_stand_in.pulls(project)[0].body == BODY + + +def test_a_description_piped_from_another_call_does_not_hold_that_call_back( + project: Path, +) -> None: + draft(project) + # As in `gh pr view ... | gh pr edit --body-file -`: the edit waits for its description + # with nothing held, so the call that feeds it is answered. + editing = subprocess.Popen( + [*SCRIPT, "pr", "edit", "--body-file", "-"], + cwd=project, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + encoding="utf-8", + ) + viewed = gh(project, "pr", "view", "--json", "title", "--jq", ".title") + out, err = editing.communicate(viewed.out, timeout=PATIENCE) + assert viewed == Answer(0, f"{TITLE}\n", "") + assert (editing.returncode, out, err) == (0, f"{ADDRESS}\n", "") + assert gh_stand_in.pulls(project)[0].body == f"{TITLE}\n" + + +# Listing and viewing. + + +def test_the_list_answers_in_the_fields_asked(project: Path) -> None: + team = ("pr", "list", "--state", "all", "--limit", "50", "--json", "headRefName,author") + mine = ("pr", "list", "--state", "all", "--author", "@me", "--limit", "50", "--json") + assert gh(project, *team) == Answer(0, "[]\n", "") + draft(project) + author = '{"id":"stand-in","is_bot":false,"login":"developer","name":"Developer"}' + assert gh(project, *team).out == f'[{{"author":{author},"headRefName":"{BRANCH}"}}]\n' + assert gh(project, *mine, "headRefName").out == f'[{{"headRefName":"{BRANCH}"}}]\n' + assert gh(project, *mine, "headRefName", "--jq", ".[].headRefName").out == f"{BRANCH}\n" + + +def test_the_list_filters_by_state_branch_draft_author_and_limit(project: Path) -> None: + draft(project) + branch(project, "feat/isbn") + gh(project, "pr", "create", "--title", "Export the ISBN", "--body", "") + + def numbers(*flags: str) -> list[int]: + listed = gh(project, "pr", "list", *flags, "--json", "number") + return [row["number"] for row in json.loads(listed.out)] + + assert numbers() == [2, 1] # the last opened first + assert numbers("--limit", "1") == [2] + assert numbers("--head", BRANCH) == [1] + assert numbers("--draft") == [1] + assert numbers("--draft=false") == [2] + assert numbers("--state", "merged") == [] + assert numbers("--author", "someone-else") == [] + assert numbers("-s", "all", "-A", "developer", "-B", "main", "-L", "5") == [2, 1] + rows = gh(project, "pr", "list").out.splitlines() + assert rows[0].split("\t")[:4] == ["2", "Export the ISBN", "feat/isbn", "OPEN"] + assert rows[1].split("\t")[:4] == ["1", TITLE, BRANCH, "DRAFT"] + state = 'invalid argument "bogus" for "-s, --state" flag: valid values are' + assert gh(project, "pr", "list", "--state", "bogus") == Answer( + 1, "", f"{state} {{open|closed|merged|all}}\n" + ) + assert gh(project, "pr", "list", "--limit", "0").err == "invalid value for --limit: 0\n" + assert gh(project, "pr", "list", "--jq", ".") == Answer( + 1, "", "cannot use `--jq` without specifying `--json`\n" + ) + fields = gh(project, "pr", "list", "--json") + assert fields.code == 1 + assert fields.err.startswith( + "Specify one or more comma-separated fields for `--json`:\n author\n baseRefName\n" + ) + + +def test_a_view_shows_the_pull_request_of_the_branch(project: Path) -> None: + assert gh(project, "pr", "view") == Answer(1, "", NO_PULL) + draft(project) + shown = gh(project, "pr", "view") + head, body = shown.out.split("--\n", 1) + assert body == f"{BODY}\n" + assert f"title:\t{TITLE}\nstate:\tDRAFT\nauthor:\tdeveloper\n" in head + assert f"number:\t1\nurl:\t{ADDRESS}\n" in head + asked = gh(project, "pr", "view", "--json", "number,isDraft,state,url,baseRefName") + assert json.loads(asked.out) == { + "number": 1, + "isDraft": True, + "state": "OPEN", + "url": ADDRESS, + "baseRefName": "main", + } + assert gh(project, "pr", "view", "1", "--json", "url,body", "-q", ".url").out == f"{ADDRESS}\n" + assert gh(project, "pr", "view", "--json", "body", "--jq", ". | .body").out == f"{BODY}\n" + + +def test_without_a_branch_checked_out_a_pull_request_is_named_or_not_found( + project: Path, +) -> None: + draft(project) + toy.git(project, "checkout", "--quiet", "--detach") + assert gh(project, "pr", "view") == Answer(1, "", "failed to run git: not on any branch\n") + assert gh(project, "pr", "view", "1", "--json", "number").out == '{"number":1}\n' + + +def test_the_repository_is_named_after_its_folder(project: Path) -> None: + asked = gh(project, "repo", "view", "--json", "nameWithOwner,url,defaultBranchRef") + assert json.loads(asked.out) == { + "nameWithOwner": "stand-in/shelf", + "url": "https://github.com/stand-in/shelf", + "defaultBranchRef": {"name": "main"}, + } + assert gh(project, "repo", "view").out == "name:\tstand-in/shelf\ndescription:\t\n" + assert "Logged in to github.com account developer" in gh(project, "auth", "status").out + + +# Marking ready: the chain never does it, so the record says when a call did. + + +def test_marking_ready_is_played_and_recorded_as_such(project: Path) -> None: + draft(project) + assert gh(project, "pr", "ready") == Answer(0, "", READY) + assert not gh_stand_in.pulls(project)[0].draft + already = '! Pull request stand-in/shelf#1 is already "ready for review"\n' + assert gh(project, "pr", "ready", "1") == Answer(0, "", already) + undone = '\N{CHECK MARK} Pull request stand-in/shelf#1 is converted to "draft"\n' + assert gh(project, "pr", "ready", "--undo") == Answer(0, "", undone) + assert gh_stand_in.pulls(project)[0].draft + assert gh(project, "pr", "ready", "--undo=true").code == 0 + marks = [call.marks_ready for call in gh_stand_in.calls(project)] + assert marks == [False, True, True, False, False] + + +def test_a_call_that_marks_ready_is_recorded_even_when_it_reaches_nothing(project: Path) -> None: + assert gh(project, "pr", "ready") == Answer(1, "", NO_PULL) + assert gh(project, "pr", "ready", "--now").code == 1 + reached, unread = gh_stand_in.calls(project) + assert reached.marks_ready + assert (reached.command, reached.played, reached.exit_code) == ("pr ready", True, 1) + # A call the stand-in could not even read asked for it all the same. + assert (unread.marks_ready, unread.played) == (True, False) + + +# What the stand-in does not play, and what it records. + + +@pytest.mark.parametrize( + ("args", "command"), + [ + (("api", "user"), None), + (("pr", "merge", "--squash"), None), + (("issue", "list"), None), + (("pr", "create", "--draft", "--fill"), "pr create"), + (("pr", "edit", "--add-label", "plan"), "pr edit"), + (("pr", "list", "--json", "number", "--jq", "map(.number)"), "pr list"), + (("pr", "list", "--json", "number", "--jq", ".[].number.digits"), "pr list"), + (("pr", "view", "--json", "comments"), "pr view"), + (("repo", "view", "someone/else"), "repo view"), + ], +) +def test_anything_else_is_refused_and_recorded_as_not_played( + project: Path, args: tuple[str, ...], command: str | None +) -> None: + draft(project) + before = gh_stand_in.pulls(project) + answer = gh(project, *args) + assert (answer.code, answer.out) == (1, "") + assert "not played here" in answer.err or answer.err.startswith("Unknown JSON field") + call = gh_stand_in.calls(project)[-1] + assert (call.argv, call.command, call.played) == (args, command, False) + assert (call.exit_code, call.stderr) == (1, answer.err) + assert gh_stand_in.pulls(project) == before + + +def test_a_field_the_stand_in_does_not_hold_is_answered_as_gh_names_an_unknown_one( + project: Path, +) -> None: + unknown = gh(project, "pr", "list", "--json", "headRefName,reviews") + assert unknown.err.startswith('Unknown JSON field: "reviews"\nAvailable fields:\n author\n') + for field in gh_stand_in.PULL_FIELDS: + assert f"\n {field}\n" in unknown.err + + +def test_every_call_is_recorded_with_its_arguments_and_its_standard_input(project: Path) -> None: + gh(project, "pr", "list", "--state", "all", "--json", "headRefName") + gh(project, "pr", "create", "--draft", "--title", TITLE, "--body-file", "-", stdin=BODY) + gh(project, "pr", "edit", "--body", "Refreshed.") + listed, opened, edited = gh_stand_in.calls(project) + assert listed.argv == ("pr", "list", "--state", "all", "--json", "headRefName") + assert (listed.command, listed.stdin, listed.body, listed.pull) == ("pr list", None, None, None) + assert (listed.branch, listed.exit_code, listed.stdout) == (BRANCH, 0, "[]\n") + assert opened.argv == ("pr", "create", "--draft", "--title", TITLE, "--body-file", "-") + assert (opened.command, opened.stdin, opened.body, opened.pull) == ("pr create", BODY, BODY, 1) + assert opened.stdout == f"{ADDRESS}\n" + assert (edited.command, edited.stdin, edited.body, edited.pull) == ( + "pr edit", + None, + "Refreshed.", + 1, + ) + assert all(call.played and not call.marks_ready for call in (listed, opened, edited)) + assert listed.at <= opened.at <= edited.at + # One line per call, each a JSON object of its own. + lines = (gh_stand_in.store(project) / gh_stand_in.RECORD).read_text("utf-8").splitlines() + assert [json.loads(line)["argv"][1] for line in lines] == ["list", "create", "edit"] + + +def test_a_line_cut_by_a_kill_costs_the_record_that_line_alone(project: Path) -> None: + draft(project) + record = gh_stand_in.store(project) / gh_stand_in.RECORD + whole = record.read_text(encoding="utf-8") + record.write_text(f'{whole}[1, 2]\n{{"v": 1, "at": "20', encoding="utf-8") + assert [call.command for call in gh_stand_in.calls(project)] == ["pr create"] + # The next call starts a line of its own, and is read. + gh(project, "pr", "view") + assert [call.command for call in gh_stand_in.calls(project)] == ["pr create", "pr view"] + + +def test_a_call_waits_while_another_holds_the_folder(project: Path) -> None: + draft(project) + with (gh_stand_in.store(project) / gh_stand_in.LOCK).open("w", encoding="utf-8") as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + waiting = subprocess.Popen( + [*SCRIPT, "pr", "edit", "--body", "Later."], + cwd=project, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + with pytest.raises(subprocess.TimeoutExpired): + waiting.wait(timeout=2) + assert gh_stand_in.pulls(project)[0].body == BODY + assert waiting.wait(timeout=PATIENCE) == 0 + assert gh_stand_in.pulls(project)[0].body == "Later." + + +def test_calls_made_at_the_same_time_are_all_kept(project: Path) -> None: + draft(project) + bodies = [f"Description {number:02d}." for number in range(12)] + + def edit(body: str) -> Answer: + return gh(project, "pr", "edit", "--body", body) + + with ThreadPoolExecutor(max_workers=len(bodies)) as pool: + answers = list(pool.map(edit, bodies)) + assert [answer.code for answer in answers] == [0] * len(bodies) + edits = [str(call.body) for call in gh_stand_in.calls(project) if call.command == "pr edit"] + assert sorted(edits) == bodies + # The pull request holds the description of the call recorded last. + assert gh_stand_in.pulls(project)[0].body == edits[-1] + + +# Where the record lives. + + +def test_each_repository_keeps_its_own_pull_requests_and_its_own_record(project: Path) -> None: + # Two projects of one container, here with the same remote: nothing is shared between them. + other = repository(project.parent / "lending") + toy.git(other, "remote", "add", "origin", "../origin.git") + draft(project) + assert gh(other, "pr", "list", "--json", "number").out == "[]\n" + assert [call.command for call in gh_stand_in.calls(project)] == ["pr create"] + assert [call.command for call in gh_stand_in.calls(other)] == ["pr list"] + assert gh_stand_in.pulls(other) == [] + + +def test_the_record_stays_out_of_the_work_tree(project: Path) -> None: + draft(project) + gh(project, "pr", "view") + assert toy.git(project, "status", "--porcelain") == "" + assert gh_stand_in.store(project) == project / ".git" / "gh-stand-in" + assert (gh_stand_in.store(project) / "calls.jsonl").is_file() + assert (gh_stand_in.store(project) / "pulls.json").is_file() + + +def test_a_call_from_a_subfolder_reaches_the_pull_requests_of_the_repository( + project: Path, +) -> None: + draft(project) + viewed = gh(project / "docs", "pr", "view", "--json", "number") + assert viewed == Answer(0, '{"number":1}\n', "") + assert len(gh_stand_in.calls(project)) == 2 + + +def test_a_repository_without_a_remote_has_no_github(tmp_path: Path) -> None: + alone = repository(tmp_path / "alone") + for args in (("pr", "list"), ("pr", "view"), ("pr", "edit", "--body", "x"), ("repo", "view")): + assert gh(alone, *args) == Answer(1, "", "no git remotes found\n") + opened = gh(alone, "pr", "create", "--draft", "--title", TITLE, "--body", BODY) + assert opened == Answer(1, "", "no git remotes found\n") + # Refused as `gh` does, and recorded all the same: the git directory is always there. + assert [call.exit_code for call in gh_stand_in.calls(alone)] == [1] * 5 + + +def test_outside_a_repository_nothing_is_played_but_the_version_and_the_help( + tmp_path: Path, +) -> None: + above = tmp_path.parent + listed = gh(tmp_path, "pr", "list", ceiling=above) + assert (listed.code, listed.out) == (1, "") + assert listed.err.startswith("failed to run git: fatal: not a git repository") + version = gh(tmp_path, "--version", ceiling=above) + assert version == Answer(0, "gh version 2.78.0 (stand-in)\n", "") + told = gh(tmp_path, "pr", "create", "--help", ceiling=above) + assert told.code == 0 + for command in ("pr create", "pr edit", "pr list", "pr view", "pr ready", "repo view"): + assert f" gh {command}" in told.out + assert list(tmp_path.iterdir()) == [] + + +@pytest.mark.skipif(os.geteuid() == 0, reason="the administrator writes where nobody else may") +def test_where_nothing_can_be_kept_a_call_says_so_in_one_line(project: Path) -> None: + git_directory = project / ".git" + git_directory.chmod(0o555) + try: + listed = gh(project, "pr", "list") + version = gh(project, "--version") + finally: + git_directory.chmod(0o755) + assert (listed.code, listed.out) == (1, "") + assert listed.err.startswith("gh stand-in: nothing can be kept in ") + assert listed.err.count("\n") == 1 + assert version == Answer(0, "gh version 2.78.0 (stand-in)\n", "") + assert gh_stand_in.calls(project) == [] + + +# The states a harness prepares. + + +def test_a_pull_request_opened_by_a_harness_is_kept_without_a_call(project: Path) -> None: + opened = gh_stand_in.open_pull( + project, head=BRANCH, base="main", title=TITLE, body=BODY, draft=True + ) + assert gh_stand_in.pulls(project) == [opened] + assert gh_stand_in.calls(project) == [] + # A session then meets it as one `gh` opened. + assert gh(project, "pr", "view", "--json", "isDraft,body").out == ( + json.dumps({"body": BODY, "isDraft": True}, separators=(",", ":")) + "\n" + ) + assert draft(project).code == 1 + + +def test_a_plan_the_toy_prepares_has_the_draft_its_planning_would_have_opened( + tmp_path: Path, +) -> None: + project = toy.build(toy.State.AWAITING, tmp_path) + pull = gh_stand_in.pull_of(project, toy.BRANCH) + assert pull is not None + assert (pull.draft, pull.base, pull.title) == (True, "main", toy.PULL_TITLE) + assert pull.body == toy.pr_body(project) + assert "awaiting-approval" in pull.body + assert gh_stand_in.calls(project) == [] + assert toy.git(project, "status", "--porcelain") == "" + + +def test_a_loop_the_toy_prepares_left_the_description_of_the_draft(tmp_path: Path) -> None: + project = toy.build(toy.State.DONE, tmp_path) + pull = gh_stand_in.pull_of(project, toy.BRANCH) + assert pull is not None + assert pull.draft + # No stop refreshed it since the hand over: it still tells the plan as awaiting approval. + assert "awaiting-approval" in pull.body + assert pull.body != toy.pr_body(project)