Repository navigation
Prompts that no longer say how an agent writes a command or changes a file - #87
Merged
Merged
Conversation
… file The prompts of the three commands, of the executor and of the reviewer told an agent how to write its commands so that the developer's permission rules could read them: no variable or function standing for a command, no command run by another, no expansion, no redirection or here-document into a file, no `sed -i`, and files changed with Write and Edit only. That rule is a general one on how an agent works, and is not the chain's to give. The chain ships no permission rule and leaves what stops for an approval to the developer's mode (ADR 0032): how an agent writes a command, and the tool it changes a file with, are its own judgment. With the rule in its prompt, an executor that changed a file through a here-document was counted as a breach, twice in three campaigns of the evaluations (#71), where it had only chosen its tool. What stays is the chain's own: every command runs from the root of the repository, with paths from there, with no `cd`, which would move the shell the dispatcher and its agents share, and no `git -C`. - `skills/surface-plan`, `skills/surface-execute`, `skills/surface-status`: the sentence "Plain commands only" goes. - `agents/surface-executor.md`: the line "Hide a command from the developer's permission rules" goes from what the executor never does. - `agents/surface-reviewer.md`: the same list goes. The reviewer still judges by reading and runs no command of its own to check behavior. - `ARCHITECTURE.md` and ADR 0032 say what the prompts hold now, and why. - The contract tests no longer hold the rule, and one holds that no prompt brings it back.
This was referenced Oct 5, 2026
PierreMardon
added a commit
that referenced
this pull request
Oct 5, 2026
Kept for the next campaign to be compared with: `evals/reports/2026-10-05-d139a1ca0dbe/`. It measures the chain on `main` after the two fixes of the day (#87, #88), with the harness of #89, #90 and #91, on the four cases that no campaign had played again, or once: `04-suspension`, `05-fine-cap` and `06-borrow-limit`, one run played whole and one stopped at the hand over each, and `02-reservations`, one run stopped at the hand over. 108 sessions, 14.23 USD at list price. `report.md` and `summary.json` are as `report --against <summary> --keep` wrote them, and nothing else changes. The summary it is compared with is the one of the first campaign, computed again from its raw data by the harness as it stands, with the count of its six blueprints: no measure the kept summary of that campaign holds differs in it. The first run of `06-borrow-limit` was played in another folder, before the stand-in `gh` was in the image, to check the option that stops a run at the hand over: its sessions and its cost were added to the ledger of the campaign. What it shows: - The three runs played whole reached `conformant` and passed all of their hidden acceptance tests. - The pull request steps are played for the first time: a draft at every hand over, its description refreshed at conformity, never marked ready. In 3 runs of 7 the draft is opened with a line of attribution under the description the state script prints, which the next refresh removes. - The count of the judge follows the fix of the blueprint (#83) on these cases: the facts said twice in prose fall from 19 to 9, from 13 to 4 and from 8 to 1.5 on `02`, `04` and `05`, and the details of implementation from 5 to 11 a page to 1 or 2, while `blueprint.no-padding` is 2 on every run of both campaigns. - On `06-borrow-limit` the interview asks 2 and 3 questions where it asked 1, and its developer hands three of them back: `interview.right-number` falls from 5 to 3.5. - Planning costs 0.94 to 2.20 USD a run here, and a run played whole 1.53 to 1.88. The conclusions, issue by issue, are in the issues #77 lists.
This was referenced Oct 5, 2026
Closed
PierreMardon
added a commit
that referenced
this pull request
Oct 7, 2026
…said in the guide (#97) Every commit a session of the chain makes ends with the `Co-Authored-By` line that `Claude Code` adds by default: all 27 of them in the fourth campaign of the evaluations, on a host whose conventions ask for none (#72). A host whose commit hook refuses an attribution, as this repository's does, would see each agent meet that refusal at its commit. The developer decided on 2026-10-07 that the chain leaves it to the settings of the host: the prompts say nothing of it, as they say nothing else of how a commit is written (#87, #88), and the guide says how a host turns it off. - `docs/guide.md`: a section, "Attribution", after "Permissions". It names the setting, `attribution` with `commit` and `pr` set to an empty string, in the `.claude/settings.json` of the project, says to write it before the first plan when a commit hook refuses an attribution, and why the chain does not do it for the host: its installer never writes those settings. - `ARCHITECTURE.md`: the same in one sentence, with what else is left to the developer. - `tests/prompts/chain_contract.py`: the comment that said the choice was still to make. The setting is the one the settings reference documents: an empty string hides the line of the commits, or the one added to a pull request description. The form given is read by every version; `"attribution": false` hides both too, on recent versions only. No prompt changes: the test that holds that none asks for an attribution or forbids one stays as it is. No session was played.
PierreMardon
added a commit
that referenced
this pull request
Oct 7, 2026
… nor git -C (#102) The prompts of the three commands, of the executor and of the reviewer held one rule in two parts: every command runs from the root of the repository, with paths from there, and never `cd` nor `git -C`, "since [...] a `cd` followed by git stops for an approval" and the permission rules do not read `git -C` as the git command it runs. #87 took the rest of how an agent writes a command out of the prompts, and left this ban. With a ban, a session has a breach to report. The last sentence of the last message of a run of `05-fine-cap`, in the fourth campaign of the evaluations: "The slice 2 executor ran one `cd` into the repository root, which the instructions forbid. It made no edits through it." The developer can do nothing with it (#67). The developer decided on 2026-10-07: the prompts keep what the rule is for, and lose the ban and its reason of permissions. - `skills/surface-plan/SKILL.md`, `skills/surface-execute/SKILL.md`: "Every command runs from the root of the repository, with paths from there, since the shell is shared with the agents." - `skills/surface-status/SKILL.md`, which launches no agent: "since one shell serves the whole session." - `agents/surface-executor.md`: the line leaves what the executor never does, and the rule stands where the state script is named, "since you share the shell with the dispatcher". - `agents/surface-reviewer.md`: the same, "with the dispatching command". - `ARCHITECTURE.md` and ADR 0032: the invariant and the sentence of the decision say what the prompts hold now, and why they ban nothing: a ban gave a session a breach to report to a developer who could do nothing with it. Left unsaid, someone would put it back. - `tests/prompts/test_agent_mandates.py`: every prompt that runs commands states the rule with its reason, the shared shell, and none names `cd` or `git -C`, bans a change of directory in other words, nor gives a reason of permissions. The commands the prompts spell themselves still change no directory. What it may cost: an agent that changes directory moves the shell for the commands after it, the dispatcher's included, which call the state script by its path from the root. The rule and its reason say so, and nothing checks it. A developer on the default mode may be asked to approve a command the ban steered an agent away from: their mode's call, as for every other command (ADR 0032). Not measured: no session was played. What to read in the next runs played whole is whether a message still reports a `cd`, and whether a command fails for running from elsewhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
The prompts of the three commands, of the executor and of the reviewer told an agent how to write its commands so that the developer's permission rules could read them: no variable or function standing for a command, no command run by another, no expansion, no redirection or here-document into a file, no
sed -i, and files changed with Write and Edit only. That rule goes from the five prompts.Why
The rule is a general one on how an agent works, and is not the chain's to give. The chain ships no permission rule and leaves what stops for an approval to the developer's mode (ADR 0032): how an agent writes a command, and the tool it changes a file with, are its own judgment.
With the rule in its prompt, an executor that changed a file through a here-document was counted as a breach, twice in three campaigns of the evaluations (#71), where it had only chosen its tool. The developer decided in #71 not to restrain the executor's tools.
What stays
What is the chain's own: every command runs from the root of the repository, with paths from there, with no
cd, which would move the shell the dispatcher and its agents share, and nogit -C. The reviewer still judges by reading and runs no command of its own to check behavior.Files
skills/surface-plan/SKILL.md,skills/surface-execute/SKILL.md,skills/surface-status/SKILL.md: the sentence "Plain commands only" goes.agents/surface-executor.md: the line "Hide a command from the developer's permission rules" goes from what the executor never does.agents/surface-reviewer.md: the same list goes.ARCHITECTURE.md: the invariant says what the prompts hold now, and why.tests/prompts/test_agent_mandates.py: the tests no longer hold the rule, and one holds that no prompt brings it back.What it may cost
A developer on the default mode may be asked to approve a command that the old sentence steered an agent away from. That is their permission mode's call, as ADR 0032 says of every other command.
Not measured
No session was played for this change: the gates pass, and no evaluation was run.