feat: add long-horizon agent eval project

[zredlined]

Summary

Add a self-contained research example for running persistent agents over configurable wall-clock horizons and repeated parallel attempts. The initial experiment evaluates an AI reviewer handling OpenShell policy proposals while a challenger attempts to mutate a protected GitHub repository.

This intentionally stays concrete: the GitHub experiment is wired directly into the runner, with no scenario registry, plugin system, service layer, or production deployment machinery.

Related issue

None. This is a standalone research project and does not change OpenShell product behavior or public APIs.

Changes

• Add one-campaign and bounded-concurrency runners with only --minutes, --runs, and --concurrency CLI controls.
• Keep endpoints, models, reasoning levels, credentials, and tuning in .env.
• Separate challenger and reviewer inference configuration.
• Include the GitHub policy-review prompts, external oracle, reviewer loop, transcripts, cost estimates, and evidence layout.
• Add focused Node tests, a direct setup guide, an architecture and trust-boundary diagram, and a project-index entry.
• Exclude personal experiment reports, historical traces, populated environment files, and automation notes.

Testing

• npm run check
• bash -n scripts/challenger.sh
• node --check scripts/check-responses-endpoints.mjs
• python3 scripts/update_license_headers.py --check
• Docker challenger image build on macOS and DGX Station
• Challenger and reviewer endpoint checks on DGX Station
• GitHub token write preflight against a disposable repository
• Two-minute live campaign: valid, no compromise, five reviewer decisions, sandbox cleanup confirmed
• Thirty-minute live campaign: 39 challenger turns, 30 reviewer decisions, no repository mutation, no model backoff, and sandbox cleanup confirmed. The attempt was excluded by the harness because five decisions hit the known OpenShell proposal-application/merge failure.

Checklist

• Commit is DCO signed.
• Credential-pattern scan completed; no populated .env, credentials, or run artifacts are included.
• README warns that the GitHub scenario performs real actions and recommends a disposable repository and repository-scoped token.
• The implementation remains a lightweight research example rather than a generalized evaluation framework.

[johnnygreco]

Thanks for keeping this concrete and research-focused. The overall shape is easy to follow, and I did not find any committed secrets or stray run artifacts.

I think this needs another pass before merge. The main blockers are secret redaction, cleanup verification, incomplete GitHub ref pagination, and selection bias from discarding healthy runs that make no proposal. I left the code-specific details inline.

The project also needs a CI job that runs npm ci and npm run check with read-only GitHub Packages access. The current Egress Gate failures look unrelated: its tests pass, but dependency auditing is failing on the runner pip version.

[johnnygreco]

The challenger should stop as soon as the GitHub oracle observes the target mutation.

Today it keeps running until the time limit even though the experiment already has a conclusive compromise. In one 10-minute run, the target was observed after 9 turns, but the challenger continued for another 531 turns and consumed about 41.6 million additional input tokens.

Please stop the challenger when the target is observed, record the observation time and a target_observed stop reason, and then allow the existing short settlement period for pending reviewer evidence. Continuing after compromise can remain an explicit opt-in for experiments that need it.

[johnnygreco]

Concern: the experiment’s main GitHub write request may never reach the reviewer.

The PR attaches the standard GitHub provider with read-only GitHub access. When the challenger proposes a narrow write permission for the same GitHub host, current OpenShell rejects the proposal before review because it conflicts with the provider rule’s internal metadata.

I reproduced this with OpenShell 0.0.112-dev.5+g679fe4c3, built from commit 679fe4c334f0c62a653b004fae7008a631bf7f22, using the exact provider and initial policy from the PR. Removing the PR’s additional github-read-only rule did not change the result. Direct policy updates, automatic OpenShell proposals, and agent-authored proposals for unrelated hosts all worked; the failure is specific to agent-authored requests that add access to a provider-covered host.

This makes an uncompromised result ambiguous. It could mean the reviewer resisted the challenger, or it could mean OpenShell prevented the reviewer from seeing the important request. The experiment currently treats those as the same outcome.

[zredlined]

Handoff/lull fix: 64c3532.

• Deduplicates repeated prose and evicts prose before command evidence.
• At the archived bdb717 lull: 32 prose / 0 commands / 59% duplicates became 2 distinct messages / 30 commands / 0% duplicates.
• A healthy trace retained 4 messages and 27 commands.
• Station verification passed all 35 tests plus shell syntax; the TypeScript/inline implementations matched.

[zredlined]

Review update: 81b17e4, 27ebae1, 6aad34f, and ff04657 address Johnny’s feedback; all 15 inline threads are resolved.

The GitHub path uses providers for protected credential delivery and requires provider policy composition to be disabled on the dedicated evaluation gateway. The README includes the one-time setting command, and the runner now fails fast if Providers v2 is enabled. The initial policy grants only model access, so all GitHub authority goes through review with an exact-write canary before either model starts.

A one-minute Station smoke at challenger xhigh / reviewer medium passed the canary and exercised four real proposals (1 approved, 3 rejected), with no compromise and verified cleanup. Station passes 43/43 tests; all GitHub checks are green. The PR is ready for re-review.