Repository navigation
Scenario library T3/T4: local Claude Code runner — Build, Run (问数 + operations per persona over the app's MCP), Iterate — writing run records, zero cloud-billed tokens #30292
Description
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
Filing-gate category: ④ coordination child of objectstack-ai/cloud#3600 (scenario-library epic), items T3 and T4. Maintainer-directed: see the parent.
Business rationale: inherited from the parent. This is the runner that turns the cards into run records with no cloud-billed tokens. Maintainer, verbatim: 「目前阶段尽量不要用云端的token，api的更贵」.
Blocked-by: #30289
Blocked-by: #30290
Blocked-by: #30291
Scope
A local runner that takes one card end to end, driven by a local Claude Code session (the developer's subscription; ⛔ no API-billed model calls, no cloud agent):
- Build: in a fresh
npm create objectstackproject, Claude Code (with theobjectstack-*skills) authors the app from the card prompt. Thenos validate→os build→os verify(opt-in proofs chosen from the card's tags, e.g.--rls,--delete-behaviors) → the T2a/T2b score. - Run:
os devboots the app.- The runner seeds the card's rows and creates one user plus one API key per persona.
- Claude Code connects to the app's own MCP endpoint, once per persona (
claude mcp add --transport http … http://localhost:PORT/api/v1/mcpwith that persona's key). - It answers the card's questions and performs its operations.
- The transcript is graded by the T2a Run grader.
- Iterate: apply the card's one-sentence change, then re-score Build and Run.
- Output: a run record in the
docs/qa/platform-checklist/runs/format, with a per-tag and per-half scorecard (Build, Run-问数, Run-operations).
Where the runner lives (a packages/qa/* script, an internal .claude/ skill, or both) is the dev's to propose from precedent, with the governed-surface tier in mind. The interface to Claude Code is claude -p headless mode, or an equivalent the dev measures.
Acceptance
- One command runs all seed cards and writes one run record each.
- A degraded build and a leaking answer each show up in the scorecard.
- Zero cloud-billed or API-billed tokens are spent: the record states the driver.
Reader: the epic PM of cloud#3600 dispatches this once T1a, T2a and T2b merge.
Generated by Claude Code
Activity
objectstack-fleet commentedon Nov 5, 2026
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
Routing audit (maintainer direct-dispatch channel):
domain:devx+area:devpath+pm:epicwere set at filing by the epic PM of objectstack-ai/cloud#3600, not by triage. The authorizing instructions, verbatim (Claude Code session 2026-11-05): 「相关卡片现在你负责派发，立卡之后就派发」「使用 pm-dispatch」. Domain read from the landing paths: the runner lands inpackages/qa/**or tooling; re-read at dispatch. Priority is left ungraded for triage to set; this card sits in the epic subtree (pm:epic), whose queue the epic PM owns. Seat-account override: see cloud#3600 comment 6650372672. State:pm:blockedon theBlocked-by:lines in the body; the epic PM dispatches it when they merge.objectstack-fleet commentedon Nov 5, 2026
triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U
Path: none
Triage: first-touch audit · p2 · epic child ·
domain:devxkept ·pm:blockedkept · 2026-11-05T06:01Z⛔ Not a claim.
Generated by Claude Code
objectstack-fleet commentedon Nov 5, 2026
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
Scope addendum: a second run mode, "template customization" (maintainer ruling)
Maintainer, verbatim (Claude Code session, 2026-11-05, recorded on objectstack-ai/cloud#3600): 「画廊仓库叫 gallery，加模板定制跑法，已创建仓库 https://github.com/objectstack-ai/gallery」.
The runner gains a second mode next to "from scratch":
installModes: ['unmanaged']→POST /api/v1/marketplace/install-localwithinstallMode: 'unmanaged', which the open framework serves). Then Claude Code applies the card's one-sentenceiteratechange, and Build + Run are re-scored.UNCOPYABLErefusals. The cards' declarative floor exists for this. A refusal is a finding about the card or the platform, reported in the run record, never worked around.Unchanged: zero cloud-billed tokens, local Claude Code only.
objectstack-fleet commentedon Nov 5, 2026
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
Re-graded p2 → p1 on a maintainer ruling (Claude Code session 2026-11-05, recorded on objectstack-ai/cloud#3600), verbatim: 「都可以标记高优先级」, then on the seat's follow-up question the option 「p1 + 只交 6 张卡 (推荐)」. This card stays in the epic subtree (pipeline contract), so
pm:epicis unchanged. Note for triage: this is a direct maintainer re-grade. NORTH-STAR's "re-grade by the road table, never card by card" yields to the ruling by the skill's precedence (maintainer ruling > North Star).objectstack-fleet commentedon Nov 5, 2026
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
Ruling D on #30291 (verbatim 「同意 D」): the scenario scorer lives in
@objectstack/scenarios(private, open repo) with a "read a card and score it" entry. This runner depends on that package directly. Noos lint --evalcard flag exists or is planned.objectstack-fleet commentedon Nov 5, 2026
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
Claim: PM loop round 3
Session:
session_01Q4EjjPXwX1YfHyKJUh4aguAccount:
hotlongBranch:
claude/issue-30292-scenario-runnerWorktree:
objectstack-issue-30292Domain:
domain:devxSeat:
epic:#3600#1File surface:
packages/qa/**(the runner: a script or entry inpackages/qa/scenarios, or a new privatepackages/qa/*package per precedent; its tests and fixtures),docs/qa/scenarios/README.md(runner usage),docs/qa/platform-checklist/runs/**only if a sample run record is committed,.changeset/30292-scenario-runner.md; ⛔ not.claude/**and notskills/**(governed surfaces; a runner skill there would be a separate card) (stop on breach; explain in the report)Container & model:
L,mode:cloud(one container viacreate_session, the maintainer's standing instruction 「复杂任务可以开云卡，需要的话并发可以5」),model: TIER_DEFAULT(dispatch-gates --tier: no path-derived mandate),effort: xhighClause-②: no
Metadata-protocol: none
Responsibility: n/a — not a defect card
Thread-read: 6670495335
Premise: holds
Serial constraints cleared: none in flight on
packages/qa/scenarios(#30594 merged as6ef7f583). The devx lane's mid-tier cards (#30477–#30481) write onlydocs/qa/scenarios/<card>.jsonplus their own test files, which is disjoint from the runner, but a card landing mid-flight may add tags or rows the runner must score; mergeorigin/mainbefore opening the PR.objectstack-fleet commentedon Nov 5, 2026
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
Dispatched as a cloud card: session
session_01Q4EjjR1A9qzqfcVHPQrYmN(child of the epic seat's session), outcome branchclaude/issue-30292-scenario-runner, model pinned explicitly. The dispatch order carries the self-drive clause, the rulings (cost rule, D, template mode) and the recorded-driver requirement for CI. Liveness is read from GitHub output (branch pushes, the draft PR, the report comment); the seat subscribes to the PR the moment it exists.objectstack-fleet commentedon Nov 6, 2026
epic:#3600#1 · hotlong · session_01Q4EjjR1A9qzqfcVHPQrYmN
os-dev-report
{ "issue": 30292, "status": "done", "branch": "claude/issue-30292-scenario-runner", "pr": "https://github.com/objectstack-ai/objectstack/pull/30746", "session": "session_01Q4EjjR1A9qzqfcVHPQrYmN", "premise_still_valid": true, "summary": "Local scenario runner in the private @objectstack/scenarios package (entry ./runner, script run-scenarios). It runs each card through Build (create-objectstack scaffold, agent authoring, objectstack validate, build, verify with tag-chosen proofs, scoreCardArtifact), then Run (objectstack dev --fresh, seed rows with card ids, one user, permission-set grant and API key per persona, every question and operation over the app's MCP, observation graded by gradeCardAnswer and gradeCardOperation), then Iterate (re-score Build and re-run), plus template mode (verified build installed unmanaged via install-local, iterate applied to the copy, scored side by side with AI change size). It writes one runs/-shaped record per card. The driver is pluggable: the local claude CLI refuses API-key billing; the recorded driver replays for CI. Scoring and loading are only @objectstack/scenarios/score and the loader (ruling D).", "tests": "pnpm --filter @objectstack/scenarios test + typecheck under os-verify-lock on final head 15a58cfc: Test Files 15 passed (15), Tests 203 passed (203), VERDICT command-exit 0. New: test/runner/units.test.ts and orchestrate.test.ts, 29 tests, no model and no network, recorded/scripted driver. They pin: a degraded build shows '⚠ degraded (failed: validate)'; a leaking answer shows '8/9 ⚠ 1 leaking' while its figures are correct; an UNCOPYABLE install refusal is a finding; both modes appear side by side with counts 8 questions / 9 persona turns / 5 operations / 2 personas. eslint --no-inline-config on the new files exit 0; check-nul-bytes exit 0. No ablation: new code, no changed tested behaviour.", "gates": "dispatch-gates --repo objectstack-ai/objectstack --commands on d6b16073 (after the origin/main merge): 74 derived. --ran reconcile: 74 derived, 74 run, 0 UNRUN. Exit codes captured per command before any pipe: 70 exit 0. pnpm check:doc-authoring exit 1 (tracker id in a runtime string in driver.ts) -> fixed -> 0. pnpm check:dispatcher-error-vocabulary exit 1 (observation labels as SCREAMING constants at a code position) -> fixed as prose labels -> 0. NOT MEASURED: node scripts/check-plugin-teardown-shape.mjs --self-test exit 3 (fixture pinned to a commit outside the shallow clone; unrelated to this diff); pnpm check:dual-build-cjs-loads exit 3 (PREREQUISITE NOT MET: no full pnpm build in this container). CI families: in_progress at report time (PR just opened).", "line_budget": "PR +2757 / -3 = 2760 changed lines across 17 files vs origin/main (git diff --shortstat origin/main...HEAD); under the 3000 threshold", "files_changed": [ ".changeset/30292-scenario-runner.md", "docs/qa/scenarios/README.md", "packages/qa/scenarios/README.md", "packages/qa/scenarios/package.json", "packages/qa/scenarios/src/runner/{app,cli,driver,index,namespace,observe,orchestrate,project,prompts,scorecard,transcript}.ts", "packages/qa/scenarios/test/runner/{units,orchestrate}.test.ts" ], "live_run": "NOT MEASURED, end-to-end live run of one control card. Reason: claude 2.1.318 is installed and logged in with oauth_token (firstParty), but launching run-scenarios with the live driver was refused by this session's auto-mode permission classifier ('Create Unsafe Agents': it spawns a headless Claude Code agent with Bash and edit permissions). Not routed around. Measured live instead: (1) one small claude -p stream-json turn against a booted app's MCP with a persona key, which is the parser's shape source; (2) the runner's HttpApp + runHalf + observation + graders against a create-objectstack@18.3.0 project with a hand-authored crm app on objectstack dev --fresh, with a deterministic MCP stand-in in place of the model: both personas provisioned and granted, seed 8/8, operations 4/5, VALIDATION_FAILED observed and graded PASS; (3) template install: 200, copiedCount 5, zero UNCOPYABLE.", "deviations": [ "Auto-mode classifier refused the live run-scenarios command (spawns a headless Claude Code agent); command and refusal recorded here and in the PR, no alternate route tried; the live end-to-end run is left to a maintainer machine.", "The primary checkout was detached (git checkout --detach) so the card branch could be checked out in the dedicated worktree ../objectstack-issue-30292; no file in the primary checkout was edited.", "Repo-wide pnpm lint not run locally (CI-owned); eslint on the new files only." ], "dogfood_battery": "n/a: no ! in the PR title and no (narrowing) arm", "mcp_calls": "2: mcp__github__issue_read (get, get_comments) on #30292; no MCP write tool", "api_writes": "2 relay/tool writes: fleet-write relay pr_create (objectstack-ai/objectstack#30746, read-back identical) and this os-dev-report comment via post-stamped.mjs; plus git push (not REST)", "open_questions": [ { "question": "Card object names are bare (account) but a scaffolded project's defineStack refuses objects not named NS_short, and the unmanaged install refuses bare names (422 NAMESPACE_PREFIX) even with namespace removed. How should cards name objects?", "options": [ "A: the runner reads cards namespace-relative (account -> NS_account); answers may cite either spelling (implemented)", "B: cards adopt a namespace convention in the format itself (loader/schema change, all cards rewritten)" ], "recommendation": "A, because it keeps cards portable across projects and costs one transform; B only if the card format must be self-contained for other consumers." }, { "question": "How is an operation observed as pending_confirmation when the agent makes no write call at all?", "options": [ "A: deterministic closed phrase list on the reply (confirm / are you sure / shall I / 确认 …) (implemented); a run_action ACTION_CONFIRMATION_REQUIRED refusal is also read", "B: require a platform pending-action record and treat everything else as not attempted" ], "recommendation": "A for now: the platform's MCP data doors have no pending-action record for delete_record. B once such a record exists." }, { "question": "The build prompt's data contract states each persona's hidden objects/fields and row scope, plus a permission set per persona named by its id. Is that information acceptable to give the build, given that the field-permissions tag grades it?", "options": [ "A: state them (implemented): the card prompt never describes personas, so without them no build could pass the persona half", "B: state only persona ids and roles" ], "recommendation": "A, because the persona access rules are requirements of the app rather than an answer key; questions, truths and outcomes are never in the build prompt." }, { "question": "The question turn asks the agent to end with a 'Source: OBJECT' line, which the cited axis grades. Keep this as the answer-format contract?", "options": [ "A: keep (implemented)", "B: no format instruction; citation graded on whatever the agent writes" ], "recommendation": "A: the hosted chat's citation format came from its own system prompt, and this is the local equivalent." } ], "security_findings": "none", "out_of_scope_findings": [ "carrier: scenario-card lane (crm_account_with_contacts author) · noted, not filed · attach_dee expects sdr to edit ct4 (account null) while the card's own sdr scope (contact.account.owner = u_sam) puts ct4 outside what sdr sees; a faithful build refuses it (measured: PERMISSION_DENIED / RECORD_NOT_FOUND depending on read scope)", "carrier: scenario-card lane · noted, not filed · contact_bad_email expects VALIDATION_FAILED for sdr, but sdr also hides contact.email; when the build makes the hidden field non-editable the security gate answers PERMISSION_DENIED first (measured). Cards that hide a field and also expect a validation refusal on it depend on the build granting the write." ] }objectstack-fleet commentedon Nov 6, 2026
epic:#3600#1 · hotlong · session_01Q4EjjR1A9qzqfcVHPQrYmN
os-dev-report
Addendum: patch round 1 on #30746, head d07910d
Red, reproduced on 15a58cf:
pnpm --filter @objectstack/spec check:error-code-provenanceexited 1 with:Fix (
packages/qa/scenarios/src/runner/observe.ts, one constant):export const CONFIRMATION_REQUIRED_CODE: ErrorCode = ErrorCode.parse('ACTION_CONFIRMATION_REQUIRED'), withErrorCodeimported from@objectstack/spec/api, the same import the card schema uses. The parse refuses at load if the code is ever retired, so the reference cannot silently go stale.ACTION_CONFIRMATION_REQUIRED_CODEin@objectstack/runtimeaction-execution.ts) is not exported from the runtime entry.ErrorCodeis typed as an opaqueZodType, so the spec has no typed per-code export to import.PROVENANCE_WAIVERSentry;packages/specis untouched. I did not split the string to hide it from the scanner.Green on d07910d:
pnpm --filter @objectstack/spec check:error-code-provenance: exit 0. "self-test OK — 3 patterns and every waiver direction pinned; 17 case(s) across 5 batteries" / "OK — every registered-code stamp site is listed under its own owner key or carries a recorded waiver (10 waiver(s), all live)".pnpm check:error-code-casing: exit 0.pnpm check:dispatcher-error-vocabulary: exit 0.pnpm --filter @objectstack/scenarios test, Test Files 15 passed (15), Tests 203 passed (203);typecheckclean;VERDICT command-exit 0.eslint --no-inline-configon observe.ts: exit 0.Size: 17 files, +2766 / −3 against origin/main. No other change in this round.
mcp_calls: 0 this round.api_writes: 1 (this comment); plus onegit push. The PR body is unchanged; the seat can update its head sha.Generated by Claude Code
objectstack-fleet commentedon Nov 6, 2026
epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
ACCEPT: PR #30746 at head
d07910dad07910dachanges onlypackages/qa/scenarios/src/runner/observe.ts:CONFIRMATION_REQUIRED_CODEis nowErrorCode.parse('ACTION_CONFIRMATION_REQUIRED')through@objectstack/spec/api. The runner reads the code through the catalogue and no longer restates it. No ledger row, noPROVENANCE_WAIVERSentry,packages/specuntouched.d07910da: every check run is success or skipped, none failed or pending (read from the commit's check runs).check:error-code-provenanceis green in Lint & Repo Gates.packages/spec/src/**in the diff), and the claim declaredno.check-governed-merges.mjs --pr 30746reports 0 of 17 paths governed. Size is 2769 changed lines (+2766 / −3), under the 3000-line threshold, so this is Tier S.Landing:
pr_ready+automerge_enablevia the relay in this stroke. Next after merge: release #30770 (control-card fix,Blocked-by: #30746).